Sr Data Engineer with CDP, GCP, Cloud Spanner

TekWissen LLC

Atlanta, GA
3 days ago

$57.91

Highlights

You will build how clickstream, care, store, and network data flows into Cloud Spanner (low-latency store), BigQuery (analytics), and the knowledge graph behind GraphQL and MCP servers while migrating and reusing proven ETL from the Azure estate (Databricks, Snowflake, Data Factory, Event Hubs, Cosmos DB). ABOUT THE ROLE: Build the Pipelines of Customer Truth: The Data Engineer, CDP builds and operates the real-time pipelines behind Client's Customer Data Platform the system of record for 100M+ customer profiles, now moving to Google Cloud to deliver real-time customer context, near-real-time signals, and sub-50ms API access.

Numbers & Facts

Location	Atlanta, GA
Industry	Computer/IT Services
Salary	$57.91
Company Size	100 to 499 employees
Year Founded	2009
Website	http://www.tekwissen.com/

Description

Overview:

TekWissen is a global workforce management provider headquartered in Ann Arbor, Michigan that offers strategic talent solutions to our clients world-wide. Our client provider of digital technology and transformation, information technology and services

Job Title: Sr Data Engineer with CDP, GCP, Cloud Spanner
Location: Atlanta, GA
Duration: 6 Months
Job Type: Temporary Assignment
Work Type: Onsite

Job Description:

ABOUT THE ROLE:

Build the Pipelines of Customer Truth:
- The Data Engineer, CDP builds and operates the real-time pipelines behind Client's Customer Data Platform the system of record for 100M+ customer profiles, now moving to Google Cloud to deliver real-time customer context, near-real-time signals, and sub-50ms API access. Any agent, store associate, or digital channel should see a customer's prior conversation and continue from there.
- You will build how clickstream, care, store, and network data flows into Cloud Spanner (low-latency store), BigQuery (analytics), and the knowledge graph behind GraphQL and MCP servers while migrating and reusing proven ETL from the Azure estate (Databricks, Snowflake, Data Factory, Event Hubs, Cosmos DB).

RESPONSIBILITIES:

What You'll Do:

Real-Time Pipeline Engineering:
- Build and run NRT ingestion with Pub/Sub and Dataflow (Beam) into Spanner and BigQuery, with data contracts, schema evolution, exactly-once semantics, dead-letter handling, and replay
- Implement Spanner loads and change streams for current-state customer profiles, tuning keys, batching, and hotspots to hold sub-50ms reads
Clickstream, CDC & Data Modeling:
- Implement reusable CDC, SCD Type 2, and late-arriving-data patterns in Beam, SQL, and PySpark; build clickstream ingestion at scale
- Partner with NBA and T-Engage to cut signal latency to sub-five minutes; build BigQuery Spanner sync and reverse ETL
Azure-to-GCP Migration:
- Move data from Azure to GCP with STS and BigQuery Spanner sync, reusing Databricks, Data Factory, and Snowflake ETL; execute migration or retirement of Cosmos DB, Redis, Event Hubs, and Functions with dual-run, reconciliation, and rollback
Knowledge Graph, API & MCP Data Feeds:
- Build the pipelines that populate the customer knowledge graph on Spanner (entity and identity resolution, Customer 360) and feed the sub-50ms GraphQL API and the MCP servers on Cloud Run used by AI agents
Quality, Operations & Delivery:
- Implement data quality, lineage, and access controls (Dataplex, Unity Catalog, TISS-310) and IAM/VPC-SC/CMEK; ship on Cloud Run, Dataproc, Dataflow, and Composer with Terraform, CI/CD, monitoring, SLOs, and cost tuning
- Own pipeline on-call and runbooks, join design reviews, mentor junior engineers, and partner with product, analytics, and AI/ML teams.

QUALIFICATIONS:

What You'll Bring Required Experience:

Overall (5+ years): Bachelor's in Computer Science or related field (or equivalent); 5+ years in data engineering, 2+ building real-time pipelines in production.
Google Cloud data platform (3+ years): Production data pipelines on GCP, including IAM, VPC Service Controls, and cost awareness.
Streaming & NRT pipelines (3+ years): Pub/Sub, Dataflow (Beam), Kafka, Event Hubs, or Spark Structured Streaming with exactly-once semantics, watermarking, and clickstream ingestion at scale.
BigQuery (3+ years): Partitioning and clustering, reservations and cost control, Spanner federation, and BigQuery-to-Spanner (reverse ETL) data movement.
Cloud Spanner (1+ year): Schema and primary key design, indexes, hotspot avoidance, change streams, and bulk and streaming loads.
Data modeling & ETL (4+ years): Dimensional, Data Vault, and medallion models; idempotent ETL/ELT (CDC, SCD Type 2); expert SQL, Python, and advanced PySpark.
Databricks, Spark & Delta Lake (3+ years): Spark tuning, Delta Lake, Delta Live Tables, and Unity Catalog, to reuse and migrate existing ETL.
Azure & Snowflake (2+ years): Data Factory, Event Hubs, Cosmos DB, Redis, and Functions, plus Snowflake, to support a hybrid-cloud migration.
Serving & APIs (1+ year): Building data feeds for REST/gRPC or GraphQL services with p99 targets under 50ms; exposure to MCP, LLM/agent, or Vertex AI data access a plus.
Platform engineering (3+ years): Terraform/IaC, CI/CD, Cloud Run, Dataproc, and Composer or Airflow; observability and SLOs.
Testing & data quality (2+ years): Unit, integration, and data-quality testing for pipelines; reconciliation, lineage, and Dataplex or similar tools.
Collaboration (2+ years): Working with product, analytics, and AI/ML teams; clearly explaining design tradeoffs; mentoring junior engineers.

TECHNICAL REQUIREMENTS:

Must-Have Skills:

Google Cloud Real-Time Pipelines & Serving:

Skill / Technology:
- Real-Time Pipelines: Pub/Sub and Dataflow (Beam) exactly-once, watermarking, ordering, dead-letter handling, schema evolution, and replay
- Cloud Spanner Loading: Key design, batching, hotspot avoidance, change streams, and stale reads at 100M+ profile scale
- BigQuery: Partitioning and clustering, reservations and cost governance, BigLake, Spanner federation, and reverse ETL to Spanner
- Graph & API Feeds: Pipelines for identity resolution and Customer 360 on Spanner Graph, feeding GraphQL (sub-50ms p99) and MCP servers on Cloud Run

Streaming, Analytics & Compute:

Skill / Technology:
- Beam & Spark Streaming: Apache Beam (Python/Java), Spark Structured Streaming, and Kafka or Event Hubs stateful processing, windowing, and backpressure.
- Orchestration & Compute:Composer/Airflow, Cloud Run, Dataproc, Dataflow, GKE, and Cloud Functions service selection and right-sizing.
- Pipeline Code Quality:Production Python, Java, SQL, and PySpark unit and integration testing, code review, and performance tuning.

Hybrid Cloud & Existing Azure Platform:

Skill / Technology:
- Cross-Cloud Data Movement:STS, BigQuery Data Transfer, Interconnect/ExpressRoute, Workload Identity Federation, egress cost, and Azure-to-GCP reconciliation.
- Databricks & Delta Lake:Databricks, Delta Lake, DLT, Unity Catalog, and medallion architecture to reuse and migrate existing ETL.
- Snowflake & Azure Services:Snowflake; Data Factory, Event Hubs, Cosmos DB, Redis, and Functions to map, replace, or retire.

Modeling, Governance & Operations:

Skill / Technology:
- Data Modeling & ETL: Dimensional, Data Vault, and medallion models; idempotent ETL/ELT (CDC, SCD Type 2, late-arriving data); expert SQL and PySpark
- Security, Quality & Ops: IAM, VPC-SC, CMEK, Cloud DLP, Dataplex, Unity Catalog, TISS-310 alignment; monitoring, SLOs, Terraform, CI/CD, and cost tuning

TekWissen Group is an equal opportunity employer supporting workforce diversity.

About Company

WE THE TEKWISSEN PEOPLE

TekWissen offers you a broader portfolio of services, industry-leading solutions, and the meaningful innovations that give you greater flexibility and speed to respond to market dynamics, reduced costs and risk to improve enterprise performance, and increased productivity to enable growth.

To keep pace with global market demands, TekWissen keeps its finger on the pulse of change. Our organized approach to guiding a project from its inception to closure. Managing projects is becoming more and more important as we enter the digital era. To cope with the pace that this transition demands, a method is required to manage projects so they can yield quality work, while incorporating efficient use of time and resources.

Project involves identifying which quality standards are relevant to the project and determining how to satisfy them.

It is important to perform quality planning during the Planning Process and should be done alongside the other project planning processes because changes in the quality will likely require changes in the other planning processes, or the desired product quality may require a detailed risk analysis of an identified problem. It is important to remember that quality should be planned, designed, then built in, not added on after the fact.

Capabilities and accomplishments in one TekWissen business enhance the opportunity for success in the others. Put simply, TekWissen's unique combination of attributes promotes success.