Sr Data Engineer

We are seeking a Senior Data Engineer for a 6-month contract role based in Atlanta, Georgia, working onsite. You will build and operate real-time data pipelines for a Customer Data Platform serving 100M+ customer profiles, migrating to Google Cloud to deliver real-time customer context, near-real-time signals, and sub-50ms API access across all customer touchpoints.

Responsibilities

• Build and run near-real-time ingestion with Pub/Sub and Dataflow (Beam) into Cloud Spanner and BigQuery, implementing data contracts, schema evolution, exactly-once semantics, dead-letter handling, and replay capabilities
• Implement Cloud Spanner loads and change streams for current-state customer profiles, tuning keys, batching, and hotspot avoidance to maintain sub-50ms read performance
• Implement reusable CDC, SCD Type 2, and late-arriving-data patterns in Beam, SQL, and PySpark; build clickstream ingestion at scale
• Partner with teams to reduce signal latency to sub-five minutes; build BigQuery-to-Spanner sync and reverse ETL pipelines
• Execute migration from Azure to GCP with Cloud Storage and BigQuery-to-Spanner sync, reusing existing Databricks, Data Factory, and Snowflake ETL; manage dual-run, reconciliation, and rollback of Cosmos DB, Redis, Event Hubs, and Functions
• Build pipelines that populate the customer knowledge graph on Cloud Spanner (entity and identity resolution, Customer 360) and feed sub-50ms GraphQL APIs and MCP servers on Cloud Run used by AI agents
• Implement data quality, lineage, and access controls; manage IAM, VPC Service Controls, and CMEK encryption; deploy on Cloud Run, Dataproc, Dataflow, and Composer using Terraform and CI/CD with monitoring, SLOs, and cost tuning
• Own pipeline on-call responsibilities and runbooks; participate in design reviews, mentor junior engineers, and collaborate with product, analytics, and AI/ML teams

Qualifications

• Required
• 5+ years of data engineering experience
• 2+ years building real-time pipelines in production
• 3+ years of production data pipelines on Google Cloud Platform, including IAM, VPC Service Controls, and cost awareness
• 3+ years with Pub/Sub, Dataflow (Beam), Kafka, Event Hubs, or Spark Structured Streaming, including exactly-once semantics, watermarking, and clickstream ingestion at scale
• 3+ years with BigQuery, including partitioning and clustering, reservations and cost control, Spanner federation, and BigQuery-to-Spanner data movement
• 1+ year with Cloud Spanner, including schema and primary key design, indexes, hotspot avoidance, change streams, and bulk and streaming loads
• 4+ years with data modeling and ETL, including dimensional, Data Vault, and medallion models; idempotent ETL/ELT with CDC and SCD Type 2 patterns
• Expert-level SQL and Python; advanced PySpark experience
• 3+ years with Databricks, Spark, and Delta Lake, including Spark tuning, Delta Live Tables, and Unity Catalog
• 2+ years with Azure and Snowflake ecosystems, including Data Factory, Event Hubs, Cosmos DB, Redis, and Functions
• 1+ year building data feeds for REST, gRPC, or GraphQL services with p99 latency targets under 50ms
• 3+ years with platform engineering tools including Terraform, CI/CD, Cloud Run, Dataproc, Composer, or Airflow; observability and SLOs
• 2+ years with testing and data quality practices for pipelines, including reconciliation, lineage, and tools like Dataplex
• 2+ years collaborating with product