Senior Databricks Data Engineer (PySpark and Delta Lake)

Senior PySpark and Delta Data Engineer

Job Summary

This is the deep build role. The Senior PySpark and Delta Data Engineer writes the production pipelines that move and reshape data at scale, and is expected to know why a job behaves the way it does rather than just that it finished. The role combines strong software engineering discipline with real Spark depth and hands on ownership of Delta Lake table design and maintenance.

Key Responsibilities

• Build and maintain production PySpark pipelines for batch and streaming workloads

• Design Delta Lake tables including partitioning, liquid clustering, Z ordering, and file sizing strategy

• Implement incremental processing using Structured Streaming, Auto Loader, and change data feed

• Apply merge, upsert, and deduplication logic that holds up under late arriving and out of order data

• Tune jobs for runtime and cost by reading the Spark UI and query plans, not by guesswork

• Write unit and integration tests, and build pipelines that fail loudly and recover cleanly

• Manage table maintenance including OPTIMIZE, VACUUM, and schema evolution

• Review code from other engineers and raise the standard of the codebase

Required Skills

• 6+ years in data engineering with 3+ years writing production PySpark

• Deep Delta Lake experience including ACID behavior, time travel, and table maintenance operations

• Strong understanding of Spark internals covering the catalyst optimizer, adaptive query execution, shuffle, skew, and spill

• Advanced SQL and strong Python software engineering fundamentals

• Experience with Structured Streaming and at least one streaming source such as Kafka, Event Hubs, or Kinesis

• Git based development, code review, and CI/CD practice

• Experience with workflow orchestration through Databricks Workflows, Airflow, or an equivalent

Preferred Skills

• Databricks Certified Data Engineer Professional

• Experience with Delta Live Tables or Lakeflow declarative pipelines

• Scala Spark experience

• Familiarity with data quality tooling such as Great Expectations or DLT expectations

• Performance tuning experience on multi terabyte workloads

Welcome to ConsultNet, a premier national provider of technology talent and solutions. Our expertise spans across project services, contract-to-hire, direct search, and managed services onshore, nearshore, and hybrid. For over 25 years, we have connected thousands of consultants with meaningful roles through a personal, communication-driven approach, partnering with a diverse client base to build high-performing teams and create lasting impact. Our comprehensive service offerings cover a wide range of technology and engineering positions across key markets nationwide. Learn more at www.consultnet.com .

We champion equality and inclusivity, proudly supporting an Equal Opportunity Employer policy. We welcome applicants regardless of Race, Color, Religion, Sex, Sexual Orientation, Gender Identity, National Origin, Age, Genetic Information, Disability, Protected Veteran Status, or any other status protected by law.