Senior Azure Databricks Data Engineer

Experience with Apache Spark, Structured Streaming, Delta Live Tables (DLT), Databricks Workflows, and Unity Catalog. Databricks & Data Engineering Strong hands-on experience with Databricks, Lakehouse Architecture, and Delta Lake.

Job Title: Senior Azure Databricks Data Engineer
Location: Frederick, MD 21704,100% Onsite
Duration: 6 Months
 
Salary Range: $52.00 - $55.00/Hour on W2 (Without Benefits).
Applicants must be willing to work on W2
 
Role Description:

• Design, develop, and maintain scalable, high-performance data engineering solutions on Azure Databricks.
• Build and support batch and real-time data pipelines using Apache Spark and Structured Streaming.
• Design and implement Lakehouse and Medallion Architecture (Bronze, Silver, Gold) using Delta Lake.
• Develop reusable ETL/ELT frameworks, standardized data processing components, and enterprise data ingestion patterns.
• Ingest and process large-scale data from diverse enterprise sources, including databases, APIs, message queues, and file-based systems.
• Develop complex data transformation logic using PySpark, Python, and SQL.
• Create trusted, governed, and consumption-ready datasets for reporting, analytics, and downstream applications.
• Implement data quality, validation, reconciliation, audit, and governance controls across data pipelines.
• Manage and optimize Databricks Workflows, Delta Live Tables (DLT), Unity Catalog, and Databricks SQL.
• Collaborate with business, analytics, architecture, and delivery teams in an Agile environment to deliver high-quality data products.

Responsibilities:

• Design, develop, and maintain scalable, high-performance data engineering solutions on Azure Databricks.
• Build and support batch and real-time data pipelines using Apache Spark and Structured Streaming.
• Design and implement Lakehouse and Medallion Architecture (Bronze, Silver, Gold) using Delta Lake.
• Develop reusable ETL/ELT frameworks, standardized data processing components, and enterprise data ingestion patterns.
• Ingest and process large-scale data from diverse enterprise sources, including databases, APIs, message queues, and file-based systems.
• Develop complex data transformation logic using PySpark, Python, and SQL.
• Create trusted, governed, and consumption-ready datasets for reporting, analytics, and downstream applications.
• Implement data quality, validation, reconciliation, audit, and governance controls across data pipelines.
• Manage and optimize Databricks Workflows, Delta Live Tables (DLT), Unity Catalog, and Databricks SQL.
• Collaborate with business, analytics, architecture, and delivery teams in an Agile environment to deliver high-quality data products.

Required Technical Skills:
Databricks & Data Engineering

• Strong hands-on experience with Databricks, Lakehouse Architecture, and Delta Lake.
• Expertise in PySpark, Python, and advanced SQL.
• Experience with Apache Spark, Structured Streaming, Delta Live Tables (DLT), Databricks Workflows, and Unity Catalog.
• Experience designing and implementing Medallion Architecture and modern Data Lake/Data Warehouse solutions.
• Strong background in data ingestion, ETL/ELT, data transformation, optimization, and deployment practices.
• Experience with event-driven and streaming technologies such as Kafka or Azure Event Hub.

Cloud Platforms:

• Microsoft Azure (preferred).
• AWS or GCP experience is also valuable.

Application Integration:

• Working knowledge of React and/or Angular for front-end integration and data-centric application development.

Domain Experience (Preferred):

• Experience in Finance, Insurance, Regulatory Reporting, or US GAAP reporting.
• Understanding of financial data models, reporting processes, and governance requirements is highly desirable.

Experience Requirements:

• Must Have 7+ years of Experience in Azure Databricks.
• Must Have 7+ years of Experience in Python.