Azure Databricks & ADF Data Engineer

Location: Michigan City, IN – Onsite
Experience: 10+ Years

Job Summary

We are seeking an experienced Azure Databricks & Azure Data Factory (ADF) Data Engineer to design, develop, and implement enterprise-grade data solutions on Microsoft Azure. The ideal candidate will have strong hands-on expertise in Azure Databricks, ADF, Synapse Analytics, Azure Data Lake, Spark, PySpark, SQL, and data warehousing.

The candidate will work closely with business stakeholders, data scientists, and engineering teams to build scalable data pipelines, modernize data platforms, and ensure data quality, governance, security, and performance.

Key Responsibilities

• Design and develop scalable, cloud-native data solutions using Azure services.

• Build and maintain end-to-end data pipelines for data ingestion, transformation, and consumption.

• Develop data integration workflows using Azure Data Factory (ADF).

• Design and implement data processing solutions using Azure Databricks, Spark SQL, and PySpark.

• Work with Azure Synapse Analytics, Azure SQL, and Azure Data Lake.

• Perform data cleansing, mapping, transformation, and validation based on business requirements.

• Implement data quality, governance, security, and compliance standards.

• Develop and optimize data warehouse solutions using dimensional modeling and OLAP concepts.

• Integrate data engineering solutions with DevOps and CI/CD processes.

• Support automation and infrastructure-as-code initiatives.

• Troubleshoot performance, data quality, and pipeline issues.

• Collaborate with stakeholders to translate business requirements into technical solutions.

• Mentor junior developers and contribute to technical best practices.

• Evaluate and recommend emerging Azure data technologies.

Required Skills

• 10+ years of experience in data engineering, data architecture, or related areas.

• 6+ years of hands-on experience with Microsoft Azure.

• Strong experience with Azure Databricks and Azure Data Factory (ADF).

• Strong knowledge of:

  • Azure Synapse Analytics

  • Azure SQL

  • Azure Data Lake

  • Logic Apps

  • Spark SQL

  • PySpark

  • Advanced SQL

• Experience with Microsoft Fabric and Unity Catalog.

• Strong understanding of data warehousing, dimensional modeling, and OLAP.

• Experience developing enterprise-scale ETL/ELT pipelines.

• Knowledge of data governance, security, and compliance frameworks.

• Experience with DevOps, CI/CD, and automation.

• Strong troubleshooting, analytical, communication, and stakeholder management skills.

Preferred Skills

• Experience modernizing legacy data platforms to Azure.

• Experience with Infrastructure as Code.

• Knowledge of Azure security and identity services.

• Experience working with large-scale enterprise data environments.

• Strong technical leadership and mentoring experience.

Key Competencies

• Azure Databricks

• Azure Data Factory

• Azure Synapse Analytics

• Azure Data Lake

• Microsoft Fabric

• Unity Catalog

• SQL / Spark SQL

• PySpark

• ETL/ELT

• Data Warehousing

• Dimensional Modeling

• Data Governance & Security

• DevOps & CI/CD

• Data Pipeline Development

• Technical Leadership

• Stakeholder Management