Job Title: Azure Data Tech Lead
Location: Alpharetta, Georgia (3-5 Days Onsite)
Berkley Heights NJ
Client prefers Diverse (Visa independent) candidates
Must Have Skills -
Skill 1 - Yrs of Exp - Azure Databricks
Skill 2 - Yrs of Exp - SQL
Skill 3 - Yrs of Exp - Spark
Good To have Skills -
Skill 1 - Yrs of Exp - Java
Skill 2 - Yrs of Exp - Python
"Core Skills: Azure, Databricks, ADLS, Spark (Python), SQL, ETL, Delta Lake, PostgreSQL, Data Architecture, Batch & Real-time Processing, Data Modelling
Overview
We are looking for an experienced Senior/Lead Data Engineer with 8+ years of expertise in designing and delivering scalable, high-performing data solutions on the Azure ecosystem. The ideal candidate will have deep hands-on experience with Databricks, Spark, modern data lakehouse architectures, data modelling, and both batch and real-time data processing. You will be responsible for driving end-to-end data engineering initiatives, influencing architectural decisions, and ensuring robust, high-quality data pipelines.
Key Responsibilities
- Architect, design, and implement scalable data platforms and pipelines on Azure and Databricks.
- Build and optimize data ingestion, transformation, and processing workflows across batch and real-time data streams.
- Work extensively with ADLS, Delta Lake, and Spark (Python) for large-scale data engineering.
- Lead the development of complex ETL/ELT pipelines, ensuring high quality, reliability, and performance.
- Design and implement data models, including conceptual, logical, and physical models for analytics and operational workloads.
- Work with relational and lakehouse systems including PostgreSQL and Delta Lake.
- Define and enforce best practices in data governance, data quality, security, and architecture.
- Collaborate with architects, data scientists, analysts, and business teams to translate requirements into technical solutions.
- Troubleshoot production issues, optimize performance, and support continuous improvement of the data platform.
- Mentor junior engineers and contribute to building engineering standards and reusable components.
Required Skills & Experience
- 10+ years of hands-on data engineering experience in enterprise environments.
- Strong expertise in Azure services, especially Azure Databricks, Functions, and Azure Data Factory (preferred).
- Advanced proficiency in Apache Spark with Python (PySpark).
- Strong command over SQL, query optimization, and performance tuning.
- Deep understanding of ETL/ELT methodologies, data pipelines, and scheduling/orchestration.
- Hands-on experience with Delta Lake (ACID transactions, optimization, schema evolution).
- Strong experience in data modelling (normalized, dimensional, lakehouse modelling).
- Experience in both batch processing and real-time/streaming data (Kafka, Event Hub, or similar).
- Solid understanding of data architecture principles, distributed systems, and cloud-native design patterns.
- Ability to design end-to-end solutions, evaluate trade-offs, and recommend best-fit architectures.
- Strong analytical, problem-solving, and communication skills.
- Ability to collaborate with cross-functional teams and lead technical discussions.
Preferred Skills
- Experience with CI/CD tools such as Azure DevOps and Git.
- Familiarity with IaC tools (Terraform, ARM).
- Exposure to data governance and cataloging tools (Azure Purview).
- Experience supporting machine learning or BI workloads on Databricks."