Senior Data Engineer Azure Databricks / PySpark

Role Summary

• Experience Required: 4–6 years

• Primary Role: Data Engineer / API Role with AI Builder

• Cloud Platform: Azure Cloud

• Role Type: Individual Contributor

Core Technical Skills

• Azure Databricks

• PySpark

• Scala

• Azure Data Factory (ADF)

• Azure Data Lake Storage Gen2 (ADLS Gen2)

• Delta Lake

• Unity Catalog

• Python

• SQL

• GitHub

• CI/CD

Data Engineering Responsibilities

• Design, develop, and optimize scalable data pipelines.

• Build cloud-based data solutions using Azure.

• Develop ETL/ELT data processing workflows.

• Work with large-scale datasets in enterprise environments.

• Perform data modeling and performance tuning.

• Implement and maintain Delta Lake solutions.

• Work with Databricks, ADF, and PySpark for data engineering workloads.

• Develop end-to-end data solutions independently.

• Apply data governance and security best practices.

API Development Skills

• Strong hands-on experience building REST or GraphQL APIs.

• Understanding of:

  • API design and contracts

  • API security

  • Error handling

  • API performance optimization

• Experience deploying APIs using Kubernetes, preferably Azure Kubernetes Service (AKS).

AI / ML Skills

• Exposure to AI/LLM technologies.

• Experience or exposure to agentic frameworks.

• Familiarity with:

  • Databricks AI/ML

  • Codex

  • Claude

  • Other LLM/AI development frameworks

Cloud & DevOps

• Strong hands-on experience with Azure Cloud.

• Experience with CI/CD practices.

• GitHub experience.

• Knowledge of cloud-native data platforms.

• Understanding of Kubernetes/AKS deployment.

Data Governance & Security

• Strong understanding of data governance.

• Knowledge of data security and access control.

• Experience with Unity Catalog is preferred.

• Ability to implement secure and governed data solutions.

Essential Skills

• Databricks

• Azure Data Factory (ADF)

• PySpark

• Azure Cloud

• ADLS Gen2

• Delta Lake

Desirable Skills

• Scala

• REST APIs / GraphQL

• Kubernetes / AKS

• AI/LLM

• Agentic frameworks

• Databricks AI/ML

• CI/CD

• Unity Catalog

Key Responsibilities

• Own end-to-end solution delivery.

• Analyze and solve complex data engineering problems.

• Collaborate with cross-functional teams and business stakeholders.

• Build scalable and reliable data pipelines.

• Optimize data processing and API performance.

• Follow enterprise standards for governance, security, and deployment.