Databricks Engineer

Role: Databricks Engineer Experience 4-8 years Location As per business requirement Employment Type Full-time Job Summary We are looking for a skilled Databricks Engineer with strong expertise in designing, developing, and optimizing modern data engineering solutions on the Databricks Lakehouse Platform. The ideal candidate should have experience building scalable ETL/ELT pipelines, working with large-scale data, and leveraging Apache Spark to deliver high-performance data solutions. Key Responsibilities Design, develop, and maintain scalable data pipelines using Databricks. Build ETL/ELT workflows for batch and streaming data processing. Develop solutions using PySpark, Spark SQL, and Delta Lake. Implement Medallion Architecture (Bronze, Silver, Gold) for data transformation. Integrate data from various sources including relational databases, APIs, cloud storage, and streaming platforms. Optimize Spark jobs for performance, scalability, and cost efficiency. Collaborate with Data Architects, Data Scientists, BI developers, and business stakeholders. Implement CI/CD pipelines and deployment automation for Databricks workloads. Ensure data quality, security, governance, and compliance. Monitor, troubleshoot, and optimize production data pipelines. Document technical solutions and follow engineering best practices. Required Skills Core Technologies Databricks Lakehouse Platform Apache Spark PySpark Spark SQL Delta Lake Python SQL Cloud Platforms (one or more) Microsoft Azure (preferred) AWS Google Cloud Platform Azure Technologies (Preferred) Azure Data Factory (ADF) Azure Data Lake Storage (ADLS Gen2) Azure Synapse Analytics Azure Key Vault Azure DevOps Data Engineering Data Warehousing Data Modeling ETL/ELT Development Batch Processing Streaming (Kafka/Event Hubs) Data Lake Architecture DevOps & Version Control Git Azure DevOps / GitHub CI/CD Pipelines Preferred Qualifications Experience with Unity Catalog. Knowledge of Databricks Workflows and Jobs. Hands-on experience with Delta Live Tables (DLT). Exposure to MLflow is an added advantage. Experience with data governance and security best practices. Familiarity with Infrastructure as Code (Terraform) is a plus. Educational Qualification Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field. Preferred Certifications Databricks Certified Data Engineer Associate Databricks Certified Data Engineer Professional Microsoft Certified: Azure Data Engineer Associate (DP-203) Azure Fundamentals (AZ-900) Good to Have Experience with real-time analytics. Knowledge of Lakehouse architecture. Experience with Agile/Scrum methodologies. Strong analytical and problem-solving skills. Excellent communication and stakeholder management abilities. Mandatory Skills Databricks PySpark Spark SQL Delta Lake Python SQL Azure/AWS/GCP (at least one cloud platform) ETL/ELT Development Data Lake Architecture Nice to Have Unity Catalog Delta Live Tables (DLT) MLflow Kafka/Event Hubs Azure Data Factory Terraform Azure DevOps/GitHub Actions