TSenior Databricks Engineer

Senior Databricks Engineer:

Total Required Experience in Years: 14%2B Years

Mode of Work:Hybrid

Job Description:

The Senior Databricks Engineer will design, develop, and optimize enterprise-scale data solutions on the Databricks platform. This role is responsible for building high-performance ETL/ELT pipelines, implementing modern Lakehouse architectures, developing scalable data models, and delivering analytics solutions using Databricks SQL, Databricks Apps, Delta Lake, and Lakeflow Declarative Pipelines. The engineer will ensure data quality, governance, security, scalability, and operational efficiency while supporting enterprise reporting and analytics initiatives.

Key Responsibilities:

• Design, develop, and optimize scalable data solutions using Databricks.

• Build and maintain ETL/ELT data pipelines using Apache Spark and PySpark or Scala.

• Develop and manage Lakeflow Declarative Pipelines (formerly Delta Live Tables/DLT).

• Design and implement Delta Lake and Medallion Architecture using:

  • Bronze Layer

  • Silver Layer

  • Gold Layer

• Develop enterprise data models using dimensional modeling techniques.

• Design Star Schema and Snowflake Schema data warehouses.

• Create and maintain Databricks SQL Dashboards and Databricks Apps.

• Develop analytical solutions that provide actionable business insights.

• Implement enterprise data governance, security, and data quality frameworks.

• Optimize Spark workloads for performance, scalability, and cost efficiency.

• Schedule and orchestrate data pipelines using:

  • Lakeflow Jobs (formerly Databricks Workflows)

  • Airflow or similar orchestration tools

• Monitor production workloads and resolve performance issues.

• Collaborate with architects, analysts, developers, and business stakeholders.

Additional Responsibilities:

• Support enterprise Lakehouse architecture initiatives.

• Develop reusable data engineering frameworks and best practices.

• Implement CI/CD pipelines for data engineering projects.

• Maintain technical documentation and operational procedures.

• Participate in architecture reviews and technical design sessions.

• Support production deployments and ongoing platform optimization.

Required Skills:

• Minimum 8 years of IT experience designing, developing, or delivering technology solutions.

• Minimum 8 years of experience with Databricks.

• Strong experience building ETL/ELT pipelines using Apache Spark.

• Proficiency with:

  • PySpark

  • Python

  • SQL

  • Scala

• Experience designing enterprise data warehouses.

• Strong knowledge of:

  • Star Schema

  • Snowflake Schema

  • Dimensional Data Modeling

• Experience implementing:

  • Delta Lake

  • Medallion Architecture

  • Bronze, Silver, Gold Layers

• Experience with Lakeflow Declarative Pipelines (formerly Delta Live Tables/DLT).

• Experience with Lakeflow Jobs (formerly Databricks Workflows) or Airflow.

• Experience creating Databricks SQL Dashboards and Databricks Apps.

• Experience implementing:

  • Data Governance

  • Data Quality

  • Data Validation

  • Data Security

• Excellent verbal and written communication skills.

Preferred Skills:

• Experience working in public-sector or state government environments.

• Databricks Certified Data Engineer Associate or Professional Certification.

• Experience implementing CI/CD pipelines.

• Experience with Git-based DevOps workflows.

• Knowledge of cloud-native data engineering best practices.

• Experience optimizing enterprise-scale Spark workloads.

Qualifications:

• Proven experience delivering enterprise data engineering solutions.

• Strong understanding of modern data lakehouse architecture.

• Excellent analytical, troubleshooting, and problem-solving skills.

• Ability to communicate effectively with technical and business stakeholders.

• Ability to manage multiple priorities in a fast-paced environment.

Education:

• Bachelor\'s degree in Computer Science, Information Systems, Data Engineering, or a related field.

• Master\'s degree preferred.

Certifications:

• Databricks Certified Data Engineer Associate Preferred

• Databricks Certified Data Engineer Professional Preferred

• Microsoft Azure Data Engineer Associate Preferred

• AWS Certified Data Analytics Specialty Preferred

• Google Professional Data Engineer Preferred