Data Engineer

Key Responsibilities

• Pipeline Development – Design, implement, and optimize scalable ETL/ELT pipelines using Databricks (PySpark, SQL, Delta Lake) to ingest structured and semi-structured data from multiple sources, including APIs, databases, and streaming platforms.

• Data Warehousing – Build and maintain cloud data warehouse solutions using Snowflake or Azure Synapse. Design star schemas, fact/dimension tables, and aggregate tables for high-performance reporting.

• Data Modeling – Create logical and physical data models for operational and analytical use cases. Implement SCD Type 2, slowly changing dimensions, and data vault methodologies where appropriate.

• Performance Tuning – Optimize Spark jobs, SQL queries, and data partitioning strategies to handle petabyte-scale data with low latency.

• Governance & Quality – Implement data quality checks, monitoring, and lineage using tools such as Great Expectations or custom frameworks. Enforce data governance policies including GDPR and CCPA.

• Collaboration – Partner with data analysts, product managers, and engineers to translate business requirements into technical data solutions.

• CI/CD & Automation – Automate deployment of data pipelines using Azure DevOps or GitHub Actions. Maintain Infrastructure as Code (IaC) using Terraform for data resources.

Required Skills & Experience

• Total Experience: 10+ years in data engineering or related roles.

• Cloud Data Platforms: Deep hands-on experience with Databricks, including notebooks, jobs, clusters, Delta Lake, and Unity Catalog. Production-level experience is required.

• Data Warehousing: Proven experience with cloud data warehouses such as Snowflake, Azure Synapse, or Redshift, including design, optimization, and administration.

• Data Modeling: Strong knowledge of dimensional modeling (Kimball/Inmon), relational database design, and experience with tools such as ER/Studio or dbt.

• Programming: Expert-level Python and SQL skills with the ability to write maintainable, production-grade code.

• Big Data: Hands-on experience with Apache Spark, PySpark, distributed computing, and performance tuning.

• Orchestration: Experience with workflow tools such as Airflow, Azure Data Factory, or Prefect for scheduling and monitoring pipelines.

• Version Control: Proficiency with Git and collaborative development workflows.

Preferred Qualifications

• Experience with streaming technologies such as Kafka, Event Hubs, or Kinesis.

• Knowledge of data mesh or data fabric architectures.

• Familiarity with BI tools such as Power BI, Tableau, or Looker.

• Databricks certification, such as Associate or Professional Data Engineer.

• Experience with dbt (data build tool) and transformation testing.

• Exposure to MLflow or MLOps practices.

Education & Soft Skills

• Bachelor's or Master's degree in Computer Science, Information Systems, or a related field, or equivalent practical experience.

• Strong communication skills.

• Self-starter with a problem-solving mindset and the ability to work independently in a hybrid environment.

Keywords

Skills: Digital – Snowflake | Digital – Databricks
Experience Required: 10+ years