Azure Data Engineer

Job Title: Azure Data Engineer Azure Data Factory (ADF)
Work Location: Redmond,WA-98052

Job Details:
Must Have Skills:
Python, SQL, (bonus: Scala), Apache Spark, PySpark, Spark SQL, Azure Databricks (notebooks, Jobs/Workflows, clusters), Azure Synapse, Delta Lake, Parquet, ADLS Gen2

Nice to have skills:
ETL/ELT, MERGE/upsert, partitioning, Spark performance tuning, Git, Azure DevOps / CI/CD

Detailed Job Description
Techniques (Data Engineering Practices)
ETL / ELT
Extract data from source systems
Transform/clean/reshape into usable form
Load into target tables (ELT = transform after loading, inside the platform)

MERGE / Upsert
Update existing rows when records already exist
Insert new rows when they don't
Handled in a single atomic operation (core Delta Lake pattern)

Partitioning
Split large tables by a column (e.g., date, source type)
Enables partition pruning - scan only relevant data
Improves query speed and reduces cost


Spark Performance Tuning
Handle data skew and manage shuffles
Caching / persistence of reused datasets
Right-size partitions; use broadcast joins
Cluster sizing and resource optimization Orm

Minimum years of experience
8-10 years