Sr. Data Engineer with AI Experience

Job Description

Job Title: Sr. Data Engineer with AI Experience

Experience: - 8+ Years

Location: -Chicago, IL (Hybrid 3 days' Work from Office)

In person interview required

Skills: Azure DataBricks ADF, DataBricks, Python, Pyspark, ETL, SQL.

Roles and Responsibilities: -

Rebuild Epic Caboodle extraction from a transformation-heavy pattern to clean incremental raw ingestion - no joins, no temp tables, no business logic at the source

Implement incremental / change-data-capture ingestion into the landing and raw layers on Azure and Databricks

Move member and facility mapping downstream, out of the acquisition step. Analyse and optimize existing SQL.

Work through the existing per-member script inventory with the Epic SME to map current Caboodle table access patterns - Identify redundant table access.

Establish the baseline table-touch count and measure the reduction the new design delivers

Implement the consolidation of transformations from three stages into two, moving redundant joins and source-specific logic into the layout-converter stage alongside the final data-model conversion

Restructure processing to run by source system rather than per member - Keep filtering out of the intermediate layer so the domain-refined mart remains the source of truth

Validate agent-generated output. Review, execute, and validate the SQL and metadata the agents generate - you are the quality gate between agent proposal and production execution

Build and run reconciliation harnesses comparing new-path output against current production extracts - Confirm downstream consumers (Databricks, SQL Server, Vertica) are not adversely affected by schema or semantic changes

Build tooling the agents depend on Data profiling routines the agents call as tools - null rates, cardinality, distributions, date coverage, code-system detection - Metadata and schema extraction jobs that seed the project's knowledge graph - Instrumentation for cost, run time, and table-touch telemetry.

Educational Qualifications: -

Engineering Degree BE/ME/BTech/MTech/BSc/MSc.

Technical certification in multiple technologies is desirable.

Skills: -

Mandatory skills

5+ years in data engineering, with recent hands-on delivery ownership.

Strong Databricks and PySpark jobs, workflows, Delta Lake, performance tuning

Expert SQL including the ability to read unfamiliar, poorly documented SQL at volume and reason about what it does and what it costs

Demonstrable query cost and performance optimization experience, this is a core requirement on this engagement, not a bonus

Incremental / CDC ingestion pattern** design and implementation - Azure data services - ADLS / Blob Storage, and Databricks on Azure.

Unity Catalog or comparable data governance and cataloguing experience - Data profiling and reconciliation testing proving two pipelines produce equivalent output

Comfort working with PHI-scoped healthcare data and the access controls that implies

Ability to work independently against ambiguous inputs and drive clarification, in a small team on a hard deadline

Preferred - Epic EHR data experience, Caboodle or Clarity schema familiarity is a significant advantage

Healthcare data domain knowledge: clinical coding systems (ICD, CPT, SNOMED, LOINC), encounter and procedure data models

Workflow orchestration experience - Orkes / Netflix Conductor, or transferable experience with Airflow, Dagster, or similar - SQL Server and/or Vertica exposure, for downstream compatibility validation

Experience working alongside AI/LLM-generated code or configuration - Knowledge-graph or metadata-management exposure (Stardog, RDF/SPARQL, or similar)

Experience with AI-assisted development tooling and spec-driven delivery practices

Someone with EPIC experience / knowledge is good to have