| Location | Indianapolis, IN |
Contribute to the enterprise data technology strategy, helping resolve complex data integration challenges (such as real-time event-based ingestion, data quality, and stewardship).
Provide technical direction and oversight for designing and implementing integrated data solutions across secure systems.
Assist with defining, implementing, and enforcing modern data management policies and engineering design standards.
Design, build, and maintain robust batch and streaming data pipelines in Azure Databricks utilizing PySpark, Spark SQL, and Delta Lake following a medallion (bronze/silver/gold) architecture.
Implement incremental ingestion workflows via Auto Loader, Structured Streaming, or Delta Live Tables (DLT).
Optimize cluster performance, right-size SQL warehouses, and troubleshoot partitioning, shuffle, skew, and caching issues using Spark UI.
Develop, load, and tune dimensional data models, star-schema facts, dimensions, stored procedures, and views (including SCD Type 1 and Type 2) in Azure Synapse Analytics (dedicated and serverless SQL pools).
Query Parquet, Delta, and CSV files in ADLS Gen2 using OPENROWSET, external tables, and views.
Utilize advanced loading features such as COPY INTO, PolyBase, and CETAS to build parameterized pipelines.
Apply rigorous data governance and security controls (Unity Catalog permissions, role-based access, row/column-level security, and data masking) to protect sensitive data in accordance with regulatory standards (e.g., HIPAA, PII/PHI guidelines).
Orchestrate and monitor data workflows using Databricks Workflows/Jobs and Azure Data Factory (ADF) pipelines, managing scheduling, alerting, and failure recovery.
Implement CI/CD best practices using Git, Azure DevOps, or GitHub for version control and artifact deployment.
Education & Experience:
Master’s degree in Data Science, Analytics, Informatics, Computer Science, Mathematics, Engineering, or a related field (with 4+ years of relevant experience); OR
Bachelor’s degree in a related field plus 2+ years of relevant experience; OR
4+ years of hands-on experience in a dedicated data engineering role. (Note: Relevant certifications and coursework may be considered.)
Azure Databricks: 4+ years developing with notebooks in PySpark, Spark SQL, and R (transformations, joins, aggregations, window functions); implementing ACID transactions, MERGE upserts, schema evolution, and table maintenance (OPTIMIZE, Z-order, VACUUM).
Lakehouse & Architecture: 4+ years working with medallion architecture, Unity Catalog (workspaces, catalogs, schemas, lineage, external locations), and incremental data ingestion.
Azure Synapse Analytics: 4+ years designing MPP warehouse tables (hash/round-robin/replicated distribution, columnstore indexes), workload management, result-set caching, and pool optimization.
Orchestration & DevOps: Proven experience scheduling dependencies via ADF Pipelines, Databricks Workflows, and managing codebases with Git, Azure DevOps, or GitHub (branching, PRs, asset bundles).
Security & Monitoring: Experience designing ADLS Gen2 folder/file-format structures, Entra ID access control, managed identities, Key Vault integration, and monitoring workloads using Azure Monitor/Log Analytics.
Databricks Certified Data Engineer Associate (Professional certification is a strong plus).
Microsoft Certified: Azure Data Engineer Associate (DP-203).
BI Integration: Experience connecting Power BI to Synapse (Import vs. DirectQuery) and optimizing model design for high-performance dashboarding.