Build and maintain scalable data pipelines on Databricks for consumer-facing IBU data products. Develop ETL/ELT pipelines using PySpark, Spark SQL and Delta Lake.
Numbers & Facts
Location
Indianapolis, IN
Description
Build and maintain scalable data pipelines on Databricks for consumer-facing IBU data products.
Develop ETL/ELT pipelines using PySpark, Spark SQL and Delta Lake
Implement medallion architecture (bronze/silver/gold), Unity Catalog and Workflows
Optimize cluster performance and cost; monitor and troubleshoot jobs in production
Requirements
Must have: 4+ yrs data engineering, Databricks, PySpark, SQL, Git/CI-CD.
Nice to have: pharma/consumer data, Databricks certification