Senior Data Engineer Jobot
Chicago, ILTodayRemote
- $150,000–$210,000 Per Year
Highlights
Data Engineering & Pipeline Development Design, develop, and maintain end-to-end data pipelines in Databricks using Spark and Delta Lake. Data Quality, Governance & Controls Implement data quality checks, reconciliation processes, and monitoring within Databricks pipelines.
Numbers & Facts
| Location | Chicago, IL |
Description
Job Title: Data Engineer - (Databricks/ Spark/ Delta Lake)
Location: Chicago, IL (onsite – Only Locals Required)
Duration: 6 Months contract+
Job Description:
Key Responsibilities:
Data Engineering & Pipeline Development
Required Qualifications:
Preferred Qualifications:
Technical Skills:
Location: Chicago, IL (onsite – Only Locals Required)
Duration: 6 Months contract+
Job Description:
Key Responsibilities:
Data Engineering & Pipeline Development
- Design, develop, and maintain end-to-end data pipelines in Databricks using Spark and Delta Lake
- Build and optimize ELT/ETL processes for structured and unstructured data ingestion into the Data Lakehouse
- Implement scalable ingestion patterns (batch and event-driven) from internal systems, third-party APIs, and cloud sources
- Develop data models (bronze, silver, gold layers) to support enterprise reporting, analytics, and downstream consumption
- Integrate the Data Lakehouse with enterprise tools such as Tableau, Alteryx, and machine learning platforms
- Design and implement data access controls, identity management, and secure data sharing mechanisms
- Support API-based integrations and downstream data consumption patterns
- Implement data quality checks, reconciliation processes, and monitoring within Databricks pipelines
- Ensure adherence to enterprise data governance standards, including lineage, metadata, and audit requirements
- Support regulatory and compliance requirements (e.g., data integrity, privacy, and security controls)
- Develop and manage workflows using orchestration tools (e.g., Airflow, Control-M)
- Automate data pipelines, deployments, and operational processes through CI/CD pipelines
- Leverage cloud-native services (AWS/Azure) for data processing, storage, and event-driven architectures
- Operations & Support Monitor, troubleshoot, and optimize data pipelines and Spark workloads for performance and reliability
- Support production data platforms, including incident resolution and root cause analysis
- Ensure high availability, data integrity, and SLA adherence across enterprise data systems
- Partner with data architects, data scientists, BI teams, and business stakeholders to deliver data solutions
- Participate in Agile ceremonies and contribute to iterative delivery of data products
- Translate business requirements into scalable technical data solutions
Required Qualifications:
- 3+ years of experience in data engineering, data platforms, or related roles
- Hands-on experience with Databricks, Apache Spark (PySpark), and Delta Lake
- Strong SQL and data modeling skills (relational and dimensional)
- Experience building and supporting data pipelines in a cloud environment (AWS or Azure)
- Experience with ELT/ETL tools (e.g., Fivetran, custom ingestion frameworks)
- Familiarity with data orchestration tools (Airflow, Control-M)
- Experience working in Agile development environments
Preferred Qualifications:
- Experience in financial services or regulated environments (e.g., banking, risk, regulatory reporting)
- Knowledge of data governance frameworks and tools (e.g., Collibra)
- Experience with real-time or streaming data pipelines
- Exposure to machine learning pipelines and feature engineering in Databricks
- Cloud certifications (AWS, Azure, or Databricks)
Technical Skills:
- Databricks (Lakehouse architecture, notebooks, jobs, Unity Catalog)
- Spark / PySpark
- SQL (advanced querying and optimization)
Similar Job Searches
Data Manager Jobs in Chicago, ILBusiness Intelligence Manager Jobs in Chicago, ILData Analyst Jobs in Chicago, ILDirector Of Information Systems Jobs in Chicago, ILData Specialist Jobs in Chicago, ILBusiness Intelligence Analyst Jobs in Chicago, ILData Consultant Jobs in Chicago, ILData Coordinator Jobs in Chicago, ILData Engineer Jobs in Riverwoods, ILData Engineer Jobs in Naperville, IL
Similar Jobs
- New!
- New!
Staff Data Engineer Jobot
Chicago, ILToday- $170,000–$205,000 Per Year
Data Engineer Jobot
Chicago, IL7 days ago- $180,000–$250,000 Per Year
- New!
Senior Healthcare Data Engineer Jobot
Chicago, IL1 day ago- $120,000–$150,000 Per Year
- New!
Commissioning Project Engineer (Data Centers) Jobot
Chicago, IL1 day agoRemote- $90,000–$160,000 Per Year
Project Executive – BAS/BMS - Hyperscale Data Center projects Jobot
Chicago, IL7 days ago- $200,000–$225,000 Per Year