Requirement ID: 94731-1
Title: Data Engineer Lead
Pay range: $40- $45/hr
Location: Kansas City, MO (Local)
Duration: 6 months
We are seeking a highly experienced Lead Data Engineer with 8 10+ years of experience in designing, developing, and supporting enterprise-scale data platforms on AWS. The ideal candidate will have strong expertise in AWS Data Services, Python, PySpark, SQL, ETL/ELT development, and Data Lake/Lakehouse architectures.
This role requires technical leadership, hands-on development, Agile delivery ownership, production support, and stakeholder management while building scalable, secure, and high-performance cloud data solutions.
Key Responsibilities
Data Engineering & Development
Design, develop, and maintain scalable data pipelines using AWS cloud services.
Build robust ETL/ELT workflows using Python, PySpark, AWS Glue, and SQL.
Develop solutions for processing structured, semi-structured, and large-scale datasets.
Implement enterprise Data Lake/Lakehouse solutions using Amazon S3.
Build reusable data ingestion and transformation frameworks.
AWS Data Platform
Develop and optimize solutions using:
Amazon S3
AWS Glue
Amazon Athena
AWS Lambda
Amazon Redshift
Amazon EMR
Design secure, scalable, and cost-efficient cloud data architectures.
Optimize storage, partitioning, compression, and query performance.
Work with Parquet, ORC, and Avro file formats.
Data Pipeline Optimization
Design high-performance batch data pipelines.
Optimize Spark jobs and SQL queries for large datasets.
Improve pipeline reliability, scalability, and operational efficiency.
Implement monitoring, logging, and alerting for data workflows.
Agile Delivery & Technical Leadership
Serve as the technical lead and Agile anchor for the data engineering team.
Lead sprint planning, backlog grooming, estimation, and delivery tracking.
Collaborate with Product Owners, Scrum Masters, Architects, and business stakeholders.
Mentor junior engineers and establish engineering best practices.
Conduct code reviews, design reviews, and technical walkthroughs.
Production Support
Provide L2/L3 production support for enterprise data platforms.
Troubleshoot pipeline failures and performance issues.
Perform Root Cause Analysis (RCA) and implement preventive solutions.
Participate in incident management and on-call support.
Utilize CloudWatch and monitoring tools to ensure platform health.
Data Governance & Quality
Implement data quality validation and reconciliation processes.
Ensure data integrity, lineage, governance, and compliance.
Develop monitoring frameworks for data quality and operational metrics.
DevOps & Automation
Implement CI/CD pipelines for data engineering solutions.
Use Git, Jenkins, AWS CodePipeline, or similar deployment tools.
Support Infrastructure as Code using Terraform or CloudFormation.
Automate deployment, testing, and operational processes.
Company Benefits & Culture