Senior Data EngineerREQ #: 111449-1
Location: Remote in Market – McLean, VA or Riverwoods, IL
Openings: 3
Rate: $62.00 - $68.00
Duration: Approximately 7 months
Job DescriptionWe are seeking a Senior Big Data / Machine Learning Engineer with strong Data Engineering experience to design and implement high-performance, scalable data pipelines capable of processing large volumes of data. This role will focus heavily on Python, PySpark, AWS Glue, Amazon EMR, and large-scale data processing.
The engineer will also be responsible for maintaining data integrity through advanced data quality checks and data lineage tracking.
Responsibilities- Design and implement high-performance data pipelines that handle large volumes of data.
- Build scalable data pipelines supporting large-scale datasets, preferably at the terabyte level.
- Develop data processing solutions using Python and PySpark.
- Work extensively within AWS environments, particularly AWS Glue and Amazon EMR.
- Implement frameworks and processes for data quality and data lineage.
- Work with S3-based data lakes and relational databases.
- Support reliable and scalable delivery of data across enterprise data environments.
Required Skills & Qualifications- Expert-level proficiency in Python and PySpark.
- Deep hands-on experience with AWS Glue and Amazon EMR.
- Proven experience designing and building scalable, high-performance data pipelines.
- Experience working with large-scale data, with terabyte-scale environments preferred.
- Demonstrated experience implementing Data Quality and Data Lineage frameworks.
- Strong experience with Amazon S3-based data lakes.
- Experience working with relational databases, including RDS, PostgreSQL, and/or MySQL.
Preferred Qualifications- Experience within the Financial Services industry.
- Experience with Kinesis and/or Kafka integration for real-time or low-latency data delivery.
- Familiarity with CI/CD practices for data pipelines.
- Familiarity with Infrastructure as Code (IaC).
#LI-MC1
#INDPRO