Highlights
Design, build, and optimize scalable batch and real-time data pipelines using Apache Spark (Scala/Python), Core Java, and Apache Kafka to process high-volume data. Implement Change Data Capture (CDC) and write complex SQL to integrate data from relational databases (Oracle, SQL Server) into dimensional data warehouses.
Numbers & Facts
| Location | Maryland Heights, MO |
| Industry | Other/Not Classified |
| Company Size | 100 to 499 employees |
| Year Founded | 1998 |
| Website | www.eteaminc.com |
Description
Job Title: Hadoop Data engineer
Work Location: Maryland Heights MO 63043 (hybrid )
Contract duration: 12
F2F interview - Yes
Job Details:
Must Have skills:
AWS, Hadoop
Key Responsibilities
Design, build, and optimize scalable batch and real-time data pipelines using Apache Spark (Scala/Python), Core Java, and Apache Kafka to process high-volume data.
Manage and maintain enterprise data Lakehouse architectures spanning AWS and Cloudera environments, leveraging S3, Apache Iceberg, and Hive for scalable storage and querying.
Implement Change Data Capture (CDC) and write complex SQL to integrate data from relational databases (Oracle, SQL Server) into dimensional data warehouses.
Automate and orchestrate pipeline workflows using Unix shell scripting and Python.
Lead and support migration of legacy Hadoop workloads to AWS cloud-native architectures.
Monitor, troubleshoot, and tune JVM and Spark job performance to ensure reliability and efficiency at scale.
Ensure all data pipelines and storage practices adhere to regulatory compliance and data governance standards.
Collaborate with cross-functional teams (data architects, analysts, platform engineers) to translate business requirements into robust data solutions.
Required Skills & Technology:
Strong programming experience in Scala/Python (Spark) and Core Java
Hands-on experience with Apache Kafka for streaming data pipelines
Working knowledge of AWS services (S3) and Cloudera platform tools (Hive, Iceberg)
Strong SQL skills, including CDC concepts, across Oracle and SQL Server
Experience with dimensional data warehouse modeling
Proficiency in Unix shell scripting and Python for automation
Experience migrating on-prem Hadoop workloads to cloud platforms
Understanding of JVM internals and Spark performance tuning techniques
Preferred Qualifications:
Prior experience in the Communications/Telecom domain (CDRs, network telemetry data)
Familiarity with data governance and regulatory compliance frameworks
Bachelor s degree in computer science, Engineering, or a related field, or equivalent practical experience
Top 3 responsibilities
Manage and maintain enterprise data lakehouse architectures spanning AWS and Cloudera environments, leveraging S3, Apache Iceberg, and Hive for scalable storage and querying. Implement Change Data Capture CDC and write complex SQL to integrate data from relational databases Oracle, SQL Server into dimensional data warehouses.
Automate and orchestrate pipeline workflows using Unix shell scripting and Python. Lead and support migration of legacy Hadoop workloads to AWS cloudnative architectures.
M
Work Location: Maryland Heights MO 63043 (hybrid )
Contract duration: 12
F2F interview - Yes
Job Details:
Must Have skills:
AWS, Hadoop
Key Responsibilities
Design, build, and optimize scalable batch and real-time data pipelines using Apache Spark (Scala/Python), Core Java, and Apache Kafka to process high-volume data.
Manage and maintain enterprise data Lakehouse architectures spanning AWS and Cloudera environments, leveraging S3, Apache Iceberg, and Hive for scalable storage and querying.
Implement Change Data Capture (CDC) and write complex SQL to integrate data from relational databases (Oracle, SQL Server) into dimensional data warehouses.
Automate and orchestrate pipeline workflows using Unix shell scripting and Python.
Lead and support migration of legacy Hadoop workloads to AWS cloud-native architectures.
Monitor, troubleshoot, and tune JVM and Spark job performance to ensure reliability and efficiency at scale.
Ensure all data pipelines and storage practices adhere to regulatory compliance and data governance standards.
Collaborate with cross-functional teams (data architects, analysts, platform engineers) to translate business requirements into robust data solutions.
Required Skills & Technology:
Strong programming experience in Scala/Python (Spark) and Core Java
Hands-on experience with Apache Kafka for streaming data pipelines
Working knowledge of AWS services (S3) and Cloudera platform tools (Hive, Iceberg)
Strong SQL skills, including CDC concepts, across Oracle and SQL Server
Experience with dimensional data warehouse modeling
Proficiency in Unix shell scripting and Python for automation
Experience migrating on-prem Hadoop workloads to cloud platforms
Understanding of JVM internals and Spark performance tuning techniques
Preferred Qualifications:
Prior experience in the Communications/Telecom domain (CDRs, network telemetry data)
Familiarity with data governance and regulatory compliance frameworks
Bachelor s degree in computer science, Engineering, or a related field, or equivalent practical experience
Top 3 responsibilities
Manage and maintain enterprise data lakehouse architectures spanning AWS and Cloudera environments, leveraging S3, Apache Iceberg, and Hive for scalable storage and querying. Implement Change Data Capture CDC and write complex SQL to integrate data from relational databases Oracle, SQL Server into dimensional data warehouses.
Automate and orchestrate pipeline workflows using Unix shell scripting and Python. Lead and support migration of legacy Hadoop workloads to AWS cloudnative architectures.
M
About Company
Looking for a great job? Join eTeam. We’re looking for talented staffing professionals to join our staff. We also provide contract assignments and full-time jobs at Fortune 2000 Companies. We’ve been named one of the best companies to work for by Staffing Industry Analysts and New Jersey Business.