Senior Data Engineer

Job Summary

We are seeking a highly skilled Data Engineer to join a team responsible for building and enhancing a large-scale decision platform that drives customer-focused business decisions. The ideal candidate will have strong expertise in Spark, Java, AWS, and large-scale data processing environments.

Key Responsibilities

• Design, develop, and optimize scalable data pipelines for ingesting and processing high-volume datasets.
• Build and maintain distributed data processing solutions using Apache Spark and Java.
• Process and manage 20M+ to 40M+ daily data records efficiently.
• Develop batch and real-time data processing workflows.
• Work with event-driven architectures and streaming platforms.
• Collaborate with cross-functional teams to enhance data platform capabilities.
• Implement and manage cloud-native solutions within AWS environments.
• Utilize Infrastructure as Code (IaC) methodologies using Terraform.

Required Skills

• Strong experience with Apache Spark (Must Have)
• Strong experience with Java (Must Have)
• Hands-on experience with AWS Cloud Services (Must Have)
• Experience with Terraform for Infrastructure as Code (Must Have)
• Experience building data ingestion and data transformation pipelines
• Experience working with large-scale distributed data environments
• Knowledge of real-time and batch data processing architectures

Preferred Skills

• Python
• Apache Kafka

Cloud & Technology Stack

• Apache Spark
• Java
• AWS EKS
• AWS EMR
• AWS S3
• AWS Aurora
• AWS MSK
• AWS SNS
• AWS SQS
• Apache Kafka
• Terraform

Ideal Candidate

• Strong background in data engineering and distributed computing.
• Experience handling high-volume data platforms.
• Comfortable working in cloud-native, event-driven architectures.
• Excellent problem-solving and analytical skills.