Job description
Job Description: Senior Data Engineer with AI
About the Role
We're hiring an exceptional Senior Data Engineer with deep data engineering expertise and strong, hands-on AI skills. You'll build scalable data infrastructure across ingestion, transformation, orchestration, and delivery — collaborating with backend, BI, and ML teams — while actively using AI tooling (Claude, Cursor, and beyond) as a core part of how you engineer and ship. Strong ownership mindset, data quality focus, on-call reliability, and a genuine drive to adopt modern AI tooling are a must.
Responsibilities
· Design, build, and maintain scalable data ingestion and processing pipelines using SQL , Spark, Hadoop, Airflow, SnapLogic, or similar tools
· Optimize existing ETL/ELT workflows and resolve bottlenecks in production jobs
· Integrate multiple structured and unstructured data sources from internal and external systems
· Design and implement event-driven data pipelines using event bus architectures (Kafka, AWS EventBridge, SNS/SQS, or similar)
· Build and maintain event producers, consumers, and stream processors for real-time and near-real-time data flows
· Manage event schema design, schema registry, and versioning to ensure backward compatibility across consumers
· Monitor event bus health, lag, throughput, and reliability; respond to failures before SLA breach
· Implement and manage CI/CD pipelines for data workflows using Git, Jenkins, or GitHub Actions
· Ensure robust job scheduling, monitoring, logging, and end-to-end testing of data pipelines
· Debug and resolve production issues before SLA breach; participate in on-call rotation
· Collaborate with backend and BI teams for API and data integration
· Design and develop schema definitions; support data warehouse/mart to integrate disparate data sources and make them available for analysis
· Support large data volumes and flexible provisioning of new data sources
· Contribute to data architecture design across the data landscape
· Write clean, reusable, well-tested Python and Shell code across the full software development lifecycle
· Gather functional requirements, develop technical specifications, and own project and test planning
· Work in Agile/Scrum environments; help align team priorities across the organization
· Leverage AI-assisted dev tools (Claude, Cursor .. etc) to accelerate delivery and code quality
Required Skills
· Strong Python, Advanced SQL, PySpark or Scala, Unix scripting
· Data orchestration: Airflow, SnapLogic, DBT, or equivalent
· Event bus & streaming: Kafka, AWS EventBridge, SNS/SQS, Kinesis, or similar
· Event-driven architecture: producers, consumers, schema registry, dead-letter queues
· AWS cloud: EC2, EMR, S3, Athena, Glue; Databricks
· Data lake & warehouse: Snowflake, Delta Lake, Redshift, BigQuery
· CI/CD tooling: Git, Jenkins / GitHub Actions, Docker
· Experience with APIs for data exchange and integration
· Data quality mindset, ownership, and SLA awareness
· NoSQL, Neo4j, GraphQL is plus
· Agile/Scrum development experience
Preferred / Nice to Have
· Experience with AI/ML models and feature pipelines
· Exposure to BI platforms (Tableau, Power BI) or data serving for dashboards
· Spark Streaming or Flink for stateful stream processing
Skills
Data Engineer, Spark, AWS, Airflow, ETL, Amazon Web Services Datalake, Python PySpark, SNS, SQS, Confluence, Jira