About the Role
We're hiring an exceptional Senior Data Engineer with deep data engineering expertise and strong, hands-on AI skills. You'll build scalable data infrastructure across ingestion, transformation, orchestration, and delivery collaborating with backend, BI, and ML teams while actively using AI tooling (Claude, Cursor, and beyond) as a core part of how you engineer and ship. Strong ownership mindset, data quality focus, on-call reliability, and a genuine drive to adopt modern AI tooling are a must.
Responsibilities
Design, build, and maintain scalable data ingestion and processing pipelines using SQL , Spark, Hadoop, Airflow, SnapLogic, or similar tools
Optimize existing ETL/ELT workflows and resolve bottlenecks in production jobs
Integrate multiple structured and unstructured data sources from internal and external systems
Design and implement event-driven data pipelines using event bus architectures (Kafka, AWS EventBridge, SNS/SQS, or similar)
Build and maintain event producers, consumers, and stream processors for real-time and near-real-time data flows
Manage event schema design, schema registry, and versioning to ensure backward compatibility across consumers
Monitor event bus health, lag, throughput, and reliability; respond to failures before SLA breach
Implement and manage CI/CD pipelines for data workflows using Git, Jenkins, or GitHub Actions
Ensure robust job scheduling, monitoring, logging, and end-to-end testing of data pipelines
Debug and resolve production issues before SLA breach; participate in on-call rotation
Collaborate with backend and BI teams for API and data integration
Design and develop schema definitions; support data warehouse/mart to integrate disparate data sources and make them available for analysis
Support large data volumes and flexible provisioning of new data sources
Contribute to data architecture design across the data landscape
Write clean, reusable, well-tested Python and Shell code across the full software development lifecycle
Gather functional requirements, develop technical specifications, and own project and test planning
Work in Agile/Scrum environments; help align team priorities across the organization
Leverage AI-assisted dev tools (Claude, Cursor .. etc) to accelerate delivery and code quality
Required Skills
Strong Python, Advanced SQL, PySpark or Scala, Unix scripting
Data orchestration: Airflow, SnapLogic, DBT, or equivalent
Event bus & streaming: Kafka, AWS EventBridge, SNS/SQS, Kinesis, or similar
Event-driven architecture: producers, consumers, schema registry, dead-letter queues
AWS cloud: EC2, EMR, S3, Athena, Glue; Databricks
Data lake & warehouse: Snowflake, Delta Lake, Redshift, BigQuery
CI/CD tooling: Git, Jenkins / GitHub Actions, Docker
Experience with APIs for data exchange and integration
Data quality mindset, ownership, and SLA awareness
NoSQL, Neo4j, GraphQL is plus
Agile/Scrum development experience
Preferred / Nice to Have
Experience with AI/ML models and feature pipelines
Exposure to BI platforms (Tableau, Power BI) or data serving for dashboards
Spark Streaming or Flink for stateful stream processing
Skills
SQL, AWS, PySpark, Data Engineering, Python PySpark, ETL & Data Analytics, Data Migration, Confluence, Jira