Kafka Engineer/ Admin

Deep knowledge of Kafka ecosystem components like ZooKeeper, Kafka Connect, Schema Registry, MirrorMaker, KRaft (Kafka without ZooKeeper). Working experience with OpenShift Kubernetes Cloud services to deploy Kafka clusters using vendor provided helm charts & docker images.

Job Summary:
The ideal candidate will be responsible for designing, deploying, maintaining, and scaling Kafka clusters in mission-critical environments, while also supporting the Linux-based infrastructure that forms the foundation of our real-time data platform.

 

Responsibilities

• Design, implement, and maintain highly available Kafka clusters in production environments.
• Perform Kafka upgrades and patching ensuring minimal downtime and seamless transitions.
• Administer security protocols and access controls to protect the platform.
• Monitor cluster health and performance using our established monitoring stack (Prometheus, Grafana)
• Troubleshoot and resolve cluster-related incidents such as broker outages, replication lag.
• Develop and enhance automation scripts to streamline operational tasks and reduce manual intervention.
• Assist with planning and managing cluster resources to support future growth.
• Develop Kafka topics, partitions, ACLs, schema registry, and optimize throughput and reliability.
• Manage a tune of Kafka Connect, Kafka Streams, and Mirror Maker as needed for data pipelines.
• Ensure platform reliability and performance on enterprise-grade Linux systems (RHEL, Ubuntu).
• Automate Kafka provisioning, configuration, and monitoring using Infrastructure as Code (IaC) and configuration management tools (e.G., Ansible, Terraform).
• Deep knowledge of Kafka ecosystem components like ZooKeeper, Kafka Connect, Schema Registry, MirrorMaker, KRaft (Kafka without ZooKeeper).
• Build custom Kafka tooling and dashboards for producers and consumers, monitor lag and health metrics.
• Work with engineering teams to onboard new Kafka use cases and ensure security and compliance standards.
• Perform root cause analysis and resolution of platform-level incidents and outages.
• Develop and maintain Linux shell scripts for automation of deployments, monitoring, and maintenance tasks.

 

Qualifications we seek in you!

 

Qualifications

• BE/B Tech/MCA
• Excellent written and verbal communication skills

 

Qualifications/ Skills

• Working experience with OpenShift Kubernetes Cloud services to deploy Kafka clusters using vendor provided helm charts & docker images
• Kafka Resiliency dc/dr setup & observability for Kafka nodes
• Data ingestion & streaming pipeline setup for other data platforms to Kafka & Kafka to data sources like Solr
• Five or more years of Kafka administration experience, ideally with Confluent Kafka
• Proficiency in Linux, Python, or Bash scripting
• Hands-on knowledge of Kafka monitoring tools and experience with security protocols.