Kafka/Flink Data Streaming Platform Engineer

We are seeking a talented and experienced Data Streaming Platform Engineer or DevOps Engineer with expertise in large cloud deployments, Kubernetes, Terraform/Ansible, Apache Kafka, and Apache Flink to join our team. In this role, you will be responsible for automating the build and maintenance scalable, high-performance data-streaming platforms that power real-time data processing and analytics.

You will work closely with data engineers, software developers, and infrastructure teams to ensure seamless data ingestion, processing, and delivery. The ideal candidate has a strong background in data-streaming technologies, cloud infrastructure, and DevOps practices, deep expertise with Kubernetes, and hands-on experience in Kafka and Flink.

Key Responsibilities:

• Automate and Implement Data Streaming Solutions:

  • Automate, deploy, and manage data streaming platforms using IaC tools such as Terraform and Ansible, onto Kubernetes or other cluster solutions

  • Develop and optimize real-time data pipelines for event-driven applications and stream processing

  • Configure and optimize Kafka Connectors for data pipelines

  • Build and improve observability tools for large data processing pipelines, perform benchmark tests and optimize performance

  • Ensure high availability, fault tolerance, and scalability of data streaming systems

• Stream Processing and Data Pipelines:

  • Implement stream processing workflows using Apache Kafka, Apache Spark and Apache Flink for real-time analytics, transformations, and aggregations

  • Integrate Kafka with Flink and other data processing frameworks to build end-to-end data pipelines

  • Monitor and optimize the performance of streaming applications and data flows

• Infrastructure as Code (IaC) and Automation:

  • Implement and maintain infrastructure as code using tools like Terraform, Ansible, or CloudFormation

  • Automate deployment, scaling, and monitoring of Kafka clusters, Flink jobs, and related services

  • Manage CI/CD pipelines for seamless integration and delivery of data streaming applications

• Monitoring and Performance Optimization:

  • Build and configure monitoring, logging, and alerting systems (e.g., Prometheus, Grafana, ELK Stack) for Cloud, Kubernetes, Kafka and Flink ecosystems

  • Troubleshoot and resolve issues related to data streaming platforms, including latency, throughput, and resource utilization

  • Optimize Kafka brokers, Flink job managers, and task managers for performance and scalability

• Collaboration and Support:

  • Work closely with data engineers, software developers, and data scientists to understand data requirements and deliver robust solutions

  • Provide technical guidance and support to teams using Kafka and Flink for real-time data processing

  • Document system architecture, processes, and best practices

• Cloud and On-Premise Infrastructure Management:

  • Manage and optimize cloud-based infrastructure (AWS, GCP, Azure) or on-premise environments for Kafka and Flink deployments

  • Ensure security, compliance, and cost-effectiveness of infrastructure

Qualifications:

• Education:

  • Bachelors or Masters degree in Computer Science, Engineering, or a related field

• Experience:

  • 3+ years of hands-on experience building large scale solutions on Kubernetes

  • 5+ years of hands-on experience with Unix/Linux systems and a variety of internet protocols

  • 2+ years of hands-on experience with Apache Kafka (e.g., cluster management, topic design, Kafka Connect, Kafka Streams)

  • 1+ years of experience with Apache Flink (e.g., stream processing, stateful computations, Flink SQL)

  • 3+ years of experience in DevOps, cloud infrastructure, or platform engineering

  • Proficiency in programming languages such as Java, Python, or Scala

  • Strong understanding of distributed systems, event-driven architectures, and microservices

• Technical Skills:

  • Experience with Kafka Streams or KSQL for stream processing

  • Knowledge of data serialization formats (e.g., Avro, Protobuf, JSON)

  • Familiarity with cloud-based data streaming services (e.g., AWS MSK, Confluent Cloud, Google Pub/Sub)

  • Experience with big data technologies (e.g., Hadoop, Cassandra, Elasticsearch)

  • Knowledge of infrastructure as code tools (e.g., Terraform, Ansible)

• Soft Skills:

  • Strong problem-solving skills and attention to detail

  • Excellent communication and collaboration skills

  • Ability to work in a fast-paced, agile environment

  • Passion for learning and staying up-to-date with emerging technologies

What We Offer:

• Competitive salary and benefits package

• Opportunities for professional growth and development

• A collaborative and innovative work environment

• The opportunity to work with a variety of clients in various industries, solving hard problems in data & platform engineering

• Flexible working hours and remote work options