Data Engineer

We are seeking an experienced Data Engineer to design, build, and optimize scalable data pipelines and cloud-based data platforms. The ideal candidate will have strong expertise in Python, SQL, Spark, ETL processes, and cloud technologies to support data-driven decision-making and advanced analytics.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines and ETL/ELT workflows.
• Build and optimize data processing solutions using Python, SQL, and Apache Spark (PySpark).
• Develop batch and real-time data ingestion pipelines.
• Work with structured and unstructured datasets from multiple data sources.
• Design and implement cloud-based data solutions using Azure, AWS, or GCP.
• Develop and maintain data models, data lakes, and enterprise data warehouses.
• Integrate data from APIs, databases, and streaming platforms.
• Optimize data pipeline performance and ensure data quality.
• Collaborate with Data Scientists, Business Analysts, and Software Engineers.
• Implement CI/CD pipelines and DevOps best practices for data engineering.
• Troubleshoot production issues and provide ongoing support.
• Ensure compliance with data governance, security, and privacy standards.

Required Skills

• 5+ years of experience as a Data Engineer.
• Strong programming skills in Python.
• Excellent SQL skills.
• Hands-on experience with Apache Spark / PySpark.
• Experience with ETL/ELT development.
• Strong knowledge of Data Warehousing concepts.
• Experience with Hadoop ecosystem (Hive, HDFS).
• Experience with Databricks.
• Experience with Apache Kafka.
• Hands-on experience with Apache Airflow or similar workflow orchestration tools.
• Experience working on Linux/Unix environments.
• Knowledge of Git and CI/CD pipelines.

Cloud Skills

Experience with one or more of the following:

• Microsoft Azure (ADF, ADLS, Synapse)
• Amazon Web Services (AWS)
• Google Cloud Platform (GCP)

Preferred Skills

• Snowflake
• Delta Lake
• Docker
• Kubernetes
• Terraform
• Scala or Java
• Power BI or Tableau
• REST APIs
• Agile/Scrum methodology
• Experience working with large-scale enterprise data platforms