We are seeking an experienced Data Engineer to design, build, and optimize scalable data pipelines and cloud-based data platforms. The ideal candidate will have strong expertise in Python, SQL, Spark, ETL processes, and cloud technologies to support data-driven decision-making and advanced analytics.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines and ETL/ELT workflows.
- Build and optimize data processing solutions using Python, SQL, and Apache Spark (PySpark).
- Develop batch and real-time data ingestion pipelines.
- Work with structured and unstructured datasets from multiple data sources.
- Design and implement cloud-based data solutions using Azure, AWS, or GCP.
- Develop and maintain data models, data lakes, and enterprise data warehouses.
- Integrate data from APIs, databases, and streaming platforms.
- Optimize data pipeline performance and ensure data quality.
- Collaborate with Data Scientists, Business Analysts, and Software Engineers.
- Implement CI/CD pipelines and DevOps best practices for data engineering.
- Troubleshoot production issues and provide ongoing support.
- Ensure compliance with data governance, security, and privacy standards.
Required Skills
- 5+ years of experience as a Data Engineer.
- Strong programming skills in Python.
- Excellent SQL skills.
- Hands-on experience with Apache Spark / PySpark.
- Experience with ETL/ELT development.
- Strong knowledge of Data Warehousing concepts.
- Experience with Hadoop ecosystem (Hive, HDFS).
- Experience with Databricks.
- Experience with Apache Kafka.
- Hands-on experience with Apache Airflow or similar workflow orchestration tools.
- Experience working on Linux/Unix environments.
- Knowledge of Git and CI/CD pipelines.
Cloud Skills
Experience with one or more of the following:
- Microsoft Azure (ADF, ADLS, Synapse)
- Amazon Web Services (AWS)
- Google Cloud Platform (GCP)
Preferred Skills
- Snowflake
- Delta Lake
- Docker
- Kubernetes
- Terraform
- Scala or Java
- Power BI or Tableau
- REST APIs
- Agile/Scrum methodology
- Experience working with large-scale enterprise data platforms