Data Engineer / Lead Data Engineer

Develop pipeline orchestration and scheduling using Airflow, Control-M, Databricks Workflows, Azure Data Factory, and AWS services . Design, develop, and maintain scalable ETL/ELT data pipelines for large-volume structured and unstructured data.

Job Title: Senior Data Engineer / Lead Data Engineer
Location: Hybrid , NJ
Employment Type: Full-Time

NEED LOCAL CANDIDATE, NY OR NJ

EXP 12+

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines for large-volume structured and unstructured data.
• Develop data processing solutions using Python, PySpark, Apache Spark, and SQL.
• Build and optimize data pipelines using Databricks, Azure Data Factory, AWS Glue, and Snowflake.
• Work with both Azure and AWS cloud platforms to implement modern data engineering solutions.
• Work with Delta Lake, Azure Data Lake, Amazon S3, Azure Blob Storage, Azure Synapse, and Snowflake.
• Develop pipeline orchestration and scheduling using Airflow, Control-M, Databricks Workflows, Azure Data Factory, and AWS services.
• Implement monitoring, logging, alerting, and troubleshooting processes for production data pipelines.
• Work with CI/CD processes using GitHub, Bitbucket, Jenkins, Docker, and Azure DevOps.

Required Skills

• 8+ years of experience in Data Engineering or related roles.
• Strong hands-on experience with Python, PySpark, Apache Spark, and SQL.
• Strong experience with Azure and/or AWS cloud environments.
• Experience with Databricks and Delta Lake.
• Strong knowledge of Azure Data Factory, Azure Synapse, Azure Data Lake/Blob Storage.
• Experience with AWS services such as S3, Glue, Athena, DMS, Lambda, SNS, SQS, and EventBridge.
• Strong experience with Snowflake and cloud data warehousing.
• Strong understanding of ETL/ELT, data warehousing, data modeling, and data integration.
• Experience working with Oracle, SQL Server, MySQL, Hive, and other relational databases.
• Strong programming and scripting experience using Python, SQL, and Shell scripting.
• Experience with Git, Bitbucket/GitHub, Jenkins, Docker, and Azure DevOps.
• .

Preferred Skills

• Experience with Kafka and real-time streaming.
• Experience with Airflow and Control-M.
• Knowledge of MLOps and machine-learning data pipelines.
• Experience with Kubernetes and containerized data workloads.
• Experience with API data integration and modernization.
• Knowledge of LLM/AI-assisted data engineering solutions.
• Experience with Power BI or Tableau for data reporting and analytics.
• Experience with data migration and reverse engineering of legacy data models.