Data Engineering (Palantir, Spark, PySpark, Python)- W2 Role

This role requires strong expertise in Palantir Foundry, Spark/PySpark, SQL, and Python, along with the ability to lead engineering teams and partner with business stakeholders across supply chain, merchandising, and store operations. We are looking for a hands-on Technical Lead in Data Engineering to drive the design, development, and delivery of scalable data solutions in a large retail enterprise.

Job Title: Technical Lead – Data Engineering (Palantir, Spark, PySpark, Python)

Please dont apply if you dont have PALANTIR experience

Onsite work :One or two days in a month
W2 role
OPT with 7 years exp is fine


Job Summary


We are looking for a hands-on Technical Lead in Data Engineering to drive the design, development, and delivery of scalable data solutions in a large retail enterprise. This role requires strong expertise in Palantir Foundry, Spark/PySpark, SQL, and Python, along with the ability to lead engineering teams and partner with business stakeholders across supply chain, merchandising, and store operations.


Key Responsibilities

•  Lead the design and implementation of scalable data platforms and pipelines using PySpark, Spark, and Python
•  Drive adoption and best practices for Palantir Foundry (data pipelines, ontology, workflows, and operational applications) 
•  Architect and optimize high-performance data processing solutions for large-scale datasets 
•  Provide technical leadership and mentorship to data engineers, ensuring code quality and best practices 
•  Collaborate with cross-functional teams (business, analytics, data science) to translate requirements into scalable solutions 
•  Design robust data models, ETL/ELT frameworks, and data integration strategies 
•  Ensure data quality, governance, security, and compliance across enterprise data platforms 
•  Lead performance tuning, troubleshooting, and optimization of data pipelines 
•  Drive CI/CD implementation, code reviews, and release management 
•  Stay current with emerging data engineering technologies and recommend improvements 

Required Qualifications

•  Bachelor’s or Master’s degree in Computer Science, Engineering, or related field 
•  8+ years of experience in data engineering, with at least 2–3 years in a technical leadership role
•  Strong hands-on experience with: 
  • Python & PySpark
  • Apache Spark
  •  Advanced SQL
•  Experience with Palantir Foundry or similar modern data platforms 
•  Deep understanding of data engineering principles (ETL/ELT, data modeling, distributed systems) 
•  Experience designing and managing large-scale data architectures
•  Strong leadership, communication, and stakeholder management skills 

Preferred Qualifications

•  Experience in retail domain (supply chain, inventory, merchandising, store analytics) 
•  Experience with cloud platforms (Azure, AWS, or GCP) 
•  Familiarity with orchestration tools (Airflow, Azure Data Factory, etc.) 
•  Experience with real-time/streaming data pipelines
•  Exposure to DevSecOps practices 

Key Skills

•  Technical Leadership 
•  Data Engineering Architecture 
•  PySpark & Spark 
•  Python Programming 
•  SQL Optimization 
•  Palantir Foundry 
•  Data Modeling & Warehousing 
•  CI/CD & DevOps

Flexible work from home options available.