Technical Lead - Data Engineering
Role Overview
We are looking for a highly skilled Tech Lead - Data Engineering to guide our engineering team in building and scaling a robust, modern data platform. In this role, you will be the bridge between architectural blueprints and engineering execution. You will remain deeply hands-on with our core stack-Databricks, dbt, and Apache Airflow-while mentoring engineers, conducting rigorous code reviews, and ensuring the delivery of high-quality data products.
Key Responsibilities
Technical Leadership & Delivery
- Lead a team of data engineers to execute sprint goals, managing code quality and delivery timelines.
- Act as the subject matter expert (SME) for the team on Databricks, dbt, and Airflow best practices.
- Enforce engineering standards, including code modularity, documentation, and version control.
- Conduct comprehensive code reviews to ensure scalability, security, and performance.
️ Hands-on Engineering & Optimization
- Build and maintain production-grade data pipelines using PySpark, Delta Live Tables (DLT), and Spark SQL on Databricks.
- Develop complex dbt models, custom macros, and tests to transform raw data into analytics-ready layers.
- Author and schedule sophisticated Apache Airflow DAGs, utilizing dynamic task mapping and custom operators.
- Optimize pipeline performance by troubleshooting bottlenecked Spark jobs, Z-Ordering Delta tables, and tuning Airflow schedulers.
DataOps & Governance
- Implement and manage CI/CD pipelines for data deployments using toolsets like GitHub Actions or Azure DevOps.
- Enforce data governance policies, access controls, and lineage tracking via Databricks Unity Catalog.
- Integrate automated data quality testing and alerting mechanisms across dbt and Airflow workflows.
Mentorship & Collaboration
- Mentor and upskill junior and mid-level data engineers through pair programming and workshops.
- Translate architectural designs into actionable, granular engineering tasks and JIRA tickets.
- Collaborate closely with data architects, product managers, and downstream data analysts.
Required Qualifications:
- 10+ years of professional experience in data engineering and backend software development.
- 2+ years of experience in a technical lead, team lead, or mentoring capacity.
Technical Proficiencies
- Databricks: Strong hands-on experience with Delta Lake, Unity Catalog, and optimizing PySpark workloads.
- dbt: Proficiency with dbt Core or Cloud, including advanced macros, packages, and custom testing.
- Airflow: Solid experience writing complex, rel iable DAGs, handling task failures, and managing dependencies.
- Languages: Advanced proficiency in Python and expert-level SQL writing.
- Cloud Platforms: Hands-on experience deploying data solutions on at least one cloud provider (AWS, Azure, or GCP).
Leadership Skills
- Proven ability to guide engineering sprint velocity and resolve technical blockers for a team.
- Excellent verbal and written communication skills to articulate technical trade-offs.
Salary Range- $130,000-$200,000 a year