Senior Data Platform / Data Product Engineering Lead

Senior Data Platform / Data Product Engineering Lead

Must Have Technical/Functional Skills

Job Description: Senior Data Platform / Data Product Engineering Lead

Role Overview

We are looking for a Senior Data Platform / Data Product Engineering Lead to drive enterprise-scale data product lifecycle enablement across modern data platforms. This role will lead the design, standardization, and adoption of a paved path for data creators, enabling self-service, governed, and scalable data product development.

The role requires deep expertise in Databricks, Airflow (Astronomer), CI/CD automation, data governance, and data marketplace constructs, along with the ability to lead platform transformation initiatives and mentor engineering teams.

____

Required Skills & Experience

• 10+ years of experience in Data Engineering / Data Platform roles
• Strong hands-on expertise in:

o Databricks (Delta Lake, workflows, DAG)

o Apache Airflow / Astronomer

o Python, SQL, DBT ,AWS

• Proven experience implementing CI/CD frameworks (Harness, GitHub Actions, Azure DevOps)
• Deep understanding of:

o Data governance (catalogs, lineage, contracts, metadata)

o Data quality and masking techniques

o Enterprise data platforms and marketplace ecosystems

• Experience with API-based integrations (e.g., entitlement systems like AccessCentral)
• Monitoring/observability tools (e.g., Datadog)

Job Description: Senior Data Platform / Data Product Engineering Lead

Role Overview

We are looking for a Senior Data Platform / Data Product Engineering Lead to drive enterprise-scale data product lifecycle enablement across modern data platforms. This role will lead the design, standardization, and adoption of a paved path for data creators, enabling self-service, governed, and scalable data product development.

The role requires deep expertise in Databricks, Airflow (Astronomer), CI/CD automation, data governance, and data marketplace constructs, along with the ability to lead platform transformation initiatives and mentor engineering teams.

____

Core Responsibilities

1. Platform Strategy & Self-Service Enablement

• Define and implement a self-service data platform strategy to reduce onboarding friction.
• Lead automated provisioning of:

o Databricks workspaces (via DevHub)

o Airflow/Astronomer environments

o Access and entitlements (AccessCentral APIs)

• Establish isolated, stable development environments for federated teams.
• Drive platform observability by integrating metrics into tools lik e Datadog.

____

2. Data Discovery, Access & Governance

• Architect and implement enterprise-wide data discovery and marketplace enablement.
• Drive adoption of:

o Data contracts

o Metadata standards

o Domain-aligned catalogs (Unity Catalog)

• Enable secure access to curated, masked datasets in dev and production environments.
• Implement tagging, access patterns, and entitlement automation.
• Partner with risk/compliance teams to enforce regulatory and governance controls (BFSI-aligned).

____

3. Data Engineering, Curation & Orchestration

• Lead design of scalable data ingestion, curation, and transformation frameworks.
• Build and standardize modular, reusable frameworks:

o LaunchLake templates

o Airflow DAG libraries

o DBT-based transformation models

• Ensure:

o Data quality and consistency

o Embedded governance and compliance policies

• Enable concurrent development using standardized patterns and environments.

____

4. CI/CD, Automation & Deployment

• Define and enforce standard CI/CD pipelines across data products:

o Harness (or equivalent)

o Databricks Asset Bundles (DAB)

• Automate:

o DAG deployments (Airflow/Astronomer)

o DBT pipeline releases

• Reduce manual interventions and ensure consistent, repeatable deployments.
• Improve release reliability with feedback loops, notifications, and monitoring.

____

5. Data Product Publishing & Marketplace Enablement

• Drive publishing of data products to:

o Unity Catalog

o Enterprise Data Marketplace

• Define and enforce:

o Documentation standards

o Data ownership models

o Versioning and contract management

• Enable cross-domain data sharing with embedded governance and access controls.

____

6. Operations, Observability & Reliability

• Establish a scalable operating model for data product support.
• Implement:

o Monitoring dashboards (Datadog)

o Data quality frameworks

o Usage and performance metrics tracking

• Improve visibility into:

o Pipeline health

o Data lineage

o Access and consumption patterns

• Lead incident management, root cause analysis, and escalation processes.

____

7. Transformation, Roadmap & Innovation

• Drive execution of platform priorities such as:

o Data contract activation strategy

o Domain catalog integration

o Data masking in development environments

o Data quality frameworks

o DBT adoption and POCs

• Lead maturity uplift from:

o Manual, fragmented workflows standardized, automated paved paths

• Champion continuous improvement and innovation in developer experience.

____

Salary Range- $100,000-$120,000 a year