Databricks Platform Engineering Lead / Technical Delivery Lead
Warren, NJ
Project Overview
We are seeking an experienced
Databricks Platform Engineering Lead
to support an enterprise-wide pricing application within a global reinsurance organization. The application leverages Databricks as its data and analytics platform and integrates with a SQL Server-based pricing application.
The resource will lead the engineering, governance, and ongoing enhancement of the Databricks platform supporting the pricing application. The role requires a strong combination of hands-on platform engineering, Databricks administration, data engineering, and technical delivery leadership.
The ideal candidate must be capable of managing the Databricks platform foundation while providing technical oversight to data-engineering teams building ETL/ELT pipelines, transformations, and pricing data products.
Technical and Delivery Expertise
10+ years of overall technology experience.
5+ years of hands-on experience with Databricks, cloud data platforms, or platform engineering.
3+ years of experience in a Platform Lead, Databricks Lead, Cloud Platform Engineer, Data Platform Lead, or similar role.
Strong hands-on experience with:
Azure Databricks
Databricks account and workspace administration
Unity Catalog
Apache Spark and PySpark
Python and SQL
Delta Lake and Lakehouse architecture
Azure Data Lake Storage
Azure Data Factory or comparable integration tools
Databricks Workflows and job orchestration
Infrastructure as code using Terraform
CI/CD and automated deployment practices
Strong understanding of:
Databricks workspace provisioning and administration
Development, test, and production environment management
Compute provisioning and cluster configuration
Cluster policies, pools, runtimes, and serverless compute
Platform security, identity, and access management
Service principals, managed identities, and secrets management
Azure networking, private endpoints, and secure connectivity
Unity Catalog architecture, permissions, lineage, and governance
Platform monitoring, logging, auditability, and observability
Databricks performance, capacity, and cost optimization
Production support and platform-incident management
Good understanding of data-engineering practices, including:
Batch and streaming data ingestion
ETL/ELT pipeline development
Medallion architecture
Data transformations and enrichment
Data quality, validation, and reconciliation
Schema management and data lineage
Spark workload and query optimization
Experience leading globally distributed platform, data-engineering, application, QA, DevOps, and production-support teams.
Ability to participate in architecture reviews, solution-design discussions, code reviews, security assessments, and production-readiness evaluations.
Strong understanding of Agile delivery, release management, risk management, dependency management, and stakeholder governance.
Preferred Qualifications
Prior experience implementing a Databricks-enabled pricing, underwriting, actuarial, insurance analytics, or financial-services application.
Experience working with global insurance or reinsurance carriers.
Understanding of reinsurance pricing, underwriting, treaty, and facultative business processes.
Experience integrating Databricks with:
Pricing engines
SQL Server applications
Policy, claims, and accounting systems
Data warehouses and reporting platforms
Internal and third-party data sources
Experience establishing reusable Databricks platform standards, engineering frameworks, and reference architectures.
Experience implementing a hybrid solution of Databricks and SQL
Experience defining the division of responsibility among platform engineering, data engineering, application development, DevOps, data science, and production-support teams.
Familiarity with MLflow, model lifecycle management, model serving, or actuarial-model integration.
Experience implementing data governance, metadata management, lineage, role-based access, audit controls, and regulatory controls.
Familiarity with Azure DevOps, Terraform, Power BI, Microsoft Purview, Azure Monitor, or comparable tools.
Databricks and Microsoft Azure certifications are preferred.
Experience supporting business-critical production platforms with defined availability, recovery, security, and performance requirements.
The Resource Will Be Expected To:
Own the engineering and operational management of the Databricks platform supporting the enterprise pricing application.
Lead end-to-end technical delivery of platform enhancements, environment changes, integrations, releases, and production improvements.
Provision and administer Databricks accounts, workspaces, compute resources, clusters, policies, and supporting Azure services.
Define and enforce platform standards covering architecture, security, governance, compute, deployment, monitoring, and cost management.
Design and maintain appropriate separation across development, test, UAT, and production environments.
Implement infrastructure as code and automated deployment practices for Databricks and supporting cloud resources.
Establish secure connectivity between Databricks, SQL Server, cloud storage, the pricing engine, the underwriting workbench, and other enterprise systems.
Configure and maintain identity, access, security, secrets, networking, and governance controls.
Define Unity Catalog structures, access models, data ownership, lineage, and auditability standards.
Establish cluster policies, runtime standards, compute controls, and workload-management practices.
Monitor and optimize platform availability, reliability, performance, capacity, and cost.
Troubleshoot workspace, authentication, networking, compute, storage, access, and platform-performance issues.
Lead root-cause analysis and corrective actions for platform incidents and recurring operational problems.
Provide technical oversight to data engineers building ingestion pipelines, transformations, curated datasets, and pricing data products.
Review pipeline designs, Spark workloads, data-processing patterns, deployment approaches, and production-readiness plans.
Ensure that data-engineering solutions comply with platform, security, governance, quality, and performance standards.
Coordinate platform engineering, data engineering, application development, QA, DevOps, infrastructure, security, and production-support teams.
Independently identify technical and delivery impacts across Databricks, the SQL Server pricing engine, the underwriting workbench, and integrated systems.
Support successful SIT, UAT, regression testing, release validation, deployment, cutover, and post-production stabilization.
Communicate technical risks, delivery dependencies, architectural decisions, and recovery plans to business and technology stakeholders.
Serve as the primary technical point of contact for the Databricks platform and its integration with the enterprise pricing application.
Serve as a subject matter expert on Databricks platform engineering, administration, governance, security, reliability, and operational best practices.
The pay range for this role is
***k - ***k
per annum including any bonuses or variable pay. *** also offers benefits like medical, vision, dental, life, disability insurance and paid time off (including holidays, parental leave, and sick leave, as required by law). Ask our recruiters for more details on our Benefits package. The exact offer terms will depend on the skill level, educational qualifications, experience, and location of the candidate.
Thanks
Regards
*** is an Equal Employment Opportunity employer. We promote and support a diverse workforce at all levels of the company. All qualified applicants will receive consideration for employment without regard to race, religion, color, sex, age, national origin or disability. All applicants will be evaluated solely on the basis of their ability, competence, and performance of the essential functions of their positions with or without reasonable accommodations. Reasonable accommodations also are available in the hiring process for applicants with disabilities.
Candidates can request a reasonable accommodation by contacting the company ADA Coordinator at
*** .