Location: New York(Hybrid)
Employment Type: Contract
Experience: 5 10 Years
Platform Experience: 3 5+ Years Hands-On Databricks Administration on AWS
Job Summary
We are seeking a Databricks Platform Manager / Engineer to design, administer, secure, monitor, and optimize an enterprise Databricks platform running on AWS. This role focuses on platform engineering and administration, ensuring scalability, security, governance, reliability, automation, and cost efficiency for data engineering, analytics, and AI teams.
Key Responsibilities
- Administer Databricks workspaces across Development, Test, UAT, and Production.
- Establish workspace standards, networking, environment isolation, and platform configurations.
- Manage Databricks Runtime upgrades and platform lifecycle activities.
- Configure cluster policies, instance pools, job clusters, and compute standards.
- Integrate Databricks with AWS IAM and manage authentication and authorization.
- Administer Unity Catalog, catalogs, schemas, tables, permissions, and RBAC.
- Manage service principals, PATs, secret scopes, and enterprise access controls.
- Configure Git integration and support CI/CD for Databricks notebooks, workflows, and infrastructure.
- Implement Infrastructure as Code using Terraform where applicable.
- Monitor workspace health, jobs, clusters, platform performance, and operational alerts.
- Troubleshoot production issues, perform RCA, and support platform releases.
- Drive Databricks compute optimization, auto-scaling, job scheduling, and cloud cost reduction.
- Maintain audit logging, security controls, operational documentation, and platform standards.
Required Qualifications
- 5 10 years of overall IT experience.
- 3 5+ years of hands-on Databricks platform administration/engineering in production AWS environments.
- Strong experience managing enterprise Databricks workspaces.
- Experience with cluster management, cluster policies, instance pools, and job clusters.
- Hands-on experience with Unity Catalog and RBAC.
- Strong experience with AWS IAM, S3, VPC, EC2, CloudWatch, KMS, and Security Groups.
- Experience with Databricks Runtime management and platform upgrades.
- Experience with Git, CI/CD, Databricks Repos, and Terraform/IaC.
- Experience with platform monitoring, incident management, troubleshooting, and production support.
- Experience with secrets management and audit logging.
- Strong understanding of Databricks security, governance, and cost optimization.
Preferred Qualifications
- Experience managing large-scale Databricks environments with 100+ users.
- Experience supporting multiple business units or enterprise data platforms.
- Strong hands-on Terraform experience.
- Knowledge of Lakehouse Architecture and Delta Lake.
- Experience with FinOps and cloud cost optimization.
- Knowledge of enterprise data governance and security frameworks.
- Experience with AWS Organizations, Control Tower, PrivateLink, or VPC endpoints.
- Experience with CloudWatch, Datadog, or Splunk.
- Exposure to Apache Spark internals and AI/ML workloads on Databricks.
Key Skills
Databricks Platform Administration, Databricks AWS, Workspace Administration, Cluster Management, Cluster Policies, Unity Catalog, AWS IAM, Databricks Runtime, Databricks Repos, CI/CD, Git, Terraform, Infrastructure as Code, Production Support, Platform Monitoring, Security, Governance, Cost Optimization
One-Line Summary
Databricks Platform Engineering AWS Unity Catalog Cluster Management AWS IAM Terraform CI/CD Git Platform Security Monitoring Production Support Cost Optimization