Senior Cloud Engineer, Databricks Platform - Hybrid or Remote

Genesis10 is currently seeking a Senior Cloud Engineer, Databricks Platform - for a Hybrid or Remote position with a Major Insurance Company located in Warren, NJ. This is a 5 month contract opportunity.

We are seeking a Senior Cloud Engineer to design, implement, and administer the Azure infrastructure behind our enterprise data and AI platform. This is a cloud infrastructure and platform engineering position centered on Azure Databricks, ADLS Gen2, Azure Data Factory, and Azure Machine Learning.
You will own the platform foundation — infrastructure as code, network and identity architecture, compute governance, observability, and cost management — and partner closely with our Databricks administration, data engineering, and data science teams. Our group defines the platform standards and reference architectures those teams build within, which means depth in Databricks administration matters here as much as depth in Azure. You should be equally at home provisioning a workspace through Terraform and advising on Unity Catalog design, cluster policy, or identity federation.
This role suits an engineer who enjoys being the technical reference point for a platform: setting the architectural direction, partnering with adjacent teams on complex problems, and translating business requirements into the platform capabilities needed to support them. 

Part One — Infrastructure Ownership:

• Design, implement, and administer the Azure foundation the platform runs on. 
• Infrastructure as Code (Terraform-first): Author and maintain the reusable Terraform modules that provision the Azure data estate — Databricks workspaces, ADLS Gen2, Data Factory, Key Vault, networking, and Azure ML. Own remote state, module versioning, drift detection, and the pipelines that deploy them.
• Network & Security Architecture: Design and implement the private networking and security posture for the platform — private endpoints, hub-and-spoke topology, VNet injection, NSGs and firewall rules, managed identities, Key Vault-backed secret scopes, RBAC, and data exfiltration controls.
• Data Lake Infrastructure: Own ADLS Gen2 at the infrastructure layer: storage account architecture, hierarchical namespace, container and ACL strategy, lifecycle and tiering policy, encryption and key management, and access patterns for Databricks and downstream consumers.
• Data Factory Platform Operations: Own ADF from the platform side — integration runtime provisioning and scaling (self-hosted and managed VNet), linked service credential and managed identity design, environment promotion, deployment automation, and monitoring. Pipeline development sits with the data engineering team.
• AI/ML Platform Infrastructure: Provision and operate the infrastructure supporting machine learning workloads: Azure ML workspaces and compute clusters, GPU quota and capacity planning, MLflow model registry integration, model serving endpoints, and the identity and networking connecting them to Unity Catalog.
• Observability & Reliability: Build and maintain the monitoring layer — Azure Monitor, Log Analytics, diagnostic settings, alerting, and operational runbooks across the estate.
• FinOps & Capacity Planning: Own cost visibility and optimization across DBU and storage spend — tagging strategy, chargeback and showback, budget alerts, reserved capacity, and forward capacity planning.
• CI/CD & Environment Management: Own the deployment pipelines and promotion path across development, test, and production environments.

Part Two — Databricks Platform Administration:

• Day-to-day Databricks operations are supported by a dedicated administration team. This role establishes the platform standards and reference configurations that team works from, and serves as a senior technical partner on architecture, design decisions, and complex escalations. That partnership requires genuine administrative depth across the following areas. 
• Account & Workspace Administration: Account console configuration, workspace provisioning and topology decisions, workspace-level settings, admin role delegation, and multi-workspace strategy.
• Unity Catalog: Metastore design and regional strategy, catalog/schema/table permission models, storage credentials and external locations, Hive metastore migration, lineage and audit, and Delta Sharing configuration.
• Identity & Access: Entra ID integration, SCIM provisioning, identity federation, groups and entitlements, service principals, and token policy.
• Compute Governance: Cluster policies, instance pools, node type and runtime standards, autoscaling and autotermination enforcement, Photon and serverless evaluation, and SQL warehouse sizing and configuration.
• Cost & Usage Analysis: System tables, usage attribution, and DBU forecasting — with the ability to assess a workspace, identify where spend is concentrated, and recommend a remediation path.
• Technical Partnership: Review platform configurations, contribute to design decisions, and support the administration and data teams on issues that span infrastructure and Databricks. Publish the standards, reference configurations, and self-service patterns that allow those teams to move quickly and safely.
• Cross-Boundary Diagnostics: Investigate job failures, cluster startup issues, permission errors, connectivity faults, and performance concerns, and determine whether the root cause sits in infrastructure, Databricks configuration, or the workload itself. 

Role Scope:

• To set clear expectations about how this position divides from adjacent teams: 
• Production ETL/ELT pipeline development and Spark transformation work sit with the data engineering team.
• Model development, training, and tuning sit with the data science team.
• Dimensional modeling, dbt development, and BI/semantic layer work sit outside this role.
• Day-to-day Databricks administration is handled by a dedicated team; this role provides the standards, architecture, and senior technical partnership behind it.

Required Qualifications:

• Experience: 6+ years in cloud infrastructure or platform engineering, with at least 4 years focused on Microsoft Azure.
• Terraform: Expert-level, hands-on. You have authored and maintained production Terraform modules, managed remote state, and run infrastructure as code through CI/CD. This is a core daily skill in this role.
• Databricks Administration: Demonstrable hands-on administration experience — account and workspace administration, Unity Catalog, cluster policies, identity federation, and cost governance. You should be able to describe administrative decisions you have made and explain the tradeoffs behind them.
• Technical Partnership: A track record as a senior technical resource to adjacent teams, including guiding stakeholders from an initial request to the underlying requirement and building consensus around the recommended approach.
• Azure Networking & Security: Deep working knowledge of private endpoints, VNets and hub-spoke design, NSGs, Entra ID, managed identities, RBAC, and Key Vault.
• Data Lake Architecture: ADLS Gen2 design and access control at enterprise scale.
• Azure Data Factory: Platform-side experience with integration runtimes, managed VNet, credential management, and deployment automation.
• Automation & Scripting: Strong Python, plus PowerShell and/or Bash, for platform tooling and automation.
• Spark & SQL Literacy: Sufficient working knowledge to diagnose infrastructure and configuration-level performance issues and engage credibly with data teams. Spark development depth is not a requirement for this role.
• Only candidates available and ready to work directly as Genesis10 employees will be considered for this position.

Preferred Qualifications:

• Databricks Asset Bundles, the Terraform Databricks provider, and workspace-as-code patterns
• Experience leading a Unity Catalog migration or multi-workspace consolidation
• Azure Machine Learning, MLflow, or model-serving infrastructure experience
• Kubernetes / AKS and containerized workloads
• Policy as code — Azure Policy, OPA, Sentinel, or Checkov
• Event-driven infrastructure — Event Hubs, Kafka, or Stream Analytics
• Formal FinOps practice experience
• Multi-region or multi-tenant Databricks deployments

Preferred Certifications:

• Databricks Certified Data Engineer Professional, or a Databricks platform administrator credential
• Microsoft Certified: Azure Solutions Architect Expert (AZ-305)
• Microsoft Certified: Azure Administrator Associate (AZ-104)
• Microsoft Certified: DevOps Engineer Expert (AZ-400)
• HashiCorp Certified: Terraform Associate

Pay rate range: $75.00 - $85.00 hourly.

If you have the described qualifications and are interested in this exciting opportunity, please apply!

Ranked a Top Staffing Firm in the U.S. by Staffing Industry Analysts for six consecutive years, Genesis10 puts thousands of consultants and employees to work across the United States every year in contract, contract-for-hire, and permanent placement roles. With more than 300 active clients, Genesis10 provides access to many of the Fortune 100 firms and a variety of mid-market organizations across the full spectrum of industry verticals. 

For contract roles, Genesis10 offers the benefits listed below. If this is a perm-placement opportunity, our recruiter can talk you through the unique benefits offered for that particular client. Benefits of Working with Genesis10:
 

• Access to hundreds of clients, most who have been working with Genesis10 for 5-20+ years. 
• The opportunity to have a career-home in Genesis10; many of our consultants have been working exclusively with Genesis10 for years. 
• Access to an experienced, caring recruiting team (more than 7 years of experience, on average.) 
• Behavioral Health Platform 
• Medical, Dental, Vision 
• Health Savings Account 
• Voluntary Hospital Indemnity (Critical Illness & Accident) 
• Voluntary Term Life Insurance 
• 401K 
• Sick Pay (for applicable states/municipalities) 
• Commuter Benefits (Dallas, NYC, SF) 

For multiple years running, Genesis10 has been recognized as a Top Staffing Firm in the U.S., as a Best Company for Work-Life Balance, as a Best Company for Career Growth, for Diversity, and for Leadership, amongst others. To learn more and to view all our available career opportunities, please visit us at our website.  

Genesis10 is an Equal Opportunity Employer. Candidates will receive consideration without regard to their race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or status as a protected veteran.