Sign upLog in
See more jobs

Azure Databricks Admin

TechDigital

  • Cincinnati, OH
  • 30+ days ago

    Highlights

    Azure Databricks Administration & Management: Deploy, configure, and manage Azure Databricks workspaces in a scalable, cost-efficient, and secure manner. Automation & Infrastructure as Code (IaC): Automate Databricks workspace deployment using Terraform, ARM Templates, or Databricks REST API .

    Numbers & Facts

    LocationCincinnati, OH
    IndustryOther/Not Classified
    Company Size100 to 499 employees

    Description

    Azure Databricks Administration & Management:
    • Deploy, configure, and manage Azure Databricks workspaces in a scalable, cost-efficient, and secure manner.
    • Administer clusters, jobs, notebooks, and workflows, ensuring high availability and performance.
    • Monitor and optimize compute resource utilization and autoscaling strategies to improve cost efficiency.
    • Manage Databricks Runtime versions, libraries, and dependencies across environments.
    Security & Compliance:
    • Implement and manage Unity Catalog for fine-grained access control and data governance.
    • Enforce Role-Based Access Control (RBAC) and integrate Databricks with Azure Active Directory (AAD).
    • Ensure compliance with SOC 2, HIPAA, GDPR, and internal security standards.
    • Set up audit logging, monitoring, and alerting for security and operational insights.

    Performance Optimization & Troubleshooting:
    • Tune Apache Spark workloads to improve query performance and resource efficiency.
    • Analyze and troubleshoot performance bottlenecks in ETL and ML workloads.
    • Optimize Delta Lake storage, caching, and indexing strategies for better query execution.
    Automation & Infrastructure as Code (IaC):
    • Automate Databricks workspace deployment using Terraform, ARM Templates, or Databricks REST API.
    • Develop and maintain CI/CD pipelines for Databricks job deployment and configuration management.
    • Implement monitoring solutions using Azure Monitor, Prometheus, or Grafana.
    Collaboration & Integration:
    • Work closely with data engineers, data scientists, and DevOps teams to support data pipelines and analytics workloads.
    • Integrate Databricks with Azure Data Lake, Azure Synapse Analytics, and Snowflake.
    • Provide technical guidance and best practices for efficient Spark job execution and cost optimization.
    Required Skills & Experience:
    • 5+ years of experience in Azure Databricks administration and performance optimization.
    • Expertise in Apache Spark, PySpark, SQL, and Scala.
    • Hands-on experience with Databricks Unity Catalog, Delta Lake, and MLflow.
    • Strong knowledge of Azure cloud services (Azure Data Lake, Azure Synapse, Azure Key Vault, etc.).
    • Experience in Infrastructure as Code (Terraform, Bicep, or ARM Templates).
    • Proficiency in CI/CD pipeline automation using Azure DevOps, GitHub Actions, or Jenkins.
    • Strong understanding of network security, identity management (AAD), and encryption best practices.
    • Excellent problem-solving skills and ability to troubleshoot complex Databricks workloads.
    • Strong communication and documentation skills.
    Preferred Qualifications:
    • Databricks Certified Associate or Professional certification.
    • Experience with Azure Kubernetes Service (AKS) and serverless computing.
    • Familiarity with Kafka, Apache Airflow, and event-driven architectures.
    • Knowledge of Python, PowerShell, or Bash scripting for automation

    Similar Jobs