Job Summary
We are seeking a highly skilled professional to serve as the go-to technical authority for all things Databricks within our organization. The ideal candidate will have deep expertise in the Databricks Lakehouse platform, including data engineering, data science, machine learning workflows, performance optimization, and best practices for scalable data solutions. This role is critical in driving the successful adoption, implementation, and optimization of Databricks across data pipelines, analytics, and AI initiatives. The successful candidate will collaborate with cross-functional teams to design robust solutions, troubleshoot complex issues, mentor team members, and ensure maximum value is derived from the Databricks ecosystem.
Key Responsibilities
- Lead the design, implementation, and optimization of end-to-end solutions on Databricks, including Unity Catalog, Delta Lake, Databricks SQL, Workflows, Jobs, and Mosaic AI.
- Architect, build, and maintain scalable data ingestion, transformation, and orchestration pipelines using Databricks tools (Spark, Delta Live Tables, Auto Loader, etc.).
- Identify and resolve performance bottlenecks in Spark jobs, clusters, and SQL queries. Implement cost optimization strategies, auto-scaling, and efficient resource management.
- Establish and enforce Databricks best practices for security, data governance, compliance (e.g., GDPR, HIPAA), and Unity Catalog implementation across the organization.
- Support data science and machine learning teams by implementing MLflow, Feature Store, Vector Search, and generative AI capabilities on Databricks.
- Act as the escalation point for complex technical issues, providing root cause analysis and rapid resolution for production workloads.
- Mentor data engineers, analysts, and developers on Databricks capabilities. Conduct training sessions, create documentation, and build internal centers of excellence.
- Design seamless integrations between Databricks and other technologies (cloud platforms like AWS/Azure/GCP, ETL tools, BI tools, data catalogs, and orchestration systems).
- Implement robust monitoring, logging, alerting, and dashboarding for Databricks workloads using Databricks features and external tools.
- Stay current with the latest Databricks features and roadmap. Evaluate new capabilities and recommend adoption strategies to drive business value.
- Work closely with stakeholders to translate business requirements into technical solutions and deliver high-impact data & AI projects.
Qualifications
- Bachelor's degree in Information Technology, Computer Science, Business Administration, or a related field, or equivalent work experience.
- US Citizenship required.
- 7 years in Data Engineering or Architecture, with at least 4 years dedicated exclusively to the Databricks ecosystem.
- Proven track record implementing Unity Catalog from scratch, including row- and column-level security and cross-workspace governance.
- Deep, low-level Apache Spark tuning expertise (e.g., managing data skew, optimizing shuffles, utilizing Adaptive Query Execution, caching strategies, and Photon engine optimization).
- Hands-on experience building production-grade Generative AI or LLM applications using Databricks Mosaic AI, Vector Search, and MLflow.
- Advanced proficiency in Python, Scala, and SQL optimized specifically for distributed computing environments.
- Production experience with Infrastructure as Code (IaC) tools like Terraform to deploy and manage Databricks workspaces, clusters, and permissions.
- Experience setting up secure network topologies for Databricks, including Private Link/VPC Peering, encryption keys (BYOK), and IP access lists.
- Strong background in CI/CD pipelines (e.g., GitHub Actions, Azure DevOps, GitLab) using Databricks Asset Bundles (DABs) or Databricks CLI.
Preferred Qualifications
- Databricks Certified Solutions Architect.
- Databricks Certified Data Engineer Professional.
- Databricks Certified Machine Learning Professional (Preferred for Mosaic AI/MLflow initiatives).
- Cloud Architecture certification (AWS Certified Solutions Architect, Azure Solutions Architect Expert).
This position is based in [City, State], offering a vibrant community with a rich history and a diverse cultural scene. Enjoy a dynamic work environment in a city known for its innovation and growth.
Benefits
PEAK's benefit offerings available for our associates include medical, dental, vision, Flexible Spending Account (FSA), Dependent Care Savings Account (DCA), and 401K plan. PEAK believes that taking care of our team is essential for success and we are proud to provide benefits that enhance both your well-being and your future. Additionally, our associates may be eligible for Paid Sick Leave as required by Federal, State, or local laws.
Equal Opportunity Employer (EEO)
PEAK Technical Staffing is committed to creating a diverse and inclusive environment and is proud to be an Equal Opportunity Employer. PEAK does not discriminate on the basis of race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, or veteran status, or any other characteristic protected by applicable law. All employment decisions are made based on qualifications, merit, and business need. We encourage all individuals to apply.
Americans Disabilities Act (ADA)
The physical and mental requirements described in this job description are representative of those that must be met by an employee to successfully perform the essential functions of the position. Reasonable accommodations may be made to enable qualified individuals with disabilities to perform the essential functions. Must be able to perform the essential physical functions of the position, including sitting, standing, walking, stooping, kneeling, and lifting up to 25 pounds, with or without reasonable accommodation.
Candidate Privacy
To read our Candidate Privacy Information Statement, which explains how we will use your information, please navigate to https://peaktechnical.com/privacy-policy/ and https://peaktechnical.com/ca-residents-privacy-rights/
AI Recruiting Disclosure
We use AI-assisted tools to help review applications and compare your experience to job requirements, but all hiring decisions are made by human recruiters. You may request a human-only process or opt out of automated communication at any time. Required notices and our latest bias audit are available on our website: www.peaktechnical.com/ai-disclosure.