Advise teams on the effective use of Databricks capabilities, including Unity Catalog, Feature Engineering, Model Serving, Lakehouse Monitoring, Delta Lake, Delta Live Tables/Lakeflow, and Databricks Workflows. Establish end-to-end ML workflows covering data preparation, feature engineering, experimentation, model training, validation, deployment, monitoring, and retraining.
Numbers & Facts
Location
Alpharetta, Georgia (Remote)
Description
Key Responsibilities:
Define the target architecture and roadmap for machine learning on AWS and Databricks.
Design and implement scalable, secure, and production-ready ML solutions.
Assess the current Databricks implementation and recommend improvements, integrations, and platform capabilities.
Establish end-to-end ML workflows covering data preparation, feature engineering, experimentation, model training, validation, deployment, monitoring, and retraining.
Implement MLOps practices using Databricks and MLflow, including experiment tracking, model management, automated deployment, governance, and monitoring.
Design batch and real-time model-inference solutions based on business and technical requirements.
Advise teams on the effective use of Databricks capabilities, including Unity Catalog, Feature Engineering, Model Serving, Lakehouse Monitoring, Delta Lake, Delta Live Tables/Lakeflow, and Databricks Workflows.
Integrate Databricks with relevant AWS services, such as Amazon S3, IAM, KMS, SageMaker, Lambda, ECR, CloudWatch, and streaming services.
Define reusable architecture patterns, development standards, and platform guardrails.
Establish appropriate security, access-control, data-governance, model-governance, and compliance practices.
Optimize workloads for scalability, reliability, performance, and cost.
Partner with data scientists, data engineers, platform teams, security teams, and business stakeholders to move ML use cases from concept into production.
Provide technical leadership, mentoring, documentation, and knowledge transfer to internal teams.
Required Qualifications:
Extensive experience designing and delivering enterprise machine learning platforms and solutions.
Strong hands-on experience with Databricks in a production environment.
Deep knowledge of the Databricks Lakehouse architecture and its machine learning capabilities.
Strong experience with MLflow, including experiment tracking, model lifecycle management, deployment, and governance.
Extensive AWS architecture and implementation experience.
Strong understanding of MLOps, CI/CD, infrastructure as code, automated testing, observability, and model monitoring.
Experience deploying both batch and real-time inference workloads.
Proficiency in Python and SQL, with experience using Apache Spark/PySpark.
Experience with feature engineering, model evaluation, drift detection, explainability, retraining, and responsible AI practices.
Strong knowledge of data security, IAM, encryption, networking, governance, and regulatory considerations in cloud environments.
Ability to translate business requirements into practical architecture and implementation plans.
Excellent communication skills and the ability to influence technical and non-technical stakeholders.