Senior Databricks AI Engineer

Senior Databricks AI Engineer

Atlanta, GA~ Smyrna, GA

Contract

Position Summary

We are seeking a highly skilled Senior Databricks AI Engineer to design, develop, and deploy scalable data and AI solutions using the Databricks Lakehouse Platform. The ideal candidate will have expertise in Data Engineering, Machine Learning, Generative AI, and cloud technologies, with hands-on experience building enterprise-grade AI applications and data platforms.

Key Responsibilities

Design and develop scalable data pipelines using Databricks, PySpark, Delta Lake, and Spark SQL.

Build ELT/ETL solutions for structured and unstructured data.

Implement Medallion Architecture (Bronze, Silver, Gold layers).

Optimize Databricks jobs, clusters, and workflows for performance and cost efficiency.

Manage data ingestion from multiple cloud and on-premise sources.

AI & Machine Learning

Develop and deploy Machine Learning models using Databricks ML and Python.

Implement MLOps practices for model training, deployment, monitoring, and governance.

Build predictive analytics solutions and recommendation engines.

Create feature stores and reusable ML pipelines.

Generative AI

Design and develop GenAI applications using:

Large Language Models (LLMs)

Retrieval Augmented Generation (RAG)

Vector Databases

AI Agents and Multi-Agent Frameworks

Integrate Databricks Mosaic AI, LangChain, LangGraph, and OpenAI/Azure OpenAI services.

Fine-tune and evaluate foundation models.

Develop enterprise chatbots, knowledge assistants, and intelligent automation solutions.

Cloud & Architecture

Implement solutions on Azure, AWS, or GCP.

Design secure and scalable Lakehouse architectures.

Work with Unity Catalog, Delta Sharing, and data governance frameworks.

Collaborate with business stakeholders, architects, and data scientists to deliver AI-driven business solutions.

Technical Skills

Databricks Lakehouse Platform

PySpark, Spark SQL

Python

Delta Lake

Databricks Workflows

Databricks Unity Catalog

Azure Data Factory / AWS Glue

SQL and Data Modeling

Git, CI/CD, DevOps

AI & GenAI Skills:

Machine Learning

Deep Learning

LLMs

RAG Architecture

Prompt Engineering

LangChain / LangGraph

Vector Databases (FAISS, Pinecone, ChromaDB)

Databricks Mosaic AI

MLflow

Cloud Platforms

Microsoft Azure (Preferred)

AWS

Google Cloud Platform

Qualifications

Bachelor''s or Master''s degree in Computer Science, Data Science, Engineering, Information Technology, or related field.

6+ years of experience in Data Engineering.

3+ years of experience with Databricks.

Experience delivering AI/ML or Generative AI solutions in production environments.

Databricks Certified Data Engineer or Machine Learning certification preferred.

Preferred Experience

Healthcare, Life Sciences, Banking, Retail, or Manufacturing domain experience.

Experience with Azure OpenAI Service.

Knowledge of Agentic AI and AI Governance.

Experience in enterprise-scale data modernization programs.

Key Deliverables

Scalable Databricks data pipelines.

Enterprise AI and GenAI solutions.

Real-time analytics and AI-powered insights.

AI-enabled automation and decision-support systems.

Secure and governed Lakehouse architecture.

Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.