Data & AI Automation Engineer (Databricks)

Position: Data & AI Automation Engineer (Databricks)

Location: Austin, TX (Locals preferred - Hybrid 3 Days Onsite Weekly)

Duration: Long-Term Contract

Required Skills: Azure Databricks, Azure Cloud, Generative AI, LLMs

Note: Looking for Permanent / Visa Independent Consultants

Overview:

We are seeking an experienced Data & AI Automation Engineer to lead the development of intelligent automation solutions within an Azure Databricks ecosystem. This role focuses on leveraging Large Language Models (LLMs), AI agents, and advanced automation techniques to enhance data engineering productivity, accelerate software delivery, and improve data quality processes.

The ideal candidate will combine strong data engineering expertise with AI-driven automation capabilities to build scalable solutions that streamline development, testing, governance, and operational activities across the data platform. Working closely with architects, engineers, and platform teams, this individual will help drive innovation through intelligent agent frameworks and automated workflows.

Key Responsibilities:

• Architect and develop AI-powered agents using LLMs, rule-based logic, or hybrid approaches to automate data engineering operations within Azure Databricks.
• Create automation solutions that generate, optimize, and refactor PySpark and SQL code to improve development efficiency and maintainability.
• Design intelligent testing and validation agents to automate data quality assessments, reconciliation processes, and QA activities.
• Implement automated governance capabilities including metadata management, lineage tracking, compliance monitoring, and policy enforcement.
• Develop autonomous CI/CD functions such as test creation, deployment verification, release validation, and recovery/rollback procedures.
• Build monitoring and diagnostic agents capable of detecting anomalies, identifying performance issues, and supporting root-cause investigations.
• Establish prompt engineering standards, reusable templates, and agent orchestration patterns tailored to enterprise data platforms.
• Partner with data engineering and architecture teams to identify automation opportunities and prioritize high-value initiatives.
• Define and enforce best practices for AI agent lifecycle management, including development, testing, deployment, observability, and maintenance.
• Produce technical documentation, workflow diagrams, operational procedures, and knowledge-transfer materials.
• Evaluate emerging AI technologies, agent frameworks, and automation tools to support continuous platform improvement.

Required Experience:

• 10+ years of experience in Data Engineering, Software Engineering, or a related technical discipline.
• Extensive hands-on experience with modern cloud data platforms such as Azure Databricks, Snowflake, or similar technologies.
• Proven background building AI-driven automation solutions utilizing LLMs, prompt engineering methodologies, and agent orchestration frameworks.
• Strong programming expertise in Python and advanced SQL development within production environments.
• Practical experience working with Azure Databricks, including notebooks, workflows, jobs, Delta Live Tables (DLT), and related services.
• Experience implementing CI/CD and Continuous Testing (CT) pipelines using tools such as GitHub Actions and Azure DevOps.
• Demonstrated success creating automated testing frameworks, data validation solutions, or quality assurance processes.
• Previous experience supporting highly regulated, compliance-driven, or risk-sensitive environments is advantageous.

Technical Skills:

• Advanced Python development skills with a strong foundation in software engineering principles, testing methodologies, documentation standards, and source control practices.
• Expertise in SQL and PySpark performance tuning and optimization.
• Hands-on experience integrating LLM services, AI APIs, prompt engineering techniques, and agent frameworks such as Databricks Agent Framework, LangChain, AutoGen, OpenAI, or Azure OpenAI.
• Deep understanding of Azure Databricks, including Delta Lake, workflows, asset bundles, and enterprise-scale data processing.
• Experience building workflow automation solutions using orchestration platforms, serverless technologies, or event-driven architectures.
• Strong analytical skills with the ability to break down complex business and technical processes into scalable automation components.
• Excellent communication skills with the ability to explain technical designs, AI agent architectures, and automation strategies to both technical and non-technical stakeholders.

Education:

• Bachelor's degree in Computer Science, Software Engineering, Data Science, Information Technology, or a related field.
• Master's degree preferred.