Position: Databricks Data Engineer
Location: Austin, TX (Preferred Locals to Austin, TX and Permanent residents of USA)
Duration: Long-Term Contract
Key Skills: Looking for a Senior Databricks Data Engineer with SQL Server + SSIS + DW modernization experience, and ideally GenAI/LLM/Agent experience
Responsibilities & Skills:
- 12yrs of Data Engineer and IT exp is mandatory
- Design and implement enterprise scale solutions on the Databricks Lakehouse Platform.
- Architect end to end data pipelines for batch and real time processing using Apache Spark and PySpark.
- Develop scalable data ingestion, transformation, and data quality frameworks.
- Design and implement Medallion Architecture (Bronze, Silver, Gold) using Delta Lake.
- Build and optimize data warehouses, data marts, and analytical solutions.
- Implement data governance, security, lineage, and access controls using Unity Catalog.
- Develop and support AI/BI dashboards, semantic models, and self-service analytics solutions.
- Configure and optimize Genie Spaces to enable natural language business queries and conversational analytics.
- Design and deploy Generative AI and RAG based solutions using Databricks Mosaic AI and Vector Search.
- Collaborate with business users to translate requirements into scalable data and AI solutions.
- Optimize Databricks workloads for performance, scalability, reliability, and cost efficiency.
- Lead cloud native implementations across Azure environments.
- Define architecture standards, best practices, and reusable design patterns.
- Mentor data engineers, analysts, and architects on Databricks technologies and platform adoption.
- Lead migration initiatives from legacy data warehouses and analytics platforms to Databricks.
- Build and maintain Genie Spaces for business self-service analytics.
- Create semantic models, metrics, and trusted data assets for AI driven reporting.
- Develop natural language to SQL analytics solutions using Databricks Genie.
- Implement RAG solutions using enterprise data and Vector Search.
- Optimize AI/BI dashboards and conversational analytics experiences.
- Troubleshoot Spark performance, query optimization, and workload management.
- Automate data validation, monitoring, and governance controls.
- Support AI use cases using Mosaic AI model serving and inference endpoints.
Technical Skills:
- Databricks Lakehouse Platform
- Apache Spark, PySpark, Spark SQL
- Python, SQL
- Delta Lake, Delta Live Tables, Lakeflow
- Unity Catalog
- Databricks AI/BI and Genie
- Mosaic AI, Vector Search, RAG
- Data Modeling (Dimensional & Data Vault)
- Structured Streaming
- Data Quality and Data Governance
- Azure
- Terraform, Git, Azure DevOps, Jenkins
- REST APIs and Data Integration
- Performance Tuning and Cost Optimization