Databricks AI Engineer

You will work closely with cross-functional teams including data owners, security engineering, and architecture to ensure reliable, high-performance data delivery while supporting modernization initiatives across the platform. A leading enterprise organization is seeking an AI Databricks Engineer to support the buildout of a centralized data lakehouse environment focused on cybersecurity and enterprise data integration.

A leading enterprise organization is seeking an AI Databricks Engineer to support the buildout of a centralized data lakehouse environment focused on cybersecurity and enterprise data integration. This role will play a key part in designing and developing scalable data pipelines that ingest and transform large volumes of structured and unstructured data.

You will work closely with cross-functional teams including data owners, security engineering, and architecture to ensure reliable, high-performance data delivery while supporting modernization initiatives across the platform.

RESPONSIBILITIES

• Design and build automated data pipelines using Azure and Databricks
• Develop ingestion frameworks for API-based and external data sources
• Optimize and tune pipeline performance for large-scale data processing
• Implement and maintain data lake and lakehouse architectures
• Collaborate with stakeholders across engineering and security teams
• Ensure data quality, reliability, and compliance with internal standards
• Support agile delivery cycles and sprint-based development

QUALIFICATIONS

• 3 5 years of data engineering experience
• Strong expertise in Python, PySpark, and SQL
• Hands-on experience with Azure Databricks
• Experience with Azure services (Data Lake Gen2, Data Factory, APIs)
• Proven experience building pipelines using REST API data sources
• Experience with performance tuning and optimization
• Strong understanding of relational and/or NoSQL databases
• Experience building APIs using Python frameworks (e.G., FastAPI)
• Strong communication and stakeholder collaboration skills

PREFERRED EXPERIENCE

• Experience in cybersecurity or security data environments
• Familiarity with Azure Synapse or Microsoft Fabric
• Experience with enterprise-scale data lakehouse platforms
• Azure certifications

How We Work:

• Agile, sprint-based development environment
• Highly collaborative across technical teams
• Focus on scalable, production-ready solutions
• Emphasis on ownership and accountability