Sign upLog in

Databricks AI Engineer

Madison-Davis

New York, NY | Jersey City, NJ | Charlotte, NC, NY

Apply
JOB DETAILS
SALARY
$70
SKILLS
Agile Programming Methodologies, Application Programming Interface (API), Artificial Intelligence (AI), Communication Skills, Cross-Functional, Data Lake, Data Management, Data Processing, Data Quality, Engineering, Enterprise Data Integration, Information/Data Security (InfoSec), Internet Security, Microsoft Windows Azure, Multiplatform/Cross-Platform, NoSQL, Performance Tuning/Optimization, Python Programming/Scripting Language, REST (Representational State Transfer), Regulatory Compliance, Relational Databases (RDBMS), SQL (Structured Query Language), Scalable System Development, Security Architecture, Structured Data, Team Player, Unstructured Data
LOCATION
New York, NY | Jersey City, NJ | Charlotte, NC, NY
POSTED
2 days ago

A leading enterprise organization is seeking an AI Databricks Engineer to support the buildout of a centralized data lakehouse environment focused on cybersecurity and enterprise data integration. This role will play a key part in designing and developing scalable data pipelines that ingest and transform large volumes of structured and unstructured data.

You will work closely with cross-functional teams including data owners, security engineering, and architecture to ensure reliable, high-performance data delivery while supporting modernization initiatives across the platform.

RESPONSIBILITIES
  • Design and build automated data pipelines using Azure and Databricks
  • Develop ingestion frameworks for API-based and external data sources
  • Optimize and tune pipeline performance for large-scale data processing
  • Implement and maintain data lake and lakehouse architectures
  • Collaborate with stakeholders across engineering and security teams
  • Ensure data quality, reliability, and compliance with internal standards
  • Support agile delivery cycles and sprint-based development

QUALIFICATIONS
  • 3 5 years of data engineering experience
  • Strong expertise in Python, PySpark, and SQL
  • Hands-on experience with Azure Databricks
  • Experience with Azure services (Data Lake Gen2, Data Factory, APIs)
  • Proven experience building pipelines using REST API data sources
  • Experience with performance tuning and optimization
  • Strong understanding of relational and/or NoSQL databases
  • Experience building APIs using Python frameworks (e.G., FastAPI)
  • Strong communication and stakeholder collaboration skills

PREFERRED EXPERIENCE
  • Experience in cybersecurity or security data environments
  • Familiarity with Azure Synapse or Microsoft Fabric
  • Experience with enterprise-scale data lakehouse platforms
  • Azure certifications

How We Work:
  • Agile, sprint-based development environment
  • Highly collaborative across technical teams
  • Focus on scalable, production-ready solutions
  • Emphasis on ownership and accountability

About the Company

M

Madison-Davis