Enterprise Architect

Seeking a Databricks Architect for an onsite contract role in Tarrytown, NY to lead enterprise-scale data platform modernization and migration initiatives on AWS. The ideal candidate will architect scalable Databricks Lakehouse solutions using Delta Lake, Unity Catalog, DLT, Workflows, Auto Loader, PySpark, and Structured Streaming. The role involves migrating EMR, NiFi, Redshift, and legacy ETL workloads, while establishing data governance, security, quality, lineage, and observability frameworks. Strong AWS expertise across S3, Glue, IAM, Redshift, EMR, and Lambda, along with Airflow, CI/CD, GitHub, and Jenkins, is required. Experience enabling AI/ML and Generative AI use cases is highly valued.

Key Responsibilities
Lead enterprise-scale Databricks Lakehouse architecture and modernization initiatives on AWS.
Design and implement scalable Medallion Architecture (Bronze, Silver, Gold) using Databricks and Delta Lake.
Lead migration of AWS EMR, Apache NiFi, Redshift, and legacy ETL workloads to Databricks.
Architect high-performance batch, streaming, CDC, and real-time data pipelines.
Develop reusable ETL/ELT frameworks using PySpark, Python, Spark SQL, Auto Loader, and Structured Streaming.
Establish enterprise data governance, security, metadata, lineage, and data quality standards using Unity Catalog.
Leverage AWS services including S3, IAM, Glue, Redshift, EMR, Lambda, and Airflow.
Drive adoption of DLT, Databricks Workflows, Unity Catalog, Serverless, and Lakehouse Federation.
Establish CI/CD, DevOps, automation, and Infrastructure-as-Code best practices.
Lead architecture reviews, technical workshops, solution assessments, and provide technical leadership and mentoring.
Define migration strategies, architecture roadmaps, standards, and governance frameworks.
Collaborate with business, data science, analytics, and engineering teams to deliver AI/ML and advanced analytics solutions.