Senior Data Architect –Databricks
Employment Type: 12 Months with Extensions
Industry: Financial Services / Enterprise Technology
Experience Required: 8+ Years
Pay Rate: $85.00 to $115.00 per hour
Location: Downtown, NY
Onsite
Position Overview
We are seeking an experienced
Senior Data Architect to design and deliver scalable, secure, and high-performance enterprise data solutions using Databricks, AWS, PySpark, and modern cloud technologies.
The ideal candidate will have extensive experience architecting cloud-based data platforms, developing ETL/ELT pipelines, implementing data governance frameworks, and optimizing large-scale analytical workloads. This role requires strong technical leadership, hands-on architecture expertise, and the ability to collaborate with engineering, analytics, and business teams.
Key Responsibilities
Architect scalable enterprise data platforms and lakehouse solutions using Databricks and AWS.
Design and optimize ETL/ELT pipelines using PySpark, Python, Spark SQL, and Databricks.
Implement Databricks Unity Catalog, AWS IAM, and secure cross-account data access.
Optimize Databricks workloads for performance, scalability, and cloud cost efficiency.
Establish data quality, governance, security, and validation frameworks.
Design incremental processing, change data capture (CDC), and historical data management solutions.
Develop dimensional data models and semantic layers supporting Tableau and enterprise BI reporting.
Implement CI/CD pipelines using GitLab and infrastructure-as-code practices.
Architect MLOps solutions leveraging AWS services, Domino Data Lab, and related technologies.
Explore generative AI capabilities and opportunities for integration into enterprise data platforms.
Establish development standards for Python, PySpark, SQL, and cloud-native architectures.
Collaborate with product owners, engineers, analysts, and architects to translate business requirements into technical solutions.
Mentor engineering teams and document architecture standards, design patterns, and best practices.
Required Qualifications
8+ years of experience in data architecture, data engineering, or enterprise database design.
Extensive hands-on experience designing and implementing Databricks-based enterprise data platforms.
Strong expertise in AWS cloud architecture, data lakes, lakehouse architectures, and cloud data services.
Advanced proficiency with Python, PySpark, Apache Spark, and complex SQL.
Experience designing scalable ETL/ELT pipelines, data integrations, and distributed processing frameworks.
Strong understanding of Databricks Unity Catalog, data governance, security, and access management.
Experience with dimensional data modeling, data warehousing, partitioning, and performance optimization.
Knowledge of GitLab CI/CD, DataOps, infrastructure-as-code, and automated deployments.
Experience architecting secure, highly available, and scalable solutions within regulated enterprise environments.
Strong technical leadership, communication, and cross-functional collaboration skills.
Bachelor's degree in Computer Science, Information Technology, or a related field.
Preferred Qualifications
5+ years of experience with Tableau and enterprise business intelligence solutions.
Experience with MLOps, machine learning model deployment, and Domino Data Lab.
Familiarity with generative AI and modern AI-driven data solutions.
Experience implementing change data capture, incremental processing, and advanced data quality frameworks.
Background supporting financial services, banking, or other highly regulated industries.
Experience leading complex enterprise modernization and cloud transformation initiatives.
Ideal Candidate
The successful candidate is a hands-on Data Architect who combines deep Databricks and AWS expertise with strong data engineering fundamentals. They should be comfortable designing enterprise architectures, optimizing Spark workloads, establishing governance standards, and providing technical direction to cross-functional engineering teams.