Databricks Architect
Must Have Technical/Functional Skills
Mandatory
Former Databricks Employee (Non-Negotiable Requirement)
12+ Years of IT Experience
5+ Years of Databricks Architecture Experience
Strong Lakehouse Architecture Expertise
Hands-on Spark/PySpark Experience
Enterprise Cloud Data Platform Design Experience
Data Governance & Security Expertise
Stakeholder Management and Leadership Experience
Preferred
BFSI Domain Experience
GenAI / AI Platform Architecture Experience
Large-scale Cloud Migration Experience
Pre-sales / Solution Consulting Experience
Databricks Technologies
- Databricks Lakehouse Platform
- Delta Lake
- Photon Engine
- Unity Catalog
- Delta Live Tables (DLT)
- Databricks Workflows
- MLflow
- Auto Loader
- Structured Streaming
- Databricks SQL
- Databricks Asset Bundles
Data Engineering
- Apache Spark
- PySpark
- Spark SQL
- Scala (preferred)
- Python
- SQL Optimization
- Data Modeling
- Data Warehousing
Cloud Platforms
- Microsoft Azure (Preferred)
- Azure Data Factory
- Azure Synapse Analytics
- Azure Storage
- Azure Key Vault
- Azure Entra ID (Azure AD)
OR
- AWS Databricks Ecosystem
- Google Cloud Databricks Ecosystem
DevOps & Automation
- Terraform
- GitHub
- Azure DevOps
- CI/CD Pipelines
- Docker
- Kubernetes
Data Governance & Security
- Unity Catalog
- Apache Ranger
- Data Lineage Tools
- Data Quality Frameworks
- Data Classification & Governance
____
Domain Experi ence (Preferred)
- Banking & Financial Services (BFSI)
- Capital Markets
- Wealth Management
- Insurance
- Lending & Consumer Finance
- Regulatory Reporting and Compliance
____
Certifications (Preferred)
- Databricks Certified Data Engineer Professional
- Databricks Certified Solution Architect
- Databricks Certified Machine Learning Professional
- Azure Solutions Architect Expert
- Azure Data Engineer Associate
Roles & Responsibilities
Architecture & Strategy
- Design and implement enterprise-scale Databricks Lakehouse platforms.
- Define end-to-end data architecture for batch, streaming, and real-time analytics solutions.
- Develop cloud-native data platform strategies aligned with business objectives.
- Establish architectural standards, governance frameworks, and best practices.
- Lead modernization initiatives involving legacy data warehouse migration to Databricks.
Data Engineering & Integration
- Design scalable ETL/ELT pipelines using Databricks, Apache Spark, and Delta Lake.
- Build medallion architecture (Bronze, Silver, Gold layers) for enterprise data platforms.
- Integrate data from multiple sources including databases, APIs, SaaS platforms, and streaming systems.
- Implement data quality, lineage, observability, and monitoring frameworks.
- Optimize data ingestion and transformation processes for performance and scalability.
Cloud & Platform Engineering
- Architect Databricks solutions on Azure, AWS, or Google Cloud Platform.
- Design secure networking, storage, identity management, and access controls.
- Implement Infrastructure as Code (Terraform, ARM, CloudFormation).
- Optimize platform usage and cloud costs through governance and monitoring.
- Lead Databricks workspace design and multi-environment deployment strategies.
AI, ML & Advanced Analytics
- Architect Databricks solutions supporting Machine Learning and Generative AI workloads.
- Design ML lifecycle management using MLflow.
- Enable enterprise AI capabilities leveraging Databricks Mosaic AI and GenAI frameworks.
- Collaborate with Data Scientists and AI teams to operationali ze ML models.
- Establish MLOps best practices and governance standards.
Security & Governance
- Implement Unity Catalog for enterprise-wide governance.
- Define data security controls including RBAC, ABAC, encryption, and access management.
- Ensure compliance with industry standards including SOX, PCI-DSS, HIPAA, GDPR, and BFSI regulatory requirements.
- Establish data governance, metadata management, and audit frameworks.
Leadership & Stakeholder Management
- Act as the primary Databricks subject matter expert.
- Engage with executive stakeholders to define data transformation roadmaps.
- Conduct architecture reviews and provide technical leadership.
- Mentor data engineers, platform engineers, and architects.
- Collaborate with business, infrastructure, security, and application teams.
Salary Range- $120,000-$150,000 a year