Job Title: Databricks Architect (Enterprise Data Platform)
Experience: 10+ Years
Location- NJ, Hybrid
Note- Need Independent candidate for C2H after 3-6 months
We are seeking a highly skilled Databricks Architect to lead the design and implementation of next-generation data platforms built on the Lakehouse paradigm.
This role goes beyond pipeline development-you will own the Databricks platform architecture end-to-end, driving scalability, governance, performance, and cost optimization across enterprise data ecosystems.
Databricks Platform Architecture
Architect and implement enterprise-scale Databricks environments (dev/test/prod)
Define workspace strategy, cluster policies, and job orchestration frameworks
Design secure, scalable Lakehouse architecture using Delta Lake
Data Engineering & Processing
Build and optimize high-performance data pipelines using Apache Spark (PySpark / Scala)
Implement Medallion Architecture (Bronze, Silver, Gold)
Develop batch and real-time streaming pipelines using Structured Streaming
Governance, Security & Compliance
Implement fine-grained access control using Unity Catalog
Define enterprise-wide data governance, lineage, and auditing frameworks
Ensure compliance with security and regulatory standards
Performance & Cost Optimization
Optimize workloads using partitioning, caching, and Photon engine
Design cost-efficient cluster strategies (autoscaling, spot instances, DBU optimization)
Monitor and improve query and pipeline performance at scale
DevOps & Automation
Implement CI/CD pipelines for Databricks using Azure DevOps, Jenkins, or GitHub Actions
Enable Infrastructure as Code using Terraform or equivalent
Standardize reusable frameworks and engineering best practices
Cloud Integration
Architect Databricks solutions on AWS / Azure / GCP
Integrate with cloud-native services (AWS S3/Glue, Azure ADLS/ADF, etc.)
Must-Have Skills
Advanced proficiency in SQL and Python
Hands-on experience with Delta Lake, Medallion Architecture, and Unity Catalog
Strong experience with at least one cloud platform (AWS / Azure / GCP)
Experience with CI/CD tools (Azure DevOps, Jenkins, GitHub Actions)
Knowledge of workflow orchestration tools (Airflow or equivalent)
Good to Have
Databricks certifications (Associate / Professional)
Experience with Databricks SQL & dashboards
Exposure to ML pipelines (MLflow)
Experience with Kafka or event-driven architectures
Domain experience in Finance / Retail / Healthcare
What Makes You a Great Fit
Strong architectural mindset with ability to design scalable data platforms
Deep understanding of performance tuning and cost optimization
Ability to balance technical depth with business impact
Excellent communication and stakeholder management skills