Databricks Data Lake Architect

Design robust lakehouse architectures that integrate Databricks Delta Lake with enterprise data sources to enable scalable analytics and generative AI solutions that advance strategic business outcomes. Requirements: Demonstrate deep expertise in Spark, Pyspark, and SparkSQL with extensive experience designing high volume data pipelines and performance tuned transformations in production environments.

Our Client, an IT Services and Consultant company, is looking for a Databricks Data Lake Architect for their Plano, TX/Hybrid location.
 
Responsibilities:

• Design robust lakehouse architectures that integrate Databricks Delta Lake with enterprise data sources to enable scalable analytics and generative AI solutions that advance strategic business outcomes.
• Architect Spark and Pyspark based data pipelines optimized for performance, reliability, and cost efficiency to support complex analytical workloads and downstream applications.
• Develop Python services and reusable components that operationalize GenAI basics and data processing logic, ensuring maintainable and testable code across projects.
• Define Databricks Workflows orchestration patterns that coordinate Spark jobs, Delta Lake operations, and data quality checks to deliver predictable and resilient data products. -Configure Databricks Jobs and Lakeflow Jobs to manage end to end batch and near real time processing while monitoring execution health and optimizing resource utilization.
• Establish Databricks SQL models, queries, and shared datasets that provide consistent and governed access to curated information for analytics and reporting teams.
• Implement Databricks Unity Catalog structures including catalogs, schemas, and access controls that enforce enterprise data governance and regulatory compliance requirements.
• Collaborate with data engineers, data scientists, and product stakeholders in a hybrid work setting to translate analytical needs into technical designs and implementation plans.
• Review solution designs and code across Spark, Python, and Databricks assets to ensure adherence to architectural standards, security practices, and performance benchmarks.
• Document end to end architecture including data lineage, dependency mapping, and operational procedures to support long term maintainability and knowledge sharing.
• Advise business partners on leveraging GenAI capabilities safely and effectively using governed data in Databricks to create measurable impact for customers and society.
• Drive continuous improvement initiatives by analyzing platform metrics, identifying bottlenecks in workflows, and implementing enhancements that increase reliability and throughput.
• Guide adoption of best practices for hybrid collaboration, ensuring that design discussions, reviews, and knowledge sessions are inclusive and accessible to distributed teams.

 
Requirements:

• Demonstrate deep expertise in Spark, Pyspark, and SparkSQL with extensive experience designing high volume data pipelines and performance tuned transformations in production environments.
• Show advanced proficiency in Python three dot x for data engineering and analytics use cases including modular coding standards, automated testing, and robust error handling.
• Apply strong hands on experience with Databricks Workflows, Jobs, and Lakeflow Jobs to build orchestrated processes that meet strict service level objectives and audit needs.
• Exhibit solid knowledge of Databricks SQL and Delta Lake for building reliable lakehouse models, incremental ingestion patterns, and optimized query structures.
• Use practical experience with Databricks Unity Catalog to design secure data governance frameworks that support role based access and consistent metadata management.
• Bring working understanding of GenAI basics including prompt driven patterns, data preparation constraints, and responsible usage aligned with organizational guidelines.
• Leverage prior background of twelve to fifteen years in data architecture or related fields to handle complex stakeholder environments and large scale platform evolution.
• Communicate clearly in hybrid work settings, using structured documentation and concise technical summaries to align cross functional teams on architectural decisions.

 
Why Should You Apply?

• Health Benefits
• Referral Program
• Excellent growth and advancement opportunities