Lead Cloud Data Engineer Data Mesh & AI

Primary Talent Partners

San Francisco, CA

Apply

JOB DETAILS

SALARY

$90–$100 Per Hour

SKILLS

AWS Lambda, Access Control, Amazon Simple Storage Service (S3), Amazon Web Services (AWS), Application Programming Interface (API), Architectural Design, Architectural Services, Artificial Intelligence (AI), Automation, Business Skills, Cisco Unity, Cloud Computing, Communication Skills, Continuous Deployment/Delivery, Continuous Integration, Cost Control, Data Management, Data Modeling, Data Quality, Data Warehousing, Database Extract Transform and Load (ETL), Disaster Recovery, Docker, Documentation, Electronic Medical Records, Financial Services, GitHub, Hybrid Cloud, Jenkins, Leadership, Mentoring, Metadata, Model Validation, Operational Support, Performance Tuning/Optimization, Policy Implementation, Programming Tools, Requirements Management, Security Compliance, Software Development Lifecycle (SDLC), Software Engineering, Technical Leadership, Technical Support, Technical Writing, Technical/Engineering Design

LOCATION

San Francisco, CA

POSTED

1 day ago

Primary Talent Partners has a new contract opening for a Lead Cloud Data Engineer – Data Mesh & AI to join a financial services company in San Francisco, CA. This is an 18-month contract to start with the potential for extensions.

Pay:$90.00 - $100.00/hr; W2 contract, no PTO, no Benefits. ACA-compliant supplemental package available for enrollment. Candidates must be legally authorized to work in the United States and must be able to sit on Primary Talent Partners W2 without sponsorship.

Description:
We are seeking an experienced Lead Data Engineer to join our team and drive the development of modern data platforms and AI-augmented solutions. This role is critical to supporting ongoing operations of Large Data Warehouse, architecting the Large Data Hub , and preparing for the enterprise data mesh modernization program.

Data Engineering & Pipeline Development:

Develop and maintain end-to-end ETL/ELT pipelines for data ingestion from multiple sources (databases, APIs, streaming), transformation, quality validation, data modeling, and consumption layer optimization using AWS services (Glue, EMR, Lambda, Kinesis, Step Functions)
Implement data governance policies, metadata management, data lineage tracking, fine-grained access controls, and automated data quality frameworks with validation rules, anomaly detection, and monitoring/alerting mechanisms

Data Mesh Architecture

Design and implement Data Mesh architecture on AWS with federated governance, defining domain boundaries, data product specifications, and self-serve infrastructure patterns across Databricks, Starburst, Collibra, and Immuta platforms
Develop reusable frameworks, accelerators, and self-serve tools enabling domain teams to independently publish, Client, and consume data products while ensuring performance optimization and cost efficiency

DevSecOps & Infrastructure

Integrate DevSecOps practices including CI/CD pipelines, infrastructure as code (Terraform/CloudFormation), automated security scanning, compliance validation, and disaster recovery strategies

AI-Augmented Development

Design and build agentic AI solutions, RAG pipelines, and intelligent automation using LLMs, orchestration frameworks, and AI-powered development tools (Copilot, Claude Code) to enhance platform capabilities and accelerate SDLC

Technical Leadership & Collaboration

Provide L3 technical support and mentor engineering teams on data mesh principles and modern data technologies
Collaborate with stakeholders to translate business requirements into technical solutions
Lead architecture design reviews and create comprehensive technical documentation

Required Qualifications
Data Mesh Architecture & AWS (5+ years required):

Deep experience designing distributed data architectures on AWS (S3, Glue, Lake Formation, EMR, Redshift)
Hands-on implementation of data mesh principles including domain-oriented data ownership, data as a product, and federated computational governance

Modern Data Platform Technologies:

Proficiency: Databricks (Unity Catalog, Delta Lake), Starburst/Trino (federated query), Collibra (data governance), or Immuta (dynamic data access control)
Ability to architect multi-technology solutions

DevSecOps & Infrastructure as Code:

Strong experience with CI/CD pipelines (GitLab/GitHub Actions, Jenkins)
Containerization expertise (Docker/ECS)
Infrastructure automation (Terraform, CloudFormation)
Security compliance frameworks for AWS GovCloud environments

AI-Augmented Development & Engineering:

Demonstrated use of generative AI tools for accelerating SDLC activities including code generation, testing, and documentation
Experience designing or implementing agentic AI solutions using Amazon Bedrock or similar frameworks

Leadership & Soft Skills

Proven ability to translate business requirements into technical solutions
Experience leading technical design sessions and mentoring engineering teams
Strong communication skills to convey complex architectural concepts to both technical and non-technical stakeholders
Experience working across hybrid cloud/on-premises environments

Primary Talent Partners is an Equal Opportunity / Affirmative Action employer committed to diversity in the workplace. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, age, national origin, disability, protected veteran status, gender identity, or any other factor protected by applicable federal, state, or local laws.

If you are a person with a disability needing assistance with the application or at any point in the hiring process, please contact us at info@primarytalentpartners.com

#PTPJobs

#LI-PTP

Lead Cloud Data Engineer Data Mesh & AI

Primary Talent Partners

San Francisco, CA

About the Company

Primary Talent Partners

Similar Job Searches