ML Performance Engineer, Distributed TrainingSkip to main content
Menu
About UsOur Products
Product OverviewzPLATFORMzWAREzFABRICzMARKETPLACEzCLOUDEnterprise GPU cloud for AI workloadszEDGEzDATA
Services
Sovereign AI Data CentersHybrid AI Data CenterEnterprise AI Data CentersAI Data CenterDistributed Cell Tower ComputingGPU Financing
Press & Publications
News & PressNewsroom GalleryPublicationsPress Inquiry
Careers
Open RolesFellowship Program
Contact
Buy Compute
Back to all roles
ML Performance Engineer, Distributed Training
Palo Alto, USAHybridEngineering
210,000-270,000USD / year
About Zettabyte
- AI computing company: we turn GPU hardware into reliable, high-utilisation AI compute through software, orchestration and full lifecycle operations
- What we offer: enterprise GPU cloud and AI inference, software for data centre owners to run their own GPU clouds, turnkey and sovereign AI data centres, edge computing and GPU financing
- Who we work with: GPU owners, data centre operators, enterprises, OEMs, integrators and financiers across multiple markets
- How we work: fast-moving and high-accountability; we value speed, ambition, clarity, ownership and measurable outcomes
Role Summary
The ML Performance Engineer, Distributed Training helps customers and internal teams train large models efficiently on Zettabytes GPU clusters.
The role analyses multi-node training jobs, finds where time is lost in compute, communication and data loading, and delivers changes that raise throughput and model FLOPs utilisation.
The ML Performance Engineer works with infrastructure, networking and solutions teams, and often directly with customer research teams.
Reporting Line
Reports to: Co-CTO and CPO
Key Responsibilities
- Performance analysis: profile distributed training jobs and identify bottlenecks across compute, communication, memory and input pipelines.
- Parallelism strategies: advise on and tune data, tensor, pipeline and expert parallelism for large models.
- Framework tuning: optimise training with PyTorch, Megatron-LM, DeepSpeed, FSDP and related frameworks.
- Benchmarks: build and maintain training benchmarks that validate new clusters and track performance over time.
- Customer engagements: support customers in onboarding and scaling training workloads on zCLOUD and zSUITE-managed clusters.
- Cluster validation: work with infrastructure and network teams to confirm that new clusters meet training performance targets.
- Checkpointing and data: improve checkpoint and data-loading performance with storage and platform teams.
- Knowledge sharing: write guides and reference configurations for common model families and cluster sizes.
Required Qualifications
- 5+ years in ML systems, HPC or performance engineering
- Hands-on experience training large models across many GPUs and nodes
- Deep knowledge of PyTorch and at least one large-scale training framework
- Strong understanding of collective communication libraries such as NCCL and of GPU interconnects
- Experience with profiling tools such as Nsight Systems and PyTorch Profiler
- Programming in Python and C++; CUDA experience is a strong plus
Bonus Qualifications
- Experience at a frontier lab, AI cloud or HPC centre
- Contributions to open-source training frameworks
- Familiarity with mixed-precision formats such as FP8
- Published work on large-scale training efficiency
What Success Looks Like (First 90 to 180 Days)
- A training benchmark suite validates every new cluster before handover
- Reference configurations exist for the most common model sizes
- At least two customer training workloads show measured throughput gains
- Performance findings feed into infrastructure and network designs
Why This Role Matters
- Customer outcomes: training efficiency decides how fast customers reach results and how much they spend.
- Platform proof: strong training performance shows what Zettabytes clusters can do.
Working Norms at Zettabyte
- Bias to action: deliver drafts quickly, then iterate on feedback
- Be precise: double-check numbers and facts; sloppy analysis creates downstream cost
- Make it usable: package outputs for decision-making, not data dumps
- Own the process: whoever drives a deliverable manages stakeholders and timelines
- Learn the ecosystem: understand GPUs, cluster operations and the buyer landscape
Details
- Workplace: Hybrid, 3 days in office and 2 days remote, Palo Alto
- Employment type: Full-time
- Compensation: USD 210,000 to 270,000 a year base salary, plus equity
- Benefits: Medical, dental and vision insurance; 401(k) retirement plan; paid time off and US public holidays; equity
- Work authorisation: Required in the United States
- Equal opportunity: Zettabyte is an equal opportunity employer; qualified applicants with arrest or conviction records will be considered in line with the California Fair Chance Act
ML Performance Engineer, Distributed Training
LocationPalo Alto, USA
Work arrangementHybrid
Employment typeFull-time
TeamEngineering
210,000-270,000USD / year
Apply
Share
ML Performance Engineer, Distributed Training
210,000-270,000USD / year
Apply
Products
Product OverviewzPLATFORMzWAREzFABRICzMARKETPLACEzCLOUDzEDGEzDATA
Services
Sovereign AI Data CentersHybrid AI Data CenterEnterprise AI Data CentersAI Data CenterDistributed Cell Tower ComputingGPU Financing
Company
About UsPress ReleasesCareersFAQPricingContact Us
Resources
PublicationsGlossary of TermsBOTT Methodology
Give us a Call
+886-2-2790-6877Taipei, Taiwan
+1 650 260 1009Palo Alto, California
Get in Touch
[email protected]
Memberships & Commitments
Zettabyte is a supplier of the United Nations Global Marketplace (UNGM #1117361) and is committed to aiding the United Nations in achieving its Sustainable Development Goals.
Zettabyte 2026. All Rights Reserved.
Privacy PolicyReturn & Cancellation PolicyTerms of Service