Senior Resiliency Architect, Hardware Diagnostics and Resiliency Modeling

We are now seeking a Senior Resiliency Architect to support the development of hardware diagnostics and resiliency models for Datacenter and Edge Computing products with applications spanning Agentic and Physical AI. In this role, you will be a key member of a team of innovators, challenging the status quo and pushing beyond boundaries. You will have the opportunity to impact the industry's leading GPUs and SoCs powering product lines ranging from the rapidly growing field of artificial intelligence to self-driving cars and robots.

What you'll be doing:

• Develop and maintain diagnostics software suite to efficiently stress test NVIDIA GPUs and SOCs to identify hardware defects, including defects that cause silent data corruption. These tests will run in large-scale deployments of Datacenter GPUs and Safety SOCs in package/board/rack configurations spanning GPUs, CPUs, and Networking SOCs.
• Analyze coverage gaps in NVIDIA diagnostic suite flagged by silicon failures on customer workloads or test suites. Enhance diagnostics to improve coverage, repeatability of failures detected, and optimize test time.
• Model and analyze RAS metrics like Failures in Time (FIT) for permanent and transient errors, and Availability from GPU to Rack to Datacenter. Use models to identify gaps and drive RAS improvements through new hardware and software features.
• Analyze field data on hardware interruptions and permanent failures, enhance RAS models to better correlate to field data. Ensure optimal fault attribution, containment, and recovery for hardware errors.
• Design, build, and maintain automation scripts and tooling to systematically kick off, monitor, and manage diagnostic jobs across multiple independent compute clusters.
• Develop and maintain dashboards that integrate diverse data sources and formats, deliver insights through analytics and visualization, and ensure reliable hosting, secure access, and multi-user performance.
• Collaborate with engineering teams to ensure system telemetries accurately categorize cluster failures into hardware faults, test-script bugs, or environment configuration issues.
• Ensure availability models and hardware diagnostics evolve to reflect AI workload changes

What we need to see:

• Master’s degree in Computer Engineering, Electrical Engineering, Computer Science or closely related degree or equivalent experience.
• At least 5+ years of relevant experience.
• Knowledge of GPU Architecture and programming, hardware diagnostics
• Good understanding of Reliability (FIT) models, AVF estimation, Availability concepts, ECC/parity/CRC strategies.
• Proficiency in data mining, automation, usage of AI tools and scripting with Python, SQLor similar.
• Proficiency in C/C++, CUDA programming, and using AI tools for coding and debug.
• Strong debugging and analytical skills.
• Excellent interpersonal skills and ability to collaborate with on-site and remote teams.
• Working knowledge of hardware diagnostics, and container-based job execution. 

Ways to stand out from the crowd:

• Strong statistical judgment - particularly around exposure-adjusted rates, sample size, and distinguishing signal from artefact.
• Strong understanding of resiliency and reliability trade-offs in AI data centers, including failure modes, mitigation and recovery strategies, and impact on large-scale training/inference workloads
• Experience in developing embedded software or visualization and data analytics tools.

Your base salary will be determined based on your location, experience, and other objective job-related factors. The expected base salary range for this position is 184,000 USD – 287,500 USD.

This posting is for an existing vacancy. 

NVIDIA uses AI tools in its recruiting processes.

NVIDIA is committed to fostering an inclusive work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.