Senior System Software Engineer, DGX Cloud AI Infrastructure
NVIDIA
Opportunity published from Workday · applications stay with the original hiring source.
Source verified 6 hours agoThe original job page was active during UpVagas' latest lifecycle check.
About the opportunity
NVIDIA’s DGX Cloud team builds the infrastructure behind large-scale AI training for NVIDIA’s own training workloads and for customers, creating services, tools and open-source software that make ML workloads more efficient and resilient.
We are looking for a Senior Software Engineer for the NVIDIA Recovery and Resilience team to own our AI infrastructure services, tools and open-source projects from design through production, mentor engineers, and guide team operations.
Key responsibilities
- Drive the design, build, launch and operation of our services, tools and open-source projects.
- Plan and assign the team’s work, mentor junior engineers and review designs.
- Build resilient AI infrastructure on Kubernetes and Slurm for NVIDIA and customer workloads across cloud supercomputers and on-premises clusters.
- Build APIs connecting to NVIDIA’s resiliency systems, including telemetry, break/fix automation, and checkpoint and execution systems.
- Build a job submission layer that hides infrastructure failures from users and automatically restarts long-running training jobs.
- Build agentic tools that help customers improve training efficiency by finding bottlenecks, diagnosing failures and recommending fixes.
- Drive open-source work by releasing components for community use and collaborating to improve them.
- Establish service operations including support, monitoring, alerts, CI/CD, integration test gates and on-call plans.
Qualifications
- BS, MS or PhD in Computer Science or a related field (or equivalent experience), with 12+ years of experience.
- Proven experience taking large systems from design through production.
- Strong experience with Kubernetes and/or Slurm at scale.
- Hands-on experience with PyTorch, TensorFlow, JAX or Ray.
- Strong understanding of NVIDIA GPUs and networking, including failure modes.
- Experience running production services, including CI/CD, monitoring, alerting and on-call.
Nice to have
- Hands-on experience with fault tolerance or checkpointing for large-scale distributed training on GPU supercomputers.
- Demonstrated experience building LLM-based agents or AI tools for performance analysis or troubleshooting.
- Background in building external services or maintaining open-source projects in ML infrastructure.
Benefits
- Eligible for equity.
- Comprehensive company benefits.
Job details
- Base salary range: 224,000 USD – 356,500 USD for Level 5; and 272,000 USD – 431,250 USD for Level 6, determined by location, experience, and other objective job-related factors.
- Applications accepted at least until November 10, 2026.
- Posting type: Existing vacancy.
Additional information
- NVIDIA uses AI tools in its recruiting processes.
Application: UpVagas lists this opportunity from the hiring company or its recruitment platform. The application is completed at the original source.
Source transparency. UpVagas organizes information provided by the company or recruitment source. Always review the original listing before submitting personal information.
