HPC Infrastructure Engineer – GPU Clusters
ElevenLabs
Opportunity published from Ashby · applications stay with the original hiring source.
About the opportunity
ElevenLabs is an AI research and product company transforming human interaction with technology through voice, creative, and API platforms used by millions of users and enterprises.
We have raised $781M in funding with a $22B valuation, backed by prominent investors like Andreessen Horowitz, ICONIQ Growth, and Sequoia.
We are looking for an engineer to join our small research infrastructure team to operate NVIDIA GPU clusters across bare metal and rented capacity, making compute fast, reliable, and efficient to accelerate research velocity and training throughput.
This is a builder-operator role offering high scope, autonomy, and rapid experimentation in a lean team environment.
Key responsibilities
- Operate and improve our GPU fleet end to end, covering provisioning, scheduling, monitoring, upgrades, and capacity planning.
- Build automation for fleet health without human intervention, including node health checks, automated draining, remediation, and burn-in pipelines.
- Own the stack beneath training code, such as OS images, NVIDIA drivers, CUDA, container runtimes, NCCL, and high-speed networking like InfiniBand and RoCE.
- Run and tune job scheduling using tools like Slurm to ensure researchers get compute fairly and fast.
- Build and maintain high-performance storage for datasets and checkpoints.
- Troubleshoot and resolve performance problems, stragglers, degraded links, thermal issues, and flaky GPUs.
- Evaluate rented GPU capacity, run benchmarks, validate performance, and hold providers to SLAs.
- Perform hands-on hardware work when needed, including racking, cabling, diagnostics, and vendor coordination.
- Maintain cluster security by default through access control, network isolation, and secrets management.
Qualifications
- Experience running large-scale Linux server or GPU environments in production with a passion for building and operating.
- Strong knowledge of the NVIDIA stack including drivers, CUDA, NCCL, and DCGM, or deep systems experience with the ability to learn hardware stacks quickly.
- Comfort with bare-metal environments, server hardware, and high-speed networking.
- Proficiency writing automation in Python and/or Bash, alongside Infrastructure as Code tools like Ansible or Terraform.
- Ability to analyze metrics, logs, and PromQL data to troubleshoot and identify root causes.
- Experience owning real scope end to end with strong reliability and accountability.
- Hands-on attitude with willingness to handle any task, including datacenter trips.
Nice to have
- Experience supporting machine learning training workloads from the infrastructure side, including distributed training failure modes and checkpointing patterns.
- Experience evaluating and working with GPU cloud providers.
- Familiarity with parallel filesystems like WEKA and VAST, or large-scale object storage.
- Knowledge of BMC, IPMI, Redfish automation, and PXE provisioning at scale.
- Awareness of power and cooling requirements for dense GPU deployments.
Benefits
- Innovative culture at the forefront of AI development.
- Growth paths and dynamic opportunities to drive impact beyond immediate responsibilities.
- Annual discretionary professional development and learning stipend.
- Annual discretionary social travel stipend to meet up with colleagues.
- Annual company offsite bringing the entire team together in new international locations.
- Monthly co-working stipend for remote team members.
Job details
- Location: Remote globally, with optional work from offices in London, New York, San Francisco, and Warsaw.
About the company
ElevenLabs is an AI research and product company founded in January 2023 that launched the first human-like AI voice model and expanded into ElevenAgents, ElevenCreative, and ElevenAPI platforms.
Backed by top-tier investors like Andreessen Horowitz, ICONIQ Growth, and Sequoia, we operate with a high-velocity, impact-driven, AI-first global team.
Application: UpVagas lists this opportunity from the hiring company or its recruitment platform. The application is completed at the original source.
