Research Member of Technical Staff- Training Systems

Rhoda AI · Mountain View · Posted Sep 18, 2026


Transparency

Dead

Online for 9 weeks and never reposted. Jobs this old or reposted this often rarely lead to a hire. It may be filled already.

First seen
Oct 10, 2026
Offered for
63 days

Overview

Category
AI & machine learning
Employment type
Permanent
Workplace type
On-site
Country
United States
Seniority
Staff
Schedule
Full time
Salary
$200,000–$300,000 / year, plus equity and cash value

Key skills

Large‑scale Distributed Training Optimization
Identify bottlenecks and improve compute, communication and memory efficiency for multimodal models across thousands of GPUs.
Parallelism Strategy Design
Define and evolve data, tensor, pipeline and sharding approaches, including hybrid schemes, to maximize scaling.
Performance Measurement & Tooling
Build observability, step‑time breakdowns, MFU metrics and automated regression detection tools.
Training Framework Extension
Contribute code to PyTorch‑based stacks, adding operators, fusions or runtime improvements as needed.
Research Collaboration for Scalable Implementations
Work directly with scientists to translate model innovations into efficient training pipelines.

Key phrases

  • large-scale multimodal training
  • bucketization
  • topology-aware mapping
  • parallelism strategies
  • data / tensor / pipeline / sharding / hybrid approaches
  • activation checkpointing
  • sequence packing/bucketing
  • MFU / throughput / scaling efficiency
  • microbenchmarks
  • performance baselines
  • FSDP / ZeRO-style
  • PyTorch
  • CUDA

From the posting

At Rhoda AI, we’re building the next generation of generalist intelligent robots. We own the full robotics stack from high-performance hardware and robot systems to the infrastructure and state-of-the-art foundation world models that control our robots. Our robots are designed to be generalists capable of operating in complex, real-world environments and handling long-tail edge cases, made possible by our cutting edge research and end-to-end system design. We've raised over $450M and are investing aggressively in model research, infrastructure, hardware development, and manufacturing scale-up to make generalist robotics a reality.

We're looking for a Staff / Principal ML Training Systems Engineer to own training systems performance end-to-end. You will define how our models train at scale — driving efficiency, scalability, and correctness across large-scale multimodal training. This is a core systems role, not infrastructure support. Your work directly determines how efficiently we use compute, how well models scale across thousands of GPUs, and how quickly research can iterate.

What You'll Do

Own training performance end-to-end

  • Diagnose and improve performance of large-scale multimodal training (vision, video, proprioception, actions, language)

  • Build systematic performance attribution: step-time decomposition (compute vs communication vs input pipeline), scaling curves across cluster sizes, and bottleneck identification and prioritization

  • Drive measurable gains in:

    • Distributed efficiency (comm/compute overlap, bucketization, topology-aware mapping, parallelism strategies)

    • Compute efficiency (kernel hotspots, operator fusion, attention optimization, framework/runtime overhead)

    • Memory efficiency (activation checkpointing, sequence packing/bucketing, fragmentation reduction)

Design training systems (not just tune them)

  • Define and evolve parallelism strategies: data / tensor / pipeline / sharding / hybrid approaches

  • Improve execution efficiency through communication scheduling and overlap, graph capture and execution optimization, and runtime-level improvements

  • Contribute to and extend training frameworks where needed

Make performance observable and measurable

  • Establish source-of-truth performance metrics: step-time breakdowns, MFU / throughput / scaling efficiency

  • Build tools to identify bottlenecks quickly, track performance across model families, and compare scaling behavior across configurations

  • Develop regression detection: microbenchmarks, performance baselines, and automated detection of efficiency regressions

Partner deeply with researchers

  • Work side-by-side with research scientists and research engineers — no silos

  • Translate model innovations into scalable, efficient implementations

  • Advise on training tradeoffs for robotics world models: long-horizon sequences, rollout/evaluation cadence, multimodal and variable-length data

Collaborate on cluster-level efficiency

  • Work with infrastructure/SRE teams to improve utilization across large distributed jobs, impact of network and collective performance on training, and topology-aware job placement and scaling behavior

What We're Looking For

  • Proven track record improving large-scale distributed training performance

  • Deep hands-on experience with modern ML stacks (PyTorch required; JAX a plus)

  • Strong understanding of data / tensor / pipeline parallelism, sharded training (FSDP / ZeRO-style), communication patterns and overlap strategies, and scaling behavior across large GPU clusters

  • Strong systems intuition — ability to reason across compute, communication, and memory bottlenecks

  • Exceptional debugging and measurement ability: turn "training is slow" into clear bottlenecks, experiments, and validated improvements

  • High ownership mindset and comfort in a fast-moving environment

Nice to Have (But Not Required)

  • GPU kernel or compiler-level experience (CUDA, Triton, graph capture, operator fusion)

  • Experience with multimodal or video training (variable-length sequences, packing/bucketing)

  • Experience working on large-scale training frameworks or distributed runtimes

  • Familiarity with cluster topology, networking, and large-scale scheduling effects

Why This Role

  • Direct leverage on research velocity — every efficiency gain you make accelerates model iteration across the entire research team

  • Own the scalability and performance of large-scale multimodal training for real-world embodied intelligence, not static benchmarks

  • Improvements you make compound across every training run the company executes — high ownership, high impact, small elite team

More jobs at Rhoda AI

Technical Talent Partner

Rhoda AI · 2 days ago | Sunnyvale, CA | On-site · Senior · Full time

Create CV

Research Intern

Rhoda AI · Last week | Mountain View | On-site · Intern · Full time

Create CV

Software QA Engineer

Rhoda AI · Last week | Mountain View | On-site · Mid level · Full time

Create CV

Optical Sensing Intern - Robot Perception

Rhoda AI · Last week | Mountain View | On-site · Intern · Full time

Create CV

Senior/Staff TPM - Hardware

Rhoda AI · 2 weeks ago | Mountain View | EquityPercentage; 175000–280000 USD / 1 YEAR | On-site · Senior · Full time

Create CV
See all 40 jobs at Rhoda AI