内容：
Modern AI infrastructure combines specialized training runtimes, GPU kernels, and communication libraries. Optimizing these components individually, however, does not necessarily meet end-to-end requirements for performance and reliability. My research examines these requirements through detailed profiling and rethinks how computation, communication, and state management work together.
In this talk, I will illustrate this approach through TrainMover, a resilient LLM training runtime. Large-scale ML training jobs are frequently interrupted by hardware and software anomalies, failures, and management events such as maintenance and resource rebalancing. Existing solutions, including checkpoint-restart and runtime reconfiguration, incur long downtimes or degrade training performance. TrainMover uses elastic and standby machines to replace affected workers while preserving the training configuration. By coordinating training initialization and communication setup, it moves expensive preparation ahead of interruptions, enabling rapid recovery and planned migration without additional GPU memory overhead. At the 1,024-GPU scale, TrainMover consistently handles interruptions with around 20 seconds of downtime. I will conclude with lessons from TrainMover for designing AI infrastructure as an integrated whole and discuss how these ideas can support elastic training.
