FlexPipe - Maximizing Training Efficiency for Transformer-based Models with Variable-Length Inputs

Learn about FlexPipe, a novel distributed system framework designed to maximize training efficiency for transformer-based models when working with variable-length inputs in this 14-minute conference presentation from USENIX ATC '25. Discover how researchers from Jilin University and University of California, Riverside address the inefficiencies caused by substantial fluctuations in computation and memory requirements across training iterations due to static partitioning in distributed frameworks. Explore the first flexible pipeline framework that dynamically adjusts pipeline parallelism through a live flexibility mechanism without compromising training loss, featuring a novel optimization problem formulation aimed at maximizing training throughput by adjusting parallel configurations and an efficient heuristic algorithm to solve it. Examine experimental results demonstrating FlexPipe's achievement of an average 1.25× training throughput improvement compared to state-of-the-art methods, moving beyond traditional single-iteration optimizations to address system-level challenges in variable-length transformer training across distributed environments.