From Throughput to Goodput: Measuring Efficiency in LLM Training

Pretraining large language models (LLMs), often with 100 billion parameters or more, is a massive undertaking requiring significant computational resources and time. While speed, measured in tokens per second, is a common metric for evaluating training progress, it doesn’t provide a complete picture of efficiency. A more nuanced approach focuses on “goodput”—the fraction of a system’s potential converted into useful training progress—and is gaining traction as a critical metric for optimizing LLM pretraining.

The Limitations of Throughput

Raw throughput (tokens/sec) is a valuable data point, but it’s context-dependent. It varies based on factors like GPU count, network topology, storage bandwidth, data modality, sequence length, model architecture, and hyperparameters. Throughput is an outcome, not a normalized measure of efficiency. A high tokens/sec rate doesn’t necessarily translate to faster overall training completion.

Introducing Goodput: A Normalized Efficiency Metric

Goodput, formally introduced by Google as ML Productivity Goodput, offers a more comprehensive view of training efficiency. It moves beyond simply measuring how many tokens per second are processed to assess what fraction of the system’s potential is actually contributing to meaningful training progress. Goodput is expressed as a value between 0 and 1, where 1.0 represents continuous, productive training without losses, and 0.5 indicates that half of the potential is being wasted.

Why Normalization Matters: Accounting for Hidden Losses

Throughput can be misleading because it doesn’t account for factors that disrupt training and reduce overall efficiency. These include:

Reliability: Job stability and frequency of restarts.
Recovery & Resiliency: The amount of progress lost during failures and the speed of recovery.
Compute Efficiency: How effectively GPUs are utilized during training.

A training run can appear “fast” based on throughput but be “slow” in terms of wall-clock completion if it’s frequently interrupted, recovers slowly, or runs with low compute efficiency.

Understanding Training Goodput: A Three-Layer Stack

To effectively measure and improve goodput, it’s helpful to consider a three-layer training stack:

Infra Layer: Cluster, orchestration, runtimes, and fault handling.
Framework Layer: Distributed training runtime, checkpointing, and state management.
Program/Model Layer: Parallelism strategy, kernels, precision, and batch/sequence regime.

This layered approach allows for clear attribution of efficiency gains and losses to specific teams and components.

Layer 1: Infra Goodput – “How Often Are We Actually Training?”

Infra goodput measures the availability of the training job—the fraction of time it’s in a healthy training state. It’s calculated as:

Infra Goodput = 1 – (Total Downtime / Measurement Window)

This metric focuses on fault detection, isolation, remediation, and restart latency.

Layer 2: Framework Goodput – “When We Fail, How Much Progress Do We Lose?”

Framework goodput assesses the impact of checkpointing and recovery on training progress. It measures the fraction of time not lost to checkpointing overhead and recovery waste. Frequent checkpointing is not “free resiliency” and can dominate I/O and coordination overhead, while infrequent checkpointing increases rollback loss.

Layer 3: Model Goodput – “Are We Turning Silicon into Math Efficiently?”

Model goodput evaluates how effectively GPUs are utilized during training, using a metric called Model FLOPs Utilization (MFU). Low MFU often results from communication overhead, poor parallelism configuration, microbatch size limitations, or memory bandwidth constraints.

Combined Training Goodput: A Stack-Level Efficiency Metric

End-to-end training efficiency can be calculated as:

Training Goodput (Gt) = Gi * Gf * Gm

Where:

Gi = Infra Goodput
Gf = Framework Goodput
Gm = Model Goodput

This single number provides a stack-aware view of overall training efficiency.

Measuring Goodput in Practice

Effective goodput measurement requires:

Establishing a consistent measurement window (e.g., 24 hours).
Explicitly logging “productive training time.”
Linking disruptions to specific fault events.
Computing MFU from steady-state training data.

Conclusion

LLM pretraining at scale is a complex distributed systems problem. While throughput remains a necessary metric, goodput offers a more stable and actionable measure of efficiency. By focusing on reducing “badput” (losses due to disruptions, recovery, and inefficiency) and treating efficiency as a stack-level property, organizations can achieve more durable gains in LLM training performance.

References:

Worth a look

Goodput: Measuring True Efficiency in Large Language Model Training