Book Notes / Machine Learning Systems at Scale (Volume II)

Vijay Janapa Reddi / Reading notes

Parallel work still has to produce one defined update

For a loss that averages independent per-example contributions, synchronous data-parallel workers can evaluate gradients on different samples from the same parameters, combine them, and apply a shared optimizer update with compatible optimizer state. Distribution is correct only if the combined result represents the intended global computation. More devices do not relax that requirement.
Suppose one worker processes one sample and reports a mean gradient of 2; another processes three samples and reports a mean of 6. The global sample mean is (1 × 2 + 3 × 6)/4 = 5. Averaging the two workers' means gives 4. Equal rank weighting matches equal sample weighting only when the local batch sizes are equal, or an equivalent correction has already been applied.
The reduction convention therefore belongs to the optimizer specification. Check whether the collective sums or averages, how each worker scales its loss, and how many samples contributed. An unnoticed normalization change can alter the update while every tensor shape and communication operation remains valid.
Batch-dependent layers or objectives need separate accounting: splitting their inputs can change the gradient itself. Correctly weighting local means cannot repair a change in what those local computations mean.

Ring AllReduce distributes the reduction and the resulting tensor without a central server. For an M-byte tensor across N workers, reduce-scatter and all-gather each take N−1 rounds, transferring M/N bytes per worker per round. Each worker sends 2M(N−1)/N bytes and receives the same amount.
The bytes sent per worker approach 2M as N grows, but the number of rounds grows. With simultaneous full-duplex sending and receiving, a simple model gives 2(N−1)α + 2M(N−1)/(NB), where α is startup time per round and B is effective bandwidth in each direction. Bounded bytes do not imply bounded completion time; startup delay and the physical path still matter.
This idealized model leaves out congestion, reduction work, unequal links, and scheduling interference. It identifies what to measure rather than predicting a cluster from its device count. Many small reductions emphasize startup cost; large tensors emphasize transfer bandwidth. Combining reductions saves startups but can delay communication until more gradients are ready.
Overlap hides communication only while independent computation is available. If a consumer needs a tensor before its transfer finishes, the remaining time is exposed on the critical path. Adding kernel and transfer durations would count overlapping work twice and obscure the wait that actually extends the step.

Increasing the worker count while keeping each local batch fixed also increases the global batch. That changes the optimization experiment. The step can process more examples while making a different amount of progress toward the quality target.
For a fixed training configuration, time to target depends on both steps to target and time per step. A hypothetical run taking 1,000 steps at 0.5 seconds finishes in 500 seconds; one taking 1,400 steps at 0.4 seconds finishes in 560 seconds. The faster step loses because it must be repeated more often.
Larger-batch behavior depends on the workload and tuned hyperparameters. Record the target, batch, optimizer, learning-rate schedule, and stopping rule when comparing configurations. A result about examples per second cannot establish a result about training time or accelerator-hours to the target.

A long distributed run can lose progress even when its ordinary steps are efficient. Checkpointing trades save overhead against the work exposed between saves. A successful save is useful only if it can restore a consistent training state, including the optimizer state and the progress information the continuation requires.
In a first-order model, let C be checkpoint cost, τ the useful-compute interval between checkpoints, and μ the job's mean time between failures. Save overhead contributes roughly C/τ; losing about half an interval per failure contributes τ/(2μ). Balancing them gives τ ≈ √(2Cμ). Making checkpoints more frequent reduces exposed work while increasing time spent saving.
This is an approximation for small overheads and infrequent failures, not a universal scheduler setting. Constant recovery time adds overhead without changing that model's optimum; checkpoint failures, correlated outages, variable save costs, and asynchronous saving can change the model itself. The failure interval must describe the job, not an individual component whose failure rate ignores the rest of its dependencies.
Measure restored progress as well as save duration. Weights that reload while optimizer history resets do not demonstrate continuation of the same training state. The useful unit of progress is computation that both advances the chosen quality target and survives the failures the run actually encounters.