Correctness report
Independent workers on their own data shards, gradients synchronised every step by ring all-reduce or a parameter server. Workers are killed, delayed and partitioned on a schedule while six invariants are checked on every step — so this page can say when something broke, not merely whether the final weights looked plausible.
The property everything rests on
For a mean-reduction loss over a partitioned batch, N-worker training is the same computation as single-process training. That makes correctness falsifiable: every step, the applied update is compared against what one process would have computed over the union of the contributing batches — rebuilt by a route that shares no arithmetic with the production path.
The weighting by shard size is the whole game. An unweighted mean of per-worker means is correct only while every batch is the same size — which stops being true the moment a worker dies or an epoch ends on a short batch. It is invisible on a loss curve, and it is the bug this project was built to catch.
Deliberately not violations
Fault scenarios
Each scenario runs both synchronisation strategies under an identical fault schedule. Loss is shown against effective batch size, so a death reads as both: the batch drops, and convergence pays for it.
The sawtooth on the batch charts is real: an epoch ends on a short batch, so the step that spans the boundary reduces over fewer samples. That is the same uneven batching that makes the correctness check non-vacuous — with equal batches a mis-weighted reduction is indistinguishable from a correct one.
Randomized sweep
The named scenarios only probe the interleavings somebody thought of. This sweep generates random fault schedules across many seeds and worker counts. Every source of randomness is seeded, so any violating seed replays exactly.
Benchmarks
A simulator cannot measure speedup; it can compute what speedup would be under stated assumptions. Those are different claims, so they are tagged and shown apart. Where they disagree, the disagreement is the finding.
Findings
Every defect this build surfaced, including the ones in the test suite and in my own analysis. Two would have produced a confidently wrong result, so they are documented rather than quietly fixed.