Plumbline: When the Loss Lies, Validating LLM Training Correctly
Organizations: Amazon.
Abstract
Correctness is a prerequisite for training large language models (LLMs) at scale, yet it is hard to ensure: a fault can sit in any part of a run, and how it shows differs across frameworks and the libraries they depend on. Some failures are loud and some are silent, and both take time to detect and trace. A single fault can waste a long run: the run crashes late, or the model scores lower only at the final evaluation. We present Plumbline, a correctness validation system for LLM training, used in production at Amazon. Plumbline checks each part of the training lifecycle, from the data a run trains on to the model it serves, down to the new layers each model brings and up to whole runs. Its diagnosis reuses these checks to trace failures to their root causes. Both rest on a systematic correctness methodology designed to carry across models and frameworks. Plumbline supports two agentic use cases: Day-0 onboarding, which validates the training of a newly released model before teams rely on it, and root-cause tracing of training failures. Across 3 frameworks and 5 models, it found 61 problems, 32 of them silent. Of these, 42 are fixed or reported upstream, 8 of them first raised by us. We further group the 61 into seven failure patterns that show where training breaks, and report how problems recur across frameworks and models.
Figures & tables
| Check | Correct training means | Reference | Rule |
| Lifecycle (§ 3.2.1 ) | |||
| Image | the job runs the framework commit under test with the dependency versions it declares | framework declarations | exact |
| Collective | each collective computes its defined result on every rank | closed form | exact on every rank |
| Data | training sees each conversation as inference does, and the loss covers the text the model generates | official template, tokenizer | exact on every row |
| Build and load | the model built and loaded is the released model | official configuration, original checkpoint | exact, and unsupported settings are refused |
| Save and resume | a resumed run continues from its saved state | state at the save | bitwise on every rank |
| Model | Model-specific architecture | Nodes |
| gpt-oss-20b ( OpenAI, 2025 ) | MXFP4 experts | 1 |
| Qwen3.8-Flash-Next ( Qiu et al., 2026 ) | linear attention, sparse attention and its indexer, hyper-connections, n-gram table, MTP | 4 |
| DeepSeek-V4-Flash ( DeepSeek-AI, 2026a ) | sparse attention, hyper-connections, MTP, FP8/FP4 weights | 8 |
| GLM-5.3-Flash ( GLM-5-Team et al., 2026 ) | sparse and linear attention, hyper-connections, MTP, FP8 weights | 8–12 |
| DeepSeek-V4.1-Flash ( DeepSeek-AI, 2026b ) | sparse attention, Engram tables, hyper-connections, DSpark draft layers, FP8/FP4 weights | 8–16 |
| Failure pattern | Lifecycle | Architecture | Code reading | Convergence | Total |
| Tokens or loss masks differ from the release | 11 (11) | – | – | 1 | 12 (11) |
| Layers break under parallelism, packing or other kernels | – | 9 (5) | 1 (1) | 4 | 14 (6) |
| Failures depend on sequence length | – | – | – | 9 (1) | 9 (1) |
| Optimizer updates are lost or misapplied | 3 (3) | 1 (1) | – | 3 | 7 (4) |
| Resumed state differs from the checkpoint | 5 (5) | – | – | – | 5 (5) |
| Conversion alters the checkpoint | 7 (4) | – | 2 (1) | 1 | 10 (5) |
LLM Reliability
Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.
See all 866 papers