ZeroGraph: Compositional Blockwise Training via Materialized Boundary Supervision
Organizations: Johns Hopkins University · Independent Researcher
Abstract
Blockwise learning can remove online gradient dependencies across depth, but independently optimized blocks need not compose into a useful model. We present ZeroGraph, a systems transformation for pretrained-model conversion and distillation that materializes a frozen teacher's coherent boundary trajectory as a versioned dataset. Each partition then trains with disjoint parameters, optimizer state, random state, and checkpoints, replacing live cross-depth dependencies with persistent supervision. Experiments separate learning quality from systems behavior. On GPT-2 Large, an unanchored objective reduces denoising error by 99.44%, yet assembled held-out cross-entropy worsens from 3.63 to 10.48 and context use becomes negative. A six-arm TinyLlama study identifies clean teacher-boundary regression as the successful interface; the tested noisy objective fails with cached or online targets. The clean recipe passes quality and context-use gates across three Qwen2.5-1.5B seeds and on a 3.086B Qwen2.5 model. In an objective-matched comparison, cached and online targets are bitwise identical. Cached jobs are faster after materialization, but a 618.31 s cache-build cost prevents wall-time break-even through 16 measured reuses. Boundary-shift tests show tolerance to intermediate representation mismatch, and an interrupted block resumes to a bitwise-identical final state without rerunning unaffected blocks. In the tested pretrained-model conversion and distillation setting, cross-depth semantic dependence can move from online optimization into persistent data.
Figures & tables
| Metric | DDP | Unanchored ZeroGraph |
|---|---|---|
| Peak allocated HBM | 14.71 GiB | 6.11 GiB |
| Critical compute | 5539.30 s | 879.06 s |
| Held-out CE | 3.6342 | 10.4811 |
| Perplexity | 37.87 | 35633.95 |
| Token accuracy | 34.89% | 5.74% |
| Teacher-gen CE | 2.1558 | 7.5821 |
| Arm | Training signal | CE | Acc. | HBM | Local s |
|---|---|---|---|---|---|
| A | FSDP2 SFT | 1.4643 | 64.78% | 5.18 | 3564.4 |
| B | unanchored local | 1.5554 | – | – | – |
| C | cached clean | 1.3059 | 67.57% | 5.13 | 536.7 |
| D | cached noisy | 1.3173 | 67.26% | 5.13 | 531.3 |
| E | online noisy | 1.3173 | 67.26% | 7.18 | 578.9 |
| F | noisy, , inner FSDP2 | 1.3366 | 66.61% | 4.75 | 2184.6 |
| Cached s/run | Online s/run | Cached $/run | Online $/run | |
|---|---|---|---|---|
| 1 | 723.982 | 116.353 | 0.231102 | 0.103321 |
| 2 | 412.891 | 117.168 | 0.160750 | 0.104045 |
| 4 | 256.587 | 117.050 | 0.124901 | 0.103941 |
| 8 | 176.465 | 117.166 | 0.105227 | 0.104043 |
| 16 | 137.773 | 116.386 | 0.096605 | 0.103351 |
| Seed | FSDP2 CE | ZeroGraph CE | FSDP2 acc. | ZeroGraph acc. | ZeroGraph wall s |
|---|---|---|---|---|---|
| 20260910 | 1.437677 | 1.277886 | 63.970% | 67.107% | 546.368 |
| 20260911 | 1.438250 | 1.276192 | 63.750% | 67.050% | 559.248 |
| 20260912 | 1.432620 | 1.275719 | 63.778% | 67.056% | 569.153 |
| CE | Acc. | Peak GiB | Critical s | Four-GPU wall s | |
|---|---|---|---|---|---|
| 2 | 1.293729 | 66.633% | 12.988 | 941.404 | 997.391 |
| 4 | 1.277886 | 67.107% | 7.448 | 512.553 | 546.368 |
| 8 | 1.274216 | 67.005% | 4.289 | 266.203 | 568.923 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Arm | First use s | Reuse s | Critical s | Wall s | Alloc GiB | Reserved GiB | GPU-h | Ckpt GiB |
|---|---|---|---|---|---|---|---|---|
| FSDP2 SFT | 5034.264 | 5034.264 | 5004.665 | 5031.180 | 7.473 | 11.188 | 5.560662 | 6.620 |
| ZeRO-3 SFT | 3496.523 | 3496.523 | 3480.688 | 3492.273 | 11.517 | 18.646 | 3.867346 | 2.875 |
| Global distill | 5275.348 | 4657.038 | 4628.651 | 4653.857 | 10.713 | 16.336 | 5.140341 | 6.620 |
| Cached clean | 1169.459 | 551.149 | 512.553 | 546.368 | 7.448 | 8.381 | 0.607075 | 4.881 |
| Online clean | – | – | 521.156 | 706.345 | 9.887 | 11.000 | 0.784828 | 4.881 |
| Variant | Mean |
|---|---|
| Frozen base | 65.000% |
| Frozen teacher | 65.078% |
| Global distillation | 63.672% |
| Ordinary FSDP2 | 62.813% |
| ZeroGraph | 64.609% |
| INT8 cached | 63.672% |
Language Model Distillation
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.
See all 562 papers