cs.AINov 1, 2026

ZeroGraph: Compositional Blockwise Training via Materialized Boundary Supervision

Authors: Omar Ramadan, Sam Siavoshian, Amir Kashif Saeed, Benjamin A. Johnson, Amin M. E. -A. Diab, Benjamin M. Rodriguez

Organizations: Johns Hopkins University · Independent Researcher

Abstract

Blockwise learning can remove online gradient dependencies across depth, but independently optimized blocks need not compose into a useful model. We present ZeroGraph, a systems transformation for pretrained-model conversion and distillation that materializes a frozen teacher's coherent boundary trajectory as a versioned dataset. Each partition then trains with disjoint parameters, optimizer state, random state, and checkpoints, replacing live cross-depth dependencies with persistent supervision. Experiments separate learning quality from systems behavior. On GPT-2 Large, an unanchored objective reduces denoising error by 99.44%, yet assembled held-out cross-entropy worsens from 3.63 to 10.48 and context use becomes negative. A six-arm TinyLlama study identifies clean teacher-boundary regression as the successful interface; the tested noisy objective fails with cached or online targets. The clean recipe passes quality and context-use gates across three Qwen2.5-1.5B seeds and on a 3.086B Qwen2.5 model. In an objective-matched comparison, cached and online targets are bitwise identical. Cached jobs are faster after materialization, but a 618.31 s cache-build cost prevents wall-time break-even through 16 measured reuses. Boundary-shift tests show tolerance to intermediate representation mismatch, and an interrupted block resumes to a bitwise-identical final state without rerunning unaffected blocks. In the tested pretrained-model conversion and distillation setting, cross-depth semantic dependence can move from online optimization into persistent data.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Probe-Space Preconditioning for Fast and Stable Zero-Order Training

    Sep 29, 2026Francois Chaubard, Mykel J. Kochenderfer, Chris RéZeroth-Order OptimizationMemory-Efficient Fine-Tuning

  2. OPRD: On-Policy Representation Distillation

    Jun 4, 2026Shenzhi Yang, Guangcheng Zhu, Bowen Song +8Cross-Architecture Knowledge DistillationLanguage Model Distillation