cs.LGFeb 23, 2026

Leap+Verify: Regime-Adaptive Speculative Weight Prediction for Accelerating Neural Network Training

Authors: Jeremy McEntire

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We introduce Leap+Verify, a framework that applies speculative execution -- predicting future model weights and validating predictions before acceptance -- to accelerate neural network training. Inspired by speculative decoding in language model inference and by the Automatically Scalable Computation (ASC) architecture for program execution, Leap+Verify decomposes training into three dynamically detected regimes (chaotic, transition, stable) using activation-space cosine similarity as a real-time Lyapunov proxy signal. Within each regime, analytic weight predictors (momentum, linear, quadratic extrapolation) attempt to forecast model parameters K training steps ahead; predictions are accepted only when validated against a held-out loss criterion. We evaluate Leap+Verify on GPT-2 124M and Qwen 2.5-1.5B trained on WikiText-103 across five random seeds, sweeping prediction depth K in {5, 10, 25, 50, 75, 100}. Momentum-based prediction (Adam moment extrapolation) fails catastrophically at both scales, with predicted losses exceeding actuals by 100-10,000x -- a universal norm explosion in optimizer-state extrapolation. Finite-difference predictors (linear, quadratic) succeed where momentum fails: at 124M, they achieve 24% strict acceptance at K=5 in stable regimes; at 1.5B, they achieve 37% strict acceptance in transition regimes. The scale-dependent finding is in regime distribution: GPT-2 124M spends 34% of training in stable regime, while Qwen 1.5B spends 64% in chaotic regime and reaches stable in only 0-2 of 40 checkpoints. Larger models are more predictable when predictable, but less often predictable -- the practical bottleneck shifts from predictor accuracy to regime availability. Cross-seed results are highly consistent (less than 1% validation loss variance), and the three-regime framework produces identical phase boundaries (plus or minus 50 steps) across seeds.

Figures & tables

ModelChaoticTransitionStableUnknown
GPT-2 124M1.6±1.71.6\pm 1.724.0±2.724.0\pm 2.713.4±3.213.4\pm 3.21.0±0.01.0\pm 0.0
Qwen 2.5-1.5B25.6±0.525.6\pm 0.512.4±1.112.4\pm 1.11.0±1.01.0\pm 1.01.0±0.01.0\pm 0.0
Table 1: Regime breakdown by model scale (mean ±\pm std across 5 seeds).
KKMomentumLinearQuadratic
GPT-2 124M — Transition regime
50.0±0.00.0\pm 0.09.3±3.89.3\pm 3.87.9±3.07.9\pm 3.0
100.0±0.00.0\pm 0.03.3±3.23.3\pm 3.23.4±3.33.4\pm 3.3
250.0±0.00.0\pm 0.00.7±1.70.7\pm 1.70.0±0.00.0\pm 0.0
GPT-2 124M — Stable regime
510.0±8.410.0\pm 8.424.3±6.824.3\pm 6.822.1±10.822.1\pm 10.8
Table 2: Strict acceptance rate (%) in transition regime: predicted loss must improve over current loss. Values are mean ±\pm std across 5 seeds. NN = total checkpoint evaluations across seeds.
KKMomentumLinearQuadratic
GPT-2 124M — Transition regime
50.0±0.00.0\pm 0.099.1±1.999.1\pm 1.9100.0±0.0100.0\pm 0.0
100.0±0.00.0\pm 0.097.3±2.597.3\pm 2.593.4±4.293.4\pm 4.2
250.0±0.00.0\pm 0.044.0±9.844.0\pm 9.837.8±12.037.8\pm 12.0
500.0±0.00.0\pm 0.014.1±6.814.1\pm 6.89.3±7.19.3\pm 7.1
GPT-2 124M — Stable regime
Table 3: Proximity (pct) acceptance rate (%) in transition regime: predicted loss within 5% of current loss. Values are mean ±\pm std across 5 seeds.
 GPT-2 124MQwen 2.5-1.5B
KKPredictedActualRatioPredictedActualRatio
52011.64122×122\times1420.82173×173\times
103601.64219×219\times2500.82305×305\times
2512841.64782×782\times5810.82709×709\times
5045051.642742×2742\times11530.821407×1407\times
7599881.646077×6077\times18190.822218×2218\times
Table 4: Momentum predictor failure at both model scales. “Predicted loss” is the validation loss of the model with momentum-extrapolated weights; “Actual loss” is the current checkpoint’s validation loss. Momentum catastrophe is universal—not scale-dependent.
KKLinear CoVQuadratic CoV
50.0%0.0%
100.0%0.0%
255.2%7.9%
5021.7%18.6%
7527.3%62.5%
10063.3%149.1%
Table 5: Coefficient of variation (%) for proximity acceptance rates across 5 seeds, Qwen 2.5-1.5B transition regime. Lower CoV indicates more consistent results across random seeds.
More in this topic

Neural Network Training Dynamics

Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.

See all 511 papers

Explore similar work

CardsList
Aug 31, 2026

Verification-Aware Training for Speculative Decoding

Speculative decoding accelerates large language model inference by using a draft model to generate candidate tokens, which are verified by the target model in a single forward pass. Verification proceeds sequentially and discards every position from the first rejection onward, yet existing draft training relies on token-level imitation of the target with a fixed per-position weighting that reflects neither property. We introduce Verification-Aware Training (VAT), a plug-in framework that simulates verification at every training step and turns the resulting accept and reject patterns into supervision. VAT consists of two components: (i) a verification head, a lightweight jointly trained binary classifier that supervises the draft model on whether each position survives sequential verification; (ii) verification-adaptive weighting, which replaces the fixed weighting schedule by keeping full weight up to each sample's first rejection point and re-anchoring the decay to start there. VAT modifies only the training objective, so it can be layered on top of existing methods without changing the draft architecture, the target model, or the inference procedure. Applied to EAGLE-3 and DFlash on Qwen3-4B, Qwen3-8B, and LLaMA-3.1-8B, VAT improves average acceptance length by up to 11.4% and wall-clock speedup by up to 8.7%, with consistent gains across math, code, and chat benchmarks. Code will be available at https://github.com/naver-ai/vat

Geonmo Gu, Byeongho Heo, HeeJae Jun +4

NAVER AI Lab · NAVER AI Search Platform · Korea University

Speculative DecodingLLM Inference Acceleration
AskPDF
Oct 9, 2026

Simulating Training Spikes of Deep Neural Networks by Layer-wise Random Feature Models

Intermittent ``spikes'' in loss, during the pre-training of Large Language Models (LLMs), remains a mysterious phenomenon both theoretically and empirically. In this paper, we show that simple quadratic models with heavy-tailed, power-law distributed inputs can replicate these spikes. We introduce a theoretical framework and simulation algorithm to model per-layer training dynamics of Deep Neural Networks (DNNs), using such quadratic models. In a case study on an Adam hyperparameter configuration (where β2β_2 is set close to 11) known to induce spikes in Transformer LMs, our simulation successfully predicts bumps indicating spiky behavior from a given checkpoint. Furthermore, we derive an appropriate β2β_2 schedule from observed metrics, which empirically improves performance.

Ran Tian, Courtney Paquette

Google Deepmind

Random Feature MethodsNeural Network Training DynamicsLanguage Model PretrainingAdaptive Gradient Methods
AskPDF
Mar 2, 2026

Quasar: Quantized Self-Speculative Acceleration for Rapid Inference via Memory-Efficient Verification

Speculative Decoding (SD) has emerged as a premier technique for accelerating Large Language Model (LLM) inference by decoupling token generation into rapid drafting and parallel verification. While recent advancements in self-speculation and lookahead decoding have successfully minimized drafting overhead, they have shifted the primary performance bottleneck to the verification phase. Since verification requires a full forward pass of the target model, it remains strictly memory-bandwidth bound, fundamentally limiting the maximum achievable speedup.In this paper, we introduce Quasar (Quantized Self-speculative Acceleration for Rapid Inference), a novel, training-free framework designed to overcome this "memory wall" by employing low-bit quantization specifically for the verification stage. Our empirical analysis reveals that while aggressive structural pruning significantly degrades verification accuracy, quantization-based verification preserves the logit distribution with high fidelity while effectively halving memory traffic. Extensive experiments on state-of-the-art models (e.g., OpenPangu and Qwen3) demonstrate that Quasar maintains a speculative acceptance length comparable to full-precision methods while achieving a 1.28×1.28\times improvement in end-to-end throughput. Being orthogonal to existing drafting strategies, Quasar offers a generic and efficient pathway to accelerate the verification leg of speculative execution. Code is available at https://github.com/Tom-HG/Quasar.

Guang Huang, Zeyi Wen

HKUST(GZ), Guangzhou, China · HKUST, Hong Kong, China

LLM Inference AccelerationSpeculative Decoding
AskPDF