On-policy distillation (OPD) is an effective post-training paradigm for large language models but requires a live teacher server throughout training, resulting in substantial infrastructure overhead. We investigate whether OPD can be performed offline by precomputing teacher log-probabilities once over SFT rollouts and reusing them during training. We find that naively doing so fails to reliably match standard OPD, and trace the root cause to a previously overlooked condition we term teacher consistency, requiring that the same teacher be used for both supervised fine-tuning and OPD. Violating this condition introduces a gradient bias that degrades performance for both offline and online OPD. Building on this insight, we propose Lightning OPD, an offline on-policy distillation framework that enforces teacher consistency and eliminates the need for a live teacher server entirely. We prove that, under teacher consistency, Lightning OPD shares the same optimum as standard OPD, with bounded gradient discrepancy and an implicit regularization effect that helps prevent policy drift. Experiments on math reasoning and code generation show that Lightning OPD achieves comparable performance to standard OPD while delivering 4.0x higher training efficiency. Starting from an SFT-initialized Qwen3-8B-Base model, Lightning OPD reaches 69.9% on AIME 2024 in just 30 GPU hours. Lightning OPD further scales to MoE architectures, training Qwen3-30B-A3B to 71.0% on AIME 2024 on a single 8xH100 node, substantially lowering the barrier for academic research on LLM post-training. Our code is released at https://github.com/jet-ai-projects/Lightning-OPD.
Figures & tables
Figure 1: (Top) Performance (Pass@1, %) and training cost of Lightning OPD compared to standard OPD and the SFT baseline on Qwen3-4B-Base and Qwen3-8B-Base models. Lightning OPD achieves comparable performance to standard OPD on both math and coding benchmarks across both scales, while eliminating the need for a live teacher server during training. At the 8B scale, Lightning OPD achieves a state-of-the-art 69.9% on AIME 2024 in just 30 GPU hours , delivering 4.0 × higher training efficiency than standard OPD. (Bottom) Intuitive comparison of GPU resource allocation. Standard OPD requires co-hosting the student and teacher, fragmenting GPU resources. Lightning OPD collects rollouts and teacher log-probabilities offline, dedicating all GPUs to student training.
Figure 2: Overview of Lightning OPD. In the SFT stage, the teacher πT generates trajectories on QSFT then the base model πbase is fine-tuned on these trajectories to obtain the SFT model πref . OPD stage proceeds in two phases. In the preprocessing phase, rollouts are sampled from πref on QOPD and the teacher is queried once to compute and store per-token log-probabilities logπT(at∣st) , forming the offline dataset DOPD . In the training phase, the student πθ is initialized from πref and trained on DOPD . At each step, the per-token advantage At=logπT(at∣st)−logπθ(at∣st) is computed by reading logπT from DOPD while computing logπθ online. During this process, we do not need a live teacher model. In contrast, standard OPD requires a live teacher server throughout training to compute advantages at every gradient step. By eliminating this requirement, Lightning OPD reduces total training cost of OPD by 4.0 × at the 8B scale (from 120 to 30 GPU hours).
Method
Math Reasoning
Code Generation
AIME 2024
AIME 2025
HMMT 2025
Avg.
LCB v5
LCB v6
Avg.
Student: Qwen3-4B-Base Teacher: Qwen3-8B
SFT
56.7
52.1
34.0
47.6
33.8
31.5
32.6
ExOPD Yang et al. (2026a)
61.0
56.0
34.4
50.5
–
29.0
–
OPD
65.4
57.9
39.9
54.4
44.2
39.3
41.8
Lightning OPD
68.1
58.4
39.8
55.4
42.8
40.3
41.5
Table 1: Pass@1 on math and code reasoning benchmarks. Lightning OPD achieves comparable performance to standard OPD across all benchmarks at both model scales, while requiring no live teacher server during training. At the 4B scale, Lightning OPD substantially outperforms ExOPD Yang et al. (2026a) , achieving 68.1% compared to 61.0% on AIME 2024 and 40.3% compared to 29.0% on LCB v6. At the 8B scale, Lightning OPD reaches 69.9% on AIME 2024 and 49.5% on LCB v5. Bold indicates the best result within each model scale.
Method
Qwen3-4B-Base
Qwen3-8B-Base
OPD
72
120
Lightning OPD
20
30
Speedup
3.6 ×
4.0 ×
Lightning OPD breakdown
└ Rollout collection
10
10
└ Teacher logprob precompute
2
4
Table 2: Training cost (GPU hours) of standard OPD vs. Lightning OPD. Lightning OPD achieves a 3.6 × speedup at 4B and a 4.0 × speedup at 8B, requiring only 20 and 30 GPU hours to train a reasoning model respectively. The lower panel breaks down Lightning OPD costs by stage, showing that the actual OPD training consumes only a moderate fraction of the total budget, highlighting how the minimal infrastructure requirement of Lightning OPD makes the training highly efficient.
Table 3: Lightning OPD applied to a Mixture-of-Experts architecture (Qwen3-30B-A3B). By precomputing teacher log-probabilities offline, Lightning OPD enables on-policy distillation of a 30B MoE model on a single 8 × H100 node, achieving state-of-the-art performance on math and code reasoning tasks. In contrast, OPD runs out of memory as it requires co-hosting both a 30B teacher and a 30B student. Bold indicates improvement over the SFT baseline.
Table 4: Teacher consistency ablation (AIME 2024 pass@1, %). Rows indicate the SFT-stage teacher and columns indicate the OPD-stage teacher. The consistent setting ( diagonal, bold ) always achieves the best performance for both standard OPD and Lightning OPD. Teacher inconsistency degrades Lightning OPD more severely than standard OPD (up to 7 points at 8B), because the fixed rollout distribution compounds the gradient bias when the SFT and OPD teachers differ.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
4B Scale
8B Scale
Training steps
3000
3000
Global batch size
256
128
Max sequence length
16384
16384
Learning rate
8×10−5
8×10−5
LR schedule
cosine
cosine
Warmup ratio
0.1
0.1
Appendix
Table 5: SFT stage hyperparameters.
Hyperparameter
4B Scale
8B Scale
Training steps
150
150
Global batch size
256
256
Max response length
4096
4096
Learning rate
2×10−6
2×10−6
LR schedule
constant
constant
Weight decay
0.1
0.1
Appendix
Table 6: OPD stage hyperparameters for standard OPD and Lightning OPD. The same configuration is used for both the math and code domains.
Figure 3: Training dynamics of Lightning OPD (Qwen3-4B-Base student). (a) The mean importance weight wt=πθ/πref drops rapidly to ≈0.94 within the first 20 steps and plateaus, with its standard deviation stabilizing concurrently, validating the implicit regularization in Theorem 3.7 . (b) AIME 2024 pass@1 rises steeply in the first 50 OPD steps and saturates thereafter, confirming that 150 steps is a sufficient training budget. (c) AIME 2024 pass@1 of the SFT model, standard OPD, and Lightning OPD as a function of SFT checkpoint quality. All three improve consistently with more SFT steps, and both OPD variants provide a large, stable gain on top of the SFT baseline at every checkpoint.