Fisher-Informed Recalibration for Feedback-Based On-Policy Self-Distillation of LLMs

Seohyun Lee; Dong-Jun Han; Seyyedali Hosseinalipour; Christopher G. Brinton

cs.LGSep 27, 2026

Fisher-Informed Recalibration for Feedback-Based On-Policy Self-Distillation of LLMs

Authors: Seohyun Lee, Dong-Jun Han, Seyyedali Hosseinalipour, Christopher G. Brinton

Organizations: Purdue University · Yonsei University · University at Buffalo-SUNY

Abstract

Feedback-based on-policy self-distillation has emerged as a promising approach for enabling foundation models, more specifically Large Language Models (LLMs), to learn from their own outputs under external feedback, with a single model serving as both teacher and student. However, such methods can exhibit unstable optimization, conducive to performance collapse during training. To address this limitation, we propose FIRE (Fisher-Informed REcalibration), a dual-branch framework that recalibrates the supervision applied to correct and incorrect on-policy outputs during fine-tuning. For correct responses, FIRE replaces self-distillation with re-weighted on-policy SFT, while for incorrect ones FIRE identifies feedback components that disproportionately influence the teacher-induced update and recalibrates the feedback-conditioned target accordingly. Both branches are influenced by a token-level radius derived in part from a softmax Fisher trace. FIRE separates which direction feedback should move the model from how far the model should move in that direction, while leaving well-behaved feedback supervision unchanged. Our experiments demonstrate that FIRE provides substantially more stable self-distillation while maintaining strong downstream performance, particularly in settings where standard feedback-conditioned distillation becomes unstable.

Figures & tables

Figure 1: The fine-tuning process of the proposed FIRE methodology. Firstly, we separate traces produced from the model into correct and incorrect groups. Next, Fisher-informed re-weighted on-policy SFT is conducted on correct traces while incorrect traces engage in recalibrated self-distillation, re-weighting based on leave-one-out counterfactuals, full feedback, and gradient insights. Both branches are influenced by the Fisher-informed term $\rho_{k}^{t}$ . The two branches are combined to form the final loss. Specifics on differences from similar works is found in Appendix E .

Model	Algorithm	GSM8K	ASDiv	AQuA-RAT	Avg.
Qwen2.5-1.5B-Instruct	Full Context	$52.43{\pm}2.10$	$76.04{\pm}3.25$	$53.47{\pm}1.67$	60.65
	On-Policy SFT	$50.35{\pm}3.35$	$77.26{\pm}0.30$	$50.17{\pm}1.83$	59.26
	Veto	$50.35{\pm}5.22$	$69.97{\pm}1.31$	$54.86{\pm}2.57$	58.39
	TOP-D	$0.00{\pm}0.00$	$0.00{\pm}0.00$	$12.50{\pm}3.25$	4.17
	TrOPD	$45.66{\pm}4.84$	$66.84{\pm}2.87$	$48.78{\pm}2.10$	53.76
	SRPO	$59.55{\pm}1.97$	$80.56{\pm}3.14$	$49.65{\pm}3.39$	63.25

Table 1: Final accuracy (%) across Qwen2.5-1.5B-Instruct and Qwen2.5-3B-Instruct and their associated datasets over three runs. FIRE exhibits superior performance to baselines across multiple datasets and model sizes.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Algorithm	Biology	Chemistry	CS	Economics	Engineering	Physics	Overall
Qwen2.5-14B-Instruct	Full Context	$75.00{\pm}2.95$	$46.88{\pm}1.47$	$54.17{\pm}2.95$	$64.58{\pm}0.00$	$40.62{\pm}1.47$	$40.62{\pm}4.42$	$53.65{\pm}0.25$
	TOP-D	$73.96{\pm}7.37$	$46.88{\pm}4.42$	$53.12{\pm}4.42$	$66.67{\pm}0.00$	$38.54{\pm}4.42$	$42.71{\pm}7.37$	$53.65{\pm}0.74$
	TrOPD	$68.75{\pm}0.00$	$48.96{\pm}7.37$	$53.12{\pm}4.42$	$62.50{\pm}0.00$	$40.62{\pm}7.37$	$43.75{\pm}0.00$	$52.95{\pm}1.72$
	SRPO	$73.96{\pm}1.47$	$45.83{\pm}2.95$	$52.08{\pm}2.95$	$65.62{\pm}1.47$	$40.62{\pm}4.42$	$42.71{\pm}1.47$	$53.47{\pm}0.98$
	DemoPSD	$77.08{\pm}2.95$	$46.88{\pm}1.47$	$54.17{\pm}2.95$	$65.62{\pm}1.47$	$39.58{\pm}0.00$	$40.62{\pm}4.42$	$53.99{\pm}1.23$
	SCOPE	$70.83{\pm}8.84$	$20.83{\pm}0.00$	$47.92{\pm}8.84$	$55.21{\pm}7.37$	$19.79{\pm}13.26$	$27.08{\pm}5.89$	$40.28{\pm}7.37$

Appendix

Table 2: Final accuracy (%) on MMLU-Pro across subject domains using Qwen2.5-14B-Instruct. FIRE achieves the strongest overall performance across the evaluated methods.

Model	Algorithm	GSM8K	ASDiv	AQuA-RAT	Avg.
Qwen2.5-1.5B-Instruct	No Attribution	$61.11{\pm}1.20$	$74.13{\pm}3.05$	$50.35{\pm}4.21$	61.86
	No Projection	$56.77{\pm}7.22$	$69.10{\pm}11.14$	$51.04{\pm}0.52$	58.97
	Hard Gradient Excision	$58.33{\pm}4.13$	$66.32{\pm}13.94$	$53.12{\pm}1.80$	59.26
	FIRE (Ours)	$60.94{\pm}2.71$	$78.30{\pm}3.84$	$52.26{\pm}2.46$	63.83
Model	Algorithm	ARC-Challenge	WinoGrande	WiC	Avg.
Qwen2.5-3B-Instruct	No Attribution	$85.94{\pm}2.27$	$65.45{\pm}5.22$	$65.28{\pm}0.80$	72.22

Appendix

Table 3: Final accuracy (%) for FIRE component ablations across Qwen2.5-1.5B-Instruct and Qwen2.5-3B-Instruct and their associated datasets over three runs. Removing attribution, projection, or replacing projection with hard gradient excision degrades average performance, demonstrating the contribution of the full FIRE objective.

Qwen2.5-1.5B-Instruct
LR Multiplier	GSM8K	ASDiv	AQuA-RAT
$0.5\times$	$54.51{\pm}5.22$	$76.91{\pm}4.55$	$54.86{\pm}1.59$
$1\times$	$63.72{\pm}3.01$	$81.94{\pm}3.35$	$53.82{\pm}1.59$
$2\times$	$61.63{\pm}4.21$	$81.77{\pm}2.39$	$52.78{\pm}2.41$
$4\times$	$54.34{\pm}4.21$	$78.47{\pm}1.67$	$50.17{\pm}1.31$
LR ( $1\times$ ): GSM8K/ASDiv $=4.0\times 10^{-6}$ ; AQuA-RAT $=2.0\times 10^{-6}$ .

Appendix

Table 4: Learning-rate ablation for FIRE. We sweep the learning rate over

\{0.5\times,1\times,2\times,4\times\}

the default learning rate utilized for each dataset-model pair.

Model	Dataset	Peak LR	$a_{0}$	$1-\mu$	Warmup	Steps	Max context	Max new	Train pool	Eval. $n$
Qwen2.5-1.5B	GSM8K	$4{\times}10^{-6}$	$4{\times}10^{-6}$	0.03	25	200	2560	384	1024	192
Qwen2.5-1.5B	ASDiv	$4{\times}10^{-6}$	$4{\times}10^{-6}$	0.03	25	200	2304	320	1600	192
Qwen2.5-1.5B	AQuA-RAT	$2{\times}10^{-6}$	$2{\times}10^{-6}$	0.03	25	250	3072	512	2048	192
Qwen2.5-3B	ARC-Challenge	$4{\times}10^{-6}$	$4{\times}10^{-6}$	0.01	20	200	2560	320	2048	192
Qwen2.5-3B	WinoGrande	$4{\times}10^{-6}$	$4{\times}10^{-6}$	0.01	15	150	2560	320	2048	192
Qwen2.5-3B	WiC	$1{\times}10^{-6}$	$1{\times}10^{-6}$	0.01	15	200	2560	320	2048	192

Appendix

Table 5: Resolved model- and dataset-specific settings. The EMA column reports the student update weight

1-\mu

in equation 4 . The nominal rate

a_{0}

is used only by FIRE.

Method	Method-specific settings
Full Context	No additional method-specific hyperparameter beyond the shared optimizer, generation, EMA, and LoRA settings.
On-Policy SFT	Group size $4$ (1.5B) / $2$ (3B).
Veto	$\beta$ decays linearly from $0.8$ to $0.0$ , forward-KL objective.
TOP-D	Group size $4$ , proximal coefficient $\alpha=0.2$ , PPO clipping $\epsilon=0.2$ , one off-policy epoch, one prompt per internal minibatch.
TrOPD	Group size $4$ , guidance coefficient $\beta=10^{-3}$ , maximum guided prefix length $384$ .
SRPO	Group size $4$ for the small-model experiments and $2$ for scaling, entropy coefficient $\beta=1.0$ , PPO clipping $\epsilon=0.2$ , ratio cap $2.0$ .

Appendix

Table 6: Important method-specific hyperparameters for run configurations. “Small” denotes the Qwen2.5-1.5B/3B experiments, and “scaling” denotes the larger model experiments.

Method	Qwen 1.5B	Qwen 3B	Qwen 14B
Full Context	$8/1$	$8/1$	$2/4$
On-Policy SFT	$2/4$	$2/4$	–
Veto	$8/1$	$8/1$	–
TOP-D	$2/4$	$2/4$	$1/8$
TrOPD	$2/4$	$2/4$	$1/8$
SRPO	$2/4$	$2/4$	$1/8$

Appendix

Table 7: Resolved micro-batch size / gradient-accumulation steps. All entries have effective batch size

8

. A dash denotes a method not used in that reported model-scale comparison.

Explore similar work

Aug 6, 2026cs.LG

On-Policy Self-Distillation without Any Supervision

On-policy (Self-)Distillation (OPD / OPSD) has shown strong potential for post-training large language models (LLMs). However, existing methods still rely heavily on external supervision, including ground-truth signals, environmental feedback, or guidance from larger models, and therefore fall short of genuine "self"-distillation. In this study, we show that on-policy self-distillation can be achieved using only a model's own generations via internal consistency. We propose unsupervised on-policy self-distillation (U-OPSD). U-OPSD first samples multiple rollouts and constructs a pseudo solution by majority vote under a self-consistency threshold. It then conditions the model's distribution on the pseudo-solution and distills itself on the disagreeing completions, allowing the model to correct itself precisely where it is confidently wrong. Across diverse benchmarks, base models, and training settings, U-OPSD consistently improves over the base models and matches or surpasses supervised methods with ground truth (GT) such as OPSD and GRPO. On five mathematical reasoning benchmarks, i.e., AIME24, AIME25, HMMT25, MATH500, and AMC23, U-OPSD improves over the base model by 8.5% and 10.7% on Qwen3 non-thinking mode at 4B and 8B scales, and outperforms OPSD by 3.2% and 2.3% on average, respectively. In thinking mode, U-OPSD stays on par with OPSD, ahead by 0.9% at 4B and level at 8B and surpassing GRPO by 0.7% and 1.1%, respectively. Code is available at https://github.com/williamium3000/u-opsd.

Yijiang Li, Bingyang Wang, Yijun Liang +3

UC San Diego · Georgia Institute of Technology · University of Maryland, College Park +1

Unsupervised On-Policy Self-Distillation Self-Distillation Framework On-Policy Self-Consistency

Ask PDF

Jun 9, 2026cs.AI

The Role of Feedback Alignment in Self-Distillation

Conditioning a language model on additional context, such as feedback on a previous attempt, typically improves its response. Self-distillation trains the model to retain this improvement when the context is not present. The method works by matching the model's output distribution under two settings: a student that sees only the question, and a self-teacher that also sees the context. What the model learns therefore depends on what context the self-teacher receives, yet the design of this context remains largely unexplored. We study context design for self-distillation by training a solver on feedback from a frozen critic. We compare three conditions: (i) a binary reward (GRPO), (ii) the reference solution, and (iii) a step-by-step critique aligned to the solver's reasoning trace. Step-aligned critique yields the largest gains, outperforming GRPO by 16.11 points and reference-solution-conditioned self-distillation by 5.27 points (Avg@12). Per-token advantage analysis reveals why: step-aligned feedback targets only the tokens where reasoning fails, leaving correct behavior intact. Conditioning on the reference solution, by contrast, pressures the model to change its behavior at every token (even correct steps) because an alternative derivation inevitably differs in phrasing and approach. This suggests that structural alignment between feedback and the solver's reasoning is a key driver of self-distillation effectiveness.

Semih Kara, Oğuzhan Ersoy

Gensyn

Self-Distillation Framework Self-Teacher Feedback Critique Role

Ask PDF

Sep 29, 2026cs.AI

Train Ahead, Distill Back: Bootstrapping On-Policy Self-Distillation for Large Language Models

On-policy self-distillation (OPSD) improves large language models by letting a self-teacher with privileged information provide dense token-level supervision on the model's own trajectories. Yet existing methods typically construct the self-teacher from the current, initial, or slowly averaged policy state, leaving the quality of supervision constrained by the teacher's ability to exploit privileged information. We ask whether the model's own optimization progress can instead be recycled into a stronger self-teacher. In this paper, we introduce Bootstrapped On-Policy Self-Distillation (B-OPSD), which temporarily trains the policy ahead to obtain a future teacher, restores the student to the original policy state, and then uses the future teacher to supervise the restarted student. The future teacher improves supervision in two complementary ways, it can generate more reliable privileged trajectories and, conditioned on them, provide more informative token-level targets along the restarted student's on-policy trajectories. Experiments on mathematical reasoning with Qwen3-4B and Qwen3-8B show consistent improvements over standard OPSD in both settings, including gains from 27.50 to 41.30 and from 48.80 to 64.44 in the rollout-privileged setting. Our findings point to a broader principle for self-improving models that future learning progress can be distilled backward, preserving acquired knowledge while bootstrapping beyond the optimization state that produced it.

Zheng Zhang, Xinyue Tan, Lufei Li +3

School of Information Science and Technology, ShanghaiTech University · State Key Laboratory of General Artificial Intelligence, BIGAI · Hefei University of Technology

Unsupervised On-Policy Self-Distillation Teacher Transport

Ask PDF