Training LLM Judges from Language Feedback via Position-Selective Self-Distillation

Ilgee Hong; Changlong Yu; Zhenghao Xu; Xin Liu; Yuwei Zhang; Qin Lu; Bing Yin; Tuo Zhao

cs.CLSep 30, 2026

Training LLM Judges from Language Feedback via Position-Selective Self-Distillation

Authors: Ilgee Hong, Changlong Yu, Zhenghao Xu, Xin Liu, Yuwei Zhang, Qin Lu, Bing Yin, Tuo Zhao

Organizations: Georgia Institute of Technology · Amazon · UC San Diego

Abstract

We study training LLM judges from natural language feedback, especially for subjective tasks where the verdict depends strongly on which evaluation criteria the judge invokes and how it weighs them. The dominant approach, outcome-supervised RL (e.g., GRPO), credits every token in the rollout with a single scalar determined only by the accuracy of the final verdict, providing no separate credit at the criterion-choice tokens and ignoring the rich language feedback (e.g., preference rationales) that naturally accompanies preference labels. Self-Distillation (SD) is one natural way to use this language feedback: the same model, conditioned on this feedback, acts as a teacher providing dense, position-level supervision. However, not all positions carry equally useful signal. Using the per-position entropy shift between teacher and student, we identify two regimes: context sharpening, where the teacher concentrates probability on a particular feedback-aligned criterion expression, and context spreading, where the teacher distributes probability across multiple feedback-aligned alternatives. We interpret these patterns as follows: sharpening encourages memorization of a particular criterion expression, whereas spreading promotes semantic understanding by preserving these alternatives. Motivated by this asymmetry, we introduce position masking based on the entropy shift that retains the lower tail of the entropy-shift distribution. Experiments show that masking higher-entropy-shift positions improves out-of-distribution generalization over naive SD. The resulting self-distilled judges outperform judges trained with outcome-supervised RL by 2-9 percentage points on the evaluated subjective subcategories, while remaining competitive on objective ones.

Figures & tables

Panel of Figure 2 — Figure 2 : Both $\Delta H_{t}$ tails account for most of the distillation loss. (a) Three-way per-sequence percentile partition (top-30% $\Delta H_{t}$ / middle 40% / bottom-30%): both tails carry $\sim$ 20–28 $\times$ the per-position KL of the neutral middle. (b) Binned mean reverse KL along $\Delta H_{t}$ using 20 equal-count bins (95% x-range shown).

Figure 3 : Context sharpening on two HelpSteer3-Preference rollouts (Qwen3-30B-A3B-Instruct-2507). Each panel shows the user task, the rationale (anchor word red ), the student rollout excerpt with the annotated position $\langle$ Pos $N$ $\rangle$ marked in blue , and student vs. teacher top-5 next-token distributions at that position. All positions sit at criterion-naming slots in the rollout’s opening evaluation-criteria list. The teacher concentrates probability on a token semantically equivalent to the rationale’s anchor while the student is uncertain across multiple plausible criteria. Two further rollouts are shown in Figure 6 .

Figure 4 : Context spreading on two HelpSteer3-Preference rollouts (Qwen3-30B-A3B-Instruct-2507). Same layout as Figure 3 . Each panel features a position where the student commits to a criterion that is non-decisive in the rationale (top-1 probability $\geq 0.79$ ), while the teacher reopens with a spread of alternatives that include semantic equivalents of the rationale’s decisive criterion. All positions sit at criterion-naming slots in the rollout’s opening evaluation-criteria list. Two further rollouts are shown in Figure 7 .

Figure 5 : Distinct semantic clusters of evaluation criteria invoked by trained 30B judges across a sweep of complete-link cosine clustering thresholds. Naive SD invokes a markedly smaller pool than SD+mask at every threshold; the gap widens with stricter clustering.

Prompting
Models	RM-Bench					RewardBench v2						Total Avg.
Models	Chat	Code	Math	Safety	Overall	Factuality	Focus	Math	Precise IF	Safety	Overall	Total Avg.
Claude-Sonnet-4-20250514	79.93	81.77	93.05	95.34	87.52	84.00	87.27	85.25	56.88	94.00	81.48	84.50
DeepSeek-R1	78.38	79.29	94.01	92.27	85.99	74.53	80.40	87.43	43.12	86.00	74.30	80.15
Training (Small Model $<$ 10B)
Think-RM-8B	65.20	54.63	71.92	91.71	70.87	49.26	62.42	58.47	26.88	75.56	54.52	62.70

Table 1 : Judge accuracy (%) on RM-Bench and RewardBench v2 . Dark gray (in bold) and light gray highlight the best and second-best performance per column, respectively. RationaleRM-30B numbers are taken from Wang et al. [2026a] as the model has not been released; RewardBench v2 cells are left empty.

Selector	Creative-writing win rate
Dr. GRPO	0.556
Naive SD	0.612
SD+mask ( $\rho=0.7$ )	0.632

Table 2 : Creative-writing win rate of responses selected through multi-round best-of-eight tournaments. Each tournament uses a different Qwen3-30B-A3B-Instruct judge as its pairwise selector; selected responses are evaluated against the Arena-Hard-v2 reference responses by Claude Sonnet 5.

$\rho$ (fraction masked)	RM-Bench					RewardBench v2						Total Avg.
$\rho$ (fraction masked)	Chat	Code	Math	Safety	Overall	Factuality	Focus	Math	Precise IF	Safety	Overall	Total Avg.
$0.00$ (naive SD)	76.74	76.56	94.45	94.26	85.50	78.53	86.26	81.42	40.62	87.33	74.83	80.17
$0.30$	77.52	75.78	95.02	94.21	85.63	80.00	83.03	85.79	43.75	85.56	75.63	80.63
$0.50$	77.17	77.49	94.85	94.15	85.92	78.95	83.43	84.15	46.88	83.78	75.44	80.68
$0.70$ (canonical)	80.53	80.07	95.63	94.81	87.76	79.16	85.66	84.15	48.75	88.44	77.23	82.50

Table 3 :

\rho

-sweep on Qwen3-30B-A3B-Instruct, all masking the top-

\rho

fraction by

\Delta H_{t}

and training on the remaining bottom

(1-\rho)

Selector	RM-Bench					RewardBench v2						Total Avg.
Selector	Chat	Code	Math	Safety	Overall	Factuality	Focus	Math	Precise IF	Safety	Overall	Total Avg.
Mask top $\rho$ fraction by $\Delta H_{t}$ (ours; drops highest-shift positions)	80.53	80.07	95.63	94.81	87.76	79.16	85.66	84.15	48.75	88.44	77.23	82.50
Mask bottom $\rho$ fraction by $\Delta H_{t}$ (direction flip; drops lowest-shift positions)	75.37	74.22	93.40	94.13	84.28	72.63	84.95	85.55	40.67	87.05	74.17	79.23
Mask bottom $\rho$ fraction by $\|\Delta H_{t}\|$ (both tails; drops smallest absolute shifts)	76.14	76.17	93.47	93.78	84.89	73.05	86.60	84.39	44.00	84.77	74.56	79.73
Random masking	75.11	75.88	93.72	94.41	84.78	77.47	85.45	81.97	40.00	86.67	74.31	79.55
Mask bottom $\rho$ fraction by student entropy	74.59	75.44	93.59	94.15	84.44	76.63	86.46	84.70	40.00	85.78	74.71	79.57

Table 4 : Selector ablation on Qwen3-30B-A3B-Instruct, all at matched mask fraction

\rho=0.7

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6 : Additional context-sharpening examples on two HelpSteer3-Preference rollouts (Qwen3-30B-A3B-Instruct-2507).

Figure 7 : Additional context-spreading examples on two HelpSteer3-Preference rollouts (Qwen3-30B-A3B-Instruct-2507).

Models	RM-Bench					RewardBench v2						Total Avg.
Models	Chat	Code	Math	Safety	Overall	Factuality	Focus	Math	Precise IF	Safety	Overall	Total Avg.
Qwen3-4B-Instruct + Dr. GRPO	68.91	73.83	94.01	90.93	81.92	62.74	75.76	86.34	43.12	83.78	70.35	76.14
Qwen3-4B-Instruct + Naive SD (rubric)	75.45	75.19	93.57	92.34	84.14	61.68	74.23	84.39	36.00	79.09	67.08	75.61
Qwen3-4B-Instruct + SD+mask (rubric)	77.26	75.97	93.28	91.91	84.61	65.89	75.05	84.97	48.00	79.55	70.69	77.65
Qwen3-4B-Instruct + Naive SD (rationale)	75.71	73.34	93.51	92.06	83.66	68.42	80.20	85.79	38.75	85.56	71.74	77.70
Qwen3-4B-Instruct + SD+mask (rationale)	77.95	75.88	94.10	90.95	84.72	70.74	77.78	84.15	45.00	89.11	73.36	79.04

Appendix

Table 5 : Judge accuracy (%) for Qwen3-4B-Instruct using preference rationales or generated per-example rubrics as language feedback. Dark gray (in bold) and light gray highlight the best and second-best performance per column, respectively.

$\rho$ (fraction masked)	RM-Bench					RewardBench v2						Total Avg.
$\rho$ (fraction masked)	Chat	Code	Math	Safety	Overall	Factuality	Focus	Math	Precise IF	Safety	Overall	Total Avg.
$0.00$ (naive SD)	75.71	73.34	93.51	92.06	83.66	68.42	80.20	85.79	38.75	85.56	71.74	77.70
$0.30$	75.54	74.51	93.97	91.38	83.85	67.79	75.96	83.06	41.88	87.33	71.20	77.53
$0.50$	76.92	75.54	93.82	91.74	84.50	67.58	76.36	83.61	46.88	88.22	72.53	78.52
$0.70$ (canonical)	77.95	75.88	94.10	90.95	84.72	70.74	77.78	84.15	45.00	89.11	73.36	79.04

Appendix

Table 6 :

\rho

-sweep on Qwen3-4B-Instruct, all masking the top-

\rho

fraction by

\Delta H_{t}

and training on the remaining bottom

(1-\rho)

Selector	RM-Bench					RewardBench v2						Total Avg.
Selector	Chat	Code	Math	Safety	Overall	Factuality	Focus	Math	Precise IF	Safety	Overall	Total Avg.
Mask top $\rho$ fraction by $\Delta H_{t}$ (ours; drops highest-shift positions)	77.95	75.88	94.10	90.95	84.72	70.74	77.78	84.15	45.00	89.11	73.36	79.04
Mask bottom $\rho$ fraction by $\Delta H_{t}$ (direction flip; drops lowest-shift positions)	76.23	74.12	93.24	92.06	83.91	68.21	80.20	83.61	43.75	88.22	72.80	78.35
Mask bottom $\rho$ fraction by $\|\Delta H_{t}\|$ (both tails; drops smallest absolute shifts)	74.33	73.25	93.85	92.49	83.48	66.53	81.03	84.39	40.67	87.27	71.98	77.73
Random masking	75.28	72.61	93.85	92.19	83.48	68.63	80.40	85.79	37.50	88.22	72.11	77.80
Mask bottom $\rho$ fraction by student entropy	73.64	73.20	93.43	92.06	83.08	69.89	81.01	83.06	38.12	87.78	71.97	77.53

Appendix

Table 7 : Selector ablation on Qwen3-4B-Instruct, all at matched mask fraction

\rho=0.7

Explore similar work

The Role of Feedback Alignment in Self-Distillation
Jun 9, 2026Semih Kara, Oğuzhan ErsoySelf-Distillation Framework Self-Teacher
Gensyn
H $^2$ SD: Hybrid Hindsight Self-Distillation
Jul 21, 2026Qiye Cai, Yichuan Ma, Peiji Li +6Reinforcement Learning With Verifiable Reward Hindsight
1Shanghai Artificial Intelligence Laboratory · 2Harbin Institute of Technology · 3Fudan University +1
Localizing Credit at the Divergence: Path-Conditioned Self-Distillation for LLM Reasoning
Jun 14, 2026Yu Li, Shu Hong, Tian LanUnsupervised On-Policy Self-Distillation Online-Policy Distillation
Department of Electrical and Computer Engineering, George Washington University

Jun 9, 2026, cs.AIJ/K move · Enter open · S save

The Role of Feedback Alignment in Self-Distillation

Semih Kara, Oğuzhan Ersoy

Gensyn

Self-Distillation Framework Self-Teacher Feedback Critique Role

Open paper AskPDF arXiv

Conditioning a language model on additional context, such as feedback on a previous attempt, typically improves its response. Self-distillation trains the model to retain this improvement when the context is not present. The method works by matching the model's output distribution under two settings: a student that sees only the question, and a self-teacher that also sees the context. What the model learns therefore depends on what context the self-teacher receives, yet the design of this context remains largely unexplored. We study context design for self-distillation by training a solver on feedback from a frozen critic. We compare three conditions: (i) a binary reward (GRPO), (ii) the reference solution, and (iii) a step-by-step critique aligned to the solver's reasoning trace. Step-aligned critique yields the largest gains, outperforming GRPO by 16.11 points and reference-solution-conditioned self-distillation by 5.27 points (Avg@12). Per-token advantage analysis reveals why: step-aligned feedback targets only the tokens where reasoning fails, leaving correct behavior intact. Conditioning on the reference solution, by contrast, pressures the model to change its behavior at every token (even correct steps) because an alternative derivation inevitably differs in phrasing and approach. This suggests that structural alignment between feedback and the solver's reasoning is a key driver of self-distillation effectiveness.

Abstract

Explore similar work

The Role of Feedback Alignment in Self-Distillation

H2^22SD: Hybrid Hindsight Self-Distillation

Localizing Credit at the Divergence: Path-Conditioned Self-Distillation for LLM Reasoning

H $^2$ SD: Hybrid Hindsight Self-Distillation