Probing to Refine: Reinforcement Distillation of LLMs via Explanatory Inversion

Zhen Tan; Chengshuai Zhao; Song Wang; Jundong Li; Tianlong Chen; Huan Liu

cs.CLFeb 26, 2026

Probing to Refine: Reinforcement Distillation of LLMs via Explanatory Inversion

Authors: Zhen Tan, Chengshuai Zhao, Song Wang, Jundong Li, Tianlong Chen, Huan Liu

Organizations: Arizona State University · University of Virginia · University of North Carolina at Chapel Hill

Abstract

Distilling robust reasoning capabilities from large language models (LLMs) into smaller, computationally efficient student models remains an unresolved challenge. Despite recent advances, distilled models frequently suffer from superficial pattern memorization and subpar generalization. To overcome these limitations, we introduce a novel distillation framework that moves beyond simple mimicry to instill a deeper conceptual understanding. Our framework features two key innovations. First, to address pattern memorization, Explanatory Inversion (EI) generates targeted ``explanatory probes'' that compel the student to articulate the underlying logic behind an answer, rather than just memorizing it. Second, to improve generalization, Explanatory GRPO (EXGRPO) uses a reinforcement learning algorithm with a novel Dialogue Structure Utility Bonus, which explicitly rewards the student for maintaining a coherent reasoning process across these probes. Extensive evaluations on 12 datasets demonstrate significant improvements. Using Gemma-7b as the student model, our method yields an average 20.39% increase over zero-shot performance and a 6.02% improvement over the state-of-the-art distillation baselines. Moreover, models distilled with our method show remarkable training efficiency (e.g., surpassing vanilla fine-tuning with 10-25% training data) and strong generalization to out-of-distribution tasks. Implementation is released at https://github.com/Zhen-Tan-dmml/ExGRPO.git.

Figures & tables

Figure 1: (a) Distilled LLMs often exhibit generalization limitations compared to teacher models (e.g., Gemini-1.5-Pro v.s. smaller distilled models on Test v.s. EI-Test set, which is the augmented version of the test set using Gemini-1.5-Pro and Explanatory Inversion (EI)). See more experimental details in Appendix D . (b) This is exemplified by the reversal curse, where a model correctly solves a forward problem (e.g., 5-2=3) but fails its inverse. (c) Prior “Reverse Thinking” approaches, like RevThink ( Chen et al., 2025a ) , attempt A-to-Q reasoning. (d) Our ExGRPO method enhances distillation by using EI probes to challenge and refine student models via RL.

Figure 2: ExGRPO framework overview. The student model learns from multi-turn explanatory probe dialogues.

Gemini-1.5-Pro (Teacher Model)
Methods	SQA	CSQA	ARC-c	MATH	GSM8K	TabMWP	ANLI	Date	Avg.
Zero-shot ( Kojima et al., 2022 )	78.60	77.90	92.00	77.20	94.50	95.10	71.00	81.50	83.48
Zero-shot-EI [Ours]	78.55 ↓0.05	78.78 ↑0.88	96.82 ↑4.82	80.59 ↑3.39	93.51 ↓0.99	97.61 ↑2.51	74.02 ↑3.02	85.75 ↑4.25	85.70 ↑2.22
Qwen2.5-7B-Instruct (Student Model)
Zero-shot ( Kojima et al., 2022 )	72.93	71.33	85.36	74.40	90.90	93.88	61.67	73.43	77.99
SKD ( Li et al., 2023 ; West et al., 2022 )	73.51 ↑0.58	73.12 ↑1.79	87.26 ↑1.90	74.88 ↑0.48	86.20 ↓4.70	94.17 ↑0.29	62.73 ↑1.06	75.10 ↑1.67	78.01 ↑0.02

Table 1: Main results comparing our ExGRPO against zero-shot performance and various knowledge distillation and data augmentation baselines across eight held-in reasoning datasets for Qwen2.5-7B-Instruct and Gemma-7B-it student models. Teacher model performance is also shown for reference. ExGRPO shows consistent improvements. Scores are reported in percentage (

\%

). * indicates the score is quoted from RevThink ( Chen et al., 2025a ) .The contribution of each rule is studied in Appendix F .

Qwen2.5-7B-Instruct (student)
Method	BoolQ	Openbook	e-SNLI	GSM8K-Rev	Avg.
Zero-shot	71.25	77.80	69.33	75.10	73.87
SKD	74.62 ↑3.37	79.80 ↑2.00	71.20 ↑1.87	78.45 ↑3.35	76.52 ↑2.65
AnsAug	75.53 ↑4.28	80.30 ↑2.50	70.80 ↑1.47	79.62 ↑4.52	76.56 ↑2.69
RevThink	77.10 ↑5.85	82.85 ↑5.05	73.91 ↑4.58	82.40 ↑7.30	79.06 ↑5.19
ExGRPO	80.30 ↑9.05	86.41 ↑8.61	78.44 ↑9.11	86.22 ↑11.12	82.34 ↑8.47

Table 2: OOD generalization on four held-out datasets. ExGRPO significantly improves generalization across both Qwen2.5-7B and Gemma-7B.

Model Configuration	SQA	CSQA	ARC-c	MATH	GSM8K	TabMWP	ANLI	Date	Avg.
Qwen2.5-7B-Instruct (Student Model)
Zero-shot	72.93	71.33	85.36	74.40	90.90	93.88	61.67	73.43	77.99
SFT (1 epoch only)	73.42 ↑0.49	72.10 ↑0.77	85.90 ↑0.54	74.80 ↑0.40	86.55 ↓4.35	94.12 ↑0.24	62.10 ↑0.43	74.02 ↑0.59	77.88 ↓0.11
SFT (3 epochs only)	76.40 ↑3.47	74.50 ↑3.17	89.30 ↑3.94	74.65 ↑0.25	86.60 ↓4.30	94.25 ↑0.37	62.75 ↑1.08	76.90 ↑3.47	79.17 ↑1.18
RL (cold start, $R_{\text{base}}$ )	10.20 ↓62.73	15.80 ↓55.53	18.60 ↓66.76	5.10 ↓69.30	20.90 ↓70.00	22.30 ↓71.58	13.40 ↓48.27	18.60 ↓54.83	15.99 ↓62.00
RL (cold start, $R_{\text{base}}$ ) + $\mathcal{L}_{\text{SFT-aux}}$	63.10 ↓9.83	62.45 ↓8.88	79.40 ↓5.96	66.55 ↓7.85	85.90 ↓5.00	88.95 ↓4.93	54.80 ↓6.87	67.10 ↓6.33	71.53 ↓6.46

Table 3: Ablation study on the impact of SFT warm-up training and RL components for ExGRPO . We evaluate performance across eight reasoning datasets. All model variants are trained using the same augmented datasets with EI probes. RL without

r_{\text{dsu}}

treats each EI probe as an independent training sample without grouping across reasoning paths.

Figure 3: RL training curves with Gemma as student. Evolution of key reward components during ExGRPO training. Left :

R_{base}

. Right :

r_{dsu}

, scaled for visualization.

Figure 4: Sample efficiency comparison on eight datasets. Our ExGRPO method achieves higher accuracy than standard SFT across all training data fractions (

p\in\{0.1,0.25,0.5,1.0\}

), often surpassing SFT trained on the full dataset with only

10-25\%

of the data.

Figure 5: Average accuracy v.s. average training token count. The dashed line shows the regression over the baselines.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Probe Type (Rule)	SQA	ARC-c	GSM8K	ANLI	Avg.
Rephrase Question ( Yu et al., 2024 )	75.3	89.1	87.2	63.9	78.88
Counterfactual (R1)	76.8	89.5	91.2	63.1	80.15
Decomposition (R2)	76.1	88.9	90.5	63.5	79.75
Justification (R3)	75.7	88.3	89.8	62.8	79.15
Analogy-Based (R4)	74.9	87.1	88.4	61.7	78.03
Contrastive (R5)	75.3	87.5	88.6	62.1	78.38

Appendix

Table 4: Performance (Accuracy %) of SFT with single EI probe types on Qwen2.5-7B-Instruct. All EI rules outperform the baseline, and the full EI combination yields the best results.

Dataset	Domain	License	Train (Original)	Train (Filtered)	Test
SQA ( Geva et al., 2021 )	Commonsense	MIT	$2061$	$1656$	$209$
CSQA ( Talmor et al., 2019 )	Commonsense	MIT	$9741$	$7770$	$1221$
ARC ( Clark et al., 2018 )	Commonsense	CC BY-SA 4.0	$1199$	$1080$	$1172$
MATH ( Hendrycks et al., 2021 )	Math	MIT	$7500$	$6228$	$5000$
GSM8K ( Cobbe et al., 2021 )	Math	MIT	$7379$	$7085$	$1319$
TabMWP ( Lu et al., 2022 )	Math (Tabular)	CC BY-SA 4.0	$23\,059$	$22\,802$	$7684$

Appendix

Table 5: The datasets used in this work are listed in the order of appearance. For each dataset, we report the domain, the number of original training samples, the number of filtered training samples, and the number of testing samples. Note that the last four datasets are held out and thus contain no filtered training samples. Due to the large size of ANLI’s training set, we randomly sampled 2,000 instances, of which 883 remained after filtering. For the Date Understanding dataset, given its small size, we randomly split the data into 200 training and 169 testing samples, and we keep all the training data.

Explore similar work

Task-Specific Knowledge Distillation via Intermediate Probes
Feb 18, 2026Ryan Brown, Chris RussellLanguage Model Distillation Knowledge Distillation
Oxford Internet Institute, University of Oxford, Oxford, United Kingdom.
Distilled Reinforcement Learning for LLM Post-training
Jul 19, 2026Chen Wang, Zhaochun Li, Jionghao Bai +4RL for Language Model Reasoning Cross-Architecture Knowledge Distillation
College of Elite Engineers, Nankai University · Zhongguancun Academy · Beijing Institute of Technology +4
Structural Rationale Distillation via Reasoning Space Compression
May 8, 2026Jialin Yang, Jiankun Wang, Jiajun Wu +3CoT Distillation Structured Reasoning
University of Calgary · Calgary, Canada · University of Michigan +1

Feb 18, 2026J/K move · Enter open · S save

Task-Specific Knowledge Distillation via Intermediate Probes

Ryan Brown, Chris Russell

Oxford Internet Institute, University of Oxford, Oxford, United Kingdom.

Language Model Distillation Knowledge Distillation

Open paper AskPDF arXiv

Knowledge distillation from large language models (LLMs) assumes that the teacher's output distribution is a high-quality training signal. On reasoning tasks, this assumption is frequently violated. A model's intermediate representations may encode the correct answer, yet this information is lost or distorted through the vocabulary projection, where prompt formatting and answer-token choices creates brittle, noisy outputs. We introduce \method{}, a distillation framework that bypasses this bottleneck by training lightweight probes on frozen teacher hidden states and using the probe's predictions, rather than output logits, as supervision for student training. This simple change yields consistent improvements across four reasoning benchmarks (AQuA-RAT, ARC Easy/Challenge, and MMLU), with gains most pronounced under limited data. Probes trained on intermediate representations provide cleaner labels than the teacher's own outputs, effectively denoising the distillation signal. \method{} requires no architectural changes to student or teacher, is architecture-agnostic, and adds minimal compute since probe training is cheap and teacher representations can be cached. By exploiting internal representations, \method{} enables practitioners to extract more value from large teacher models without additional training data or architectural complexity.

Abstract

Explore similar work

Task-Specific Knowledge Distillation via Intermediate Probes

Distilled Reinforcement Learning for LLM Post-training

Structural Rationale Distillation via Reasoning Space Compression