Hindsight-Guided Rationale Distillation for Rare Disease Diagnosis

Aarav Singh; Animesh Pathak; Navyansh Singh

cs.CLOct 2, 2026

Hindsight-Guided Rationale Distillation for Rare Disease Diagnosis

Authors: Aarav Singh, Animesh Pathak, Navyansh Singh

Organizations: Dr. Shyama Prasad Mukherjee International Institute of Information Technology, Naya Raipur

Abstract

We study hindsight-guided distillation for rare disease diagnosis on ZebraMap: a 1.5B student is fine-tuned on chain-of-thought traces from a 8B teacher that observes the ground-truth diagnosis during generation. Absolute accuracy remains low for all models - the task is hard at this scale - but within this ceiling a filtered variant (StudentF) achieves a small, statistically significant accuracy advantage over the teacher (p < 0.001), concentrated in better-represented diseases. The unfiltered student does not significantly outperform the teacher (p = 0.129), establishing that contamination filtering - not hindsight distillation alone - drives the gain. The gap traces to an artifact we term GT hallucination. Label-visible generation causes the teacher to embed "ground truth is X" phrases in its reasoning chain; SFT copies the pattern. At inference, the unfiltered student reproduces the phrase in 33.9% of cases, with severe accuracy degradation when the hallucinated label is wrong. A regex filter removing these slots reduces contamination to near-zero, producing the observed gain - though the effect remains small. We precisely quantify this gain-cost tradeoff, document frequency-dependent knowledge transfer absent from the RL-trained teacher, and characterize a calibration gap that SFT does not close - identifying both as directions for future work.

Figures & tables

Metric	Base	Student	StudentF	Teacher
Mentioned (proxy only)	0.191 [0.162, 0.221]	0.255 [0.223, 0.288]	0.255 [0.222, 0.287]	0.253 [0.220, 0.287]
Accuracy (primary)	0.069 [0.054, 0.083]	0.172 [0.147, 0.198]	0.196 [0.170, 0.223]	0.155 [0.133, 0.179]
Accuracy, upper bound	—	0.174 [0.149, 0.200]	0.197 [0.170, 0.224]	0.163 [0.140, 0.187]
Accuracy, format-compliant ( $n=1887{}$ )	—	0.175 [0.149, 0.202]	0.198 [0.171, 0.226]	0.163 [0.139, 0.187]
<diagnosis> present	—	0.987	0.991	0.962
Mean response length (chars)	1950	4098	3753	3421

Table 1: All metrics for 2,000 cases across 250 diseases. 95% clustered bootstrap CIs shown below each point estimate. Accuracy is the primary metric. Mentioned is a search-space proxy and is never used as an accuracy measure. Base (tagged): base model in zero-shot tag-prompting mode, no chain-of-thought.

Comparison	Metric	RD [95% CI]	Cohen’s $h$ [95% CI]
Primary (M3/Accuracy)
StudentF vs. Teacher	Acc.	$+0.040{}$ [ $+0.019{}$ , $+0.061{}$ ]	$+0.105{}$ [ $+0.052{}$ , $+0.160{}$ ]
Student vs. Teacher	Acc.	$+0.016{}$ [ $-0.004{}$ , $+0.037{}$ ]	$+0.045{}$ [ $-0.012{}$ , $+0.101{}$ ]
StudentF vs. Student	Acc.	$+0.024{}$ [ $+0.006{}$ , $+0.043{}$ ]	$+0.061{}$ [ $+0.010{}$ , $+0.115{}$ ]
StudentF vs. Base	Acc.	$+0.127{}$ [ $+0.105{}$ , $+0.150{}$ ]	$+0.386{}$ [ $+0.327{}$ , $+0.446{}$ ]
Conservative bounds

Table 2: Pairwise comparisons. RD = risk difference (A–B); Cohen’s

h

= effect size on arcsine scale. All CIs: 95% clustered bootstrap;

p

-values: Holm–Bonferroni adjusted clustered permutation. StudentF exceeds the teacher under every Accuracy variant (

p<0.001

primary;

p<0.01

conservative bounds). Student alone does not significantly outperform the teacher (

p=0.129{}

). StudentF vs. Student (

p=0.015{}

) is significant, confirming that filtering—not distillation alone—drives the gain; see Section 5.2 .

Stratum (defined by Student’s reasoning chain)	$n$	Student Accuracy	StudentF Accuracy	Teacher Accuracy
No GT phrase	1322	0.2050 [0.1735, 0.2373]	0.2194 [0.1891, 0.2504]	0.1793 [0.1502, 0.2094]
GT phrase, label correct	74	0.7703 [0.6712, 0.8611]	0.5000 [0.3803, 0.6164]	0.2973 [0.1867, 0.4143]
GT phrase, label wrong	604	0.0265 [0.0142, 0.0408]	0.1060 [0.0810, 0.1329]	0.0960 [0.0721, 0.1219]

Table 3: Accuracy stratified by GT phrase presence in the Student’s reasoning chain. Strata are defined by Student behaviour; StudentF and Teacher values cover the same case subsets for comparability. StudentF produces near-zero GT phrases at inference (3/2,000); Teacher produces none. Because strata are defined retrospectively by Student’s own outputs and the models are unmatched on these cases, comparisons within strata are descriptive, not causal. Clustered permutation test (Student: GT-wrong vs. GT-absent):

p{<}0.001

Panel of Figure 1 — Figure 1: GT contamination cascade for the unfiltered Student. Of 2,000 inferences, 33.9% [0.3185, 0.3595] contain a GT phrase; of those, 89.1% [0.8646, 0.9156] name the wrong disease; of those, 74.5% [0.7079, 0.7810] are copied into the <diagnosis> tag. Error bars: 95% clustered bootstrap CIs.

Panel of Figure 2 — Figure 2: M3 accuracy stratified by GT phrase presence in Student’s reasoning chain. StudentF: zero GT phrases at inference (filter removed all from training). Teacher: zero GT phrases (zero-shot; no label access). Strata are defined by Student’s own reasoning, so cross-model comparisons within a stratum are descriptive rather than causal. Clustered permutation, Student GT-wrong vs. GT-absent: $p{<}0.001$ .

Q	Freq	Stud Acc.	SF Acc.	Teach Acc.
Q1	7	0.104 [0.068, 0.143]	0.133 [0.096, 0.174]	0.148 [0.107, 0.191]
Q2	13	0.115 [0.075, 0.159]	0.139 [0.095, 0.188]	0.137 [0.091, 0.188]
Q3	25	0.185 [0.141, 0.234]	0.214 [0.159, 0.270]	0.159 [0.113, 0.208]
Q4	59	0.286 [0.226, 0.349]	0.298 [0.238, 0.359]	0.177 [0.131, 0.226]
$\rho$		+0.306 ∗∗	+0.242 ∗	+0.029 (ns)
OLS		+0.064 ∗∗	+0.058 ∗	-0.000 (ns)

Table 4: Mean per-disease Accuracy by training-frequency quartile with 95% clustered bootstrap CIs. The StudentF–Teacher gap grows from

-0.016{}

in Q1 (where Teacher marginally leads) to

+0.121{}

in Q4. Teacher shows no frequency dependence.

{}^{**}p{<}0.01

{}^{*}p{<}0.05

, ns:

p{\geq}0.05

Metric	Student	StudentF	Teacher
Think length (chars)	$3252.7235$	$2984.1540$	$3201.5420$
Self-corrections	$0.8290$	$0.1495$	$0.2765$
Expl. uncertainty	4.53	2.12	4.09
Confident assertions	0.31	0.51	0.67
SC ratio ( $\times$ / $\checkmark$ )	1.5143 [1.2969, 1.7884]	1.3365 [0.9646, 1.9537]	1.4650 [1.0984, 2.0598]

Table 5: Mean reasoning-style features (overall) and SC ratio (95% clustered bootstrap CI). All models hedge more when wrong; SC ratio

>1

directionally for all three, though StudentF’s CI

[0.9646{},1.9537{}]

straddles 1.0.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: Error taxonomy following the pipeline A.6 priority cascade (see Appendix A for category definitions). GT-poisoned is exclusive to Student ( $22.5$ %). ICD-10 uncategorized buckets (lavender shades): Both uncategorized = neither predicted nor true disease has an ICD-10 code; Pred. uncategorized = predicted disease lacks an ICD-10 code but true disease has one; True uncategorized = true disease lacks an ICD-10 code but predicted disease has one. These cover rare diseases not yet assigned ICD-10 codes and constitute a large share of wrong-answer cases.

Panel of Figure 4 — Figure 4: All three models’ Accuracy (M3) across bidirectional match strictness levels (mc = 0–10). The StudentF advantage over Teacher is directionally consistent at every setting; the SF $-$ T gap grows from $0.0220{}$ at mc = 0 to $0.0285{}$ – $0.0255{}$ at mc = 5–7, peaking at $0.0400{}$ at the exact/synonym default (mc = None, dashed reference line in annotation). At low mc = 0–4, any-length substring matches inflate all three models’ scores equally, compressing the detectable gap. The default setting is used throughout the main paper.

Dataset — ZebraMap Islam et al. (2026b)
Eval set	250 diseases $\times$ 8 cases $=$ 2,000
Train set	1,090 diseases, 18,388 cases (4 quality filters)
Splits	Raw CoT $\to$ Student; GT-redacted CoT $\to$ StudentF
Hindsight trace generation
Teacher	DeepSeek-R1-Distill-LLaMA-8B Guo et al. (2025) (GGUF Q4_K_M) via llama.cpp
Input	Case narrative + ground-truth label

Appendix

Table 6: Reproducibility settings.

Explore similar work

MedThink: Enhancing Diagnostic Accuracy in Small Models via Teacher-Guided Reasoning Correction
Apr 9, 2026Xinchun Su, Chunxu Luo, Lipeng Ma +2Language Model Distillation Knowledge Distillation
School of Computer Science, Fudan University, Shanghai, China
Audit Before You Distill: A Controlled Workflow for Reasoning Transfer into Compact Medical Language Models
Oct 12, 2026Tiago Martinelli, João Paulo Papa, Adriano José PereiraLanguage Model Distillation Medical LLMs
Einstein Global Advanced Technologies for Equity (E-GATE), Hospital Israelita Albert Einstein, São Paulo, Brazil · Institute of Science and Technology Itaú (ICTi), Itaú Unibanco, São Paulo, SP, Brazil · School of Sciences, São Paulo State University, São Paulo, Brazil
Better Accuracies, Worse Reasoning: A Step-Level Audit of Medical Chain-of-Thought Distillation
May 27, 2026Zhaoyang Jiang, Xuanqi Peng, Fei Teng +5CoT Distillation CoT Evaluation
School of Health & Wellbeing, University of Glasgow, Glasgow, UK · Department of Respiratory and Critical Care Medicine, Shanghai Sixth People’s Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China · School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China +1

Apr 9, 2026J/K move · Enter open · S save

MedThink: Enhancing Diagnostic Accuracy in Small Models via Teacher-Guided Reasoning Correction

Xinchun Su, Chunxu Luo, Lipeng Ma, Yixuan Li+1

School of Computer Science, Fudan University, Shanghai, China

Language Model Distillation Knowledge Distillation Medical LLMs

Open paper AskPDF arXiv

Accurate clinical diagnosis requires extensive domain knowledge and complex clinical reasoning capabilities. Although large language models (LLMs) hold great potential for clinical reasoning, their high computational and memory requirements limit their deployment in resource-constrained environments. Knowledge distillation (KD) can compress LLM capabilities into smaller models, but traditional KD merely transfers superficial answer patterns and fails to preserve the structured reasoning required for reliable diagnosis. To address this, we propose a two-stage distillation framework, MedThink, designed to cultivate robust clinical reasoning in small language models (SLMs). In the first stage, a teacher LLM screens data and injects domain-knowledge explanations to fine-tune a student model, establishing a knowledge foundation. In the second stage, the teacher evaluates the student's errors, generates reasoning chains linking knowledge to correct answers, and refines the student's diagnostic reasoning through a second round of fine-tuning. We evaluate MedThink on general medical benchmarks and a gastroenterology dataset comprising 955 question-answer pairs. Experiments demonstrate that MedThink outperforms six distillation strategies in all benchmarks: achieving an improvement of up to 12.7% over the student baseline in general tasks, and reaching a total top accuracy of 56.4% in gastroenterology evaluation. This indicates that iterative distillation centered on reasoning can significantly enhance the diagnostic accuracy and generalization capabilities of SLMs whilst maintaining computational efficiency. Our code and data are publicly available at https://github.com/destinybird/PrecisionBoost.

Abstract

Explore similar work

MedThink: Enhancing Diagnostic Accuracy in Small Models via Teacher-Guided Reasoning Correction

Audit Before You Distill: A Controlled Workflow for Reasoning Transfer into Compact Medical Language Models

Better Accuracies, Worse Reasoning: A Step-Level Audit of Medical Chain-of-Thought Distillation