Audit Before You Distill: A Controlled Workflow for Reasoning Transfer into Compact Medical Language Models

Tiago Martinelli; João Paulo Papa; Adriano José Pereira

cs.CLOct 12, 2026

Audit Before You Distill: A Controlled Workflow for Reasoning Transfer into Compact Medical Language Models

Authors: Tiago Martinelli, João Paulo Papa, Adriano José Pereira

Organizations: Einstein Global Advanced Technologies for Equity (E-GATE), Hospital Israelita Albert Einstein, São Paulo, Brazil · Institute of Science and Technology Itaú (ICTi), Itaú Unibanco, São Paulo, SP, Brazil · School of Sciences, São Paulo State University, São Paulo, Brazil

Abstract

Teacher rationales can help a smaller model answer health professions questions, but they can also reinforce errors. We examined when they help and how this evidence can guide training using Brazilian Portuguese questions from medical and multiprofessional residency examinations. First, we measured how a teacher rationale changed the student's probability of selecting the correct answer and compared this effect with rationales drawn from other questions. The complete rationale increased this probability by 21~~p.p. when the teacher was correct and decreased it by 14~~p.p. when the teacher was wrong. By intervening on the reasoning content, we showed that shuffled rationales did not produce the same pattern. Guided by this result, we trained only with rationales attached to correct teacher answers and compared answer-only training, step-by-step distillation, and our proposed Distill-SFT method. Distill-SFT showed a modest advantage over step-by-step distillation on previously missed questions under reasoning, while using fewer training instances by combining the answer and rationale into a single target. We also investigated catastrophic forgetting using the out-of-domain ENEM dataset. Answer-only training reduced reasoning accuracy by 12~~p.p. relative to baseline, compared with approximately 4~~p.p. for either rationale-based training method. Together, these findings support a workflow that audits rationale content before distillation, verifies teacher answers before supervision, and trains compact models efficiently for health professions question answering.

Figures & tables

Figure 1: Study workflow. Panel A shows how the baseline student read increasing portions of a teacher rationale. After each portion, we measured the probability assigned to the correct answer among options A to E. We compared each rationale with one assigned from another question and separated cases in which the teacher was correct from those in which it was wrong. Panel B shows the training targets. Pure SFT uses the answer only. Distill-SFT places the rationale and answer in one target and gives rationale tokens weight

\alpha

. Step-by-step distillation uses separate answer and rationale tasks, with weight

\alpha

for the rationale task. Only rationales linked to correct teacher answers were used for training.

Figure 2: Rationales move the student toward the teacher’s answer. Panel A shows an English translation of one rationale originally written in Portuguese. The example illustrates how successive parts of the rationale were shown to the student. The curves in panels B and C were calculated using the original Portuguese rationales after removing the explicit final answer. They show the mean probability assigned to the correct answer among the five alternatives when the teacher was correct (B) or wrong (C). Solid lines use the original rationales. Dashed lines use rationales assigned from other questions. Shaded regions show 95% bootstrap intervals from resampling questions. With the complete rationale, the effects relative to rationales from other questions were

+0.22

and

-0.14

, respectively, a difference of

0.36

Technique	$\alpha$	Steps	B dir.	B reas.	D dir.	D reas.
Pure SFT	1.0	382	0.30	–	0.19	–
distill-SFT	1.0	382	0.27	0.43	0.15	0.24
distill-SFT	0.3	382	0.28	0.40	0.18	0.22
distill-SFT	0.1	382	0.29	0.41	0.18	0.24
Step-by-step	1.0	678	0.31	0.40	0.19	0.23
Step-by-step	0.3	678	0.33	0.39	0.21	0.23

Table 1: Stage 2 accuracy and optimizer steps. Means across ten data splits. B is the transfer subset and D the held-out difficult subset; “dir.” and “reas.” denote direct and reasoning inference. A dash marks a regime not evaluated.

Figure 3: A localized distill-SFT advantage under reasoning. Points are ten-split means and lines are 95% two-level bootstrap intervals on transfer subset B under reasoning. Color encodes the rationale loss weight

\alpha

and marker shape the technique. Distill-SFT is numerically ahead of step-by-step at every

\alpha

, with a detected difference at

\alpha=1.0

. The vertical dashed line marks pure-SFT direct answering. Table 1 shows the complementary direct-inference advantage of step-by-step.

Figure 4: Rationale supervision mitigates forgetting of pretrained reasoning behavior. Paired change in out-of-domain ENEM reasoning accuracy relative to the baseline model (ten data splits,

\alpha=1.0

). Answer-only SFT shows the largest degradation; distill-SFT and step-by-step retain more of the pretrained reasoning behavior. The dashed line marks no change from baseline.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Contrast	Set	Inference	Difference, p.p. (95% CI)	$t$ - $p$	Wilcoxon $p$
Technique and inference regime at $\alpha=1.0$ , unless specified
distill-SFT minus step-by-step	B	reasoning	$+2.17$ [ $+0.33$ , $+4.00$ ]	0.025	0.035
distill-SFT minus step-by-step	B	direct	$-3.57$ [ $-4.76$ , $-2.38$ ]	$<0.001$	0.002
distill-SFT minus step-by-step	ABC	reasoning	$+1.04$ [ $-0.40$ , $+2.48$ ]	0.137	0.221
distill-SFT minus step-by-step	D	reasoning	$+0.59$ [ $-1.59$ , $+2.78$ ]	0.553	0.711
Technique-by-inference interaction	B	both	$+5.73$ [ $+3.00$ , $+8.47$ ]	0.001	0.004

Appendix

Table 2: Paired training comparisons. Differences are in p.p. across ten paired splits, with paired

t

intervals. The interaction is the reasoning contrast minus the direct contrast for distill-SFT versus step-by-step. Tests are two-sided and unadjusted; Wilcoxon tests provide a sensitivity check.

Sensitivity set	Records	Stem groups	$C_{\mathrm{correct}}$	$C_{\mathrm{wrong}}$	$I$ (95% CI)
All records	4,260	3,792	$+0.22$	$-0.14$	$+0.36$ [ $+0.33$ , $+0.39$ ]
Exclude exact option wording	3,145	2,850	$+0.18$	$-0.12$	$+0.30$ [ $+0.27$ , $+0.33$ ]
Exclude high overlap margin	3,825	3,432	$+0.21$	$-0.14$	$+0.35$ [ $+0.32$ , $+0.38$ ]
Exclude both types of cue	2,920	2,661	$+0.18$	$-0.12$	$+0.30$ [ $+0.27$ , $+0.33$ ]

Appendix

Table 3: Lexical sensitivity. Records count question entries; stem groups collect repeated question stems. Effects follow Eqs. B.2 and B.3 , with bootstrap intervals over the remaining groups.

Set	Generator outcomes	$n$	Distill	Step	Difference (95% CI)	$t$ - $p$	Wilcoxon $p$
B	All	1,430	$+0.21$	$+0.19$	$+0.02$ [ $-0.02$ , $+0.05$ ]	0.183	0.232
B	Both correct	417	$+0.59$	$+0.56$	$+0.03$ [ $-0.02$ , $+0.09$ ]	0.120	0.193
B	Both wrong	658	$-0.11$	$-0.09$	$-0.02$ [ $-0.06$ , $+0.02$ ]	0.101	0.065
D	All	2,020	$+0.08$	$+0.08$	$-0.01$ [ $-0.04$ , $+0.03$ ]	0.630	0.770
D	Both correct	259	$+0.64$	$+0.62$	$+0.02$ [ $-0.07$ , $+0.10$ ]	0.553	0.322
D	Both wrong	1,333	$-0.11$	$-0.10$	$-0.01$ [ $-0.04$ , $+0.01$ ]	0.132	0.049

Appendix

Table 4: Generated rationales. Distill and Step show the change in correct-answer probability with the question’s own rationale versus another question’s rationale. Differences are distill-SFT minus step-by-step, with bootstrap intervals.

n

counts paired evaluations across splits. Outcome groups are descriptive; tests are unadjusted.

Sub.	Regime	$\alpha$	Step	Distill	$\Delta$ (p.p.)	$t$ - $p$
A	direct	1.0	93.1	93.9	$+0.8$	0.20
A	direct	0.3	93.4	93.4	$+0.0$	0.95
A	direct	0.1	92.7	93.1	$+0.4$	0.46
A	reasoning	1.0	82.0	82.7	$+0.7$	0.40
A	reasoning	0.3	80.0	82.2	$\mathbf{+2.2}$	0.02
A	reasoning	0.1	81.9	82.5	$+0.6$	0.55

Appendix

Table 5: Retention of previously correct answers. Mean accuracy on subsets A and C in percent across ten splits. Differences are distill-SFT minus step-by-step in p.p., with paired

t

tests. Bold rows indicate

p<0.05

Set	Offline policy	Acc.	Latency (%)
ABC	answer all directly	0.72	1
ABC	route subset B to reasoning	0.77	31
ABC	reason on all items	0.68	100
ABC	per-question oracle	0.81	10
D	answer all directly	0.15	1
D	reason on all items	0.24	100

Appendix

Table 6: Selective reasoning. Distill-SFT at

\alpha=1.0

across ten splits. Latency is relative to reasoning on every item. Subset and oracle policies use known answer outcomes and cannot be applied prospectively as tested.

Explore similar work

Better Accuracies, Worse Reasoning: A Step-Level Audit of Medical Chain-of-Thought Distillation
May 27, 2026Zhaoyang Jiang, Xuanqi Peng, Fei Teng +5CoT Distillation CoT Evaluation
School of Health & Wellbeing, University of Glasgow, Glasgow, UK · Department of Respiratory and Critical Care Medicine, Shanghai Sixth People’s Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China · School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China +1
Structural Rationale Distillation via Reasoning Space Compression
May 8, 2026Jialin Yang, Jiankun Wang, Jiajun Wu +3CoT Distillation Structured Reasoning
University of Calgary · Calgary, Canada · University of Michigan +1
Which Medical Questions Deserve Rationales? Perturbation-Sensitive Selection for Robust QA
Sep 9, 2026Yuexin Wu, Dayou Yu, Vasile RusData Selection Medical QA
Department of Computer Science, University of Memphis

May 27, 2026J/K move · Enter open · S save

Better Accuracies, Worse Reasoning: A Step-Level Audit of Medical Chain-of-Thought Distillation

Zhaoyang Jiang, Xuanqi Peng, Fei Teng, Zhizhong Fu+4

School of Health & Wellbeing, University of Glasgow, Glasgow, UK · Department of Respiratory and Critical Care Medicine, Shanghai Sixth People’s Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China · School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China+1

CoT Distillation CoT Evaluation Clinical QA Medical Reasoning Teacher-Student Learning

Open paper AskPDF arXiv

Chain-of-thought (CoT) distillation trains a smaller model to imitate a teacher's reasoning trace, but it is typically evaluated by final-answer metrics including accuracy. We ask whether gains in answer quality are accompanied by improvements in the trace. In medical QA, where short answer options can leave a richer clinical justification under-specified, a Qwen3-8B student distilled from a DeepSeek-V3-family teacher improves on MedQA-USMLE answer metrics (SC@64 74.7% to 84.4%; expected calibration error (ECE) 0.096 to 0.034). Yet under a Kimi-K2.6 style-blind LLM-judge audit, its error rate over non-abstained steps rises from 30.6% to 50.3%. In this primary medical setting, answer quality and trace factuality move in opposite directions. This before--after pattern persists across evaluators, teacher strengths, student scales and families, medical benchmarks, and style, segmentation, and answer-correctness controls. A 150-step blinded audit by a clinical expert reproduces the same ordering. Boundary checks narrow the scope of the claim: the risk appears when a compact answer under-constrains the rationale and a capable student can imitate expert-like form without reliably grounding each local claim. Standard answer metrics and aggregate hedging rates do not reveal the shift. When such traces are released or reused, answer-level metrics alone are insufficient.

Abstract

Explore similar work

Better Accuracies, Worse Reasoning: A Step-Level Audit of Medical Chain-of-Thought Distillation

Structural Rationale Distillation via Reasoning Space Compression

Which Medical Questions Deserve Rationales? Perturbation-Sensitive Selection for Robust QA