cs.CLNov 3, 2026

Where Do Distilled Students Store Thinking?

Authors: Taeyeong Kim, Jaemin Jung, Ahhyun Kim, Unggi Lee

Organizations: Chosun University · Korea University Sejong Campus

Abstract

Small students degrade on large reasoning models' (LRMs') long solutions, so recent work shortens them by interpolating an LRM with a short-answer model at ratio λλ. We ask what student weights retain when this teacher stops thinking, comparing LoRA adapters of three students from three teacher pairs at each λλ. In each pair, the adapters' update subspace shifts most where the teacher stops thinking and solutions shorten most, locating the change without attributing it. We identify a shared adapter subspace tied to thinking-like behavior (reflective markers, monologue format and long solutions). Shared by adapters before that point, it keeps about half (post hoc) while one pair's teacher halves its solutions but keeps its think tag, and largely empties after thinking stops. Removing it largely turns this behavior off for every pair, student and benchmark. Only in that pair and one other student does it leave clearly fewer markers than removing the update's top components; elsewhere, same-energy random directions often nearly match it. Grafted into a short-answer student at twice its weight, it turns early markers (a post hoc count) on, like the whole update, but the student rarely finishes. Our work identifies a weight subspace tied to distilled thinking-like behavior.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Solving Without Stopping: On-Policy Distillation at Small Scale

    Sep 29, 2026Hongyang Li, Yiming Zhu, Xiao Li +3Reasoning DistillationSmall Language Models

  2. Learning from Think-Mode Advantage via On-Policy Distillation

    Sep 29, 2026Wanqi Ren, Jianxiang Wang, Danxuan Liu +2CoT DistillationLanguage Model Distillation