Where Do Distilled Students Store Thinking?
Organizations: Chosun University · Korea University Sejong Campus
Abstract
Small students degrade on large reasoning models' (LRMs') long solutions, so recent work shortens them by interpolating an LRM with a short-answer model at ratio . We ask what student weights retain when this teacher stops thinking, comparing LoRA adapters of three students from three teacher pairs at each . In each pair, the adapters' update subspace shifts most where the teacher stops thinking and solutions shorten most, locating the change without attributing it. We identify a shared adapter subspace tied to thinking-like behavior (reflective markers, monologue format and long solutions). Shared by adapters before that point, it keeps about half (post hoc) while one pair's teacher halves its solutions but keeps its think tag, and largely empties after thinking stops. Removing it largely turns this behavior off for every pair, student and benchmark. Only in that pair and one other student does it leave clearly fewer markers than removing the update's top components; elsewhere, same-energy random directions often nearly match it. Grafted into a short-answer student at twice its weight, it turns early markers (a post hoc count) on, like the whole update, but the student rarely finishes. Our work identifies a weight subspace tied to distilled thinking-like behavior.
Figures & tables
| GSM8K | GPQA-Diamond | |||
|---|---|---|---|---|
| Condition | Tokens | Acc. | Tokens | Acc. |
| 3,596 | 34.2 | 3,319 | 13.7 | |
| Region removed | 353 | 47.6 | 938 | 20.1 |
| Other removed | 3,268 | 39.3 | 2,687 | 17.6 |
| 411 | 48.5 | 1,226 | 22.0 | |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Output | First 30 and last 40 words | Tokens | Markers | End |
| student | Alright, so I need to figure out how many positive whole-number divisors 196 has. Hmm , okay, let me think about this step by step. I remember that a divisor of … 16 = 12.25, 196 / 24.5 = 8.0625, 196 / 49 = 4, 196 / 98 = 2, 196 / 196 = 1, which is 1 more. So, that’s 196 / 1 = 2, 196 / 2 = 98, 196 | 7,680 | 2.4 | truncated, loop |
| Region removed | Alright, let’s think about this step by step. We’re looking for the number of positive whole-number divisors of 196. 1. Understanding the Problem: - We need to find the number … . - For [math] , the exponents are [math] and [math] . - Applying the formula, we get: [math] 4. Conclusion: - The number of positive whole-number divisors of 196 is [math] . Therefore, the final answer is [math] . | 458 | 0.0 | answered (correct) |
| Length-matched cut | Alright, so I need to figure out how many positive whole-number divisors 196 has. Hmm , okay, let me think about this step by step. I remember that a divisor of … 6.1875, not an integer. 33: 196 / 33 approx. 5.838, not an integer. 34: 196 / 34 = 5.714, not an integer. 35: 196 / 35 approx. 5.514, not an integer. 36: 196 / 36 approx. 5.472 Final Answer [math] | 1,502 | 8.1 | cut, then answered (correct) |
| Graft, | Alright, so I need to find how many positive whole-number divisors 196 has. Hmm , okay, so dividing 196 by some numbers and seeing how many times it divides evenly. Let … So, 196 is 2 squared times 49 squared, which is 4 times 2001. So, 196 is 2 squared times 49 squared, which is 4 times 2001. So, 196 is 2 squared times 49 squared, which is 4 times 2001. So | 7,680 | 1.1 | truncated, loop |
| student | To determine the number of positive whole-number divisors of 196, we first need to find its prime factorization. We start by dividing 196 by the smallest prime number, which is … 2 + 2 = 4. Therefore, the number of positive whole-number divisors is: [math] The positive whole-number divisors of 196 are 1, 2, 4, 7, 14, 49, and 196. Thus, the number of positive whole-number divisors of 196 is: [math] | 397 | 0.0 | answered (wrong) |
| Teacher | Okay, so I have this vector problem here. Let me read it again and try to understand what’s being asked. We have three vectors: a, b, and c. Their magnitudes … This simplifies to: [math] Taking the magnitude of both sides: [math] We compute the magnitude squared: [math] Substituting the known values: [math] This equals [math] since [math] : [math] Solving for [math] : [math] Thus, the final answer is: [math] | 4,866 | 13.3 | training solution |
CoT Distillation
Chain-of-thought distillation transfers a model’s intermediate reasoning to smaller models so they can solve tasks with less compute.
See all 83 papers