Disposition Distillation at Small Scale: A Three-Arc Negative Result

Hari Sadasivan

cs.LGApr 13, 2026

Disposition Distillation at Small Scale: A Three-Arc Negative Result

Authors: Hari Sadasivan

Organizations: Tinman Lab

Abstract

We set out to train behavioral dispositions (self-verification, uncertainty acknowledgment, feedback integration) into small language models (0.6B to 2.3B effective parameters) through a four-stage all-MIT distillation pipeline, with follow-on experiments on inference-time attention-head interventions and a frozen-base confidence-gated sidecar. An internal draft reported +33.9-point MCAS and +15.3-point HumanEval gains on a Qwen3-0.6B student; a second-pass sanity check falsified both numbers before publication. The HumanEval delta was a truncation artifact (n_predict=512) that inverted to -8.0 points at n_predict=1024; the MCAS gain disappeared under apples-to-apples scoring. That falsification triggered three subsequent arcs. Across (1) SFT/DPO LoRA on three model families and two domains, (2) inference-time attention-head tempering on o_proj, and (3) a training-free frozen-base sidecar reading the final-token hidden state h_last, we find no operator that moves judge-measured disposition without damaging content or collapsing into stylistic mimicry. The failure is consistent across five models (Qwen3-0.6B, Qwen3-1.7B, Qwen3.5-0.8B, Gemma 4 E2B, and SmolLM2-1.7B-Instruct). A within-distribution cross-validation pass (AUC=0.683) collapsed to chance on fresh prompts (AUC=0.516). We contribute a three-arc negative result with mechanism, a two-failure-mode taxonomy for linear h_last probes, and an honest falsification pipeline that converts the class of false positives we ourselves produced into publishable negatives. As an independent finding, Gemma 4 E2B exhibits near-complete confidence-correctness decoupling on the Chef domain (assertion asymmetry -0.009; the model asserts at 91% regardless of correctness).

Figures & tables

Figure 1: Arc 1 hook. Our first-pass evaluation on Qwen3-0.6B reported

+15.3

HumanEval and

+33.9

MCAS; these are the numbers the withheld draft was built around. Under honest re-evaluation at

n_{\text{predict}}{=}1024

with apples-to-apples judge prompting, both results invert. The same four-stage pipeline applied to Qwen3-1.7B produces a

-15.8

-point v2-harness regression driven by a training-distribution reasoning-compression failure.

Model	Method	Baseline	DD	$\Delta$
Qwen3-0.6B	SFT LoRA (all mods), HumanEval	$36.0\%$	$28.0\%$	$\mathbf{-8.0}$
Qwen3-0.6B	Correction recovery	$10.5\%$	$7.5\%$	$\mathbf{-3.0}$
Qwen3-0.6B	MCAS apples-to-apples (/25)	$12.39$	$12.07$	$\mathbf{-0.32}$
Qwen3-1.7B	Attn-only LoRA, v2 harness	$83.5\%$	$67.7\%$	$\mathbf{-15.8}$
Qwen3-1.7B	All-modules LoRA ( $r{=}32$ )	$83.5\%$	$62.8\%$	$\mathbf{-20.7}$
Gemma 4 E2B	SFT+DPO LoRA, Chef checklist	$0.452$	$0.379$	$\mathbf{-7.3}$

Table 1: Arc 1 summary: honest re-evaluation of SFT/DPO LoRA across three model families and two domains.

#	Model	Disposition	Attrib.	Disp. moves	Content
1	Gemma 4 E2B	Humble	mag.	frozen	$-7$ pt
2	Gemma 4 E2B	Adv-self	mag.	wrong direction	flat
3	Qwen3.5-0.8B	Humble	mag.	non-monotonic	drops/recovers
4	Qwen3.5-0.8B	Humble (rerun)	mag.	wrong direction	worse
5	Qwen3.5-0.8B	Humble	dir.	no separable axis	N/A

Table 2: Arc 2 tempering sweep. Five variants across two bases and two dispositions. Zero of five produce a clean positive.

Figure 2: Arc 3 headline. The within-distribution CV ceiling (Step 1,

\text{AUC}{=}0.683

) collapses to near chance the moment any escape hatch is closed: held-out fresh generation on the same distribution (Step 2), multi-layer pooling (Step 4), and cross-model replication on SmolLM2-1.7B (Step 5) all land below the

\text{AUC}{\geq}0.63

gate. Dashed line: chance; dotted line: gate threshold.

	Assertive	Hedged	Total
Correct (coverage $\geq 0.5$ )	53	5	58
Wrong (coverage $<0.5$ )	38	4	42
Total	91	9	100

Table 3: Gemma 4 E2B baseline disposition: the

2\times 2

matrix. The model asserts at

\sim

91% regardless of correctness.

Axis	Disposition	pos (s16/fresh)	CV AUC	Transfer AUC	Pass
factual accuracy	capability anchor	49 / 101	0.533	0.480	$\times$
hedging approp.	Humble	79 / 123	0.599	0.515	$\times$
pedagogical framing	Deliberate	81 / 147	0.481	0.439	$\times$
self-verification	Adversarial-self	26 / 76	0.594	0.518	$\times$
completeness	Persistent	47 / 130	0.510	0.442	$\times$

Table 4: Per-axis disposition probe sweep on Gemma 4 E2B. Zero of four pass; three of five transfer AUCs are below chance.

Figure 3: Arc 3 Step 3: per-axis disposition sweep on Gemma 4 E2B. Gold bars are within-distribution CV AUC on the step16 set; red bars are transfer AUC on the 193 fresh prompts. Zero of four locked dispositions pass; three of five transfer AUCs are below chance.

Figure 4: SmolLM2-1.7B final-token residual stream on the 193 fresh Chef prompts, projected to the top two principal components. Blue: factually correct (

n{=}9

); red: factually wrong (

n{=}184

). No linear or visible nonlinear separation exists.

Metric	Gemma 4 E2B	SmolLM2-1.7B
Step16 pos rate	$0.58$	$0.56$
Fresh pos rate	$0.285$	$\mathbf{0.047}$
Fresh AUC (best $C$ )	$0.516$	$0.519$
Fresh best asym	$-0.043$	$-0.188$ (deg.)
Gate pass	$\times$	$\times$
Primary failure mechanism	Single output distribution	Capability cliff on fresh

Table 5: Two distinct failure modes on two frozen bases, same verdict.

Arc	Operator class	Bases tested	What moved	What didn’t
1	Imitation into weights (SFT/DPO LoRA)	Qwen3-0.6B, Qwen3-1.7B, Gemma 4 E2B	Surface style (openers, hedges)	HumanEval, MCAS, recovery, checklist
2	Inference-time $o_{\text{proj}}$ tempering	Gemma 4 E2B, Qwen3.5-0.8B	Nothing stable	Humble, Adv-self; content damaged
3	Frozen-base $\mathbf{h}_{\text{last}}$ sidecar	Gemma 4 E2B, SmolLM2-1.7B	Nothing (AUC $\approx$ chance)	Factual anchor + 4 dispositions

Table 6: Three arcs, three operator classes, same bottom line.

Language Model Distillation

Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.

See all 512 papers

What the Abstention Option Measures: Refusal Loss, Sampling Churn, and Option Capture in Distilled Small Language Models
May 19, 2026Plawan Kumar RathRefusal Behavior in Language Models LLM Abstention
Meta
Self-Verified Distillation: Your Language Model Is Secretly Its Own Synthetic Data Pipeline
May 20, 2026Tony Lee, Percy LiangLanguage Model Self-Verification Self-Training for Language Models
Stanford University
Bounded Behavioral Indistinguishability for Black-Box LLM Distillation
May 28, 2026Munawar HasanLLM Evaluation Language Model Distillation
Michigan Technological University, Houghton, MI, USA

May 19, 2026J/K move · Enter open · S save

What the Abstention Option Measures: Refusal Loss, Sampling Churn, and Option Capture in Distilled Small Language Models

Plawan Kumar Rath

Meta

Refusal Behavior in Language Models LLM Abstention Language Model Bias Evaluation Language Model Distillation

Open paper AskPDF arXiv

Small instruction-tuned language models reach phones, cars, and offline assistants through knowledge distillation, and whether a distilled model is fair enough to ship is often decided by a multiple-choice bias score on a benchmark that offers an abstention option. Three mechanisms move behavior around the abstention option: refusal loss, where the student stops declining on items its pre-distillation reference declined; sampling churn, the same change observed between two runs of the reference itself; and option capture, where the student takes an abstention option whenever one is offered. Injecting each mechanism into a reference's own records shows that aggregate scores see only outright loss, per-item scores see churn and remapping, and only conditioning on committed answers stops moved refusal mass from reading as fairness. Resampling a reference across inference seeds already turns 10% to 27% of its refusals into stereotype answers, so a per-item refusal metric reported without that floor overstates damage. Option capture is what corpus repair teaches: students trained with refusal exemplars take an offered abstention option on up to 95% of science questions where abstention is never correct, and the habit carries to three further benchmarks. Free-form evaluation shows that the refusals multiple choice loses are not where free-form stereotyping occurs. We distill these findings into a per-condition report anchored to the reference, with measured floors and ceilings, and apply it to four student families against a fine-tuning control: distillation keeps fewer of the reference's refusals than fine-tuning on the same prompts where the reference's refusals are reliable, by a few points, while the common metrics report improvements whenever refusal mass moves. A bias score quoted for a distilled model should say which of these it reports.

Abstract

Language Model Distillation

Explore similar work

What the Abstention Option Measures: Refusal Loss, Sampling Churn, and Option Capture in Distilled Small Language Models

Self-Verified Distillation: Your Language Model Is Secretly Its Own Synthetic Data Pipeline

Bounded Behavioral Indistinguishability for Black-Box LLM Distillation