A language model's confidence on a physics problem deserves trust only if it ignores details irrelevant to the answer. We introduce DECOY (Distractor-Embedded Certified cOunterfactuals for phYsics): 155 families of four prompts differing in one irrelevant detail. A symbolic solver certifies that each edit preserves the answer. Families are labelled by the largest drop in the model's log-probability of the certified answer, using a threshold fixed before probe training. Qwen2.5-7B-Instruct is sensitive on 50.0% of the 140 families yielding labels, and Llama-3.1-8B-Instruct on 22.9%. Effects are largest when distractor values lie numerically close to the answer. A linear probe on Qwen's layer-18 hidden state, averaged over four renderings, predicts sensitivity before generation at AUROC 0.760 with whole families held out. A separately tuned single-rendering probe reaches 0.747. A frozen layer-16 probe reaches 0.820 pooled AUROC on 15 families built after training, though within-cue-type AUROC is lower (0.667-0.6875). Under a paired family bootstrap, the probe performs on par with a fitted token-entropy baseline (AUROC 0.727) and achieves the lowest Brier score among tested predictors (0.204 versus 0.214). Knocking out four attention heads at layer 16 cancels sensitivity on a 20-family panel, localising it to a narrow mechanism. These results show that Qwen's confidence can track irrelevant details, that this failure is detectable before generation, and that targeted intervention can suppress it. DECOY provides certified ground truth and paired-tested baselines for evaluating hidden-state monitors.
Figures & tables
Figure 1: Overview of the pipeline, following the three levels of Section 3.1 . A solver certifies that the cue cannot affect the answer (Level 1). One teacher-forced pass per rendering gives the family’s sensitivity label (Level 2). A probe on the layer-18 hidden state of Qwen2.5-7B-Instruct predicts that label before generation (Level 3), and the probe is evaluated against non-activation baselines and by head knockout.
Cue type
Families
Labelled
Sensitive (%)
Mean drop
A: irrelevant variable
35
35
12 (34.3)
0.515
B: unit-matched distractor
105
100
53 (53.0)
1.330
C: equivalent units
15
5
5 (100.0)
9.022
Domain
Mechanics
91
86
46 (53.5)
1.575
Electrostatics and circuits
43
37
17 (45.9)
1.172
Table 1: DECOY composition and the sensitivity of Qwen2.5-7B-Instruct at the pre-registered 0.5-nat threshold. Families : certified families. Labelled : certified families included in the primary Qwen panel (excludes 15 families held out to test transfer: 0 Cue A, 5 Cue B, 10 Cue C). Sensitive : labelled families whose log-probability drop reaches 0.5 nats. Mean drop : mean Sθlp in nats over labelled families.
Figure 2: Left: distribution of the log-probability drop Sθlp over the 140 labelled families for both models on identical prompts. The dashed line is the pre-registered 0.5-nat threshold, and Qwen’s tail is much heavier. Right: probe AUROC at each layer of Qwen, with families held out. The signal rises after layer 14, peaks at layer 18 (0.760), and the shaded area is its margin over a classifier on the prompt’s words (0.651).
Figure 3: (a) AUROC of every predictor on the 135-family panel. Blue predictors read hidden states and orange ones use only the prompt or the model’s outputs. (b) Paired differences in AUROC with 95% family-bootstrap intervals, both predictors scored on the same resampled families in each of 10,000 iterations.
Figure 4: The knockout intervention and its six controls on the Qwen2.5-7B-Instruct Cue B panel, with the normalised recovery (or, for the damage control, the change in accuracy) each returned. The knockout effect is roughly ninety times the random-direction control and three to five times the random-site and donor controls.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Rendering
Cue
Qwen generated
Qwen log p
Llama generated
Llama log p
0 (base)
3.5
Answer: 13.86 N.m
−45.531
Answer: 8 N.m.
−32.938
1
6.9
Answer: 6.84 N.m
−55.562
Answer: 8 N.m.
−34.938
2
10.0
Answer: 8.0 N.m
−45.000
Answer: 8 N.m.
−32.156
3
14.0
Answer: 8.0 N.m
−46.469
Answer: 8.0 N.m.
−33.125
Appendix
Table 2: The four behavioural records for CM_B_UM_059 , both primary models. Log-probability is of the certified answer “6.928 N*m” under teacher forcing. The generated text is what the model produced when allowed to answer freely.
Model
Layers
dmodel
Backend
Patch Δ logit
Status
Qwen2.5-7B-Instruct
28
3584
TransformerLens
30.55
extraction, patching, determinism pass
Llama-3.1-8B-Instruct
32
4096
TransformerLens
25.72
extraction, patching, determinism pass
DeepSeek-R1-Distill-Qwen-32B
64
5120
direct hooks
n/a
log-probability and determinism pass
Appendix
Table 3: Instrumentation validation. All checks are pass/fail gates run before experiments.
Figure 5: Left: monitor AUROC against the binarisation threshold applied to Sθlp , from the Stage 9 sweep. The pre-registered 0.5 is marked, and the sweep’s maximum (0.782 at 1.5) lies above it. Right: reliability diagram of the layer-18 monitor in ten probability bins (expected calibration error, ECE, 0.093 in this run). Marker area is proportional to the number of families in the bin, and the diagonal is perfect calibration.
Figure 6: Variance-probe AUROC by layer for the two other models, leave-one-family-out against each model’s own behaviour. DeepSeek peaks at layer 44 of 64 (0.703). Llama’s best layer is layer 0, the token embeddings before any processing (0.702), a lexical signal distinct from a late-layer representation, which is why no Llama monitoring claim is made in the body.
Configuration
AUROC
Layer
Site
95% CI
Mean probe, full sweep (primary)
0.7596
18
RP-LP
[0.6733,0.8405]
Variance-ensemble probe (PCA-50, L15–20)
0.7475
18
RP-LP
[0.6657,0.8290]
Per-variant probe sweep
0.7465
16
RP-LP
[0.7035,0.7871]
MLP probe
0.7424
18
RP-LP
[0.6546,0.8207]
Correctness-residualised monitor
0.7376
18
RP-LP
[0.6504,0.8185]
Domain-generalisation probe
0.7343
18
RP-LP
[0.6453,0.8151]
Appendix
Table 4: All probe configurations evaluated, leave-one-family-out unless noted. Site abbreviations: RP-LP is residual stream at the last prompt token, RP-CT is residual stream at the cue token.
Comparator
AUROC
Access required
Generic correctness probe (L24)
0.7657
hidden states
Token entropy proxy, fitted
0.7270
one forward pass
Combined non-activation (9 raw features)
0.7048
one forward pass
Black-box two-query counterfactual check
0.6684
two queries
Surface TF–IDF on prompt text
0.6514
prompt only
Answer-and-rationale TF–IDF classifier
0.6464
generated text
Appendix
Table 5: All non-activation and textual comparators on the 135-family monitoring panel, with the hidden-state correctness probe shown for reference.
Table 6: Calibration and operating-point metrics on the 135-family panel. Brier score is the mean squared error of the predicted probability. ECE (expected calibration error) is the average gap between predicted probability and observed frequency over probability bins. [email protected] is the fraction of sensitive families missed when the threshold flags 10% of stable families. Lower is better for all three.
Figure 7: Every intervention and control on the 20-family Cue B panel in Qwen2.5-7B. The knockout effect exceeds the norm-matched random-direction control by roughly two orders of magnitude and exceeds the random-site, stable-donor and same-answer-donor controls by a factor of three to five. The unrelated-family donor is correctly negative.
Intervention
Site
Recovery
Reading
Qwen2.5-7B-Instruct
Multi-head knockout, held-out Cue A/C panel
L16
2.177
Generalises to Cue A only
Multi-head knockout {26,24,13,11}
L16
1.870
Primary result; clipped 0.969
Head-11 layer sweep
L16
1.848
Most of the effect is head 11
Cue-present stable donor
L16
0.590
Specificity ratio 3.13
Same-answer donor
L16
0.405
Specificity ratio 4.57
Appendix
Table 7: All interventions and controls, ordered by mean normalised recovery of Sθlp on the 20-family Cue B panel unless noted. Recovery divides by the original Sθlp and is heavy-tailed when that denominator is small.
Figure 8: Left: knocking out head 11 alone at each layer from 10 to 22 on the 20-family Cue B panel. The causal window is one layer wide: 1.848 at layer 16 against 0.540 at layer 15 and −0.049 at layer 17. Right: every subset of the four heads knocked out at layer 16. Every subset containing head 11 recovers between 1.71 and 2.01. No subset without it exceeds 0.24.
Figure 9: The random-direction control family by family, sorted by the knockout effect. For each family the actual four-head knockout recovery is compared with the mean and 95th percentile of twenty norm-matched random perturbations at the same site. The knockout exceeds the 95th percentile on 20 of 20 families. Recovery is heavy-tailed because it divides by the family’s original Sθlp .
Figure 10: Left: the same four-head knockout applied to ten held-out families it was not selected on, five Cue A (above the line) and five Cue C (below), by layer. Every Cue A family recovers above 0.5 at layer 16 and only one Cue C family does (the other four all recover below 0.35), so the 2.177 panel mean is carried by Cue A. Right: direction-injection dose response on 30 stable families: the mean Sθlp rises monotonically with α but the fraction crossing the threshold reaches only 0.367 at α=2 , twice the probe’s own scale.
Faced with an unanswerable question, small language models can either refuse or confabulate: confidently produce a wrong answer. Can the hidden state before the first generated token provide a clue? Can we flip confabulations into refusals? I introduce ConfabQA, a 784-item benchmark whose obscure-pre-cutoff category breaks the typical correctness/knowledge-cutoff confound and map the geometry of the hidden state with linear probes, a sparse autoencoder (SAE), and direct optimization of intervention directions. On Qwen3-1.7B and Gemma 2 2B the probe separates upcoming refusals from confabulations well beyond prompt features (+7.4/+16.3 pp); on Llama 3.2 3B the ConfabQA margin is within fold-to-fold spread, but a direct refusal probe on PopQA and TriviaQA recovers +18-25 pp. On Qwen I closely inspect three candidate directions for the refusal vector: probe-recovered, Qwen-Scope SAE feature, and directly optimized. Although the three directions are pairwise 50-81 degrees apart, a push at or below the state's own norm along any of them flips most confabulations into refusal openings dominated by the "As..." family. Forcing the opener without touching the hidden state yields the same one-in-three judged-refusal rate as any push. The opposite push collapses refusals mostly into confabulations, yet leaves originally correct answers intact, so the direction gates the opening speech act rather than the knowledge behind it. Confabulations do not resolve into a concentrated feature in a dictionary or a token signature on Qwen3-1.7B. Scoring every probe on the margin over the strongest of four prompt-feature baselines extracts the hidden state's own contribution. Correctness signal is model-dependent and portable only for Llama 3.2 3B: 81% of the probe's margin over test majority survives transfer between ConfabQA, PopQA, and TriviaQA. Benchmark, judge, and code are released.
Whether a language model "knows what it knows" matters for trustworthy deployment and for deciding when it should abstain. Prior work usually measures this through one channel: verbalized confidence, a linear probe on hidden states, or agreement among sampled answers (semantic entropy), and often treats these as interchangeable. We compare all three in a pre-registered audit of five Qwen2.5 models (0.5B-7B, including a base/instruct pair at 7B) on factual retrieval (PopQA, SimpleQA) and multi-step reasoning (GSM8K, MATH). First, probes trained on prefill states to predict whether sampled answers will disagree become informative earlier for retrieval (about 7% of layer depth on average) than for reasoning (about 28%). The pre-registered five-layer test supported this only marginally; a finer 21-layer follow-up, designed after those results, supports it more clearly. In most retrieval cells, however, a prompt-only TF-IDF classifier matches the probe, so the early signal may reflect surface features of the question. Second, verbalized confidence is unstable: slightly different prompt templates yield scores that barely correlate. Third, Expected Calibration Error is misleading: verbal confidence looks well calibrated because it clusters near the base rate while discriminating poorly, whereas sampling agreement has worse ECE but about twice the resolution. Of five pre-registered hypotheses, two were falsified, one was partly supported, one was not supported, and one could not be tested because the base model's verbal confidence was degenerate.
The study of test-time reasoning in large language models (LLMs) has attracted increasing interests due to its profound impact on many tasks, with early work tracing back to approaches such as chain-of-thought reasoning. The basic mechanism underlying the test-time reasoning process, however, is still under-studied: from the same input prompt, and even the same partial solution, LLMs can produce varied answers if sampled multiple times. In this paper, we leverage question-asking as an inference-time intervention that articulates information about the model's hidden state. Specifically, we present a student-teacher setting where a student asks questions to a teacher. We train a probe on the student's hidden state before and after asking a question and find it is predictive of the trajectory's final correctness, even before generating the teacher's answer. This suggests there is a meaningful signal from the self-diagnosis that occurs during question generation rather than information transfer from the teacher. We then frame question-asking as a sequential decision problem, using this probe as a quality score, and define a gating policy to ask questions that maximize likelihood of correctness. We find that the success of question-asking as an intervention is largely dependent on the model's self-consistency. Our empirical results show a gap between detection and recovery; while our gating policy captures model correctness and uncertainty, interventions are equally likely to harm correct trajectories as they are to recover incorrect ones. This gap between diagnosis and correction has broader implications on language models' capacity for self-refinement under uncertainty.
Chu Fei Luo, Samuel Dahan, Xiaodan Zhu
Department of Electrical and Computer Engineering & Ingenuity Labs, Queen’s University · Conflict Analytics Lab, Queen’s University · Cornell Law School +1