A language model's confidence on a physics problem deserves trust only if it ignores details irrelevant to the answer. We introduce DECOY (Distractor-Embedded Certified cOunterfactuals for phYsics): 155 families of four prompts differing in one irrelevant detail. A symbolic solver certifies that each edit preserves the answer. Families are labelled by the largest drop in the model's log-probability of the certified answer, using a threshold fixed before probe training. Qwen2.5-7B-Instruct is sensitive on 50.0% of the 140 families yielding labels, and Llama-3.1-8B-Instruct on 22.9%. Effects are largest when distractor values lie numerically close to the answer. A linear probe on Qwen's layer-18 hidden state, averaged over four renderings, predicts sensitivity before generation at AUROC 0.760 with whole families held out. A separately tuned single-rendering probe reaches 0.747. A frozen layer-16 probe reaches 0.820 pooled AUROC on 15 families built after training, though within-cue-type AUROC is lower (0.667-0.6875). Under a paired family bootstrap, the probe performs on par with a fitted token-entropy baseline (AUROC 0.727) and achieves the lowest Brier score among tested predictors (0.204 versus 0.214). Knocking out four attention heads at layer 16 cancels sensitivity on a 20-family panel, localising it to a narrow mechanism. These results show that Qwen's confidence can track irrelevant details, that this failure is detectable before generation, and that targeted intervention can suppress it. DECOY provides certified ground truth and paired-tested baselines for evaluating hidden-state monitors.
Figures & tables
Figure 1: Overview of the pipeline, following the three levels of Section 3.1 . A solver certifies that the cue cannot affect the answer (Level 1). One teacher-forced pass per rendering gives the family’s sensitivity label (Level 2). A probe on the layer-18 hidden state of Qwen2.5-7B-Instruct predicts that label before generation (Level 3), and the probe is evaluated against non-activation baselines and by head knockout.
Cue type
Families
Labelled
Sensitive (%)
Mean drop
A: irrelevant variable
35
35
12 (34.3)
0.515
B: unit-matched distractor
105
100
53 (53.0)
1.330
C: equivalent units
15
5
5 (100.0)
9.022
Domain
Mechanics
91
86
46 (53.5)
1.575
Electrostatics and circuits
43
37
17 (45.9)
1.172
Table 1: DECOY composition and the sensitivity of Qwen2.5-7B-Instruct at the pre-registered 0.5-nat threshold. Families : certified families. Labelled : certified families included in the primary Qwen panel (excludes 15 families held out to test transfer: 0 Cue A, 5 Cue B, 10 Cue C). Sensitive : labelled families whose log-probability drop reaches 0.5 nats. Mean drop : mean Sθlp in nats over labelled families.
Figure 2: Left: distribution of the log-probability drop Sθlp over the 140 labelled families for both models on identical prompts. The dashed line is the pre-registered 0.5-nat threshold, and Qwen’s tail is much heavier. Right: probe AUROC at each layer of Qwen, with families held out. The signal rises after layer 14, peaks at layer 18 (0.760), and the shaded area is its margin over a classifier on the prompt’s words (0.651).
Figure 3: (a) AUROC of every predictor on the 135-family panel. Blue predictors read hidden states and orange ones use only the prompt or the model’s outputs. (b) Paired differences in AUROC with 95% family-bootstrap intervals, both predictors scored on the same resampled families in each of 10,000 iterations.
Figure 4: The knockout intervention and its six controls on the Qwen2.5-7B-Instruct Cue B panel, with the normalised recovery (or, for the damage control, the change in accuracy) each returned. The knockout effect is roughly ninety times the random-direction control and three to five times the random-site and donor controls.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Rendering
Cue
Qwen generated
Qwen log p
Llama generated
Llama log p
0 (base)
3.5
Answer: 13.86 N.m
−45.531
Answer: 8 N.m.
−32.938
1
6.9
Answer: 6.84 N.m
−55.562
Answer: 8 N.m.
−34.938
2
10.0
Answer: 8.0 N.m
−45.000
Answer: 8 N.m.
−32.156
3
14.0
Answer: 8.0 N.m
−46.469
Answer: 8.0 N.m.
−33.125
Appendix
Table 2: The four behavioural records for CM_B_UM_059 , both primary models. Log-probability is of the certified answer “6.928 N*m” under teacher forcing. The generated text is what the model produced when allowed to answer freely.
Model
Layers
dmodel
Backend
Patch Δ logit
Status
Qwen2.5-7B-Instruct
28
3584
TransformerLens
30.55
extraction, patching, determinism pass
Llama-3.1-8B-Instruct
32
4096
TransformerLens
25.72
extraction, patching, determinism pass
DeepSeek-R1-Distill-Qwen-32B
64
5120
direct hooks
n/a
log-probability and determinism pass
Appendix
Table 3: Instrumentation validation. All checks are pass/fail gates run before experiments.
Figure 5: Left: monitor AUROC against the binarisation threshold applied to Sθlp , from the Stage 9 sweep. The pre-registered 0.5 is marked, and the sweep’s maximum (0.782 at 1.5) lies above it. Right: reliability diagram of the layer-18 monitor in ten probability bins (expected calibration error, ECE, 0.093 in this run). Marker area is proportional to the number of families in the bin, and the diagonal is perfect calibration.
Figure 6: Variance-probe AUROC by layer for the two other models, leave-one-family-out against each model’s own behaviour. DeepSeek peaks at layer 44 of 64 (0.703). Llama’s best layer is layer 0, the token embeddings before any processing (0.702), a lexical signal distinct from a late-layer representation, which is why no Llama monitoring claim is made in the body.
Configuration
AUROC
Layer
Site
95% CI
Mean probe, full sweep (primary)
0.7596
18
RP-LP
[0.6733,0.8405]
Variance-ensemble probe (PCA-50, L15–20)
0.7475
18
RP-LP
[0.6657,0.8290]
Per-variant probe sweep
0.7465
16
RP-LP
[0.7035,0.7871]
MLP probe
0.7424
18
RP-LP
[0.6546,0.8207]
Correctness-residualised monitor
0.7376
18
RP-LP
[0.6504,0.8185]
Domain-generalisation probe
0.7343
18
RP-LP
[0.6453,0.8151]
Appendix
Table 4: All probe configurations evaluated, leave-one-family-out unless noted. Site abbreviations: RP-LP is residual stream at the last prompt token, RP-CT is residual stream at the cue token.
Comparator
AUROC
Access required
Generic correctness probe (L24)
0.7657
hidden states
Token entropy proxy, fitted
0.7270
one forward pass
Combined non-activation (9 raw features)
0.7048
one forward pass
Black-box two-query counterfactual check
0.6684
two queries
Surface TF–IDF on prompt text
0.6514
prompt only
Answer-and-rationale TF–IDF classifier
0.6464
generated text
Appendix
Table 5: All non-activation and textual comparators on the 135-family monitoring panel, with the hidden-state correctness probe shown for reference.
Table 6: Calibration and operating-point metrics on the 135-family panel. Brier score is the mean squared error of the predicted probability. ECE (expected calibration error) is the average gap between predicted probability and observed frequency over probability bins. [email protected] is the fraction of sensitive families missed when the threshold flags 10% of stable families. Lower is better for all three.
Figure 7: Every intervention and control on the 20-family Cue B panel in Qwen2.5-7B. The knockout effect exceeds the norm-matched random-direction control by roughly two orders of magnitude and exceeds the random-site, stable-donor and same-answer-donor controls by a factor of three to five. The unrelated-family donor is correctly negative.
Intervention
Site
Recovery
Reading
Qwen2.5-7B-Instruct
Multi-head knockout, held-out Cue A/C panel
L16
2.177
Generalises to Cue A only
Multi-head knockout {26,24,13,11}
L16
1.870
Primary result; clipped 0.969
Head-11 layer sweep
L16
1.848
Most of the effect is head 11
Cue-present stable donor
L16
0.590
Specificity ratio 3.13
Same-answer donor
L16
0.405
Specificity ratio 4.57
Appendix
Table 7: All interventions and controls, ordered by mean normalised recovery of Sθlp on the 20-family Cue B panel unless noted. Recovery divides by the original Sθlp and is heavy-tailed when that denominator is small.
Figure 8: Left: knocking out head 11 alone at each layer from 10 to 22 on the 20-family Cue B panel. The causal window is one layer wide: 1.848 at layer 16 against 0.540 at layer 15 and −0.049 at layer 17. Right: every subset of the four heads knocked out at layer 16. Every subset containing head 11 recovers between 1.71 and 2.01. No subset without it exceeds 0.24.
Figure 9: The random-direction control family by family, sorted by the knockout effect. For each family the actual four-head knockout recovery is compared with the mean and 95th percentile of twenty norm-matched random perturbations at the same site. The knockout exceeds the 95th percentile on 20 of 20 families. Recovery is heavy-tailed because it divides by the family’s original Sθlp .
Figure 10: Left: the same four-head knockout applied to ten held-out families it was not selected on, five Cue A (above the line) and five Cue C (below), by layer. Every Cue A family recovers above 0.5 at layer 16 and only one Cue C family does (the other four all recover below 0.35), so the 2.177 panel mean is carried by Cue A. Right: direction-injection dose response on 30 stable families: the mean Sθlp rises monotonically with α but the fraction crossing the threshold reaches only 0.367 at α=2 , twice the probe’s own scale.
Department of Electrical and Computer Engineering & Ingenuity Labs, Queen’s University · Conflict Analytics Lab, Queen’s University · Cornell Law School +1