A frozen language model on reasoning tasks has two coupled weaknesses: it under-uses evidence its own residual stream already encodes, and it fails to detect when the input is insufficient to answer, so it confabulates. This paper consolidates two research lines that address these on the same residual stream: a conditional steering probe writes the stream at mid-stack layers and recovers reasoning accuracy from a frozen backbone, and a zero-shot sufficiency direction reads the stream and abstains when information is insufficient. Deployed in one forward pass they interfere: the steering write shifts the state the direction reads, costing up to 8 AUROC points of cross-domain transfer on small models; a separate clean pass doubles inference cost. We keep the direction fixed and train a small network to reconstruct the pre-steering residual from the steered one -- mean-squared error on (steered, clean) pairs, no sufficiency labels -- and read the direction on the reconstruction. The resulting system, YOPO (You Only Pass Once), answers, steers, and abstains in one forward pass of a frozen Qwen2.5 backbone (1.5B/3B/7B). End to end, three-way accuracy more than doubles the frozen baseline (0.375->0.798 on 1.5B alphaNLI) and one pass beats the two-pass reference at every scale (0.798/0.830/0.893 vs 0.753/0.790/0.863) and on ten backbones across six model families. We chart the capacity-transfer frontier quantifying the principle that abstention should not be trained in; a source-side audit catches our own alphaNLI construction leaking a surface artifact, so architectural claims are anchored on native-label replications (SQuAD2, RepLiQA, MuSiQue); and on the standard four-domain suite we contribute, to our knowledge, the first answer-or-abstain benchmark, where our gate tops every in-domain dataset and the label-free direction is the only gate family to survive domain transfer.
Figures & tables
Figure 1: The conditional steering probe (the writer): at each injection layer a per-layer sub-probe reads the residual at spread-5 prompt positions, emits an input-conditioned unit direction (MLP) and a gated magnitude, and writes r+a∥r∥u(r) back through forward hooks. The backbone stays frozen; the edit is hot-swappable and fully reversible.
model
read
in-domain
transfer (HellaSwag)
1.5B
clean pass (2 passes)
0.962
0.918
1.5B
steered pass (1 pass)
0.913
0.836
3B
clean pass (2 passes)
0.967
0.922
3B
steered pass (1 pass)
0.942
0.891
7B
clean pass (2 passes)
0.985
0.968
7B
steered pass (1 pass)
0.995
0.955
Table 1: Contamination (AUROC, full vs. replace; α NLI in-domain, HellaSwag zero-shot transfer): the fixed zero-shot direction d read on the steered vs. clean pass. The steering write costs the gate 8.2 transfer points at 1.5B and 3.1 at 3B; at 7B the transfer cost is 1.3 points and the in-domain effect inverts (the steered residual is a better substrate for the read than the clean one). The conflict is concentrated at small scale.
Figure 2: YOPO (You Only Pass Once): one-pass fusion with a label-free correction. The steering probe (blue, the writer) writes the residual stream at layers {12,16,20} ; the same steered pass is read at a mid layer, passed through the reconstruction map M (ours, orange—trained only to reconstruct the clean residual, never on sufficiency labels), and scored against the FIXED zero-shot direction d (the sufficiency gate, purple, the reader). Below threshold τ (a percentile cut) the system abstains; otherwise it emits the steering-amplified answer. One forward pass serves both the write and the read. Layer indices shown for Qwen2.5-1.5B/7B; 3B uses {16,20,24} , read layer 24.
model
one-pass gate
in-domain
transfer (HellaSwag)
1.5B
naive d⊤hsteer (floor)
0.913
0.836
1.5B
de-contam →d [recon]
0.944
0.888
1.5B
ML de-contam →d [recon]
0.959
0.869
1.5B
ML de-contam + BCE boost
0.982
0.859
1.5B
ref: 2-pass clean read
0.962
0.918
3B
naive d⊤hsteer (floor)
0.942
0.891
Table 2: The one-pass sufficiency gate across scale (AUROC full vs. replace; α NLI in-domain, HellaSwag zero-shot transfer; single steered pass except the italic reference). De-contamination is label-free; the BCE boost is supervised. At 1.5B capacity trades transfer for in-domain along a clean frontier; at 3B the boosted map reaches in-domain above the 2-pass ceiling at a 0.006 transfer cost; at 7B the identity-anchored map learns ≈ identity (floor = de-contam row) because there is no contamination left to undo. See Figure 3 .
Figure 3: The one-pass capacity–transfer frontier across scale. Each hue is a frozen Qwen2.5 backbone; points are one-pass gates in increasing capacity order (line); the haloed ★ is our proposed flagship (ML de-contamination + BCE boost) and × is the two-pass clean-read reference ( 2× cost). At 1.5B the frontier slopes down—capacity buys in-domain and costs transfer; at 3B it flattens; at 7B all one-pass gates cluster at the ceiling. The 7B multi-layer reconstruction point is omitted (convergence failure, § 11 ).
policy
1.5B
3B
7B
α NLI
Hella
α NLI
Hella
α NLI
Hella
frozen baseline B0 (never abstains)
0.375
0.500
0.563
0.400
0.628
0.604
steering-only (never abstains)
0.590
0.502
0.625
0.502
0.668
0.592
gate-only (clean read, no steering)
0.560
0.602
0.743
0.560
0.828
0.780
steering + clean gate (2-pass ref.)
0.753
0.598
0.790
0.642
0.863
0.768
steering + naive steered gate (1 pass)
0.698
0.580
0.760
0.616
0.880
0.754
Table 3: Combined-system benchmark: end-to-end 3-way accuracy (answer correctly on answerable conditions / abstain on the no-information condition), 4-cond validation sets, abstain percentile p=25 via quantile transfer. “Label-free” scopes to the read path : no abstention labels are optimized against (the steering probe in every row is answer-label trained; see § 5 ). The 7B ML de-contam cells inherit that variant’s convergence failure (§ 11 ) and are reported for completeness.
Qwen2.5-1.5B (tested on)
Qwen2.5-7B (tested on)
trained on
α NLI
Hella
SQuAD2
RepLiQA
MuSiQue
α NLI
Hella
SQuAD2
RepLiQA
MuSiQue
α NLI
.986
.748
.648
.465
.677
.995
.928
.805
.795
.766
HellaSwag
.923
.905
.648
.480
.657
.967
.959
.930
.905
.808
SQuAD2
.766
.872
.863
.941
.817
.801
.828
.932
.961
.815
RepLiQA
.919
.599
.552
.999
.616
.867
.914
.913
.988
.830
MuSiQue
.799
.864
.865
.949
.832
.918
.935
.916
.969
.850
Table 4: Direction-transfer cross-matrix, computed entirely from our stored states. Each row: the sufficiency direction d is fit on that dataset’s train split (the sufficiency line’s v2 protocol: per-layer difference of means on an 80% subset, L∗ by held-out AUROC, frozen); each column: ranking AUROC ( ↑ ) of that frozen direction on the target dataset’s evaluation split, clean states, no recalibration. Diagonal (gray) = in-domain. α NLI/HellaSwag are our constructed 4-condition sets (sufficient = non- replace ); SQuAD2/RepLiQA carry native labels; MuSiQue (multi-hop, supports-first windows, native answerability labels) is both the hardest target column and, at 7B, the strongest source row. Layer sweep restricted to the capture set shared by all five extractions.
Figure 4: YOPO-2: one pass, write and read on separate channels. The answer-only steering write (orange) fires through a self-suppressing magnitude gate trained with unlabeled out-of-distribution negatives—active in-domain, near-zero under shift. Abstention never touches the write: a small read-head (blue) scores sufficiency from the layer-18 state of the same pass (stop-gradient), and a label-free quantile threshold turns the score into the abstain decision (green). In-domain this configuration beats the two-pass reference on every seed ( 0.877 – 0.883 ); under shift it holds parity with the best alternative ( 0.781±0.012 vs. 0.784 ) while the read-head’s score is the best transfer gate we measure ( 0.955 – 0.964 AUROC).
backbone
size
baseline
steer only
flagship (1 pass)
2-pass ref.
gate naive → flagship
Qwen2.5
0.5B
0.318
0.420
0.620
0.545
0.720 → 0.981
OLMo-2
1B
0.485
0.535
0.743
0.662
0.920 → 0.987
TinyLlama (Llama arch.)
1.1B
0.375
0.375
0.565
0.500
0.830 → 0.963
Qwen2.5
1.5B
0.375
0.590
0.797
0.752
0.903 → 0.989
StableLM-2
1.6B
0.375
0.480
0.695
0.625
0.823 → 0.987
SmolLM2
1.7B
0.375
0.580
0.787
0.777
0.754 → 0.982
Table 5: Cross-family end-to-end validation: ten backbones, six families, 0.5B–7B. Same pipeline as Table 3 on the α NLI 4-condition policy benchmark (answer/abstain accuracy, ↑ , abstention budget p=25% ): zero-shot baseline; steering only; the proposed one-pass flagship (steer + multi-layer map + BCE boost); the two-pass reference (clean re-read, 2× cost). Last column: gate ranking AUROC read from the steered pass, naive → flagship ( = echo size and its repair). Steering probes for non-Qwen backbones come from our earlier backbone studies or are retrained by the same recipe (layers mapped by proportional depth); single seed per cell. Bold = one-pass flagship ≥ two-pass reference.
in-domain decision acc.
cross-domain (12 arenas)
system
params
SQuAD2
RepLiQA
MQ-C
MQ-F
mean
vs. self
vs. target zs
zero-shot spoken judgment
0
0.854
0.873
0.588
0.535
—
—
—
+ forced chain-of-thought
0
0.817
0.886
0.555
0.516
—
—
—
read-only direction
0
0.860
0.953
0.785
0.705
0.799
−2.7
+8.6 (9/12)
constrained write (steering)
8.3M
0.853
0.976
0.849
0.695
0.681
−16.3
−3.2 (6/12)
free write (LoRA)
7.5M
0.888
0.994
0.949
0.803
0.678
−23.1
−3.5 (5/12)
Table 6: The three-family comparison on the standard four-domain suite (Qwen2.5-7B, judgment decision accuracy; the sufficiency line’s measurement). In-domain, capacity ranks the families (LoRA sweeps all four); across the 12 cross-domain arenas the ranking inverts: only the read-only direction exports a gain over the target’s own zero-shot spoken judgment ( +8.6pp mean, 9 of 12 arenas), both write families landing below it. Forced chain-of-thought lowers the spoken judgment on three of four domains.
raw QA
hybrid accuracy
answered-precision
domain
(answerable)
zero-shot
+ calib. zs
+ sufficiency gate
oracle
zs →+ gate
SQuAD2
0.890
0.826
0.830
0.834
0.945
0.749→0.786
RepLiQA
0.632
0.622
0.748
0.805
0.816
0.459→0.636
MuSiQue-C
0.319
0.625
0.622
0.619
0.660
0.637→0.645
MuSiQue-F
0.062
0.505
0.503
0.502
0.531
0.275→0.275
Table 7: Task C (hybrid answer-or-abstain) on the standard four-domain suite with the zero-shot sufficiency gate, Qwen2.5-7B. Columns: raw answerable-QA accuracy; hybrid accuracy under zero-shot, calibrated zero-shot, and OR-composition with the sufficiency gate; the oracle upper bound; and the precision of answers actually given, zero-shot →+ gate. OR-composition on RepLiQA gains +18.3pp ( 98.6% of the oracle gap) and lifts answered-precision 0.459→0.636 ; the multi-hop domains are answering-bound.
in-domain
cross-domain
target
gen-only
zero-shot d
read-head (flagship)
hardest-source d
oracle
SQuAD2
0.826
0.834
0.845
0.828
0.945
RepLiQA
0.622
0.805
0.805
0.745
0.816
MuSiQue-C
0.625
0.619
0.642
0.625
0.660
MuSiQue-F
0.505
0.502
0.505
–
0.531
Table 8: Task C (hybrid answer-or-abstain) on the standard four-domain suite, Qwen2.5-7B. Hybrid accuracy: answerable rows scored on the answer (normalized containment or token-F1 ≥0.5 ), unanswerable rows on abstention; OR-composition with the generation’s own abstention. In-domain gates are fit on the target’s own split (read-head cells are three-seed means, ±≤0.002 ); the cross-domain column applies the direction fit on the hardest source (MuSiQue-Full, label-free threshold) to each target (its own row is in-domain, marked “–”). The supervised read-head leads every in-domain cell; cross-domain, the label-free direction from the hardest source matches in-domain on two of three transferable targets while the supervised gate collapses off-diagonal. Multi-hop targets are answering-bound (oracle within 3.5/2.7pp of gen-only).
Figure 5: Task C (hybrid answer-or-abstain) on the standard four-domain suite, Qwen2.5-7B (Table 8 ). Blue: the in-domain flagship read-head, top on all four datasets. Orange: the label-free direction fit on the hardest source (MuSiQue-Full), deployed cross-domain with no target labels—it reaches the in-domain bar on SQuAD2 and MuSiQue-C and trails only on RepLiQA (MuSiQue-F is the source, so it has no cross bar). The oracle cap (dashed) and zero-shot floor (solid gray) bound each domain: the wide oracle gap on SQuAD2 is answering headroom, while on the multi-hop domains the oracle nearly meets the floor—these are answering-bound, and no gate has room there.
configuration (shared injection module)
RepLiQA
MuSiQue-C
zero-shot baseline
0.636
0.626
(B) injection off (read-only ablation)
0.822
0.628
(C) layer ordering (sufficiency line, replicated)
0.852
0.717
(B) read-only, zero-training direction
0.874
0.734
(B) read-only, BCE read-head
0.887
0.800
Table 9: Three one-pass fusion architectures on one shared injection module (answer-trained probe, layers {16,20,24} , α=0.3 ), three-way hybrid accuracy, in-domain. (A) reconstruction is the discriminative-set optimum; under this generative protocol the simplest read-only design (B, BCE read-head) leads both cells, and (C) layer-ordering reproduces the sufficiency line’s reported gains. The write–read interference is protocol-dependent: severe on the controlled set, mild here. Row (B) numbers are seed-checked on both trained components (footnote in text); (C) is single-seed.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
config
αmax
α NLI
HellaSwag
PIQA
Qwen2.5-1.5B frozen B0
–
50.8
74.1
68.6
1 layer, last
0.3
65.6 ( +14.8 )
74.9 ( +0.8 )
70.6 ( +2.0 )
1 layer, last
0.5
65.5 ( +14.6 )
74.5 ( +0.4 )
72.6 ( +4.0 )
1 layer, spread-5
0.3
69.1 ( +18.3 )
75.8 ( +1.7 )
73.4 ( +4.8 )
1 layer, spread-5
0.5
70.4 ( +19.5 )
76.2 ( +2.1 )
75.8 ( +7.2 )
3 layers, last
0.3
65.5 ( +14.7 )
74.7 ( +0.6 )
72.7 ( +4.1 )
Appendix
Table 10: Steering benchmark grid: accuracy (% Δ vs. the frozen baseline B0 ) across injection layer count, position scheme, and αmax , per dataset and backbone.
Latent reasoning replaces natural-language chains of thought with compact continuous "soft-token" representations, reducing the number of intermediate tokens that must be decoded autoregressively. We use a small external encoder to induce latent reasoning in a frozen backbone model. The encoder iteratively maps the backbone's hidden states to new latent blocks, constructing a latent reasoning prefix, after which the frozen model writes its remaining reasoning and its answer as text. Our method, INTACT, trains this encoder through distillation to reproduce the effect of written reasoning steps while keeping the model's behavior close to that of the frozen backbone, and then reinforces it on answer correctness. Across four frozen backbones from 1B to 8B, INTACT improves both GSM8K and average out-of-domain accuracy, including GSM8K gains from 75.8% to 79.3% on Llama-3.2-3B and from 77.6% to 79.7% on Qwen2.5-3B. It also reduces end-to-end latency per problem by 19% on Llama-3.2-3B and outperforms the evaluated latent-reasoning baselines at the 3B and 8B scales. Beyond the mathematics setting, the same recipe trained on a multi-domain corpus improves the frozen backbone on twelve evaluation benchmarks spanning mathematics, science, and commonsense, without task-specific adaptation.
Repeating Transformer layers can improve language-model performance without adding parameters. A pretrained model can be adapted to support both single-pass and looped inference by reusing middle-layer weights for additional computation. Because both paths share weights, training to improve looped predictions can degrade single-pass quality. We study joint training strategies using weighted losses on both outputs, with guidance from the frozen original model. Joint supervision substantially reduces single-pass degradation compared to looped-only training, while preserving looped gains over ordinary continued training. Most of this benefit comes from adjusting the single-pass loss weight, with guidance from the frozen original model further improving the looped output. With validation-selected weights, our method preserves single-pass quality within a predefined tolerance of ordinary continued training while improving looped predictions.
Jiarun Shen, Lei Zhao, Xiao Li
The Chinese University of Hong Kong, Shenzhen · Shanghai Jiao Tong University
Language models expose internal signals that predict whether an answer is correct, readable from a single forward pass of a frozen model without additional generations. Yet existing probes often commit to one signal family or layer and can be brittle under distribution shift; in retrieval-augmented settings, many specialized detectors instead target passage faithfulness, which can diverge from correctness when retrieved evidence is unhelpful or conflicting. We therefore ask where answer correctness is readable, which internal signal families carry it, and how they should be combined. We search over hidden states, token probabilities, residual-stream features, attention, and their fusion, treating the selected readouts as a predictive measurement rather than a mechanistic localization. We run this analysis separately in closed-book and with-context settings, since context can change which readouts are informative. A consistent anatomy emerges: correctness concentrates in the answer span, recovered from the answer tokens even under retrieval, and the families carry it complementarily, so fusing them helps most out of distribution, where a single signal is weakest. The protocol is effective across two backbones and gates a retrieval controller as one downstream use.
Yuansen Liu, Yixuan Tang, Anthony Kum Hoe Tung
School of Computing, National University of Singapore