Large language models reason in high-dimensional hidden-state spaces, while users observe only final outputs. We introduce Bypass Observation, a non-intrusive layer-wise readout architecture that attaches read-only observation heads to selected Transformer layers without feeding their outputs back into the backbone. We consider three variants: a shared LM head across layers, layer-specific heads, and a layer- or step-adaptive head. For full-vocabulary readout, we derive a closed-form overhead approximation governed primarily by V/(12d), with representative estimates ranging from about 30% to 240%, and discuss cost reductions via sparse observation, low-rank factorization, reduced vocabularies, top-k readout, and selective positions. We argue that Bypass Observation can make model computation more observable while remaining only a partial, potentially misleading projection of hidden states. We further distinguish bypass chain-of-thought from conventional chain-of-thought: conventional reasoning tokens enter the autoregressive computation, whereas bypass readouts remain causally external at inference time, although they can still provide training signals in reinforcement learning. Finally, we discuss applications to looped and recurrent-depth Transformers, where iteration-wise readout may expose convergence, oscillation, and potential halting signals. The proposal is conceptual and analytical; systematic empirical validation remains future work.
Figures & tables
Figure 1: Three architectural variants of Bypass Observation. (a) A shared LM head is reused across multiple reasoning layers, providing a parameter-efficient common readout. (b) Each reasoning layer is equipped with its own LM head, allowing layer-specific calibration at increased parameter cost. (c) A k -adaptive LM head conditions its readout on the layer index or inference step, enabling one observer to adapt across depth or recurrent computation. Solid black arrows denote the main forward computation, while dashed blue arrows denote the read-only observation (bypass) pathway; the displayed “observed text” should be interpreted as a schematic readout that may correspond to top- k vocabulary predictions or, with an additional semantic translator, a textual summary. In all variants, observation outputs are not fed back into the backbone.
Model
d
V
h
dhead
hkv
Attention
dff
QKVO/ d2
FFN/ d2
Total/ d2
ρ
GPT-3 175B
12288
50257
96
128
96
MHA
49152 (4.0 d )
4.00
8.00
12.00
≈+34%
Llama-2 7B
4096
32000
32
128
32
MHA
11008 (2.69 d )
4.00
8.06
12.06
≈+65%
Llama-2 70B
8192
32000
64
128
8
GQA
28672 (3.5 d )
2.25
10.50
12.75
≈+31%
Llama-3 8B
4096
128256
32
128
8
GQA
14336 (3.5 d )
2.50
10.50
13.00
≈+241%
Qwen2.5 7B
3584
152064
28
128
4
GQA
18944 (5.29 d )
2.29
15.86
18.14
≈+234%
Table 1: Analytical estimates of the per-layer overhead of full-vocabulary bypass decoding. Values are computed from representative model configurations under the accounting assumptions described in Section 3.1.
Dimension
Conventional CoT
Bypass chain-of-thought
Enters the computation loop
Yes; generated text becomes part of subsequent context
No; read-only open-loop observation
Causal role of text
Part of the computation itself
Projection of hidden computation with no direct causal contribution
Generation mechanism
Autoregressive, serial, context-dependent
Layer-wise readout parallel to the main generation stream
Observation granularity
Typically one textual trajectory
Potentially token × layer observations
Text form
Free-form intermediate reasoning
Progressively changing predictions or semantic summaries
Can provide an RL training signal
Yes
Yes; even with a frozen observer, scalar rewards can update the backbone through policy gradients
Table 2: Causal differences between conventional chain-of-thought and bypass chain-of-thought.
Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.
We propose a new architectural change, and post-training pipeline, for making LLMs more verbose reasoners by teaching a model to truncate forward passes early. We augment an existing transformer architecture with an early-exit mechanism at intermediate layers and train the model to exit at shallower layers when the next token can be predicted without deep computation. After a calibration stage, we incentivise the model to exit as early as possible while maintaining task performance using reinforcement learning. We provide preliminary results to this effect for small reasoning models, showing that they learn to adaptively reduce computations across tokens. We predict that, applied at the right scale, our approach can minimise the amount of excess computation that reasoning models have at their disposal to perform non-myopic planning using their internal activations, reserving this only for difficult-to-predict tokens.
Elizabeth Pavlova, Mariia Koroliuk, Karthik Viswanathan +3
MARS · Geodesic Research, equal contribution · University of Amsterdam
Large reasoning models (LRMs) often suffer from overthinking, continuing with redundant verification and repeated attempts even after reaching a correct solution. This wastes computation and can overturn an initially correct answer. We show that the transition from productive to redundant reasoning is decodable from late-layer hidden states. These representations separate the two phases around first-correct-solution boundaries beyond what position and lexical controls explain, and distinguish correct from incorrect solution endings. Building on this finding, we propose ROM (Real-time Overthinking Mitigation), which uses a lightweight streaming detector (0.07--0.25% of backbone parameters) to monitor a frozen LRM and intervene at well-formed reasoning boundaries without answer extraction or probe decoding. Counterfactual Self-Correction (CSC) augments detector training with wrong→correct trajectories to preserve useful self-correction. Heads are trained separately for each backbone using shared MATH500 supervision and transfer zero-shot to four unseen benchmarks. Evaluated against ten baselines across five backbones from three families and five benchmarks, ROMCSC reduces response length by 28--77% (45% on average), with significant reductions in all 25 settings and accuracy matching or exceeding vanilla decoding in 22. Measurements on two backbones further show 20--70% lower end-to-end latency with less than 0.5% detector overhead per decoding step. Code is available at https://github.com/SaFo-Lab/ROM.
Xinyan Wang, Xiaogeng Liu, Ming Pei +1
University of Wisconsin–Madison · Johns Hopkins University
Can a language model read the quality of its ongoing computation, and can an external intervention turn that readout into better outcomes? We test both questions in a frozen 2.6B looped transformer, Ouro-RLTT. On GSM8K, a strict pre-answer probe excludes the answer region and gold value yet predicts success: hidden states plus length/log-probability features reach AUROC 0.797 versus 0.731 for those surface features alone (increment +0.066; task-clustered 95% CI [+0.021,+0.112]; 170 tasks). On Horizon Logic, a prospectively extended task-disjoint study gives an increment of +0.111 (CI [+0.056,+0.169]), independently replicated on the new cohort (+0.095) and robust to an adversarial malformed-sibling shortcut. Recurrence also moves candidate-quality readability to progressively earlier physical depth; the trend replicates across the Ouro family and qualitatively in out-of-family Huginn, although their transfer geometry differs. The readout converts into validated decision-level gains. Hidden-state-based scores improve risk-coverage over shortcut-only scores in four sealed selective-prediction arms, and terminal selection beats matched random even when every candidate is well formed (27/32 correct selections versus 64.8% expected; p = 0.0086). Generative control does not convert: directional steering is negative, a branch screen is bounded, and exact-compute loop allocation and minimal LoRA direction-binding detect no gain. These tests run through bit-exact branch/carry/prune machinery over Ouro's 192-slot recurrent cache, including a suffix-recompute splice saving up to 88% of per-branch layer passes. We call this decision-usable but not generatively controllable property operational proto-introspection. All load-bearing values use source-item-disjoint splits and antisymmetrized pairwise evaluation.