Hidden future-token leakage can create false architectural gains: an autoregressive model may appear substantially better under teacher-forced evaluation while its free-running generation deteriorates. We investigate this failure in an adaptive residual-routing decoder whose sequence-level routing summary exposes prefix positions to future tokens despite a correct causal attention mask. An anomalous 24-layer checkpoint reduces validation loss from the baseline's 3.5958 to 1.4259, yet it fails a prefix-invariance audit and has substantially higher reference-model perplexity during generation. Layer-wise diagnostics and implementation inspection trace the violation to a full-sequence summary broadcast through the dynamic routing-query path. Two independently trained 12-layer causal repairs eliminate detectable leakage and return validation loss and reference perplexity to the control scale, removing the large apparent advantage. As diagnostic tools, we use the established prefix-invariance contract, maximum prefix-output discrepancy (LeakScore), complementary behavioral metrics, and layer-wise profiles. Supporting experiments cover six synthetic implementation pairs spanning five module families, clean pretrained controls, controlled fault injection, and cache consistency across numerical precisions. These results illustrate how hidden future-token access can produce misleading claims of architectural progress and why lower teacher-forced loss should be interpreted together with whole-model causal checks and free-running evaluation.
Figures & tables
Figure 1: Mechanism of hidden future-token leakage. Solid paths represent permitted dependencies on past and present inputs, while the orange dashed path highlights a forbidden dependency on future inputs through a non-causal auxiliary operation. This is a schematic dependency diagram rather than a particular architecture: pooling, temporal mixing, state aggregation, routing, and cross-time normalization can introduce such paths when implemented non-causally.
Figure 2: Prefix-invariance audit. Two inputs share a prefix p=x1:t but have different suffixes s and s′ . Both are evaluated by the same model under the same evaluation settings, with stochastic effects controlled. LeakScore is the largest absolute prefix-logit difference across positions and vocabulary entries. A score above the numerical tolerance ϵ indicates an audit failure; passing provides evidence only for the tested inputs and conditions.
Module
Non-causal operation
Causal repair
Before
After
Pooling
Full-sequence mean
Token-wise routing
0.0755
0.0000
Pooling
Last-token broadcast
Prefix-mean routing
0.3232
0.0000
Conv1d
Symmetric padding
Left-only padding
0.6625
0.0000
Linear attention
Full-sequence KV aggregation
Prefix-only KV aggregation
0.1168
0.0000
Dynamic routing
Full-sequence context
Prefix-only context
0.0033
0.0000
Normalization
Full-sequence statistics
Prefix-only statistics
0.0255
0.0000
Table 1: Six synthetic non-causal/causal implementation pairs. Score before and score after denote LeakScore for the non-causal implementation and its causal repair, respectively, under the same audit ( ϵ=10−5 ). Scores are reported on each module’s own output scale and should not be used to rank different modules.
Module
8
16
32
64
noncausal_pool_mean
0.0484
0.0425
0.0328
0.0212
noncausal_pool_last
0.6819
0.5959
0.6969
0.6041
noncausal_samepad_conv1d
0.8203
0.7067
0.9781
0.9717
noncausal_fullkv_linear_attn
0.1343
0.1134
0.0958
0.0711
noncausal_global_route
0.00421
0.00466
0.00359
0.00217
noncausal_global_norm
0.0358
0.0306
0.0199
0.0178
Table 2: LeakScore of non-causal modules under different prefix lengths (sequence length 80, random seed 42, ϵ=10−5 ). All causally repaired variants remain at 0.0 and are omitted for clarity.
Module
Passed runs
LeakScore (mean ± std)
noncausal_pool_mean
0/3
0.0760 ± 0.0211
noncausal_pool_last
0/3
0.5104 ± 0.1420
noncausal_samepad_conv1d
0/3
0.6173 ± 0.0175
noncausal_fullkv_linear_attn
0/3
0.1269 ± 0.0213
noncausal_global_route
0/3
0.00347 ± 0.00008
noncausal_global_norm
0/3
0.0316 ± 0.0104
Table 3: Seed stability of the audit over three random seeds (sequence length 32, prefix length 16, ϵ=10−5 ). All causally repaired variants achieve 3/3 pass with LeakScore 0.0±0.0 .
Model
α
Max diff
Norm. max
JS div
Affected-token rate
Pass rate
GPT-2
0.01
2.1e-4
1.3e-6
1.9e-8
0.993
0.000
GPT-2
0.05
1.0e-3
6.3e-6
1.8e-8
1.000
0.000
GPT-2
0.10
2.1e-3
1.3e-5
2.0e-8
1.000
0.000
Qwen2.5-0.5B
0.01
1.6e-3
7.1e-5
1.1e-8
1.000
0.000
Qwen2.5-0.5B
0.05
7.9e-3
3.5e-4
9.9e-8
1.000
0.000
Qwen2.5-0.5B
0.10
1.6e-2
7.0e-4
3.9e-7
1.000
0.000
Table 4: Fault-injection sensitivity with a sequence-level summary-broadcast wrapper and unchanged pretrained weights. Max diff and Norm. max are maxima over 72 pairs; JS divergence and affected-token rate are per-pair means, and pass rate is the fraction of passing pairs ( ϵ=10−5 ).
Model
Precision
P50 max diff
P95 max diff
Max diff
Top-1 agreement
GPT-2
fp32
1.9×10−4
4.5×10−4
7.6×10−4
32/32
GPT-2
fp16
0.125
0.25
0.625
32/32
GPT-2
bf16
0.75
2.0
4.0
31/32
Qwen2.5-0.5B
fp32
2.5×10−5
3.9×10−5
8.1×10−5
32/32
Qwen2.5-0.5B
fp16
0.0313
0.0476
0.1475
32/32
Qwen2.5-0.5B
bf16
0.2461
0.3703
0.5
32/32
Table 5: Cache-consistency audit between full-prefix forward and incremental decoding, summarized over the 32 prefix steps by the discrepancy distribution of the per-step maximum logit difference. The key calibration quantities are the clean-control P50/P95/max values for each runtime setting.
Model
Layers
Inject → FirstLeakLayer
Final-layer discrepancy
GPT-2
12
3→3
3.47
Qwen2.5-0.5B
24
3→3
0.86
Mamba-130M
24
3→3
18.15
Table 6: Hidden-state localization on one representative pair per model ( t=32 ) after injection at zero-based layer 3 ( α=0.05 ). Final-layer discrepancy is measured at the last monitored layer.
Figure 3: Layer-wise hidden-state discrepancy after fault injection at zero-based layer 3 ( α=0.05 ). Dotted lines indicate the injection depth. Zero discrepancies are displayed at 10−7 on the logarithmic axis.
Variant
FirstLeakLayer
Final-layer discrepancy
Baseline
–
0.00
Static AttnRes
–
0.00
CDR-gated
0
2.20
CDR-mean
0
3.55
CDR-last
0
4.64
CDR-last 24L
0
16.96
Table 7: CDR hidden-state localization on a representative input pair with prefix length 32. Final-layer discrepancy is measured at the last monitored layer; – denotes no detected leaking layer. Unmarked variants are 12-layer.
Variant
Val loss
pplref
Entropy
Distinct-2
Repeat-3
Static AttnRes
3.6676
17.2
1.64
0.804
0.174
Baseline
3.6900
19.8
1.85
0.808
0.153
Baseline 24L
3.5958
16.8
1.90
0.827
0.139
CDR-gated
2.6015
198.5
2.31
0.822
0.136
CDR-mean
2.8473
299.9
2.80
0.725
0.198
CDR-last
2.5488
695.5
2.87
0.784
0.134
Table 8: Validation loss and generation diagnostics for controls and pre-fix CDR checkpoints. Generation metrics are averaged over 50 prompts, with one continuation per prompt; validation loss is the checkpoint validation result. Unmarked variants are 12-layer.
Variant
Val loss
LeakScore
FirstLeakLayer
pplref
CDR-token-fix
3.6766
0.0
–
15.3
CDR-prefix-mean-fix
3.6731
0.0
–
18.1
Table 9: Audit and evaluation results for the two independently trained 12-layer causal repairs. – denotes no detected leaking layer. Both repairs pass the audit and have validation loss and reference perplexity on a similar scale to the controls.
Figure 4: Diagnostics for seven 12-layer checkpoints in a shared order: (a) validation loss, (b) checkpoint-level LeakScore, and (c) mean reference perplexity over 50 prompts. Open and shaded bars denote passing and failing checkpoints, respectively. Panel (c) uses a logarithmic axis with untransformed perplexity labels. CDR-token and CDR-prefix-mean abbreviate the two repaired variants.
Variant
Val loss
LeakScore
Top-1 flip rate
Verdict
Baseline
3.6900
0.0
0.0
pass
Static AttnRes
3.6676
0.0
0.0
pass
Baseline 24L
3.5958
0.0
0.0
pass
CDR-token-fix
3.6766
0.0
0.0
pass
CDR-prefix-mean-fix
3.6731
0.0
0.0
pass
CDR-gated
2.6015
8.52
0.483
fail
Table 10: Prefix-invariance audit on nine core CDR checkpoints ( ϵ=10−5 ). LeakScore is the maximum over 72 pairs; top-1 flip rate is the mean of per-pair rates. Controls and repairs pass on all tested pairs, while all pre-fix variants fail. Unmarked variants are 12-layer.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Data
OpenWebText, GPT-2 tokenizer (vocabulary padded from 50257 to 50304), block size 512
3×10−4 , cosine decay to 3×10−5 , warmup 1000 iterations
Iterations
20,000
Effective batch
64 sequences/step (batch size 4 per device, gradient accumulation 16/2 GPUs =8 per device, 2 devices)
Precision
bfloat16
Training seed
1337 (all nine core checkpoints)
Appendix
Table 1: Shared CDR training configuration. Parameter counts are unique trainable parameters; the input embedding and the output head share a single weight matrix, counted once. CDR adds rank- r down- and up-projections for dynamic routing-query corrections in the residual aggregation paths before both the attention and MLP sublayers.
Hybrid sequence models must satisfy prefix invariance: representations at position t must not depend on future inputs, yet this is rarely verified. We formalize prefix invariance and give a lightweight audit, two forward passes, no training or gradients, yielding a per-layer score localizing where causality breaks. Attention-mask inspection, the field's default check, is incomplete: causality is a graph-level property, and leaks can occur via scans, aggregations, or normalization despite correct masks. Across 192 injected-fault trials on eight checkpoints, mask inspection detected none, while our audit localized all 192/192 to the exact layer. Static/dynamic analysis of chunked-scan code in transformers found the same defect in Zamba2 and Nemotron-H, an inter-chunk axis error fixed via the reference implementation. The method fits on one page and runs in seconds.
Taebong Kim, Youngsik Hong, Minsik Kim +3
VIDRAFT AI Research · QuantumOS, Seoul, Republic of Korea
Autoregressive pretraining increasingly draws on heterogeneous data, making it important to understand how a model learns from an individual token. The next-token prediction objective naturally identifies a token's contribution with its own loss. However, each token is not only a prediction target but also context for what follows. Using controlled corruption, we decouple these two roles and find a reversal: making a noisy token easier to predict reduces its damage as a target but increases it as context. The same decoupling helps explain text generated by language models: generation selects each token by its fit to the prefix, while its role as context is never tested against an independently determined continuation, because that continuation is generated to fit it. At known corrupted positions, acting through the context can reduce damage that removing the token's own loss does not. Understanding and controlling what a model learns from a token therefore requires decoupling its roles.
Suqin Yuan, Runqi Lin, Kevin Qinghong Lin +4
University of Sydney · University of Oxford · Southeast University
This work presents LeakyLMs, a set of attacks that leak proprietary model, architecture, and deployment information from production language models. LeakyLMs is the first to demonstrate that key model and deployment details can be inferred using only token generation timing, even when interacting through remote APIs. LeakyLMs introduces two core attacks. The first attack targets inference optimizations and deployment strategies. For example, our attack detects whether a provider uses speculative decoding, a widely deployed inference-time optimization, and further identifies the context length of the draft model used in the pipeline. Our measurements show that Google Gemini Flash 2.5 uses speculative decoding with a draft context window of approximately 128K tokens. The second attack recovers key architectural properties, including the number of transformer layers, hidden dimension size, and number of attention heads. To achieve this, LeakyLMs builds a detailed and accurate model of token-generation timing on modern NVIDIA GPUs, characterizing how latency scales with model configuration and hardware parameters. The attack then performs a search over the architecture space using this timing model. In experiments with Llama models, the near-correct architectural configuration appears in the top-10 guesses more than 90% of the time.