Beyond Causal Masks: Auditing and Localizing Hidden Future-Token Leakage in Adaptive Sequence Models

Huiting Liu; Jinsong Wu

cs.AIOct 22, 2026

Beyond Causal Masks: Auditing and Localizing Hidden Future-Token Leakage in Adaptive Sequence Models

Authors: Huiting Liu, Jinsong Wu

Organizations: Department of Electrical Engineering, University of Chile, Santiago, 8370451, Chile

Abstract

Hidden future-token leakage can create false architectural gains: an autoregressive model may appear substantially better under teacher-forced evaluation while its free-running generation deteriorates. We investigate this failure in an adaptive residual-routing decoder whose sequence-level routing summary exposes prefix positions to future tokens despite a correct causal attention mask. An anomalous 24-layer checkpoint reduces validation loss from the baseline's 3.5958 to 1.4259, yet it fails a prefix-invariance audit and has substantially higher reference-model perplexity during generation. Layer-wise diagnostics and implementation inspection trace the violation to a full-sequence summary broadcast through the dynamic routing-query path. Two independently trained 12-layer causal repairs eliminate detectable leakage and return validation loss and reference perplexity to the control scale, removing the large apparent advantage. As diagnostic tools, we use the established prefix-invariance contract, maximum prefix-output discrepancy (LeakScore), complementary behavioral metrics, and layer-wise profiles. Supporting experiments cover six synthetic implementation pairs spanning five module families, clean pretrained controls, controlled fault injection, and cache consistency across numerical precisions. These results illustrate how hidden future-token access can produce misleading claims of architectural progress and why lower teacher-forced loss should be interpreted together with whole-model causal checks and free-running evaluation.

Figures & tables

Figure 1: Mechanism of hidden future-token leakage. Solid paths represent permitted dependencies on past and present inputs, while the orange dashed path highlights a forbidden dependency on future inputs through a non-causal auxiliary operation. This is a schematic dependency diagram rather than a particular architecture: pooling, temporal mixing, state aggregation, routing, and cross-time normalization can introduce such paths when implemented non-causally.

Figure 2: Prefix-invariance audit. Two inputs share a prefix

p=x_{1:t}

but have different suffixes

s

and

s^{\prime}

. Both are evaluated by the same model under the same evaluation settings, with stochastic effects controlled. LeakScore is the largest absolute prefix-logit difference across positions and vocabulary entries. A score above the numerical tolerance

\epsilon

indicates an audit failure; passing provides evidence only for the tested inputs and conditions.

Module	Non-causal operation	Causal repair	Before
Pooling	Full-sequence mean	Token-wise routing	0.0755
Pooling	Last-token broadcast	Prefix-mean routing	0.3232
Conv1d	Symmetric padding	Left-only padding	0.6625
Linear attention	Full-sequence KV aggregation	Prefix-only KV aggregation	0.1168
Dynamic routing	Full-sequence context	Prefix-only context	0.0033
Normalization	Full-sequence statistics	Prefix-only statistics	0.0255

Table 1: Six synthetic non-causal/causal implementation pairs. Score before and score after denote LeakScore for the non-causal implementation and its causal repair, respectively, under the same audit (

\epsilon=10^{-5}

). Scores are reported on each module’s own output scale and should not be used to rank different modules.

Module	8	16	32	64
noncausal_pool_mean	0.0484	0.0425	0.0328	0.0212
noncausal_pool_last	0.6819	0.5959	0.6969	0.6041
noncausal_samepad_conv1d	0.8203	0.7067	0.9781	0.9717
noncausal_fullkv_linear_attn	0.1343	0.1134	0.0958	0.0711
noncausal_global_route	0.00421	0.00466	0.00359	0.00217
noncausal_global_norm	0.0358	0.0306	0.0199	0.0178

Table 2: LeakScore of non-causal modules under different prefix lengths (sequence length 80, random seed 42,

\epsilon=10^{-5}

). All causally repaired variants remain at

0.0

and are omitted for clarity.

Module	Passed runs	LeakScore (mean $\pm$ std)
noncausal_pool_mean	0/3	0.0760 $\pm$ 0.0211
noncausal_pool_last	0/3	0.5104 $\pm$ 0.1420
noncausal_samepad_conv1d	0/3	0.6173 $\pm$ 0.0175
noncausal_fullkv_linear_attn	0/3	0.1269 $\pm$ 0.0213
noncausal_global_route	0/3	0.00347 $\pm$ 0.00008
noncausal_global_norm	0/3	0.0316 $\pm$ 0.0104

Table 3: Seed stability of the audit over three random seeds (sequence length 32, prefix length 16,

\epsilon=10^{-5}

). All causally repaired variants achieve

3/3

pass with LeakScore

0.0\pm 0.0

Model	$\alpha$	Max diff	Norm. max	JS div	Affected-token rate
GPT-2	0.01	2.1e-4	1.3e-6	1.9e-8	0.993
GPT-2	0.05	1.0e-3	6.3e-6	1.8e-8	1.000
GPT-2	0.10	2.1e-3	1.3e-5	2.0e-8	1.000
Qwen2.5-0.5B	0.01	1.6e-3	7.1e-5	1.1e-8	1.000
Qwen2.5-0.5B	0.05	7.9e-3	3.5e-4	9.9e-8	1.000
Qwen2.5-0.5B	0.10	1.6e-2	7.0e-4	3.9e-7	1.000

Table 4: Fault-injection sensitivity with a sequence-level summary-broadcast wrapper and unchanged pretrained weights. Max diff and Norm. max are maxima over 72 pairs; JS divergence and affected-token rate are per-pair means, and pass rate is the fraction of passing pairs (

\epsilon=10^{-5}

Model	Precision	P50 max diff	P95 max diff	Max diff	Top-1 agreement
GPT-2	fp32	$1.9\times 10^{-4}$	$4.5\times 10^{-4}$	$7.6\times 10^{-4}$	$32/32$
GPT-2	fp16	0.125	0.25	0.625	$32/32$
GPT-2	bf16	0.75	2.0	4.0	$31/32$
Qwen2.5-0.5B	fp32	$2.5\times 10^{-5}$	$3.9\times 10^{-5}$	$8.1\times 10^{-5}$	$32/32$
Qwen2.5-0.5B	fp16	0.0313	0.0476	0.1475	$32/32$
Qwen2.5-0.5B	bf16	0.2461	0.3703	0.5	$32/32$

Table 5: Cache-consistency audit between full-prefix forward and incremental decoding, summarized over the 32 prefix steps by the discrepancy distribution of the per-step maximum logit difference. The key calibration quantities are the clean-control P50/P95/max values for each runtime setting.

Model	Layers	Inject $\rightarrow$ FirstLeakLayer	Final-layer discrepancy
GPT-2	12	$3{\rightarrow}3$	3.47
Qwen2.5-0.5B	24	$3{\rightarrow}3$	0.86
Mamba-130M	24	$3{\rightarrow}3$	18.15

Table 6: Hidden-state localization on one representative pair per model (

t=32

) after injection at zero-based layer 3 (

\alpha=0.05

). Final-layer discrepancy is measured at the last monitored layer.

Figure 3: Layer-wise hidden-state discrepancy after fault injection at zero-based layer 3 (

\alpha=0.05

). Dotted lines indicate the injection depth. Zero discrepancies are displayed at

10^{-7}

on the logarithmic axis.

Variant	FirstLeakLayer	Final-layer discrepancy
Baseline	–	0.00
Static AttnRes	–	0.00
CDR-gated	$0$	2.20
CDR-mean	$0$	3.55
CDR-last	$0$	4.64
CDR-last 24L	$0$	16.96

Table 7: CDR hidden-state localization on a representative input pair with prefix length 32. Final-layer discrepancy is measured at the last monitored layer; – denotes no detected leaking layer. Unmarked variants are 12-layer.

Variant	Val loss	$\mathrm{ppl}_{\mathrm{ref}}$	Entropy	Distinct-2	Repeat-3
Static AttnRes	3.6676	17.2	1.64	0.804	0.174
Baseline	3.6900	19.8	1.85	0.808	0.153
Baseline 24L	3.5958	16.8	1.90	0.827	0.139
CDR-gated	2.6015	198.5	2.31	0.822	0.136
CDR-mean	2.8473	299.9	2.80	0.725	0.198
CDR-last	2.5488	695.5	2.87	0.784	0.134

Table 8: Validation loss and generation diagnostics for controls and pre-fix CDR checkpoints. Generation metrics are averaged over 50 prompts, with one continuation per prompt; validation loss is the checkpoint validation result. Unmarked variants are 12-layer.

Variant	Val loss	LeakScore	FirstLeakLayer	$\mathrm{ppl}_{\mathrm{ref}}$
CDR-token-fix	3.6766	0.0	–	15.3
CDR-prefix-mean-fix	3.6731	0.0	–	18.1

Table 9: Audit and evaluation results for the two independently trained 12-layer causal repairs. – denotes no detected leaking layer. Both repairs pass the audit and have validation loss and reference perplexity on a similar scale to the controls.

Figure 4: Diagnostics for seven 12-layer checkpoints in a shared order: (a) validation loss, (b) checkpoint-level LeakScore, and (c) mean reference perplexity over 50 prompts. Open and shaded bars denote passing and failing checkpoints, respectively. Panel (c) uses a logarithmic axis with untransformed perplexity labels. CDR-token and CDR-prefix-mean abbreviate the two repaired variants.

Variant	Val loss	LeakScore	Top-1 flip rate	Verdict
Baseline	3.6900	0.0	0.0	pass
Static AttnRes	3.6676	0.0	0.0	pass
Baseline 24L	3.5958	0.0	0.0	pass
CDR-token-fix	3.6766	0.0	0.0	pass
CDR-prefix-mean-fix	3.6731	0.0	0.0	pass
CDR-gated	2.6015	8.52	0.483	fail

Table 10: Prefix-invariance audit on nine core CDR checkpoints (

\epsilon=10^{-5}

). LeakScore is the maximum over 72 pairs; top-1 flip rate is the mean of per-pair rates. Controls and repairs pass on all tested pairs, while all pre-fix variants fail. Unmarked variants are 12-layer.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Data	OpenWebText, GPT-2 tokenizer (vocabulary padded from 50257 to 50304), block size 512
Optimizer	AdamW ( $\beta=(0.9,0.95)$ , weight decay 0.1, gradient clip 1.0), dropout 0.0, no bias
Learning rate	$3\times 10^{-4}$ , cosine decay to $3\times 10^{-5}$ , warmup 1000 iterations
Iterations	20,000
Effective batch	64 sequences/step (batch size 4 per device, gradient accumulation 16/2 GPUs $=8$ per device, 2 devices)
Precision	bfloat16
Training seed	1337 (all nine core checkpoints)

Appendix

Table 1: Shared CDR training configuration. Parameter counts are unique trainable parameters; the input embedding and the output head share a single weight matrix, counted once. CDR adds rank-

r

down- and up-projections for dynamic routing-query corrections in the residual aggregation paths before both the attention and MLP sublayers.

Look-Ahead Bias in Language Models

See all 10 papers

The Mask Is Not the Model: Auditing Prefix Invariance in Attention, State-Space, and Hybrid Sequence Models
Aug 24, 2026Taebong Kim, Youngsik Hong, Minsik Kim +3LLM Auditing Sequence Modeling
VIDRAFT AI Research · QuantumOS, Seoul, Republic of Korea
Decoupling Token Roles in Autoregressive Pretraining
Sep 27, 2026Suqin Yuan, Runqi Lin, Kevin Qinghong Lin +4Language Model Pretraining Autoregressive Language Modeling
University of Sydney · University of Oxford · Southeast University
Leaky Language Models: Stealing Architecture and Inference Optimizations via Per-Token Timing
Jul 22, 2026Sadegh Majidi, Niloofar Mireshghallah, Kazem TaramModel Extraction Attacks Language Model Fingerprinting
Purdue University West Lafayette, IN, USA · Carnegie Mellon University Pittsburgh, PA, USA

Aug 24, 2026J/K move · Enter open · S save

The Mask Is Not the Model: Auditing Prefix Invariance in Attention, State-Space, and Hybrid Sequence Models

Taebong Kim, Youngsik Hong, Minsik Kim, Sunyoung Choi+2

VIDRAFT AI Research · QuantumOS, Seoul, Republic of Korea

LLM Auditing Sequence Modeling State Space Models

Open paper AskPDF arXiv

Hybrid sequence models must satisfy prefix invariance: representations at position t must not depend on future inputs, yet this is rarely verified. We formalize prefix invariance and give a lightweight audit, two forward passes, no training or gradients, yielding a per-layer score localizing where causality breaks. Attention-mask inspection, the field's default check, is incomplete: causality is a graph-level property, and leaks can occur via scans, aggregations, or normalization despite correct masks. Across 192 injected-fault trials on eight checkpoints, mask inspection detected none, while our audit localized all 192/192 to the exact layer. Static/dynamic analysis of chunked-scan code in transformers found the same defect in Zamba2 and Nemotron-H, an inter-chunk axis error fixed via the reference implementation. The method fits on one page and runs in seconds.

Abstract

Look-Ahead Bias in Language Models

Explore similar work

The Mask Is Not the Model: Auditing Prefix Invariance in Attention, State-Space, and Hybrid Sequence Models

Decoupling Token Roles in Autoregressive Pretraining

Leaky Language Models: Stealing Architecture and Inference Optimizations via Per-Token Timing