While transformers achieve strong performance, their internal decision-making processes remain opaque. We investigate whether architectural constraints can enforce interpretability by design through architectural stream independence: maintaining a token stream (carrying symbolic structure) and contextual semantics in separated streams that remain independently observable throughout processing, with integration delayed until output. We validate this principle through the Late Fusion Architecture (LFA), which demonstrates interpretable symbolic heads through all the final layers, while standard transformers show dissolution by the third of six layers; we quantify this effect by introducing the Token-Position Dependence Score (PDS), with PDSmax = 0.276 and 0.058, respectively. Crucially, intervention experiments demonstrate functional modularity: suppressing LFA's recency heads causes minimal semantic damage (Cohen's d = -0.158) versus catastrophic entanglement in baselines (d = -0.672). LFA demonstrates that architectural constraints improve underlying learning mechanisms, averaging 42% stability versus 19% and 11% for baseline comparisons, with extremes from 50% on LFA's best pairs (6 of 12 heads position-invariant) down to 0% complete collapse in over-constrained cases. By preventing premature entanglement, architectural independence steers models toward semantic understanding over positional heuristics, establishing interpretability as an architectural design criterion enforceable through structural constraints rather than post-hoc analysis.
Figures & tables
Property
Immediate
Stream
Integration
Independence
(Std-T)
(LFA)
Position encoding
Added at L0
Frozen in XT
Stream separation
None
XT / XE distinct
Information flow
Bidirectional
XT→XE only
Position observable L5
No (PDS=0.058)
Yes (PDS=0.276)
Table 1: Immediate integration vs. architectural stream independence. LFA maintains independent observability through asymmetric information flow and delayed symmetric combination.
Model
Attn
FFN
Params
Val Loss
Std-T
dns
dns
22.2M
1.8114
D-Cas
dns
dns
21.2M
1.8399
LFA
ind
dns
18.5M
1.9063
CFM
ind
ind
13.4M
2.1019
Table 2: Model configurations and training performance. All models have 6 layers, 6 heads, 384d embeddings, trained on 2M samples from TinyStories for 2 epochs.
Head
Mean Attn
Top1 Acc
LFA
Std-T
LFA
Std-T
L4.H3
0.323
0.033
48.3%
6.9%
L3.H5
0.265
0.047
48.3%
6.9%
L4.H0
0.211
0.045
44.1%
13.1%
L4.H1
0.286
0.060
37.2%
13.1%
L3.H0
0.225
0.049
46.9%
4.8%
Table 3: Coreference specialization: LFA vs. Std-T. LFA’s top 5 coreference heads (ranked by mean attention) compared to the same head positions in Std-T. LFA concentrates specialists in L3-L4; these same positions in Std-T show weak, diffuse patterns. Std-T’s actual best heads (L1.H5: 46.9%, L3.H2: 44.8%) are distributed across different layers, requiring search to locate.
Model
Total
L4-L5
Max L5
Avg PDS
PDS > 0.075
Heads
PDS
LFA
7
5
0.276
0.061
Std-T
3
1
0.058
0.032
CFM
5
0
0.032
0.042
Table 4: Token-Position dependence by layer. LFA maintains distinct symbolic channels in deep layers (5 of 7 heads in L4-L5, max L5 PDS = 0.276), validating architectural independence is maintained. Std-T dissolves by L5 (max = 0.058), CFM fails to integrate (max = 0.032).
Figure 1: Temporal validation of architectural stream independence. LFA concentrates positional processing in layers 4-5, maintaining complete stream separation throughout all transformer layers. Std-T processes position in layers 0-1, causing immediate entanglement and dissolution by mid-layers. CFM’s mid-layer concentration reflects failed integration—excessive constraint prevents late-layer coordination. This layer distribution directly validates the architectural hypothesis: separated streams preserve distinct symbolic channels in deep layers.
Figure 2: Functional transparency via intervention. SPS under baseline and recency head suppression (PDS > 0.075). LFA demonstrates functional independence ( d=−0.158 ), Std-T shows moderate entanglement ( d=−0.298 ), CFM reveals complete opacity ( d=−0.672 ). Error bars: SEM.
Model
Layers
# Heads
Cohen’s d
p -value
LFA
L5
7
-0.158
0.028
Std-T
L1, L4
3
-0.298
< 0.0001
CFM
L2
5
-0.672
< 0.0001
Table 5: Functional transparency via intervention: effect sizes for recency head suppression. Negative Cohen’s d indicates performance degradation. LFA demonstrates functional independence ( d=−0.158 ) while CFM reveals entangled opacity ( d=−0.672 ).
Figure 4: PDS distribution across all heads. LFA shows bimodal separation between position-dependent (PDS >0.075 ) and position-invariant heads. Std-T and CFM show unimodal distributions centered near zero, indicating position signals dissolve or fail to form.
Figure 5: Gate response curves for top-1 recency head intervention. LFA shows shallow, linear response (functional independence). Std-T and CFM show steep, non-linear collapse (entanglement).
Figure 6: Gate response curves for top-3 recency head intervention. Pattern consistent with k=1 : LFA maintains linearity, baselines show catastrophic entanglement.
Figure 7: PDS heatmap acrss all heads. Each cell shows PDS for layer ℓ , head h . LFA concentrates high-PDS heads in layers 4-5 (late fusion). Std-T shows weak position signals in layers 0-1 that dissolve by mid-layers (early fusion). CFM shows mid-layer concentration without late-layer integration (failed fusion).
Standard transformers entangle all computation in a single residual stream, obscuring which components perform which functions. We introduce the Dual-Stream Transformer, which decomposes the residual stream into two functionally distinct components: a token stream updated by attention and a context stream updated by feed-forward networks. Information flow between attention heads is controlled through a hierarchy of mixing strategies, from fully independent (maximum interpretability) to dense (standard transformer behavior). This design exposes a tunable tradeoff between interpretability and performance. We measure this tradeoff on language modeling tasks at 29M parameters. Fully independent head mixing increases validation loss by 8% relative to dense baselines. The recommended Kronecker mixing strategy, which permits scalar communication between heads while preserving within-head structure, costs only 2.5%. All configurations maintain functional generation under attention amplification (scaling logits by factors up to 16 at inference time), with degradation ranging from 16% to 27%. This robustness suggests the architectures learn discrete algorithms that operate independently of soft probabilistic mixing. The architecture provides a foundation for interpretable language models where internal structure is exposed by design. \footnote{This work was partially supported by DARPA Contract HR001125C0302.}
Transformers resist surgical control. Ablating an attention head identified as critical for capitalization produces minimal behavioral change because distributed redundancy compensates for damage. This Hydra effect renders interpretability illusory: we may identify components through correlation, but cannot predict or control their causal role. We demonstrate that architectural interventions can expose hidden modularity. Our approach combines dual-stream processing separating token and contextual representations, per-layer supervision providing independent gradient signal at each depth, and gated attention regularizing toward discrete activation patterns. When trained with per-layer supervision, models produce ablation effects 5 to 23 times larger than architecturally identical controls trained with standard objectives. This enables 4 times greater control leverage on targeted behaviors: scaling identified attention heads produces smooth, predictable changes in model output. The key finding is architectural. Without per-layer supervision, ablation damage concentrates near zero with low variance (Winograd standard deviation 0.63%). With per-layer supervision, effects spread widely (standard deviation 6.32%), revealing which predictions depend on which circuits. The larger variance is not measurement noise but the signature of unmasked modularity. We validate our approach through three components: engineered features that capture computational dynamics rather than vocabulary structure (validated by near-zero correlation with raw activation clustering), an architecture providing positive control for modularity, and causal experiments demonstrating functional reorganization where different tasks route through different attention heads. This es tablishes a methodology for transforming interpretability from passive observation to active control.
Language models are thought to exhibit the phenomenon of superposition, representing many more features than dimensions in their residual streams. Sparse autoencoders (SAEs) are designed to recover such features post-hoc, but training models that are interpretable by construction has remained impractical, as a per-layer over-complete bottleneck is prohibitively expensive in both memory and compute. To overcome this issue, we introduce the ParityTransformer, a GPT-2-scale architecture whose intermediate representations are efficient and wide / sparse by design. At each layer, a Deep Parity Bottleneck (DPB) replaces a learned over-complete basis with a parameter-free algebraic dictionary, providing a deterministic incoherence guarantee and eliminating the memory requirements that have prevented per-layer interpretable bottlenecks at scale. A DPB is a hierarchically structured sparse bottleneck which efficiently enforces sparsity using a multi-level mixture-of-experts approach: a hardware-aware implementation that closes the cost gap between activation sparse and dense training to a manageable interpretability tax. Empirically, ParityTransformers perform at least as well as post-hoc SAEs on sparse probing tasks, while out-performing on measures of feature absorption, steering effectiveness, and fine-grained causal interventions. Because subsequent computation acts only on features that survive the sparse bottleneck, the ParityTransformer's features are native to the model's forwards pass by construction, addressing the question of whether SAEs probe features the model actually uses during computation. We see this as a step toward training models whose internal representations are interpretable by design rather than recovered post hoc.