While transformers achieve strong performance, their internal decision-making processes remain opaque. We investigate whether architectural constraints can enforce interpretability by design through architectural stream independence: maintaining a token stream (carrying symbolic structure) and contextual semantics in separated streams that remain independently observable throughout processing, with integration delayed until output. We validate this principle through the Late Fusion Architecture (LFA), which demonstrates interpretable symbolic heads through all the final layers, while standard transformers show dissolution by the third of six layers; we quantify this effect by introducing the Token-Position Dependence Score (PDS), with PDSmax = 0.276 and 0.058, respectively. Crucially, intervention experiments demonstrate functional modularity: suppressing LFA's recency heads causes minimal semantic damage (Cohen's d = -0.158) versus catastrophic entanglement in baselines (d = -0.672). LFA demonstrates that architectural constraints improve underlying learning mechanisms, averaging 42% stability versus 19% and 11% for baseline comparisons, with extremes from 50% on LFA's best pairs (6 of 12 heads position-invariant) down to 0% complete collapse in over-constrained cases. By preventing premature entanglement, architectural independence steers models toward semantic understanding over positional heuristics, establishing interpretability as an architectural design criterion enforceable through structural constraints rather than post-hoc analysis.
Figures & tables
Property
Immediate
Stream
Integration
Independence
(Std-T)
(LFA)
Position encoding
Added at L0
Frozen in XT
Stream separation
None
XT / XE distinct
Information flow
Bidirectional
XT→XE only
Position observable L5
No (PDS=0.058)
Yes (PDS=0.276)
Table 1: Immediate integration vs. architectural stream independence. LFA maintains independent observability through asymmetric information flow and delayed symmetric combination.
Model
Attn
FFN
Params
Val Loss
Std-T
dns
dns
22.2M
1.8114
D-Cas
dns
dns
21.2M
1.8399
LFA
ind
dns
18.5M
1.9063
CFM
ind
ind
13.4M
2.1019
Table 2: Model configurations and training performance. All models have 6 layers, 6 heads, 384d embeddings, trained on 2M samples from TinyStories for 2 epochs.
Head
Mean Attn
Top1 Acc
LFA
Std-T
LFA
Std-T
L4.H3
0.323
0.033
48.3%
6.9%
L3.H5
0.265
0.047
48.3%
6.9%
L4.H0
0.211
0.045
44.1%
13.1%
L4.H1
0.286
0.060
37.2%
13.1%
L3.H0
0.225
0.049
46.9%
4.8%
Table 3: Coreference specialization: LFA vs. Std-T. LFA’s top 5 coreference heads (ranked by mean attention) compared to the same head positions in Std-T. LFA concentrates specialists in L3-L4; these same positions in Std-T show weak, diffuse patterns. Std-T’s actual best heads (L1.H5: 46.9%, L3.H2: 44.8%) are distributed across different layers, requiring search to locate.
Model
Total
L4-L5
Max L5
Avg PDS
PDS > 0.075
Heads
PDS
LFA
7
5
0.276
0.061
Std-T
3
1
0.058
0.032
CFM
5
0
0.032
0.042
Table 4: Token-Position dependence by layer. LFA maintains distinct symbolic channels in deep layers (5 of 7 heads in L4-L5, max L5 PDS = 0.276), validating architectural independence is maintained. Std-T dissolves by L5 (max = 0.058), CFM fails to integrate (max = 0.032).
Figure 1: Temporal validation of architectural stream independence. LFA concentrates positional processing in layers 4-5, maintaining complete stream separation throughout all transformer layers. Std-T processes position in layers 0-1, causing immediate entanglement and dissolution by mid-layers. CFM’s mid-layer concentration reflects failed integration—excessive constraint prevents late-layer coordination. This layer distribution directly validates the architectural hypothesis: separated streams preserve distinct symbolic channels in deep layers.
Figure 2: Functional transparency via intervention. SPS under baseline and recency head suppression (PDS > 0.075). LFA demonstrates functional independence ( d=−0.158 ), Std-T shows moderate entanglement ( d=−0.298 ), CFM reveals complete opacity ( d=−0.672 ). Error bars: SEM.
Model
Layers
# Heads
Cohen’s d
p -value
LFA
L5
7
-0.158
0.028
Std-T
L1, L4
3
-0.298
< 0.0001
CFM
L2
5
-0.672
< 0.0001
Table 5: Functional transparency via intervention: effect sizes for recency head suppression. Negative Cohen’s d indicates performance degradation. LFA demonstrates functional independence ( d=−0.158 ) while CFM reveals entangled opacity ( d=−0.672 ).
Figure 4: PDS distribution across all heads. LFA shows bimodal separation between position-dependent (PDS >0.075 ) and position-invariant heads. Std-T and CFM show unimodal distributions centered near zero, indicating position signals dissolve or fail to form.
Figure 5: Gate response curves for top-1 recency head intervention. LFA shows shallow, linear response (functional independence). Std-T and CFM show steep, non-linear collapse (entanglement).
Figure 6: Gate response curves for top-3 recency head intervention. Pattern consistent with k=1 : LFA maintains linearity, baselines show catastrophic entanglement.
Figure 7: PDS heatmap acrss all heads. Each cell shows PDS for layer ℓ , head h . LFA concentrates high-PDS heads in layers 4-5 (late fusion). Std-T shows weak position signals in layers 0-1 that dissolve by mid-layers (early fusion). CFM shows mid-layer concentration without late-layer integration (failed fusion).