Residual connections are a fundamental component of transformer architectures, yet the roles of the attention and feed-forward residual pathways remain poorly understood when considered independently. This paper presents a reproducibility study of partial residual ablations in Pre-LN GPT-style transformers trained at two scales (10M and 124M parameters). I compare four architectural configurations by selectively removing the attention residual connection, the feed-forward residual connection, or both. At 10M scale, a controlled 8-seed deterministic sweep shows a clear asymmetry: removing the attention residual (FFNOnly) reaches the No-Residual collapse regime (3.350 +/- 0.002), whereas removing the feed-forward residual (AttnOnly) remains well separated from it (1.580 +/- 0.003). A subsequent controlled 124M sweep establishes that the direction of this asymmetry persists under matched seeds: across five seeds each, AttnOnly reaches 6.037 +/- 0.355 versus 7.569 +/- 0.001 for FFNOnly, with no overlap between the observed ranges. AttnOnly nevertheless remains substantially degraded relative to FullResidual (4.577 +/- 0.012) and exhibits markedly greater seed sensitivity at 124M. During the investigation, I identified and corrected an experimental measurement confound in runtime gain scaling and retained an intermediate reproduction failure rather than excluding it. I propose cross-position routing through self-attention as a falsifiable hypothesis for the observed asymmetry; no clean causal test of this mechanism has yet been completed. To support reproducibility, I release the source code, experiment configurations, training logs, and experimental results, including intermediate non-reproducing runs.
Figure 1: Validation loss curves at 10M (left, TinyShakespeare) and 124M (right, OpenWebText) using the nanoGPT implementation. Both partial-residual configurations and NoResidual plateau in nearly the same high-loss regime at both scales.
Figure 2: Mechanistic diagnostics from the original 10M investigation. (a) In nanoGPT, the recorded Layer-0 fc1 parameter gradient norm rounds to 0.000 from step 300 onward for the ablated configurations while FullResidual remains nonzero. (b) Historical nanoGPT hidden-norm growth ratios: FullResidual 1.4 × , AttnOnly 14.0 × , FFNOnly 5.4 × , and NoResidual 8.4 × . (c) The originally recorded ResidualGPT outcomes: FullResidual 1.502, AttnOnly 1.600, FFNOnly 3.349, and NoResidual 3.356. Panel (c) does not show the later Phase-2 reproduction.
Config
Best val loss ± std
PPL
Seeds
vs. Full Residual
Full Residual
1.5017 –
4.49
1337
– (baseline)
AttnOnly
1.5999±0.0149
4.95
1337, 42, 123
+6.5% [ORIGINALLY RECORDED BELOW FLOOR]
FFNOnly
3.3493±0.0009
28.48
1337, 42, 123
+123.0% [COLLAPSES]
No Residual
3.3564 –
28.69
1337
+123.5% [COLLAPSES]
Table 3: ResidualGPT implementation, 10M params, TinyShakespeare. Historical mean ± population standard deviation across three seeds for AttnOnly and FFNOnly; FullResidual and NoResidual are single-seed anchors. AttnOnly values are as originally recorded; see Section 8.1 for a subsequent reproduction attempt that did not recover this exact statistic.
Figure 3: Historical 10M ResidualGPT asymmetry as originally recorded. The bars show FullResidual 1.5017, AttnOnly 1.5999±0.0149 across three seeds, FFNOnly 3.3493±0.0009 across three seeds, and the NoResidual seed-1337 anchor 3.3564. This figure does not visualize the later reproduction attempt; that reproduction is retained in the text as experimental history. Numerical annotations report mean ± population SD, whereas plotted error bars extend to ±2 population SD for the multi-seed configurations. FullResidual and NoResidual are single-seed anchors.
Figure 4: Historical three-seed 10M ResidualGPT stability result as originally recorded. AttnOnly gives mean 1.5999 ± 0.0149 across seeds 1337, 42, and 123, while FFNOnly gives 3.3493 ± 0.0009. Individual points show each seed, and dashed lines show the means. This figure is retained as experimental history; the later controlled Phase-3 sweep in Figure 5 provides the primary 10M estimate. Historical standard deviations use the population convention; shaded bands represent ±1 population SD.
Figure 5: Controlled Phase-3 10M ResidualGPT confirmation. Points show the reported aggregate best-validation-loss statistics: FullResidual anchor 1.499, AttnOnly 1.580 ± 0.003 across eight seeds, FFNOnly 3.350 ± 0.002 across three seeds, and NoResidual anchor 3.356. Error bars denote sample standard deviation where multi-seed statistics are available. The AttnOnly–FFNOnly mean separation is 1.770 validation-loss units. Individual eight-seed values are not reconstructed here because the retained paper artifacts support the aggregate statistics rather than all per-seed values.
Config
Best val loss
PPL
Ratio / Result
Full Residual
3.2630
26.13
1.00 × / Baseline
AttnOnly
4.8994
134.21
1.50 × / originally below floor (+50.1% above Full Res)
FFNOnly
7.6662
2,135
2.35 × / high-loss regime
No Residual
7.6887
2,184
2.36 × / floor
Table 4: Historical ResidualGPT 124M results on OpenWebText, seed 1337. These values are retained unchanged as experimental history and are not pooled with the later controlled sweep.
Figure 6: Historical 124M ResidualGPT scale-verification figure retained as experimental history. Panel (a) shows the originally recorded seed-1337 comparison (FullResidual 3.2630, AttnOnly 4.8994, FFNOnly 7.6662, NoResidual 7.6887); panel (b) shows the historical FFNOnly/AttnOnly loss ratios (2.09 at 10M and 1.56 at 124M). These values are not pooled with the later controlled matched-seed sweep.
Config
Best val loss
n
Range
Interpretation
Full Residual
4.577±0.012
5
4.560–4.592
Baseline; tight across seeds
AttnOnly
6.037±0.355
5
5.662–6.508
Degraded vs. baseline, but below FFNOnly in every matched run
FFNOnly
7.569±0.001
5
7.568–7.570
Tight high-loss regime
No Residual
7.584
1
–
Seed-1337 floor anchor
Table 5: Controlled 124M ResidualGPT sweep on OpenWebText. Mean ± sample standard deviation across five matched seeds (1337, 42, 123, 0, 7). NoResidual is a contemporaneous seed-1337 anchor.
Figure 7: Controlled 124M ResidualGPT matched-seed summary. Points show mean best validation loss; error bars show sample standard deviation, and the lighter vertical bars show the observed min–max range for the five-seed FullResidual, AttnOnly, and FFNOnly sweeps. NoResidual is the contemporaneous seed-1337 anchor. The figure uses reported aggregate and range statistics and does not reconstruct unavailable individual seed values. The closest observed AttnOnly–FFNOnly pair is separated by 1.060 validation-loss units.
Scale
Impl.
AttnOnly best val
FFNOnly best val
AttnOnly gap
Asym. ratio
10M
nanoGPT
3.353 (high-loss)
3.348 (high-loss)
+127.3%
1.00 ×
10M
ResidualGPT historical
1.600±0.015
3.349±0.001
+6.5%
2.09 ×
10M
ResidualGPT Phase 3 (8-seed sweep)
1.580±0.003
3.350±0.002
+5.4%
2.12 ×
124M
nanoGPT
7.433 (high-loss)
7.430 (high-loss)
+113.5%
1.00 ×
124M
ResidualGPT historical
4.899
7.666
+50.1%
1.56 ×
124M
ResidualGPT controlled sweep
6.037±0.355
7.569±0.001
+31.9%
1.25 ×
Table 6: Cross-scale asymmetry summary. Historical rows are retained unchanged; controlled rows provide the primary multi-seed evidence. The historical 10M ResidualGPT row retains population SDs; controlled-sweep SDs follow the conventions stated in their respective tables and captions.
k
Accessible context
Seed 1337
Seed 42
Seed 123
Mean ± std
0
self only
2.490095
2.490724
2.491776
2.490865±0.000849
1
self + 1 previous
1.562058
1.571663
1.575357
1.569693±0.006865
3
self + 3 previous
1.527295
1.527308
1.517451
1.524018±0.005687
4
self + 4 previous
1.518006
1.522560
1.510614
1.517060±0.006029
6
self + 6 previous
1.520973
1.525632
1.520637
1.522414±0.002792
8
self + 8 previous
1.521617
1.539797
1.526760
1.529392±0.009371
Table 7: 10M communication-radius mechanism study. Here k is the number of previous tokens accessible per attention layer in addition to the current token. All reported rows use three matched seeds. Individual seed values are rounded for display; summary statistics retain the notebook-reported values computed from the underlying run results. Lower validation loss is better.
Figure 8: Replicated 10M communication-radius intervention. Points show mean best validation loss across seeds 1337, 42, and 123; error bars show sample standard deviation. The left panel shows all replicated radii; the right panel enlarges the nonzero-radius results. Radii are displayed at equally spaced categorical positions. The dominant transition occurs from k=0 (self only) to k=1 (one previous token accessible per attention layer). Increasing the radius beyond a small local window gives comparatively modest changes and does not improve monotonically.
Looped transformers increase effective depth by repeatedly applying a shared set of transformer layers, offering a parameter-efficient way to scale computation. However, standard residual connections aggregate information across recurrent depth into a single stream, providing limited control over which intermediate representations from preceding layers should be reused. In this paper, we introduce ARLoop, a looped transformer architecture based on attention residuals, where the input of each layer is selectively aggregated through attention over model depth. Distinct from standard transformers, we find that looped transformers do not require access to the full residual history: sliding-window depth attention matches or outperforms full-history attention at larger recurrent depths. Across controlled experiments with multiple parameter scales (140M and 370M) and recurrent depths (R = 1 to 5), ARLoop consistently outperforms prior residual designs for looped transformers. More importantly, we further scale ARLoop to 0.6B and 1.2B parameters under a realistic LLM pretraining recipe, demonstrating that the gains transfer to realistic pretraining, with particularly strong improvements on math and coding tasks. Our 1.2B model, pretrained on only 0.5T tokens with three recurrences, outperforms Huginn-3.5B (the only fully-open looped LLM as of our release date) across all evaluated tasks and is competitive with similar-sized non-looped models such as Qwen3.5-2B despite being trained on substantially less data.
Attention-only dynamical theories model Transformer residual directions as particles aggregating on a sphere. We extend this framework by incorporating the feed-forward network (FFN) term as a local steering field acting on each token state. The resulting theory predicts that the tangential component of the FFN field is necessary for motion in residual-direction space, that critical residual directions correspond to nonlinear projective equilibria, and that a commutator defect determines when a finite attention--FFN block can be accurately approximated by a parallel, additive flow. Across GPT-2, Pythia, Mistral, and Llama models, the extended theory improves one-step angular prediction relative to an attention-only baseline, with the contribution of the FFN increasing from GPT-2 to Llama-3-8B. Intervention experiments show that retaining only the tangential FFN component preserves most model quality, whereas retaining only the radial component causes performance to collapse. The tangential component also preserves output diversity under aggregation pressure. As a practical application, layers with small commutator defects can be approximately parallelized with only a modest increase in loss, whereas layers with large defects degrade rapidly. These findings support the interpretation of FFN layers as directional steering fields that shape Transformer residual geometry and govern the feasibility of block-level interventions.
Timur Mudarisov, Mikhail Burtsev, Radu State
University of Luxembourg · 2London Institute of Mathematical Sciences
Residual connections add every sublayer's proposed update with a fixed coefficient of one; the network never evaluates whether an update is reliable before committing it. Drawing on the human-factors principle of independent verification, we introduce Review Residuals, which scale each update by a learned, input-dependent gate conditioned on both the current state and the proposed update: h_l = h_{l-1} + r_l * u_l with r_l = sigmoid(W[RMSNorm(h_{l-1}), RMSNorm(u_l)]). Conditioning the gate on the update is the property that distinguishes it from prior gated and scaled residuals. We report two findings. First, a depth-stability result: a convex (Highway-style) form of the gate reintroduces vanishing gradients and fails to train beyond ~20 layers, whereas the additive, identity-preserving form trains stably at all depths we tested. Second, an emergence-with-scale result: trained from scratch across five sizes (60M-1B parameters, multi-seed), Review Residuals show no advantage at small scale but at 590M significantly outperform both a parameter-matched Highway gate and a parameter-matched standard residual (p<0.05), with a larger advantage at 1B. The benefit grows with model size rather than shrinking.