A Reproducibility Study of Partial Residual Ablations in Pre-LN Transformers
Organizations: Independent Research Project
Abstract
Residual connections are a fundamental component of transformer architectures, yet the roles of the attention and feed-forward residual pathways remain poorly understood when considered independently. This paper presents a reproducibility study of partial residual ablations in Pre-LN GPT-style transformers trained at two scales (10M and 124M parameters). I compare four architectural configurations by selectively removing the attention residual connection, the feed-forward residual connection, or both. At 10M scale, a controlled 8-seed deterministic sweep shows a clear asymmetry: removing the attention residual (FFNOnly) reaches the No-Residual collapse regime (3.350 +/- 0.002), whereas removing the feed-forward residual (AttnOnly) remains well separated from it (1.580 +/- 0.003). A subsequent controlled 124M sweep establishes that the direction of this asymmetry persists under matched seeds: across five seeds each, AttnOnly reaches 6.037 +/- 0.355 versus 7.569 +/- 0.001 for FFNOnly, with no overlap between the observed ranges. AttnOnly nevertheless remains substantially degraded relative to FullResidual (4.577 +/- 0.012) and exhibits markedly greater seed sensitivity at 124M. During the investigation, I identified and corrected an experimental measurement confound in runtime gain scaling and retained an intermediate reproduction failure rather than excluding it. I propose cross-position routing through self-attention as a falsifiable hypothesis for the observed asymmetry; no clean causal test of this mechanism has yet been completed. To support reproducibility, I release the source code, experiment configurations, training logs, and experimental results, including intermediate non-reproducing runs.
Figures & tables
| Parameter | 10M | 124M | Shared |
| n_layer | 6 | 12 | lr: 3 10 -4 (nanoGPT: constant; ResidualGPT 10M: cosine, 100-step warmup, floor 3 10 -5 ) / 6 10 -4 6 10 -5 cosine, 2,000-step warmup (ResidualGPT 124M) |
| n_embd | 384 | 768 | =0.9, =0.95, weight decay=0.1 |
| n_head | 6 | 12 | Grad clip=1.0, dropout=0.2 (10M) / 0.1 (124M) |
| block_size | 256 | 1024 | Optimizer: AdamW |
| Batch size | 64 | 16 | Grad accum: 1 (10M) / 8 (124M) |
| Train steps | 3,000 | 20,000 | Parameters 10.7M / 124.4M |
| Config | Attn | FFN | Best val | PPL | Final val | Ratio |
| Full Residual | 1.4751 | 4.37 | 1.4751 | 1.00 | ||
| AttnOnly | 3.3534 | 28.60 | 3.3534 | 2.27 | ||
| FFNOnly | 3.3475 | 28.43 | 3.3475 | 2.27 | ||
| No Residual | 3.3523 | 28.57 | 3.3523 | 2.27 |
| Config | Best val loss std | PPL | Seeds | vs. Full Residual |
| Full Residual | 1.5017 – | 4.49 | 1337 | – (baseline) |
| AttnOnly | 4.95 | 1337, 42, 123 | +6.5% [ORIGINALLY RECORDED BELOW FLOOR] | |
| FFNOnly | 28.48 | 1337, 42, 123 | +123.0% [COLLAPSES] | |
| No Residual | 3.3564 – | 28.69 | 1337 | +123.5% [COLLAPSES] |
| Config | Best val loss | PPL | Ratio / Result |
| Full Residual | 3.2630 | 26.13 | 1.00 / Baseline |
| AttnOnly | 4.8994 | 134.21 | 1.50 / originally below floor (+50.1% above Full Res) |
| FFNOnly | 7.6662 | 2,135 | 2.35 / high-loss regime |
| No Residual | 7.6887 | 2,184 | 2.36 / floor |
| Config | Best val loss | Range | Interpretation | |
| Full Residual | 5 | 4.560–4.592 | Baseline; tight across seeds | |
| AttnOnly | 5 | 5.662–6.508 | Degraded vs. baseline, but below FFNOnly in every matched run | |
| FFNOnly | 5 | 7.568–7.570 | Tight high-loss regime | |
| No Residual | 7.584 | 1 | – | Seed-1337 floor anchor |
| Scale | Impl. | AttnOnly best val | FFNOnly best val | AttnOnly gap | Asym. ratio |
| 10M | nanoGPT | 3.353 (high-loss) | 3.348 (high-loss) | +127.3% | 1.00 |
| 10M | ResidualGPT historical | +6.5% | 2.09 | ||
| 10M | ResidualGPT Phase 3 (8-seed sweep) | +5.4% | 2.12 | ||
| 124M | nanoGPT | 7.433 (high-loss) | 7.430 (high-loss) | +113.5% | 1.00 |
| 124M | ResidualGPT historical | 4.899 | 7.666 | +50.1% | 1.56 |
| 124M | ResidualGPT controlled sweep | +31.9% | 1.25 |
| Accessible context | Seed 1337 | Seed 42 | Seed 123 | Mean std | |
| 0 | self only | 2.490095 | 2.490724 | 2.491776 | |
| 1 | self + 1 previous | 1.562058 | 1.571663 | 1.575357 | |
| 3 | self + 3 previous | 1.527295 | 1.527308 | 1.517451 | |
| 4 | self + 4 previous | 1.518006 | 1.522560 | 1.510614 | |
| 6 | self + 6 previous | 1.520973 | 1.525632 | 1.520637 | |
| 8 | self + 8 previous | 1.521617 | 1.539797 | 1.526760 |
Transformer
Transformers are neural architectures that use attention to model relationships among sequence elements in language, vision, and scientific data.
See all 579 papers