We identify a systematic attention collapse pathology in the BLOOM family of transformer language models, where ALiBi positional encoding causes 31-44% of attention heads to attend almost entirely to the beginning-of-sequence token. The collapse follows a predictable pattern across four model scales (560M to 7.1B parameters), concentrating in head indices where ALiBi's slope schedule imposes the steepest distance penalties. We introduce surgical reinitialization: targeted Q/K/V reinitialization with zeroed output projections and gradient-masked freezing of all non-surgical parameters. Applied to BLOOM-1b7 on a single consumer GPU, the technique recovers 98.7% operational head capacity (242 to 379 of 384 heads) in two passes. A controlled comparison with C4 training data confirms that reinitialization -- not corpus content -- drives recovery, and reveals two distinct post-surgical phenomena: early global functional redistribution that improves the model, and late local degradation that accumulates under noisy training signal. An extended experiment reinitializing mostly-healthy heads alongside collapsed ones produces a model that transiently outperforms stock BLOOM-1b7 by 25% on training perplexity (12.70 vs. 16.99), suggesting that pretrained attention configurations are suboptimal local minima. Code, checkpoints, and diagnostic tools are released as open-source software.
Figures & tables
Model
Params
Heads/Layer
Sick Band
Sick %
Layers Affected
BLOOM-560m
560M
16
H9–H15
44%
56–94% per index
BLOOM-1b7
1.7B
16
H9–H15
36%
56–94% per index
BLOOM-3b
3B
32
H20–H30
38%
63–94% per index
BLOOM-7b1
7.1B
32
H21–H30
31%
66–94% per index
Table 1: BOS-sink collapse patterns across the BLOOM model family.
Figure 1: Bimodal distribution of BOS mass across all 384 heads in stock BLOOM-1b7. Heads cluster near 0.0 (healthy, content-dependent attention) or above 0.8 (collapsed, BOS-fixated), with very few in the intermediate range. Our 0.50 threshold falls in the sparse valley, making classification robust.
Figure 2: Cross-scale BOS-sink band pattern across the BLOOM family (560M, 1.7B, 3B, 7.1B). Each panel shows a layer × head heatmap colored by BOS mass. The sick band (upper head indices, darker color) appears consistently across all scales.
Metric
Stock
Pass 1 (E3)
Pass 2 (E1)
Healthy
242 (63.0%)
341 (88.8%)
379 (98.7%)
BOS-sink
136
39
1
DEAD
3
0
0
Low-entropy
3
4
4
Training PPL
16.99
15.13
15.10
Held-out PPL
21.45
27.52
28.76
Table 2: Head health and perplexity metrics across surgical passes.
Model
C4 Validation PPL
Eval Prompt PPL
Stock BLOOM-1b7
32.42
21.45
C4 Surgical (E3)
29.30
29.59
Curated Surgical (Pass 2)
74.11
28.76
Table 3: C4 validation perplexity (50 held-out C4 texts, 4,024 tokens). The C4-trained surgical model improves over stock on C4 data, confirming that held-out PPL increase on our evaluation prompts is distribution shift, not capacity loss.
Metric
Curated (E3)
C4 (E3)
C4 (E15)
Heads woken
108/108
108/108
108/108
Training PPL
∼ 15.4
20.80
36.31
Held-out PPL
27.52
29.59
—
Table 4: Comparison of curated vs. C4 training corpus for Pass 1 surgery.
Figure 3: Three-way attention topology comparison. Stock BLOOM-1b7 ( left ), curated surgery E3 ( center ), C4 baseline E3 ( right ). Each panel shows the 24×16 head grid colored by BOS mass. Curated surgery drives more global redistribution outside the surgical zone while achieving lower perplexity.
Table 6: In-band frozen head drift (H9–H15 healthy heads, kept frozen during surgery).
Figure 4: Two-phenomenon comparison: curated E3 vs. C4 E3 redistribution patterns at matched epochs. Phenomenon 1 (functional redistribution): early, global, beneficial—curated corpus drives more outside-zone redistribution. Phenomenon 2 (local degradation): late, local, pathological—C4 at E15 shows doubled in-band frozen drift relative to E3.
Column
Frozen Heads
E3 Mean δ
E15 Mean δ
Trend
H15
4
0.083
0.154
Spreading
H13
2
0.046
0.070
Spreading
H9
4
0.034
0.051
Stable
H10
3
0.046
0.046
Stable
H12
3
0.007
0.019
Stable
Table 7: Column-wise frozen head drift in C4 baseline across training epochs.
Layer
Stock BOS Mass
Stock Status
Post-Surgery BOS Mass
Change
L5
0.166
Healthy
0.056
−66%
L7
0.317
Healthy
0.034
−89%
L10
0.374
Healthy
0.040
−89%
L11
0.517
Collapsed
0.079
−85%
L12
0.666
Collapsed
0.027
−96%
L14
0.483
Healthy
0.023
−95%
Table 8: H5 column head status before and after surgery (selected layers).
Condition
Training PPL
Stock BLOOM-1b7
16.99
Band-only surgery (Pass 1 E3)
15.13
Band + H5 surgery (E1)
13.90
Band + H5 sub-epoch best (step 42/140)
12.70
Table 9: Training perplexity comparison for extended surgery conditions.
Figure 5: H5 sub-epoch PPL trajectory. Perplexity drops monotonically from post-reinitialization (19.25) to 12.70 at step 42 (30% of epoch 1), crossing the stock BLOOM-1b7 baseline (16.99) by step 10. Overfitting begins after step 42, causing perplexity to rise above the stock baseline after approximately one full epoch.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Head
Slope
Distance Penalty at pos 10
At pos 100
H0
0.7071
−7.07
−70.71
H4
0.1768
−1.77
−17.68
H8
0.0442
−0.44
−4.42
H12
0.0110
−0.11
−1.10
H15
0.0039
−0.04
−0.39
Appendix
Table 10: ALiBi slope values and distance penalties for BLOOM-1b7 ( H=16 ).
Figure 6: Frozen in-band head drift heatmap across training epochs (C4 baseline). Each cell shows the absolute BOS mass change ∣δ∣ for a frozen head (H9–H15 band, healthy at surgery time) relative to stock. Drift accumulates progressively in the H15 column while H12 and H10 remain largely stable, consistent with the column-specific propagation pattern described in Section 4.5.
Figure 7: Column-wise drift trajectories across training epochs (C4 baseline). Each line tracks the mean ∣δ∣ for frozen heads within a given head-index column. H15 shows a spreading pattern (mean delta nearly doubles from E3 to E15), while H9, H10, and H12 remain stable. The differential vulnerability tracks the ALiBi slope schedule: steeper-slope columns are more susceptible to residual stream perturbation.