Breakdown of Autoregressive Multi-Step Reasoning Under Concentrated vs. Diffuse Attention Head Dropout
Organizations: Institute of Information Technology, Warsaw University of Life Sciences – SGGW Warsaw, Poland
Abstract
During extended wakefulness, mammalian neocortical networks experience localized periods of transient neuronal silence termed off-states. Here, we examine how analogous spatiotemporally concentrated disruptions affect sequential reasoning in autoregressive Large Language Models (LLMs). Evaluating 250 paired derivations on the GSM8K benchmark across two 8B models (Qwen3-8B and LLaMA-3.1-8B-Instruct), we contrast clustered bursts of attention head masking against budget-matched diffuse i.i.d. dropout. Concentrated bursts induce substantial reasoning degradation: at onset parameter (scheduling of vulnerable head-token volume), Qwen3-8B accuracy drops from under diffuse noise to under burst dropout ( percentage points [pp], exact McNemar ), while LLaMA-3.1-8B-Instruct drops from to ( pp, ). Conversely, dispersing the identical cumulative budget uniformly across generation steps via stratified quintile matching yields substantially milder degradation (retaining in Qwen and in LLaMA). In an exploratory factorial dissociation (), single-step dropout density is associated with an acute drop in a single schedule draw ( pp on Qwen, ), while multi-token continuity exhibits an inconclusive compounding trend ( pp, ). Error analysis reveals that Qwen failures reflect derivation errors within formatted responses (conditional accuracy drop of pp), whereas in LLaMA of discordant losses lack a final answer format line (conditional gap narrows to pp). These findings demonstrate that autoregressive reasoning chains are particularly vulnerable to concentrated structural disruptions, providing empirical benchmarks for fault tolerance in transformer architectures.
Figures & tables
| Model | Arm | Budget | Realized Heads | Answer Rate | Raw Acc | Cond Acc | Paired | Discordant | 95% Bootstrap CI | ||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3-8B | Clean | 0.00 | 0.0 | 99.6% | 93.6% | 94.0% | Baseline | — | — | — | — |
| i.i.d. | 0.05 | 4,969.8 | 98.4% | 92.4% | 93.5% | — | — | — | — | — | |
| Burst | 0.05 | 4,868.6 | 98.8% | 88.0% | 88.7% | pp | 0.0614 | 0.0614 | |||
| i.i.d. | 0.10 | 8,631.3 | 98.8% | 91.2% | 91.9% | — | — | — | — | — | |
| Burst | 0.10 | 8,897.3 | 96.4% | 78.8% | 80.5% | pp | |||||
| LLaMA-3.1-8B-Instruct | Clean | 0.00 | 0.0 | 98.8% | 87.2% | 87.9% | Baseline | — | — | — | — |
| Model | Arm | Accuracy | Realized Heads | Contrast | (pp) | Discordant | ||
|---|---|---|---|---|---|---|---|---|
| Qwen3-8B | Clean | 93.0% | 0.0 | Baseline | — | — | — | — |
| i.i.d. | 94.0% | 7,485.9 | Baseline i.i.d. | — | — | — | — | |
| Spaced | 84.0% | 7,585.4 | Spaced vs. i.i.d. | pp | 0.0213 | 0.0425 | ||
| Burst | 77.0% | 7,700.4 | Burst vs. Spaced | pp | 0.1892 | 0.1892 | ||
| Burst vs. i.i.d. | pp | |||||||
| LLaMA-3.1-8B-Instruct | Clean | 88.0% | 0.0 | Baseline | — | — | — | — |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Arm | Budget | Mean Sched | Mean Realized | Accuracy | Count | Paired | Discordant | ||
|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3-8B | Clean | 0.00 | 0.0 | 0.0 | 90.0% | 54/60 | Baseline | — | — | — |
| i.i.d. | 0.15 | 21,666 | 11,353.8 | 90.0% | 54/60 | — | — | — | — | |
| Burst | 0.15 | 21,666 | 11,381.2 | 66.7% | 40/60 | pp | 0.0013 | 0.0026 | ||
| i.i.d. | 0.20 | 25,921 | 13,508.2 | 88.3% | 53/60 | — | — | — | — | |
| Burst | 0.20 | 25,921 | 14,463.2 | 73.3% | 44/60 | pp | 0.0225 | 0.0225 | ||
| LLaMA-3.1-8B-Instruct | Clean | 0.00 | 0.0 | 0.0 | 90.0% | 54/60 | Baseline | — | — | — |
| Model | Duration | Mean Sched | Realized (B) | Realized (I) | Burst Acc | i.i.d. Acc | Paired | Discordant | ||
|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3-8B | 13,214 | 7,122.3 | 6,699.9 | 71.7% | 93.3% | pp | 0.0002 | 0.0007 | ||
| 15,502 | 7,969.5 | 7,990.1 | 78.3% | 90.0% | pp | 0.0654 | 0.1309 | |||
| 18,323 | 9,644.7 | 9,267.1 | 75.0% | 85.0% | pp | 0.1796 | 0.1796 | |||
| LLaMA-3.1-8B-Instruct | 11,709 | 5,217.2 | 5,125.5 | 85.0% | 85.0% | pp | 1.0000 | 1.0000 | ||
| 14,104 | 6,225.2 | 6,074.3 | 80.0% | 90.0% | pp | 0.0703 | 0.2109 | |||
| 15,290 | 6,662.5 | 6,434.1 | 88.3% | 95.0% | pp | 0.2891 | 0.5781 |
| Model | Experiment | Arm | Budget | Final Answer Rate | Raw Acc | Conditional Acc | |
|---|---|---|---|---|---|---|---|
| Qwen3-8B | Confirmatory | Clean | 0.00 | 250 | 99.6% | 93.6% | 94.0% |
| Confirmatory | i.i.d. | 0.05 | 250 | 98.4% | 92.4% | 93.5% | |
| Confirmatory | Burst | 0.05 | 250 | 98.8% | 88.0% | 88.7% | |
| Confirmatory | i.i.d. | 0.10 | 250 | 98.8% | 91.2% | 91.9% | |
| Confirmatory | Burst | 0.10 | 250 | 96.4% | 78.8% | 80.5% | |
| Dissociation | Clean | 0.00 | 100 | 100.0% | 93.0% | 93.0% |
Language Model Robustness
Language model robustness studies how reliably language models preserve correct, safe behavior under input changes, distribution shifts, and adversarial pressure.
See all 519 papers