cs.AINov 10, 2026

Breakdown of Autoregressive Multi-Step Reasoning Under Concentrated vs. Diffuse Attention Head Dropout

Authors: Karol Struniawski

Organizations: Institute of Information Technology, Warsaw University of Life Sciences – SGGW Warsaw, Poland

Abstract

During extended wakefulness, mammalian neocortical networks experience localized periods of transient neuronal silence termed off-states. Here, we examine how analogous spatiotemporally concentrated disruptions affect sequential reasoning in autoregressive Large Language Models (LLMs). Evaluating 250 paired derivations on the GSM8K benchmark across two 8B models (Qwen3-8B and LLaMA-3.1-8B-Instruct), we contrast clustered bursts of attention head masking against budget-matched diffuse i.i.d. dropout. Concentrated bursts induce substantial reasoning degradation: at onset parameter B=0.10\mathcal{B}=0.10 (scheduling ∼5.7%\sim 5.7\% of vulnerable head-token volume), Qwen3-8B accuracy drops from 91.2%91.2\% under diffuse noise to 78.8%78.8\% under burst dropout (Δ=−12.4Δ= -12.4 percentage points [pp], exact McNemar p=5.5×10−6p = 5.5 \times 10^{-6}), while LLaMA-3.1-8B-Instruct drops from 84.0%84.0\% to 76.8%76.8\% (Δ=−7.2Δ= -7.2 pp, p=0.0198p = 0.0198). Conversely, dispersing the identical cumulative budget uniformly across generation steps via stratified quintile matching yields substantially milder degradation (retaining 91.2%91.2\% in Qwen and 84.0%84.0\% in LLaMA). In an exploratory factorial dissociation (N=100N=100), single-step dropout density is associated with an acute drop in a single schedule draw (−10.0-10.0 pp on Qwen, pholm=0.0425p_{\text{holm}} = 0.0425), while multi-token continuity exhibits an inconclusive compounding trend (−7.0-7.0 pp, pholm=0.1892p_{\text{holm}} = 0.1892). Error analysis reveals that Qwen failures reflect derivation errors within formatted responses (conditional accuracy drop of −11.4-11.4 pp), whereas in LLaMA 22.2%22.2\% of discordant losses lack a final answer format line (conditional gap narrows to −5.0-5.0 pp). These findings demonstrate that autoregressive reasoning chains are particularly vulnerable to concentrated structural disruptions, providing empirical benchmarks for fault tolerance in transformer architectures.

Figures & tables

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Lexical Perturbations Disrupt LLM Reasoning: An Empirical Study of Attention Diversion

    Aug 22, 2026Jiaqian Zhu, Yang Zhang, Junhua Ding +1Language Model RobustnessCausal Interventions in Language Models

  2. Pruning via Causal Attribution Preserves Reasoning Performance in Large Language Models

    Apr 27, 2026Amogh Sheth, Andrew Lin, Biruk Assefa +2LLM PruningSelf-Attention

  3. Attention-Aligned Reasoning for Large Language Models

    Oct 3, 2025Hongxiang Zhang, Yuan Tian, Tianyi ZhangLLM ReasoningStructured Reasoning in Language Models