Summarize Before You Speak with ARACH: A Training-Free Inference-Time Plug-In for Enhancing LLMs via Global Attention Reallocation

Jingtao Wang; Yucong Wang; Jun Ding; Rui Cai; Xun Wang

cs.CLMar 10, 2026

Summarize Before You Speak with ARACH: A Training-Free Inference-Time Plug-In for Enhancing LLMs via Global Attention Reallocation

Authors: Jingtao Wang, Yucong Wang, Jun Ding, Rui Cai, Xun Wang

Organizations: Meakins-Christe Laboratories, Research Institute of McGill University Health Centre, 1001 Decarie Blvd, Montreal, H4A 3J1, Quebec, Canada · Department of Medicine, Division of Experimental Medicine, McGill University, 1001 Decarie Blvd, Montreal, H4A 3J1, Quebec, Canada · The College of Computer Science and Technology, Zhejiang Gongshang University, Hangzhou, China · Zhejiang Key Laboratory of Big Data and Future E-Commerce Technology, Hangzhou, China ·

Abstract

Large language models (LLMs) achieve remarkable performance, yet further gains often require costly training. This has motivated growing interest in post-training techniques-especially training-free approaches that improve models at inference time without updating weights. Most training-free methods treat the model as a black box and improve outputs via input/output-level interventions, such as prompt design and test-time scaling through repeated sampling, reranking/verification, or search. In contrast, they rarely offer a plug-and-play mechanism to intervene in a model's internal computation. We propose ARACH(Attention Reallocation via an Adaptive Context Hub), a training-free inference-time plug-in that augments LLMs with an adaptive context hub to aggregate context and reallocate attention. Extensive experiments across multiple language modeling tasks show consistent improvements with modest inference overhead and no parameter updates. Attention analyses further suggest that ARACH mitigates the attention sink phenomenon. These results indicate that engineering a model's internal computation offers a distinct inference-time strategy, fundamentally different from both prompt-based test-time methods and training-based post-training approaches.

Figures & tables

Figure 1: ARACH overview. (a) Summarize-then-generate intuition. At each decoding step, an index-aligned context hub stream $\{c_{i}\}$ aggregates the causally available prefix and supports next-token prediction alongside the verbal token stream $\{x_{i}\}$ . (b) Hub-based attention routing realized by a two-stream attention layout. Self-attention is partitioned into four blocks under hub-specific visibility constraints. A scalar logit offset $b$ is applied to regulate the strength of hub-mediated attention reallocation at inference time.

Dataset	Metric	Baseline	$b=-0.3$		$b=-0.4$		$b=-0.5$
			+ARACH	$\Delta$	+ARACH	$\Delta$	+ARACH	$\Delta$
LAMBADA	Acc (%) $\uparrow$	46.89	49.93	+3.04	50.24	+3.35	50.42	+3.53
StoryCloze	Acc (%) $\uparrow$	57.46	57.51	+0.05	57.56	+0.10	57.56	+0.10
SQuAD	EM $\uparrow$	50.41	50.90	+0.49	50.73	+0.32	50.54	+0.13
SQuAD	F1 $\uparrow$	59.71	60.48	+0.77	60.32	+0.61	60.18	+0.47
WikiText-103	PPL $\downarrow$	29.37	29.17	+0.20	29.21	+0.16	29.29	+0.08

Table 1: Paired evaluation of GPT-2 small with and without ARACH under identical settings (no weight updates). We report results for

b\in\{-0.3,-0.4,-0.5\}

. We define

\Delta

such that positive values indicate improvement.

Figure 2: Attention sink and hub-mediated reallocation analyses on PG-19, comparing the baseline and ARACH (default logit offset setting). (a) Layerwise sink score, defined as the mean attention mass assigned to the first verbal token, averaged over heads and samples in the test set. $L^{*}$ denotes the layer with the largest sink score under the baseline. (b,c) Heatmaps of mean attention weights among verbal tokens at layer $L^{*}$ for the baseline and ARACH . The red boxes highlight early verbal-token columns that are most associated with sink-like concentration in the baseline. For readability, only the first and last $K=64$ tokens are shown. (d) Layerwise attention-mass decomposition, reporting the fraction of attention assigned to the first verbal token, to hub tokens, and to the remaining verbal tokens (averaged over samples in the test set and heads). (e) Routing summary at layer $L^{*}$ for ARACH, reporting the fraction of attention mass allocated to each attention block.

Figure 3: Sensitivity to the hub-attention logit offset

b

across evaluations (sweep from

0

-1.0

). Curves show the task metric versus

b

; the dashed horizontal line is the baseline and the red dashed vertical line marks the default

b=-0.5

. The blue area indicates values of

b

for which all tasks improve over the baseline in our sweep. (a) LAMBADA (Accuracy

\uparrow

). (b) StoryCloze (Accuracy

\uparrow

). (c) SQuAD (F1

\uparrow

). (d) WikiText-103 (Perplexity

\downarrow

). (e) PG-19 (Perplexity

\downarrow

). (f) SQuAD (Exact Match

\uparrow

Variant	LAMBADA (Acc $\uparrow$ )	SQuAD (EM/F1 $\uparrow$ )	WT-103 (PPL $\downarrow$ )	PG-19 (PPL $\downarrow$ )	StoryCloze (Acc $\uparrow$ )
Baseline (no hub)	46.89	50.41/59.71	29.37	37.33	57.45
Hub-only ( $b=0$ )	48.46	50.26/59.66	29.21	37.21	57.45
Full ARACH ( $b=-0.5$ )	50.42	50.54 / 60.18	29.29	33.11	57.56
$\Delta$ Hub-only	+1.57	-0.15/-0.05	+0.16	+0.12	0.00
$\Delta$ Full	+3.53	+0.13/+0.47	+0.08	+4.21	+0.11

Table 2: Ablation of the hub-attention offset term

b

under paired evaluation. Hub-only sets

b=0

(hub stream retained), while Full uses the default

b=-0.5

. We define

\Delta

such that positive values indicate improvement.

LLM Inference

Large language model inference studies how models generate outputs efficiently, reliably, and securely across hardware and workloads.

See all 267 papers

In-Place Test-Time Training
Apr 7, 2026Guhao Feng, Shengjie Luo, Kai Hua +4Test-Time Training Continual Learning for LLMs
ByteDance Seed · Peking University
Training Language Models to Cooperate with Inference-Time Controllers
Jul 26, 2026Moumita Choudhury, Vanshaj Khattar, Jing Liu +4RL for Language Model Reasoning LLM Post-Training
College of Information and Computer Sciences, University of Massachusetts Amherst, Amherst, Massachusetts, USA · Mitsubishi Electric Research Laboratories, Cambridge, Massachusetts, USA
Nudging Hidden States: Training-Free Model Steering for Chain-of-Thought Reasoning in Large Audio-Language Models
Mar 15, 2026Lok-Lam Ieong, Chia-Chien Chen, Chih-Kai Yang +3Language Model Steering Audio-Language Models
National Taiwan University, Taiwan

Apr 7, 2026J/K move · Enter open · S save

In-Place Test-Time Training

Guhao Feng, Shengjie Luo, Kai Hua, Ge Zhang+3

ByteDance Seed · Peking University

Test-Time Training Continual Learning for LLMs Long-Context Language Model Inference

Open paper AskPDF arXiv

The static

train then deploy" paradigm fundamentally limits Large Language Models (LLMs) from dynamically adapting their weights in response to continuous streams of new information inherent in real-world tasks. Test-Time Training (TTT) offers a compelling alternative by updating a subset of model parameters (fast weights) at inference time, yet its potential in the current LLM ecosystem is hindered by critical barriers including architectural incompatibility, computational inefficiency and misaligned fast weight objectives for language modeling. In this work, we introduce In-Place Test-Time Training (In-Place TTT), a framework that seamlessly endows LLMs with Test-Time Training ability. In-Place TTT treats the final projection matrix of the ubiquitous MLP blocks as its adaptable fast weights, enabling a

drop-in" enhancement for LLMs without costly retraining from scratch. Furthermore, we replace TTT's generic reconstruction objective with a tailored, theoretically-grounded objective explicitly aligned with the Next-Token-Prediction task governing autoregressive language modeling. This principled objective, combined with an efficient chunk-wise update mechanism, results in a highly scalable algorithm compatible with context parallelism. Extensive experiments validate our framework's effectiveness: as an in-place enhancement, it enables a 4B-parameter model to achieve superior performance on tasks with contexts up to 128k, and when pretrained from scratch, it consistently outperforms competitive TTT-related approaches. Ablation study results further provide deeper insights on our design choices. Collectively, our results establish In-Place TTT as a promising step towards a paradigm of continual learning in LLMs.

Abstract

LLM Inference

Explore similar work

In-Place Test-Time Training

Training Language Models to Cooperate with Inference-Time Controllers

Nudging Hidden States: Training-Free Model Steering for Chain-of-Thought Reasoning in Large Audio-Language Models