cs.CLMar 10, 2026

Summarize Before You Speak with ARACH: A Training-Free Inference-Time Plug-In for Enhancing LLMs via Global Attention Reallocation

Authors: Jingtao Wang, Yucong Wang, Jun Ding, Rui Cai, Xun Wang

Organizations: Meakins-Christe Laboratories, Research Institute of McGill University Health Centre, 1001 Decarie Blvd, Montreal, H4A 3J1, Quebec, Canada · Department of Medicine, Division of Experimental Medicine, McGill University, 1001 Decarie Blvd, Montreal, H4A 3J1, Quebec, Canada · The College of Computer Science and Technology, Zhejiang Gongshang University, Hangzhou, China · Zhejiang Key Laboratory of Big Data and Future E-Commerce Technology, Hangzhou, China · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language models (LLMs) achieve remarkable performance, yet further gains often require costly training. This has motivated growing interest in post-training techniques-especially training-free approaches that improve models at inference time without updating weights. Most training-free methods treat the model as a black box and improve outputs via input/output-level interventions, such as prompt design and test-time scaling through repeated sampling, reranking/verification, or search. In contrast, they rarely offer a plug-and-play mechanism to intervene in a model's internal computation. We propose ARACH(Attention Reallocation via an Adaptive Context Hub), a training-free inference-time plug-in that augments LLMs with an adaptive context hub to aggregate context and reallocate attention. Extensive experiments across multiple language modeling tasks show consistent improvements with modest inference overhead and no parameter updates. Attention analyses further suggest that ARACH mitigates the attention sink phenomenon. These results indicate that engineering a model's internal computation offers a distinct inference-time strategy, fundamentally different from both prompt-based test-time methods and training-based post-training approaches.

Figures & tables

Figure 1: ARACH overview. (a) Summarize-then-generate intuition. At each decoding step, an index-aligned context hub stream {ci}\{c_{i}\} aggregates the causally available prefix and supports next-token prediction alongside the verbal token stream {xi}\{x_{i}\} . (b) Hub-based attention routing realized by a two-stream attention layout. Self-attention is partitioned into four blocks under hub-specific visibility constraints. A scalar logit offset bb is applied to regulate the strength of hub-mediated attention reallocation at inference time.
DatasetMetricBaselineb=−0.3b=-0.3b=−0.4b=-0.4b=−0.5b=-0.5
   +ARACHΔ\Delta+ARACHΔ\Delta+ARACHΔ\Delta
LAMBADAAcc (%) ↑\uparrow46.8949.93+3.0450.24+3.3550.42+3.53
StoryClozeAcc (%) ↑\uparrow57.4657.51+0.0557.56+0.1057.56+0.10
SQuADEM ↑\uparrow50.4150.90+0.4950.73+0.3250.54+0.13
SQuADF1 ↑\uparrow59.7160.48+0.7760.32+0.6160.18+0.47
WikiText-103PPL ↓\downarrow29.3729.17+0.2029.21+0.1629.29+0.08
Table 1: Paired evaluation of GPT-2 small with and without ARACH under identical settings (no weight updates). We report results for b∈{−0.3,−0.4,−0.5}b\in\{-0.3,-0.4,-0.5\} . We define Δ\Delta such that positive values indicate improvement.
Figure 2: Attention sink and hub-mediated reallocation analyses on PG-19, comparing the baseline and ARACH (default logit offset setting). (a) Layerwise sink score, defined as the mean attention mass assigned to the first verbal token, averaged over heads and samples in the test set. L∗L^{*} denotes the layer with the largest sink score under the baseline. (b,c) Heatmaps of mean attention weights among verbal tokens at layer L∗L^{*} for the baseline and ARACH . The red boxes highlight early verbal-token columns that are most associated with sink-like concentration in the baseline. For readability, only the first and last K=64K=64 tokens are shown. (d) Layerwise attention-mass decomposition, reporting the fraction of attention assigned to the first verbal token, to hub tokens, and to the remaining verbal tokens (averaged over samples in the test set and heads). (e) Routing summary at layer L∗L^{*} for ARACH, reporting the fraction of attention mass allocated to each attention block.
Figure 3: Sensitivity to the hub-attention logit offset bb across evaluations (sweep from 00 to −1.0-1.0 ). Curves show the task metric versus bb ; the dashed horizontal line is the baseline and the red dashed vertical line marks the default b=−0.5b=-0.5 . The blue area indicates values of bb for which all tasks improve over the baseline in our sweep. (a) LAMBADA (Accuracy ↑\uparrow ). (b) StoryCloze (Accuracy ↑\uparrow ). (c) SQuAD (F1 ↑\uparrow ). (d) WikiText-103 (Perplexity ↓\downarrow ). (e) PG-19 (Perplexity ↓\downarrow ). (f) SQuAD (Exact Match ↑\uparrow ).
VariantLAMBADA (Acc ↑\uparrow )SQuAD (EM/F1 ↑\uparrow )WT-103 (PPL ↓\downarrow )PG-19 (PPL ↓\downarrow )StoryCloze (Acc ↑\uparrow )
Baseline (no hub)46.8950.41/59.7129.3737.3357.45
Hub-only ( b=0b=0 )48.4650.26/59.6629.2137.2157.45
Full ARACH ( b=−0.5b=-0.5 )50.4250.54 / 60.1829.2933.1157.56
Δ\Delta Hub-only+1.57-0.15/-0.05+0.16+0.120.00
Δ\Delta Full+3.53+0.13/+0.47+0.08+4.21+0.11
Table 2: Ablation of the hub-attention offset term bb under paired evaluation. Hub-only sets b=0b=0 (hub stream retained), while Full uses the default b=−0.5b=-0.5 . We define Δ\Delta such that positive values indicate improvement.
More in this topic

LLM Inference

Large language model inference studies how models generate outputs efficiently, reliably, and securely across hardware and workloads.

See all 268 papers

Explore similar work

CardsList
Apr 7, 2026

In-Place Test-Time Training

The static train then deploy" paradigm fundamentally limits Large Language Models (LLMs) from dynamically adapting their weights in response to continuous streams of new information inherent in real-world tasks. Test-Time Training (TTT) offers a compelling alternative by updating a subset of model parameters (fast weights) at inference time, yet its potential in the current LLM ecosystem is hindered by critical barriers including architectural incompatibility, computational inefficiency and misaligned fast weight objectives for language modeling. In this work, we introduce In-Place Test-Time Training (In-Place TTT), a framework that seamlessly endows LLMs with Test-Time Training ability. In-Place TTT treats the final projection matrix of the ubiquitous MLP blocks as its adaptable fast weights, enabling a drop-in" enhancement for LLMs without costly retraining from scratch. Furthermore, we replace TTT's generic reconstruction objective with a tailored, theoretically-grounded objective explicitly aligned with the Next-Token-Prediction task governing autoregressive language modeling. This principled objective, combined with an efficient chunk-wise update mechanism, results in a highly scalable algorithm compatible with context parallelism. Extensive experiments validate our framework's effectiveness: as an in-place enhancement, it enables a 4B-parameter model to achieve superior performance on tasks with contexts up to 128k, and when pretrained from scratch, it consistently outperforms competitive TTT-related approaches. Ablation study results further provide deeper insights on our design choices. Collectively, our results establish In-Place TTT as a promising step towards a paradigm of continual learning in LLMs.

Guhao Feng, Shengjie Luo, Kai Hua +4

ByteDance Seed · Peking University

Test-Time TrainingContinual Learning for LLMsLong-Context Language Model Inference
AskPDF
Jul 26, 2026

Training Language Models to Cooperate with Inference-Time Controllers

Large language model (LLM) performance increasingly depends not only on the base model, but also on the inference-time controller used to organize reasoning. Existing post-training methods, however, typically optimize for a single fixed interaction pattern, despite real deployments relying on diverse controllers such as Chain-of-Thought, self-consistency, debate, planning, and verification pipelines. This creates a training--deployment mismatch and limits transfer to new workflows. We introduce CALM (Controller-Aware Language Models), a post-training framework that explicitly places controllers in the training loop. We formulate controller-aware post-training as multi-task reinforcement learning over controller-induced interaction protocols, where controllers are compositions of reusable local reasoning modules. This structure also induces a module-level decomposition of mixed-controller training under a turn-level GRPO objective, enabling a systematic study of controller and module-aware training strategies. We evaluate CALM on held-out controller compositions and broader controller shifts, showing that controller-aware post-training improves generalization across inference-time workflows beyond single-controller optimization.

Moumita Choudhury, Vanshaj Khattar, Jing Liu +4

College of Information and Computer Sciences, University of Massachusetts Amherst, Amherst, Massachusetts, USA · Mitsubishi Electric Research Laboratories, Cambridge, Massachusetts, USA

RL for Language Model ReasoningLLM Post-TrainingLanguage Model Post-Training
AskPDF
Mar 15, 2026

Nudging Hidden States: Training-Free Model Steering for Chain-of-Thought Reasoning in Large Audio-Language Models

Chain-of-thought (CoT) prompting has been extended to large audio-language models (LALMs) to elicit reasoning, yet enhancing its effectiveness without training remains challenging. We study inference-time model steering as a training-free approach to improve LALM reasoning. We introduce three strategies using diverse information sources and evaluate them across four LALMs and four benchmarks. Results show general accuracy gains up to 4.4% over CoT prompting. Notably, we identify a cross-modal transfer where steering vectors derived from few text samples effectively guide speech-based reasoning, demonstrating high data efficiency. We also examine hyperparameter sensitivity to understand the robustness of these approaches. Our findings position model steering as a practical direction for strengthening LALM reasoning.

Lok-Lam Ieong, Chia-Chien Chen, Chih-Kai Yang +3

National Taiwan University, Taiwan

Language Model SteeringAudio-Language Models
AskPDF