When Should an Agent Refresh? Adaptive Index Reuse for Efficient Long-Horizon Reasoning

Guangyuan Wang; Yifan Zhang; Shilong Liu

cs.LGNov 1, 2026

When Should an Agent Refresh? Adaptive Index Reuse for Efficient Long-Horizon Reasoning

Authors: Guangyuan Wang, Yifan Zhang, Shilong Liu

Organizations: Mila – Québec Artificial Intelligence Institute · McGill University · Princeton University · Columbia University

Abstract

Language agents read code, tool output, and dialogue in contexts that grow at every step. Dynamic sparse attention lets a small indexer choose the earlier tokens each query attends to, but the indexer still scores the whole prefix at every layer. Cross-layer reuse methods such as IndexCache rerun the indexer at only some layers, with one fixed pattern for every input. In an agent, the relevant evidence can change abruptly when a traceback or test result arrives, so fresh selections matter more at some steps than at others. We propose Adaptive Index Reuse (AIR), which keeps IndexCache's execution and learns, for each segment of an agent episode, how many layers refresh their selection. A small controller reads signals from completed computation and the latest tool event, chooses a pattern from a nested library, and learns from terminal task reward under an expected compute budget. On Qwen3-4B-Instruct-2507 with a 128K context, AIR improves success over the strongest static schedule at matched cost on SWE-bench Verified (27.0% to 32.4%), Terminal-Bench 2.0, and three $τ^2$ -bench domains, and it lowers time per successful task on all five. Across 50 backbone, context, and benchmark settings, it matches or exceeds that schedule in 44.

Figures & tables

Panel of Figure 1 — Figure 1: AIR ’s decision loop. Before segment $u$ computes, the controller maps state from finished work to a budget; the segment runs under that budget’s nested mask. Full layers select position IDs; Shared layers reuse them. Every layer keeps its own KV cache, and the next observation feeds decision $u{+}1$ .

	Success (%) $\uparrow$			Min/success $\downarrow$
Benchmark	Static	AIR	$\Delta$	Static	AIR
SWE-bench Verified	27.0	32.4	$+5.4$	64.6	52.7
Terminal-Bench 2.0	22.5	24.7	$+2.2$	61.9	56.4
$\tau^{2}$ retail	53.3	57.2	$+3.9$	14.1	13.1
$\tau^{2}$ airline	42.5	46.5	$+4.0$	14.1	12.9
$\tau^{2}$ telecom	42.5	46.5	$+3.9$	19.5	17.9

Table 1: AIR raises success and lowers time per successful task on every benchmark. Qwen3-4B-Instruct-2507, 128K cap; Static is the strongest feasible static schedule (

R=18

) at matched expected cost.

Panel of Figure 2 — Figure 2: Offline preparation and training. Each stage uses its own owner-disjoint fitting partition and is frozen before the next; validation selects a learned controller or a static fallback before any final run.

References
	SWE-bench V.		Terminal-Bench		$\tau^{2}$ retail		$\tau^{2}$ airline		$\tau^{2}$ telecom
Method	Succ. $\uparrow$	Min/s. $\downarrow$	Succ. $\uparrow$	Min/s. $\downarrow$	Succ. $\uparrow$	Min/s. $\downarrow$	Succ. $\uparrow$	Min/s. $\downarrow$	Succ. $\uparrow$	Min/s. $\downarrow$
Native attention	32.0	71.5	28.1	67.3	57.7	20.8	50.0	19.1	46.5	27.8
All-refresh	31.6	56.8	27.0	54.1	57.0	14.6	48.5	13.8	45.4	20.2
Static allocation
Static ( $R=18$ )	27.0	64.6	22.5	61.9	53.3	14.1	42.5	14.1	42.5	19.5

Table 2: AIR -based methods have the highest success on two benchmarks and the lowest time per successful task on three, and AIR improves over the matched static schedule on all five. Qwen3-4B-Instruct-2507, 128K; bold: best frozen-backbone value (ties included); tint: AIR -based; gray: native attention, unranked.

Figure 3: AIR on SWE-bench Verified and in training (Qwen3-4B-Instruct-2507, 128K). (a) Success against minutes per attempt for 24 selector settings. (b) Validation success against normalized cost for three seeds; dashed: the 0.50 budget. (c) All-refresh time divided by each method’s time, per attempt. (d) Success on long-history continuations at prefix occupancy

\varphi

Static and state-free
Variant	SWE	Term.	Retail	Airl.	Tel.
Greedy mask, $R=18$	27.0	22.5	53.3	42.5	42.5
Staggered mask, $R=18$	24.0	21.3	50.4	40.5	40.1
State-free mixture	27.0	22.5	52.9	42.5	42.5
Rules and local learning
Event trigger	29.4	23.6	53.3	48.0	41.7

Table 3: Among the variants shown, AIR leads on SWE-bench Verified, retail, and telecom, ties the best Terminal-Bench 2.0 score, and the supervised router leads on airline. Success (%), Qwen3-4B-Instruct-2507 at 128K,

\beta=0.50

; bold: best per column, ties included.

Figure 4: Decision-interval and warm-start ablations on SWE-bench Verified (Qwen3-4B-Instruct-2507, 128K). Points are mean success over the 96 factorial configurations; gray marks show min–max ranges in (a) and the 48 matched pairs in (b).

Figure 5: AIR ’s success minus the analysis comparator’s (percentage points) for every backbone, cap, and benchmark; blue favors AIR . Gemma-3-4B-it at 32K deploys the static fallback $R=2$ .

Figure 6: Tools and the environment bound the whole-agent speedup, which grows with the prefix (Qwen3-4B-Instruct-2507, 128K). (a) Time per attempt relative to all-refresh; (b) AIR ’s speedup against prefix length.

Appendix figures & tables74 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Dynamic sparse attention at a Full layer: (a) the indexer scores the prefix and keeps the top- $k$ positions plus sink and local positions; (b) core attention gathers the layer’s own keys and values there.

Axis	IndexCache	AIR
Layer action	Full or Shared	Full or Shared
Patterns	One deployed pattern	One pattern per budget, nested
Decision state	None (global)	Current agent segment
Pattern selection	Calibration LM loss	Calibration LM loss
Controller training	None	NLL-regret warm start, then terminal reward and cost
Compute target	Fixed retention ratio	Expected trajectory budget

Appendix

Table 4: What AIR shares with IndexCache and what it adds. Both methods execute the same Full and Shared layer roles.

Panel of Figure 8 — Figure 8: IndexCache execution inherited by AIR : Full layers overwrite the position buffer $\mathcal{T}_{\mathrm{cache}}$ , Shared layers reuse it, and AIR appends index keys at every eligible layer so any of them can later become Full.

Panel of Figure 9 — Figure 9: Offline library construction: one greedy elimination path from all-refresh to aggressive reuse supplies a nested mask for each refresh count in the menu.

Panel of Figure 10 — Figure 10: Three execution timescales: one library mask per segment of the agent loop, cache appends per query within a segment, and the $\mathcal{T}_{\mathrm{cache}}$ buffer per query across depth.

Symbol	Meaning
$N$	Number of decoder layers ( $N=36$ for the primary backbone)
$\mathcal{N}_{\mathrm{idx}}$ , $N_{\mathrm{idx}}$ , $\ell_{1}$	Ordered set of eligible layers, its size, and its first layer, which is always Full
$L$	Sequence length
$\ell,\,j,\,u,\,t,\,p$	Indices for layers, agent steps, controller decisions, query positions, and key positions
$k$ , $k_{t}$	Selected positions per query, and $k_{t}=\min\{k,\|\mathcal{E}_{t}\|\}$
$\mathcal{S}$ , $\mathcal{S}_{t}$	Fixed sink positions, and those available at query $t$ : $\mathcal{S}_{t}=\mathcal{S}\cap\{1,\ldots,t\}$

Appendix

Table 5: Core notation for agentic adaptive index reuse.

Panel of Figure 11 — Figure 11: One AIR decision and one query on the Gemma-3-4B-it layout: (A) the mask is fixed from completed history; (B) Full layers write position IDs that Shared layers read, while sliding-window layers bypass the buffer; (C) IDs are released per query and caches persist.

Block	Example fields	Requirement and cost treatment
Trajectory position	Segment index, context length, elapsed tool steps	Read from history at the boundary and normalize with training-only statistics
Interface event	Traceback, test result, file read, edit, API reply	Derive from the latest completed tool event, with no verifier or future outcome
Budget history	Previous budget, cumulative normalized cost, target gap	Compute only from executed actions and the frozen cost model
Cache state	Previous mask, distance to last Full layer, completed index age	Record source token, layer, timestamp, cache version, and action-history dependence
Lagged uncertainty	Completed-segment entropy and logit margin	Read retained logits from segment $u-1$ , with a missing flag at $u=0$
Lagged index state	Completed-query overlap, score entropy, selection margin	Reuse statistics already produced in segment $u-1$ , with no new prefix scoring

Appendix

Table 6: Controller observation groups and the timing and cost requirements each meets. Every value exists before the segment’s budget is chosen.

Panel of Figure 12 — Figure 12: Offline structural-change diagnostic: selected positions of matched queries before and after event $j$ are compared on their common prefix by Jaccard distance.

Term	Floating-point operations	Bytes
$F_{\mathrm{key}}(t,\ell)$	$2dH_{\mathrm{idx}}d_{\mathrm{idx}}$ for key projections, plus input normalization	$H_{\mathrm{idx}}d_{\mathrm{idx}}b_{\mathrm{idx}}$ written, plus amortized weight reads
$F_{\mathrm{sel}}(t,\ell)$	$2dH_{\mathrm{idx}}(d_{\mathrm{idx}}+1)$ for query and gate projections; $2H_{\mathrm{idx}}d_{\mathrm{idx}}E_{t}$ for dot products; $3H_{\mathrm{idx}}E_{t}$ for scaling, ReLU, and gating; $E_{t}$ for top- $k$ selection	$H_{\mathrm{idx}}d_{\mathrm{idx}}b_{\mathrm{idx}}E_{t}$ of index keys read, plus amortized weight reads
$F_{\mathrm{w}}(t,\ell)$ , $F_{\mathrm{r}}(t,\ell)$	None	$4k_{t}$ written at a Full layer or read at a Shared layer
$F_{\mathrm{sw}}(b_{u-1},b_{u})$ , per segment	None	$\lceil N/8\rceil$ for the new mask when $b_{u}\neq b_{u-1}$
$F_{\phi}(\mathcal{H}_{u})$ , per segment	Entropy and margin over the retained final-token logits; PCA projections; overlap statistics	Logits, stored position sets, and projection weights read
$F_{\mathrm{probe}}(\mathcal{H}_{u})$ , per segment (paid-probe comparison only)	For the appended tokens: embeddings and native layers below $\ell_{1}$ (standard per-token counts); $F_{\mathrm{key}}$ at $\ell_{1}$ ; $F_{\mathrm{sel}}$ at $\ell_{1}$ ; entropy, margin, and overlap statistics	Weight and embedding reads; temporary writes and reads of layer inputs and index keys; prefix index keys of $\ell_{1}$ read; previous selection read

Appendix

Table 7: Cost breakdown by operation, per query position

t

and layer

\ell

unless stated otherwise.

Panel of Figure 13 — Figure 13: Primal-dual training: each batch of episodes gives one actor step and several critic steps, the price moves on a slower timescale, and validation bounds select the deployed policy.

Checkpoint (role)	Params (total/ active)	Attention	$N$ / $N_{\mathrm{idx}}$	Menu $\mathcal{R}_{m}$ (fractions)	Supported context	Context caps	Reasoning mode	Precision
Qwen3-4B-Instruct-2507 (primary)	4.0B	GQA, 32 Q / 8 KV heads	36 / 36	3, 5, 9, 18, 36 (.08, .14, .25, .50, 1)	262,144 native	32K, 128K	non-thinking only	BF16
Qwen3-1.7B (smaller)	1.7B	GQA, 16 Q / 8 KV	28 / 28	3, 4, 7, 14, 28 (.11, .14, .25, .50, 1)	32,768	32K	hybrid; thinking off	BF16
Qwen3-8B (larger)	8.2B	GQA, 32 Q / 8 KV	36 / 36	3, 5, 9, 18, 36 (.08, .14, .25, .50, 1)	32,768; 131,072 with YaRN	32K; 128K with YaRN $\times 4$	hybrid; thinking off	BF16
Gemma-3-4B-it (second family)	4B	5 local (1,024) : 1 global; 8 Q / 4 KV	34 / 5	1, 2, 3, 5 (.20, .40, .60, 1)	128K (131,072)	32K, 128K	none	BF16
MiniMax-M2 (MoE extension)	230B / 10B	GQA, 48 Q / 8 KV; 256 experts, 8 active	62 / 62	6, 9, 16, 31, 62 (.10, .15, .26, .50, 1)	196,608	32K, 128K	interleaved thinking, retained	FP8 weights
Qwen3-4B (release control)	4.0B	GQA, 32 Q / 8 KV	36 / 36	as primary	32,768; 131,072 with YaRN	32K	hybrid; thinking off	BF16

Appendix

Table 8: Backbones. “Supported context” follows each model card; caps, menus, and settings are our choices (Hugging Face repositories pinned to fixed revisions).

N

: decoder layers;

N_{\mathrm{idx}}

: eligible layers; menus follow Equation 9 .

Population	Tasks (published / pinned)	Release, harness, and scorer	Endpoint	Role
SWE-bench Verified	500 / 500	Pinned dataset revision; official swebench harness (version and image digests fixed)	Official resolved rate: all FAIL_TO_PASS and PASS_TO_PASS tests pass	Primary claim; repair controller
Terminal-Bench 2.0	89 / 89	Harbor, dataset [email protected]	Official task-verifier success	Transfer
$\tau^{2}$ retail	115 / 114	tau2-bench v0.2.0, base tasks	pass@1 and pass 4 over 4 trials	Transfer
$\tau^{2}$ airline	50 / 50	tau2-bench v0.2.0, base tasks	pass@1 and pass 4 over 4 trials	Transfer
$\tau^{2}$ telecom	114 / 114	tau2-bench v0.2.0, base tasks, default dual control	pass@1 and pass 4 over 4 trials	Transfer

Appendix

Table 9: Final evaluation populations. Counts are the published size and the size of the pinned task list; the pinned count is authoritative for analysis. Releases and scoring definitions are fixed.

Panel of Figure 14 — Figure 14: Experimental pipeline per backbone: separate fitting partitions train the indexer, calibrate the masks, and fit and select the controller, each frozen before the next stage.

Track	Methods	Extension rule	What must be controlled
Shared indexer	Anchor, static, periodic, staggered; mixture, temporal, event, staleness, supervised, bandit, AIR	Every accepted backbone	Frozen backbone and indexer; paired budget and state controls
Alternative indexer	Training-aware IndexCache; LatentIndex; MISA; HISA; LongCat CLI / HI	Every compatible injected indexer	Separate calibration; key, pooling, routing and training costs
Temporal adaptation	ReTopK with injected indexer	Every compatible injected indexer	Exact prefill; decode reuse; cache invalidation on prefix rebuild
Native-score methods	Published ReTopK; HShare; SFI; TidalDecode; Quest; Kascade	Qwen family; other families after adapter check	Different selector; own scores, caches and complete runtime charged
AIR composition	AIR with HISA, MISA-4 or LongCat HI	Where both components qualify	Own all-refresh reference, static library and fitted controller
Retrained backbone	Full LongCat Sparse Attention	Primary model only	Continual training and changed support; upfront cost disclosed

Appendix

Table 10: Comparison tracks: each answers its own question and holds different factors fixed; read numerical rows within their track.

Backbone	Cap	$b_{0}$	Dev. %	Dev. cost	Deployment	Final R values
Qwen3-1.7B	32K	R=14	9.4	0.485	Learned	14
Qwen3-4B-Instruct-2507	32K	R=18	23.0	0.485	Learned	18, 36
Qwen3-4B-Instruct-2507	128K	R=18	23.0	0.485	Learned	18
Qwen3-8B	32K	R=18	30.6	0.485	Learned	18
Qwen3-8B	128K	R=18	30.6	0.485	Learned	18
Gemma-3-4B-it	32K	R=2	12.2	0.388	Static R=2	2, 3

Appendix

Table 11: The three static selections come from different data: the development reference

b_{0}

(static-selection partition), the deployed policy (validation), and the analysis comparator (each final population; last column).

Partition	Target	Tasks	Repos	Owners	Tasks	Share	Tasks	Share
		All tasks			SWE-Gym		SWE-smith
Indexer training	20%	7,592	24	23	484	19.9%	7,108	20.0%
Schedule calibration	10%	3,788	13	12	228	9.4%	3,560	10.0%
Static selection	10%	3,801	12	12	251	10.3%	3,550	10.0%
Controller training	45%	17,087	52	51	1,101	45.2%	15,986	45.0%
Controller validation	15%	5,700	21	17	374	15.3%	5,326	15.0%

Appendix

Table 12: Repair-pool partitions by owner group (seed 20261007). Counts are eligible tasks. Shares are of the whole pool and of each source.

Comparison	Held fixed	Question
Native vs. all-refresh sparse	Backbone, prompts, tools, limits	Does the injected sparse attention preserve agent capability?
All-refresh vs. static IndexCache	Backbone, indexer, $k$ , runtime	How much indexer work does static reuse remove, and at what quality?
Dynamic-ready vs. specialized static runtime	Masks and model outputs	What does keeping all index keys cost?
Anchor-only, periodic, staggered vs. greedy static	Refresh count	Does layer placement matter within a budget?
Static vs. mixture and temporal schedule	Action frequencies or time sequence	Does varying the budget without state help?
Heuristics vs. supervised router	State, menu	Do simple rules capture the allocation signal?

Appendix

Table 13: Comparisons and the question each isolates. Rows apply to every evaluated backbone and population unless stated otherwise.

Method	What changes	Fixed	Calibration or training	Question
Native backbone	Unmodified attention, no indexer	Harness, decoding	None	Does the injected sparse attention preserve agent success?
All-refresh sparse	Every eligible layer Full	Indexer, $k$	None	Upper end of refresh cost
Anchor-only ( $R=1$ )	Only $\ell_{1}$ Full	Indexer, $k$	None	Is any depth refresh needed?
Periodic depth mask	Full layers at a fixed interval, same $R_{b}$	Refresh count	None	Does greedy placement matter?
Staggered depth mask	Full layers at seeded random positions, same $R_{b}$	Refresh count	Preregistered mask seeds	Is the greedy pattern better than chance?
Static IndexCache	One greedy library pattern for the episode	Library	Greedy calibration search	Static reuse at each budget

Appendix

Table 14: Native reference and refresh controls on the shared indexer. “Fixed” lists what is held constant beyond the shared setup described in the text.

Method	What changes	Fixed	Calibration or training	Question
Phase / model / index / model+index / full state	Controller inputs	Training recipe	As AIR	Which signals carry allocation value?
Decision interval	$Q$ and model-call-only decisions	Menu	As AIR	Responsiveness vs. overhead
Library construction	Uniform, staggered, or independently searched patterns	Refresh counts	As AIR	Outer policy vs. inner schedule
Budget sweep	Ratio $\beta$	Everything else	As AIR	Shape of the trade-off
Overhead	Feature blocks added one at a time	Everything else	As AIR	Does added overhead remove the saving?

Appendix

Table 15: Controller and library ablations. Each varies one factor of AIR and keeps the rest of its recipe.

Method	Source	Code we start from	Reproduction
Training-aware IndexCache	arXiv 2603.12201 v1	THUDM/IndexCache 08d22d69 (DSA models only)	Ours, on injected indexer
LatentIndex	arXiv 2610.04635 v1	None in the linked TransArch repository	Ours; targets are our indexer’s keys
MISA (head routing)	arXiv 2605.07363 v1	TransArch, directory MISA_preprint (decode router differs from paper)	Ours, paper’s router
HISA	arXiv 2603.28458 v4	TileLang example dsa_hisa ; SGLang branch hisa_pr (DSA models)	Ours, on injected indexer
ReTopK	arXiv 2607.27692 v1	No public code	Published per-head version and indexer-based adaptation, both ours
LongCat cross-layer indexing	arXiv 2608.01662 v2	Inference reuse in SGLang main 9ddbba5 ; distillation code unpublished	Adapter to our indexer and distillation, ours

Appendix

Table 16: Alternative-method documentation. “Ours” marks a reimplementation or adaptation. Table 17 continues with calibration, persistent state, and counted cost.

Method	Calibration or training	Persistent state and invalidation	Extra cost counted
Training-aware IndexCache	Multi-layer distillation per pattern on indexer-training data	One indexer bank per pattern; caches as flat indexer	Adaptation compute; bank memory
LatentIndex	Closed-form reduced-rank ridge fit on fitting data	One latent per token per layer group; append-only	Latent projections, latent cache, per-layer selection, candidate rescoring (hierarchical)
MISA (head routing)	None; active heads $\{1,2,4,8\}$ preregistered	Block-pooled keys, appended per block	Router scoring and pooled-key maintenance
HISA	None	Block-pooled keys, appended per block	Block scoring, pooled-key maintenance
ReTopK	None; defaults fixed in advance	Per-head FIFO of queries and supports; cleared on prefix change	Similarity lookup, cache storage, candidate scoring, fallbacks, refreshes
LongCat cross-layer indexing	Cross-layer distillation of the indexer	As flat indexer	Distillation compute

Appendix

Table 17: Alternative-method documentation, continued from Table 16 : calibration or training, persistent state and its invalidation, and the extra cost each method is charged.

Population	Tasks	Comparator	$\Delta_{V}$ (pp)	$\Delta_{\overline{C}}$
SWE-bench Verified	500	$R=18$	$+5.4$	$+0.022$
Terminal-Bench 2.0	89	$R=18$	$+2.2$	$+0.020$
$\tau^{2}$ retail	114	$R=18$	$+3.9$	$+0.017$
$\tau^{2}$ airline	50	$R=18$	$+4.0$	$+0.013$
$\tau^{2}$ telecom	114	$R=18$	$+3.9$	$+0.017$

Appendix

Table 18: AIR ’s success gain over the analysis comparator is positive on every population at a cost difference of at most 0.022.

\Delta_{V}

: success difference (pp);

\Delta_{\overline{C}}

: normalized cost difference; differences are calculated before rounding.

Backbone	Cap	Deployed type	Candidate	Success LB %	Cost UB
Qwen3-1.7B	32K	Learned	s47_n12288	6.2	0.477
Qwen3-4B-Instruct-2507	32K	Learned	s11_n12288	22.0	0.481
Qwen3-4B-Instruct-2507	128K	Learned	s47_n12288	22.6	0.473
Qwen3-8B	32K	Learned	s11_n12288	27.2	0.476
Qwen3-8B	128K	Learned	s11_n6144	29.0	0.488
Gemma-3-4B-it	32K	Static fallback	static_r2	5.7	0.412

Appendix

Table 19: Validation deploys a learned controller in nine of ten configurations and the static fallback

R=2

(shaded) for Gemma-3-4B-it at 32K. Feasible: cost upper bound at most 0.53; candidate names give seed and rollout count.

Figure 15: Paired by task and trial, AIR solves attempts that the strongest static schedule fails and spends a smaller share of its time on failed attempts. Top: outcome shares against each comparator; bottom: share of time spent on failed attempts.

Figure 16: At similar time per attempt, AIR reaches higher success than static schedules on all five benchmarks (Qwen3-4B-Instruct-2507, 128K). Blue: static

R\in\{3,\dots,36\}

, comparator open; red: AIR at

\beta\in\{0.35,0.50,0.70\}

; dashed: all-refresh; arrows: AIR minus the static comparator at the deployed budget (pp).

References
	Minutes per attempt (speedup over all-refresh, $\times$ )
Method	SWE	Terminal	Retail	Airline	Telecom
Native attention	22.9 (0.78)	18.9 (0.77)	12.0 (0.70)	9.6 (0.70)	12.9 (0.71)
All-refresh sparse	17.9 (1.00)	14.6 (1.00)	8.3 (1.00)	6.7 (1.00)	9.2 (1.00)
Strongest feasible static	17.4 (1.03)	13.9 (1.05)	7.5 (1.11)	6.0 (1.11)	8.3 (1.10)
Alternative injected selectors

Appendix

Table 20: Mean minutes per attempt (speedup over all-refresh) for every method; bold marks the fastest per column. AIR is 4 to 5% faster than all-refresh on SWE-bench Verified and Terminal-Bench 2.0 and 9 to 10% faster on

\tau^{2}

Figure 17: Few settings are both faster than all-refresh and at least as successful; on SWE-bench Verified only AIR and LatentIndex are. Left: success change (pp); right: speedup over all-refresh.

Figure 18: AIR ’s speedup over all-refresh changes little with episode length: across five workload strata it stays at 1.03 to 1.05 on SWE-bench Verified and Terminal-Bench 2.0 and 1.09 to 1.12 on

\tau^{2}

. Ribbon: interquartile range of per-attempt ratios.

Backbone	Caps	Speedup	Success change (pp)
Qwen3-4B-Instruct-2507	32K, 128K	0.99–1.11	$-2.2$ to $+1.1$
Qwen3-8B	32K, 128K ‡	1.00–1.13	$-1.5$ to $+0.9$
Gemma-3-4B-it	32K † , 128K	0.96–1.03	$-5.0$ to $-1.3$
MiniMax-M2	32K, 128K	1.01–1.12	$-2.6$ to $+1.1$
Qwen3-4B (original)	32K	0.99–1.08	$-2.0$ to $+0.2$

Appendix

Table 21: AIR relative to all-refresh on each backbone: ranges of whole-agent speedup and success change over all evaluated caps and benchmarks. † static fallback

R=2

; ‡ YaRN extension.

Figure 19: At 32K, AIR matches or exceeds the comparator in successful tasks per hour in 21 of 25 cells, by up to 23%. Left: success difference (pp); right: ratio of successful tasks per hour.

Figure 20: AIR ’s whole-agent speedup on SWE-bench Verified is 0.96 to 1.05, smaller than its prefill and decode gains because tools and the environment take about half of each episode.

Figure 21: On Qwen3-8B at 32K, AIR lies on the success-yield boundary (dotted: settings no other displayed setting beats on both axes) in two of the five benchmarks. Blue: static menu.

Placement and state-free variation
Method	SWE	Terminal	Retail	Airline	Telecom	Speedup
All-refresh reference	31.6	27.0	57.0	48.5	45.4	1.00
Strongest feasible static ( $R=18$ )	27.0	22.5	53.3	42.5	42.5	1.03
Staggered mask, $R=18$	24.0	21.3	50.4	40.5	40.1	1.00
State-independent mixture	27.0	22.5	52.9	42.5	42.5	1.03
Rules and local learning

Appendix

Table 22: Among the

\beta=0.50

variants, AIR has the highest success on SWE-bench Verified, retail, and telecom and ties the best Terminal-Bench 2.0 score; on airline five locally trained or restricted-state variants score higher, the supervised router most. The larger budget

\beta=0.70

is a separate experiment and scores higher on four populations. Success (%); last column: SWE-bench Verified speedup over all-refresh.

Figure 22: Every state ablation and both locally trained alternatives lower SWE-bench Verified success relative to AIR , by 1.0 to 4.2 points; on airline five of the six score higher. Left: success difference (pp); right: time change (%), positive is slower.

Figure 23: The full controller state gives the highest success on all five populations (tied on Terminal-Bench 2.0 and airline with model plus index state); removing all three groups costs 2.2 to 4.5 points. Rows: the other seven combinations of event ( $E$ ), model ( $M$ ), and index ( $I$ ) state.

Figure 24: Budget, decision interval, and library each move SWE-bench Verified success by up to 4 points, and the warm start adds 1.4 points on average. Dots: mean over the 96 controllers at each level; bars: range;

Q

in tokens (call: call-only); libraries G greedy, U uniform, S staggered, I independent.

Figure 25: The paid probe raises or matches success at every decision interval, by up to 2.3 points, and adds 1 to 5 seconds per attempt. Curves join

Q=128

, 512, 2048, and call-only decisions with the probe off (red) or on (cyan); open markers fail the cost check.

Backbone	Cap	Success change (pp)	Added cost	Time change (%)
Qwen3-1.7B	32K	0.0 to +2.0	0.019 to 0.031	0.0 to +0.4
Qwen3-4B-Instruct-2507	32K	+0.4 to +2.0	0.013 to 0.024	0.0 to +0.4
Qwen3-4B-Instruct-2507	128K	+0.7 to +2.2	0.014 to 0.024	+0.1 to +0.6
Qwen3-8B	32K	0.0 to +1.5	0.014 to 0.024	0.0 to +0.4
Qwen3-8B	128K	+1.0 to +1.6	0.014 to 0.024	+0.3 to +0.6
Gemma-3-4B-it	32K	+2.0 to +3.9	0.217 to 0.352	-1.5 to +3.6

Appendix

Table 23: At

Q=512

the paid probe raises or matches success everywhere; its cost is small except on Gemma-3-4B-it, whose probe runs five native layers. Entries: ranges over populations of probe-on minus probe-off.

Figure 26: Controller cost barely moves end-to-end time: removing it shortens episodes by 0.3 to 0.6%, and an eight-fold cost lengthens them by 2.4 to 3.9%. Left: time change; right: normalized index and control cost.

Figure 27: Keeping a static mask dynamic-ready leaves success unchanged and costs up to 2.3% more time, 5.6% lower decode throughput, and 12 times the index-cache memory at small

R

(SWE-bench Verified; blue 32K, red 128K).

Figure 28: AIR shortens most attempts relative to all-refresh: median paired speedups are 1.06 on SWE-bench Verified, 1.05 on Terminal-Bench 2.0, and 1.10 to 1.11 on

\tau^{2}

, and 96 to 100% of attempts finish faster. Curves: distributions of per-attempt time ratios.

Figure 29: The prefix effect holds on all five populations: AIR ’s speedup over all-refresh rises from 0.97–1.00 at 4,096 tokens to 1.20–1.33 at 126,976 tokens, most on

\tau^{2}

References
	Success (%) $\uparrow$					Speedup over all-refresh $\uparrow$
Method	SWE	Term.	Retail	Airl.	Tel.	SWE	Term.	Retail	Airl.	Tel.
Native attention	32.0	28.1	57.7	50.0	46.5	0.78	0.77	0.70	0.70	0.71
All-refresh sparse	31.6	27.0	57.0	48.5	45.4	1.00	1.00	1.00	1.00	1.00
Strongest feasible static	27.0	22.5	53.3	42.5	42.5	1.03	1.05	1.11	1.11	1.10
Alternative injected selectors

Appendix

Table 24: Among the frozen-backbone methods shown, AIR has the highest success on SWE-bench Verified, HISA with AIR on telecom, and LongCat CLI on retail, and HISA with AIR is the fastest or tied-fastest on every population; Table 20 lists configurations that are faster on some populations. Gray: native attention, unranked.

Figure 30: AIR has the highest success of 24 selector settings on SWE-bench Verified, retail, and telecom; on Terminal-Bench 2.0 and airline other settings score up to 2.2 and 2.5 points higher. Solid:

R

sweeps per selector; dashed: MISA head sweep.

Figure 31: Routing fewer of the eight indexer heads (MISA) cuts index cost by about 70% at one head and costs 5.0 to 7.0 points of success.

Figure 32: AIR at

\beta=0.50

gives the most successful tasks per hour of 17 settings on SWE-bench Verified and ranks second on telecom; other settings yield more on Terminal-Bench 2.0 and airline. Dotted: settings no other setting beats on both axes.

Figure 33: Learned allocation helps most with the flat indexer; with cheaper selectors its success gain depends on the population. (a) AIR minus the selector’s static schedule (pp); (b) speedup over the selector’s all-refresh run.

	Success (%)			Speedup		Break-even attempts
Population	Unadapted	LongCat	+ AIR	LongCat	+ AIR	LongCat	+ AIR
SWE	31.6	30.8	31.8	1.09	1.07	6,403	7,290
Terminal	27.0	27.0	25.8	1.10	1.08	7,369	8,390
Retail	57.0	56.6	57.9	1.17	1.15	8,987	10,231
Airline	48.5	48.5	49.5	1.17	1.15	11,294	12,858
Telecom	45.4	45.0	46.7	1.15	1.14	8,652	9,851

Appendix

Table 25: Full LongCat Sparse Attention keeps success within 0.8 points of the unadapted all-refresh model at 1.08 to 1.17 times its speed, and adding AIR raises success on four of five populations; repaying 180 GPU-hours (192 with AIR ) takes 6,403 to 12,858 attempts.

Subset	Tasks	Transfer	In-domain	$\Delta$ (pp)	Transfer	In-domain
		Held-out success (%)			Minutes per attempt
Terminal-Bench 2.0	20	15.0	20.0	$+5.0$	15.5	15.5
$\tau^{2}$ retail	40	56.9	58.1	$+1.3$	7.5	7.5
$\tau^{2}$ airline	20	35.0	40.0	$+5.0$	6.5	6.5
$\tau^{2}$ telecom	40	44.4	45.6	$+1.3$	9.0	9.0

Appendix

Table 26: In-domain training raises held-out success over the transferred repair controller by 1.3 to 5.0 points at equal time per attempt (minutes). Both controllers run on the same held-out tasks; differences are calculated before rounding.

	32K			128K
Method	Retail	Airline	Telecom	Retail	Airline	Telecom
Native attention	16.7	16.0	10.5	21.1	10.0	10.5
All-refresh sparse	16.7	16.0	10.5	21.1	10.0	10.5
Strongest feasible static	13.2	16.0	8.8	15.8	8.0	7.9
Training-aware IndexCache	16.7	14.0	10.5	21.1	10.0	10.5
LatentIndex + HS	14.0	14.0	11.4	18.4	10.0	10.5

Appendix

Table 27: AIR ’s pass 4 (all-four success, %) exceeds the comparator’s in five of six

\tau^{2}

cells; on 32K airline the comparator is higher (16.0 against 12.0).

Population	Tasks	$\varphi=0.60$	$\varphi=0.75$	$\varphi=0.90$
SWE	500	392	271	172
Terminal	89	71	50	32
Retail	114	96	75	52
Airline	50	41	31	18
Telecom	114	99	77	49

Appendix

Table 28: Eligible long-history snapshots per occupancy band; fewer tasks reach the higher bands (SWE-bench Verified: 392, 271, and 172 of 500).

Figure 34: In long-history continuations, AIR ’s success exceeds the comparator’s in 12 of 15 population and occupancy bands and ties it in 3; all-refresh scores higher in 5 bands. Success covers only the continuation from a shared snapshot.

Figure 35: In long-history continuations, AIR finishes the remaining work 4 to 10% faster than all-refresh in every band and within 2.2% of the comparator. Snapshots, bands, and methods as in Figure 34 .

Figure 36: Training brings every 128K backbone below the 0.50 budget (dashed; dotted: tolerance) within about 4,000 rollouts while validation success rises by 6.2 to 7.6 points. (d) Budget mix of the primary model.

Figure 37: Each seed of the primary controller moves toward the budget while raising validation success, explores before it commits, and keeps a mixed allocation. (a) success against cost, with markers at 0, 6,144, and 12,288 rollouts; (b) action entropy; (c) expected Full-layer fraction; (d) dual updates.

Figure 38: At 32K, cost falls from 0.79 to 0.44–0.45 on five of six backbones and their violations reach zero; Gemma-3-4B-it ends at 0.59 with its dual multiplier still rising. Dashed: the 0.50 budget.

Figure 39: At 32K, seed-mean validation success rises by 5.6 to 7.2 points for every backbone over 12,288 rollouts.

Figure 40: Every 32K controller still uses at least three refresh options more than 10% of the time at the end of training. Options run from fewest to most Full layers within each backbone.

Figure 41: Controllers trained in each of the five fitting domains converge to a cost of 0.44 to 0.46, below the 0.50 budget, and raise validation success by 5.7 to 9.7 points.

Panel of Figure 42 — Figure 42: SWE-bench Verified event card for Section 5.2 : after a new test diagnostic, AIR raises its budget from $R=3$ to $R=18$ at the next model call, while fixed IndexCache keeps $R=9$ . Counts are cumulative Full-layer selections and index-key updates.

Panel of Figure 43 — Figure 43: Airline case, Event 1: both methods read the current booking, AIR at $R=5$ and fixed IndexCache at $R=9$ .

Panel of Figure 44 — Figure 44: Airline case, Event 2: both methods request the same quote, AIR at $R=3$ ; the quote adds a $35 bag charge that needs the customer’s confirmation.

Panel of Figure 45 — Figure 45: Airline case, Event 3: after the bag condition appears, AIR answers at $R=3$ and needs a second call at $R=18$ , while fixed IndexCache stays at $R=9$ .

Panel of Figure 46 — Figure 46: Telecom case, Event 1: all three methods read the line state, which reports stale provisioning; AIR runs at $R=5$ .

Panel of Figure 47 — Figure 47: Telecom case, Event 2: after the user confirms the backend refresh, AIR raises its budget to $R=18$ ; the API acknowledges the request, which does not restore connectivity.

Panel of Figure 48 — Figure 48: Telecom case, Event 3: once the user-side actions are observed, all three methods read the line status again, which now shows active provisioning; AIR runs at $R=9$ .

LLM Agent Efficiency

See all 85 papers

Oct 8, 2026

When Should Agents Think? Adaptive Reasoning via Cross-Turn Estimation

Large language model (LLM)-based agents have demonstrated strong capabilities on complex tasks. They typically perform reasoning before each action throughout an interaction trajectory. However, reasoning may not be necessary at every turn, as reasoning produced earlier can continue to support subsequent actions. A key challenge is therefore to determine when existing reasoning remains sufficient and when a new reasoning step is needed, without relying on costly generation-based verification. We find that decreases in the likelihood of subsequent reference actions after removing additional reasoning closely track whether those actions remain recoverable given earlier reasoning, providing an effective and lightweight signal for estimating cross-turn action support. Based on this observation, we propose Reasoning Adaptation through Cross-Turn Estimation (RACE), a training approach for adaptive agent reasoning. RACE introduces a Likelihood-Guided Progressive Reasoning Cover Detection (LoGiC) procedure that progressively identifies reasoning turns whose removal has limited impact on the current and subsequent reference actions. The resulting removal signals are incorporated into both supervised fine-tuning and agentic reinforcement learning, enabling the policy to learn when to reason and when to act directly. Extensive experiments on four representative agent benchmarks show that RACE substantially reduces reasoning cost while maintaining or improving task performance.

Yiruo Cheng, Shen Huang, Xiaoshuai Song +5

Gaoling School of Artificial Intelligence, Renmin University of China · Alibaba Token Hub, Alibaba Group

Adaptive Reasoning in Language Models Agentic Reasoning Efficient Language Model Reasoning

Ask PDF

Oct 26, 2026

AT-AT: Automatic Tool Optimization from Agent Traces

Efforts to improve LLM-based agents typically focus on modifying the agent itself, yet success also depends on the environment in which an agent acts. Tools mediate this interaction: a tool's description shapes how an agent calls it, and its implementation determines the results and errors the agent receives afterwards. Many tools, such as APIs and MCP servers, are shared by many agents and maintained by providers who cannot change those agents. We introduce AT-AT (Automatic Tool Optimization from Agent Traces), an agentic tool optimizer that improves a shared toolset offline from historical agent executions while holding the agents fixed. A lightweight coding agent reads agent trajectories and their evaluation results, and selectively edits tool descriptions and, when permitted, implementations. On

τ^2

-bench-verified, AT-AT achieves the highest task completion rates in the Airline and Telecom domains at lower cost than strong prompt- and tool-optimizer baselines, and its Telecom gain is statistically significant under paired permutation tests. Optimizing the tools within a coding agent harness also improves held-out performance on TerminalBench-2 and TBLite. When applied to three production MCP servers at a large enterprise, AT-AT lowered cost or tool error rates on every server and increased held-out task passes on one. By turning accumulated execution traces into updates to shared tool environments, AT-AT provides a path toward continual improvement of agent systems through environment-side learning.

Maya Ravichandran, Pragya Tripathi, Dharamendra Kumar +2

Netflix

Tool-Using Agents LLM Agent Efficiency

Ask PDF

May 24, 2026

AgentIR: A Workload-Adaptive Cascade Retrieval Substrate for Long-Term Conversational Memory

Long-term conversational memory is a retrieval workload classical IR was not built for: the index grows during the query stream, query types shift intra-session, and the latency budget per retrieval is sub-10 ms. Lucene-class engines treat the index as static and the query as stateless, leaving the workload's structure unexploited. AgentIR treats fusion as a per-query decision along two axes: which fusion to apply (BM25, Dense, RRF, or agent-aware RRF), and whether the ~52 ms dense channel is worth running at all. The second axis is a confidence-triggered cascade router that decides from the BM25 top-k margin alone and re-tunes across workloads without retraining. On LongMemEval (n=500), where the dense channel does add information, the cascade skips 63% of queries at parity LLM-judged accuracy (2.67x faster under two judges, paired bootstrap p>=0.88); per-qtype thresholds extend this to 5.76x under 5-fold cross-validation. On LoCoMo (n=1,982), where BM25 alone is already the strongest single system, the same trigger auto-tunes to a 100% skip rate (132x faster, +0.089 Hit@5). Capacity on a shared 8-core VM rises from ~154 to ~1,400 concurrent agents (9x). Underneath the cascade, a time-partitioned index does O(log 1/epsilon) work independent of corpus size: 1234x corpus growth costs only 3.6x latency, ending in 1769x over sequential at sub-100 us p50 on 5M records. At parity quality with Lucene on 9 BEIR datasets up to 8.8M docs, the substrate runs 10x geo-mean over Pyserini 8T and 11x over PISA-1T BlockMax-WAND; an A100 reaches 1.8-39x over Pyserini 8T; chunked index build sustains 56.8K docs/sec on MS MARCO. Three subtle BM25/GPU correctness pitfalls that silently regress nDCG@10 by 6-8x are documented and fixed; post-fix CPU and GPU agree within 0.0002 nDCG@10 on all eight datasets that fit a single A100.

Aojie Yuan, Haiyue Zhang, Shahin Nazarian

University of Southern California Los Angeles, CA, USA

Lexical Retrieval Long-Term Conversational Memory Agent Memory Management

Ask PDF

Abstract

LLM Agent Efficiency

Explore similar work

When Should Agents Think? Adaptive Reasoning via Cross-Turn Estimation

AT-AT: Automatic Tool Optimization from Agent Traces

AgentIR: A Workload-Adaptive Cascade Retrieval Substrate for Long-Term Conversational Memory