LEAP: Learning Efficient Action Proposals For LLM Agents

Zhen Xu; Qizheng Zhang; Gerry Wan; Shang Zhu; Ce Zhang

cs.LGOct 2, 2026

LEAP: Learning Efficient Action Proposals For LLM Agents

Authors: Zhen Xu, Qizheng Zhang, Gerry Wan, Shang Zhu, Ce Zhang

Organizations: University of Chicago · Stanford University · Together AI

Abstract

LLM agents are known to be slow in rollouts. An agent completes a task one step at a time. At each step, it reasons and then chooses an action to execute. The next step and action cannot start until the previous one has finished. Speculative decoding accelerates the rollouts at the reason phase by drafting and verifying the inference tokens. Recent works have also started to apply similar ideas at the action phase. These works use off-the-shelf models, usually large, to draft action proposals for target model to verify. Large drafters match the target more often but take longer to propose, while small off-the-shelf models are fast but rarely make the same decision as the target. We ask a more general question: what determines the end-to-end speedup of action speculation? To answer it, we develop a latency framework for the speculative round. The framework compares what a round gains with what it costs. The gain depends on how well the drafter predicts the target and on how many steps the task can take before it ends. The cost comes from drafting, from waiting for target verification and from executing tools. Guided by the framework, we introduce LEAP (Learning Efficient Action Proposals) which keeps the drafter small and makes it accurate by training it on the target actions sequences. With a small 0.6B model, LEAP agrees with the target on most decisions and makes agents up to 60% faster in end-to-end wall clock time, with no systematic change in task success. Across various datasets, target models and draft models, the framework accounts for most of the measured speedups. We also show the draft model can be online trained with no prior trace collection and match the performance of offline training, making LEAP practical to deploy in the real world.

Figures & tables

Panel of Figure 1 — Figure 1: LEAP trains and uses an action draft. Offline target traces or online commits supply the training pairs. The adapter proposes actions; the target verifies their contexts concurrently. In this $k=2$ example, one proposal matches and the target corrects the next. The remaining verification is cancelled, and the two committed actions can become further training data.

Benchmark	Target	0.6B	1.7B	4B
OpenAGI	Qwen3-32B	$1.50\,\pm\,0.07$	$\mathbf{1.55}\,\pm\,0.06$	$1.46\,\pm\,0.08$
	Gemma-4-31B	$\mathbf{1.55}\,\pm\,0.11$	$\mathbf{1.55}\,\pm\,0.11$	$1.52\,\pm\,0.10$
TaskBench	Qwen3-32B	$1.38\,\pm\,0.08$	$\mathbf{1.47}\,\pm\,0.07$	$1.36\,\pm\,0.08$
	Gemma-4-31B	$1.60\,\pm\,0.07$	$\mathbf{1.68}\,\pm\,0.07$	$1.62\,\pm\,0.06$
$\tau^{2}$ -bench	Qwen3-32B	$1.61\,\pm\,0.09$	$\mathbf{1.63}\,\pm\,0.09$	$1.56\,\pm\,0.08$
	Gemma-4-31B	$\mathbf{1.63}\,\pm\,0.07$	$1.55\,\pm\,0.06$	$1.55\,\pm\,0.06$

Table 1: Whole-task speedup across benchmarks, targets and draft sizes. Values are pooled speedup factors

\pm

one task-bootstrap SE (10,000 paired resamples); bold marks the largest rounded point estimate per row.

k=3

for planning and

k=2

for tool benchmarks; test sizes are 63/60/40/60. Qwen3 planning entries pool two runs. Full 95% intervals and components: Table 9 .

Benchmark	0.6B trained	0.6B base	32B direct	32B reason
	( LEAP )	( Liu et al., 2026 )	( Hua et al., 2025 ; Guan et al., 2026 )	( Bai et al., 2026 ; Ji et al., 2026 )
OpenAGI	$75.0\,\pm\,3.1$	$12.8\,\pm\,1.6$	$76.6\,\pm\,2.3$	$97.3\,\pm\,0.9$
TaskBench	$75.2\,\pm\,3.0$	$25.0\,\pm\,1.8$	$84.8\,\pm\,2.2$	$96.9\,\pm\,0.9$
$\tau^{2}$ -bench	$63.7\,\pm\,4.4$	$8.0\,\pm\,1.1$	$42.5\,\pm\,2.6$	$65.0\,\pm\,4.0$
BFCL	$66.7\,\pm\,6.0$	$2.0\,\pm\,1.8$	$62.6\,\pm\,3.7$	$91.9\,\pm\,3.2$

Table 2: Offline action agreement (%), Qwen3 target. Estimate

\pm

one task-bootstrap SE. Draft types used in the cited work are evaluated here on common held-out contexts. “Direct” disables reasoning; “reason” enables it. BFCL includes turn ends. Label sampling, repeated predictions and the bootstrap protocol are detailed in Table 7 .

		offline agreement
objective	supervision	TaskBench	BFCL
SFT on target actions (ours)	target’s committed actions	75.2	66.7
SFT on gold labels	benchmark ground truth	69.9	50.5
GRPO from the SFT draft	exact-match reward	77.1	65.7

Table 3: Supervision and training objective. Offline agreement (%) for the Qwen3 target; BFCL counts all decisions, including turn ends. Gold supervision also changes the training contexts. No variant improves agreement by the three points required to trigger an end-to-end run.

Figure 2: Measured acceptance and drafting cost, Qwen3 target. (a) Trained and untrained 0.6B controls. Acceptance is the share of compared proposals that match the target. (b) Trained 0.6B on all four benchmarks, plus the reasoning-disabled target on the two planning benchmarks where this control was run;

D

is proposing time per round normalized by a target step. All points use the control runs of Table 10 .

Figure 3: Draft-size ablation, Qwen3 target. Measured speedup, independent planning repeats, and the predictions used in Fig. 4 b. BFCL has one measured size study and no offline-hit prediction. Agreement and cost values are in Table 14 ; pooled results are in Table 1 .

Figure 4: Predicted versus measured speedup and verification wait. (a) 21 Qwen3 controls with measured hit rates and draft cost. (b) Nine size-study configurations per target, using offline hit rates and trace-replay

\beta

; draft timing sources are in Table 12 . (c) Verification wait

\beta

for the points of (b): the trace-replay prediction used in (b) against the measured value. Points above the diagonal waited longer than predicted, which raises the predicted speedup. Axes have equal scales within each panel; shading marks

\pm 0.10

in (a) and (b).

Figure 5: Learning online, Qwen3 target. (a) Held-out agreement after each adapter update. (b) Speedup on each stream batch, placed at the batch midpoint: sequential time per step over speculative time per committed step. Dashed lines are offline drafts. Each batch uses an adapter trained on earlier tasks. Training time is reported separately; final test results are in Table 5 .

Benchmark	Offline	Online: retrain	Online: continual
OpenAGI	$1.51\,\pm\,0.07$	$1.47\,\pm\,0.08$	$1.51\,\pm\,0.06$
$\tau^{2}$ -bench	$1.61\,\pm\,0.09$	$1.61\,\pm\,0.10$	$1.63\,\pm\,0.10$

Table 5: Final online adapters on the test split, Qwen3 target. Whole-task inference speedup

\pm

one task-bootstrap SE, using trained 0.6B drafts with matched offline and online serving settings. Training time is separate; 95% intervals are in Table 20 .

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

$T$ , $D$	target agent; draft
$s_{i}$ , $a_{i}$	context and target action at step $i$ ; $\hat{a}_{j}$ : $j$ -th proposal of a round
$k$	proposals per round (depth)
$A$	accepted proposals in a round; the round commits $A{+}1$ steps
$L$	agent steps in a segment (between user turns or up to the end of the task)
$q_{j}$	$\Pr[A\geq j]$ , hit rate at position $j$ ; $q_{1}$ is the first-position acceptance
$c$ , $e$	committed steps per round; expected bonus-step loss at segment ends (Eq. 3 )

Appendix

Table 6: Notation.

Panel of Figure 6 — Figure 6: Round timelines with $k=2$ . Illustrative durations. A successful round waits for the slowest required verification; on a mismatch, unused requests are cancelled. Tool execution for proposals occurs while proposing, and accepted environment states are adopted.

		drafts used in prior work			ours
benchmark	contexts	untrained 0.6B	32B-direct	32B-reason	trained 0.6B
		( Liu et al., 2026 )	( Hua et al., 2025 ; Guan et al., 2026 )	( Bai et al., 2026 ; Ji et al., 2026 )
OpenAGI	263	12.8	76.6	97.3	75.0
TaskBench	277	25.0	84.8	96.9	75.2
$\tau^{2}$ -bench	226	8.0	42.5	65.0	63.7
BFCL	99	2.0	62.6	91.9	66.7

Appendix

Table 7: Offline action agreement with the Qwen3-32B target (exact match on the target’s own held-out contexts; repeated predictions averaged per context, except the

\tau^{2}

-bench 32B-reason reference, which uses the mode of three passes). “32B-direct” is the target with reasoning disabled; “32B-reason” is the target itself with reasoning on, replayed on the same contexts, i.e. on the planning benchmarks its agreement with its own earlier pass, a reference set by the engine’s nondeterminism rather than a bound. The three prior-work columns are an off-the-shelf small model, a cheaper configuration of the same model, and the target as its own draft, for which “32B-reason” is the reference, all measured in one setting. BFCL rows are on 99 development contexts and count every decision, a turn end or an exact tool group; the tool-group rate used for the acceptance comparison is in Table 8 .

\tau^{2}

-bench labels were sampled from the target at temperature 0.6, and all

\tau^{2}

-bench columns are measured against them; there, as on BFCL, the 32B-reason column replays the target (reasoning on, temperature 0) on the same contexts.

benchmark	offline	$q_{1}$	$q_{2}/q_{1}$	$q_{3}/q_{2}$	$a_{2}$	$a_{3}$	per proposal
OpenAGI	75.0	0.71	0.63	0.63	0.81	0.80	0.75
TaskBench	75.2	0.66	0.51	0.53	0.66	0.73	0.68
$\tau^{2}$ -bench	63.7	0.64	0.54	–	0.63	–	0.64
BFCL	49.2	0.38	0.20	–	0.62	–	0.53

Appendix

Table 8: From offline agreement to live acceptance for the trained 0.6B drafts. Offline: exact match on the target’s held-out contexts (Table 7 : mean over repeated predictions; BFCL: tool-group rate). Live, from the end-to-end runs of Table 9 :

q_{1}

, the fraction of rounds whose first proposal is accepted;

q_{2}/q_{1}

and

q_{3}/q_{2}

, the ratios of successive hit rates, which also count the rounds that stop proposing before the position;

a_{2}

and

a_{3}

, the acceptance of the second and third proposal among the rounds that made it and accepted the earlier ones; and the pooled per-proposal acceptance. BFCL live rates are per tool group. A dash marks a position that does not exist:

\tau^{2}

-bench and BFCL run with

k{=}2

benchmark	draft	steps	$t_{\mathrm{gen}}$ (s)	$k$	agreem.	accept.	$c$	speedup	95% CI
Target Qwen3-32B
OpenAGI	0.6B	4.2	1.1	3	75.4	0.75	2.05	1.50 $\times$	[1.36, 1.65]
	1.7B	4.2	1.1	3	77.6	0.78	2.18	1.55 $\times$	[1.43, 1.68]
	4B	4.2	1.1	3	78.7	0.79	2.26	1.46 $\times$	[1.30, 1.62]
TaskBench	0.6B	5.0	1.8	3	74.7	0.70	1.94	1.38 $\times$	[1.24, 1.55]
	1.7B	5.0	1.8	3	81.0	0.76	2.07	1.47 $\times$	[1.34, 1.61]

Appendix

Table 9: Whole-task end-to-end speedup of LEAP: four benchmarks, three draft sizes, two targets (one H100 shared by target and draft;

\tau^{2}

-bench and BFCL execute tools; 63, 60, 40 and 60 test tasks). Each benchmark serves its three drafts in one configuration (App. I ). Agreement is the draft’s offline exact-match rate (%) on the target’s held-out contexts (BFCL: decision level); acceptance is the share of verified proposals that were accepted in the runs (the per-proposal rate of Table 8 );

c

is the measured number of committed steps per round (Eq. 3 ); the 95% interval is a task-level bootstrap of the pooled ratio (10,000 resamples). Bold marks the fastest of the three drafts for a benchmark and target. Task success per arm shows no systematic change (App. M ).

Figure 7: Component checks of the latency framework. (a) Measured per-position hit rates (solid) and

q_{1}^{j}

(dotted) for the four main configurations. (b) Trace-replay

\beta

(filled) and the independent-step closed form (hollow), against measured values. (c) Committed steps from the single-rate replay against measured commits. Configuration details are in Table 10 .

			hit rates				$\beta$			predicted
benchmark	draft	$k$	$q_{1}$	$q_{2}$	$q_{3}$	$c$	pred.	meas.	draft	i.i.d.	Eq. 4	measured
OpenAGI	trained 0.6B	1	0.74	–	–	1.54	1.09	1.08	0.08	1.35	1.32	1.34
OpenAGI	trained 0.6B	3	0.71	0.44	0.28	2.05	1.21	1.19	0.18	1.48	1.47	1.51
OpenAGI	trained 0.6B	5	0.70	0.47	0.31	2.25	1.24	1.21	0.22	1.47	1.54	1.57
OpenAGI	untrained 0.6B	3	0.15	0.00	0.00	1.14	1.03	1.05	0.13	0.97	0.98	0.98
OpenAGI	32B-direct	3	0.70	0.50	0.33	2.19	1.21	1.27	0.61	1.12	1.20	1.16

Appendix

Table 10: Predicted vs. measured speedup.

q_{j}

: measured per-position hit rates;

c

: committed steps per round;

\beta

: round-wait inflation, predicted from the sequential arm’s step times and measured from the speculative arm; draft: the measured draft time per round divided by the mean sequential step time (the

D

term of Eq. 4 , with rounds that stop proposing early counted as run). “i.i.d.” replaces the measured hit rates by

q_{j}=q_{1}^{\,j}

. Every row was run with cancellation, so the harness never waits on already-discarded verifications (Sec. 3.2 , assumption (e)) and the measured column is the speedup as run. Rows with other

k

exercise the depth terms of the model; Table 4 exercises its tool terms. The 32B-direct rows use the target itself, with reasoning disabled, as the draft; the untrained rows use the 0.6B base without an adapter. A dash marks a position beyond the row’s

k

	replay, before the run				live, in the run				speedup
benchmark	$q_{1}$	$q_{2}$	$q_{3}$	$c$	$q_{1}$	$q_{2}$	$q_{3}$	$c$	ex-ante	measured
OpenAGI	0.70	0.45	0.26	2.03	0.71	0.44	0.28	2.05	1.48 $\times$	1.51 $\times$
TaskBench	0.73	0.42	0.24	2.04	0.66	0.34	0.18	1.88	1.56 $\times$	1.37 $\times$
$\tau^{2}$ -bench	0.59	0.34	–	1.85	0.64	0.35	–	1.88	1.59 $\times$	1.58 $\times$

Appendix

Table 11: Ex-ante inputs from teacher-forced predictions. Hit rates and committed steps per round from replaying the loop on the held-out sequential trajectories with the offline predictions of Table 7 (“replay”), against the live values of the runs of Table 10 ; the ex-ante speedup uses the replayed values with the trace-replay

\beta

and the draft’s single-stream latency measured offline on the held-out contexts.

\tau^{2}

-bench obtains hit rates from the 14 validation tasks and sequential timing from the 40 test tasks. BFCL is omitted: its rounds are bounded by user turns with a terminal probe, which the replay does not reproduce. A dash marks a position beyond

k{=}2

Target	Benchmark	Draft	Draft timing	Predicted	Measured	$\beta$ replay	$\beta$ meas.
Qwen3	OpenAGI	0.6B	probe	1.48	1.50	1.22	1.19
Qwen3	OpenAGI	1.7B	run	1.53	1.55	1.22	1.21
Qwen3	OpenAGI	4B	run	1.47	1.46	1.24	1.25
Qwen3	TaskBench	0.6B	probe	1.53	1.38	1.14	1.22
Qwen3	TaskBench	1.7B	run	1.58	1.47	1.16	1.21
Qwen3	TaskBench	4B	run	1.53	1.36	1.16	1.26

Appendix

Table 12: Predictions using offline hit rates, both targets. The eighteen size-study points in Fig. 4 (b), using each draft’s offline predictions and the original trace-replay

\beta

. Serving configurations and measured speedups match Table 1 . Draft timing is either an offline single-stream probe or measured per-proposal latency from the speculative run (“run”); the latter rows are conditional on that cost. Sequential timing is from the test trajectories;

\tau^{2}

-bench hit rates are from validation trajectories. The last two columns give the trace-replay and measured

\beta

plotted in Fig. 4 (c).

draft	overall	seen	unseen
untrained 0.6B	12.8	11.5	14.8
LoRA $r{=}64$ , 3 epochs, epoch 2 by validation loss (Table 7 )	75.0	82.2	64.5
LoRA $r{=}64$ , ep 7 by validation exact match	77.1	86.4	63.2
LoRA $r{=}64$ , epoch 12	74.1	83.4	60.4
full fine-tuning, ep 7	75.2	84.7	61.0
LoRA $r{=}64$ , ep 7, 50% of the pairs (535)	72.0	80.5	59.4

Appendix

Table 13: OpenAGI offline agreement (%) of 0.6B drafts under recipe variations. Rows marked “ep 7” train for 12 epochs and pick the checkpoint by validation exact match; the Table 7 draft trains for 3 epochs and picks by validation loss (epoch 2). Seen/unseen as defined in the text.

	offline agreement				draft term $D$				predicted speedup
draft	OpenAGI	TaskBench	$\tau^{2}$ -bench	BFCL	OpenAGI	TaskBench	$\tau^{2}$ -bench	BFCL	OpenAGI	TaskBench	$\tau^{2}$ -bench
0.6B	75.4	74.7	63.3	66.7	0.17	0.16	0.03	0.04	1.48 $\times$	1.53 $\times$	1.58 $\times$
1.7B	77.6	81.0	68.1	72.7	0.20	0.19	0.03	0.04	1.53 $\times$	1.58 $\times$	1.64 $\times$
4B	78.7	80.5	69.5	79.8	0.28	0.29	0.04	0.07	1.47 $\times$	1.53 $\times$	1.62 $\times$

Appendix

Table 14: Draft size. Three drafts trained with the same recipe. Offline agreement (%) on the held-out contexts of Table 7 ; the measured per-round draft term

D

of Eq. 4 in the common serving configurations of Table 9 ; the speedup Eq. 4 predicts from the sequential trace and each draft’s own offline hit predictions, with the draft timing sources listed in Table 12 . BFCL has no offline-hit prediction (Fig. 3 ). Measured speedups are in Table 9 .

Figure 8: Online training of the draft. The draft is trained online, from the commits of the running speculative system: the untrained 0.6B draft starts the system on the training tasks, and after every batch a new adapter replaces the previous one in the draft server without a restart. Colour marks the benchmark. Solid lines retrain the adapter on all commits so far; dash-dot lines with hollow markers continue the previous adapter (continual). (a) Offline agreement of each adapter version on the held-out contexts. (b) Acceptance on each batch of the stream. (c) Speedup on each OpenAGI batch and, as stars with bootstrap intervals, the final adapters on the test tasks. Dashed: the offline drafts (Tables 7 and 9 ); grey line: no speedup.

	OpenAGI	TaskBench	$\tau^{2}$ -bench	BFCL multi-turn
task type	tool-selection plan	tool calls with args	ticket-driven support	multi-turn function calling
tools	14 $+$ TERMINATE	DailyLife APIs	44	per-task catalog
target format	ReAct, <tool> tag with tool name	ReAct, <tool> tag with name({json})	single-call JSON, guided	<tool_call> tags, ordered group
target reasoning	thought $\leq 512$ tok.	thought $\leq 512$ tok.	on, budget 2,048	on, budget 4,096
target action budget	32 tok.	128 tok.	(in budget)	(in budget)
draft format	direct, tool name	direct, name $+$ args	grammar, JSON call	grammar, call group / <respond/>

Appendix

Table 15: Per-benchmark settings of the end-to-end runs.

benchmark	draft	$k$	$q_{1}$	$q_{2}$	$q_{3}$	$q_{4}$	$q_{5}$	$e$	$c$
OpenAGI	trained 0.6B	3	0.71	0.44	0.28	–	–	0.38	2.05
OpenAGI	trained 0.6B	1	0.74	–	–	–	–	0.20	1.54
OpenAGI	trained 0.6B	5	0.70	0.47	0.31	0.14	0.08	0.46	2.25
OpenAGI	untrained 0.6B	3	0.15	0.00	0.00	–	–	0.01	1.14
OpenAGI	32B-direct	3	0.70	0.50	0.33	–	–	0.34	2.19
TaskBench	trained 0.6B	3	0.66	0.34	0.18	–	–	0.30	1.88

Appendix

Table 16: Measured per-position hit rates

q_{j}=\Pr[A\geq j]

, end-of-segment loss

e

, and committed steps per round

c=1+\sum_{j}q_{j}-e

. A dash marks a position beyond the row’s

k

	OpenAGI	TaskBench	$\tau^{2}$ -bench	BFCL
max model length (tokens)	2,048	4,096	16,384	24,576
GPU memory fraction, target / draft	0.82 / 0.10	0.85 / 0.10	0.88 / 0.08	0.88 / 0.06
max concurrent sequences, target / draft	8 / 2	4 / 2	4 / 4	4 / 1
guided decoding	–	–	xgrammar	xgrammar
reasoning parser	–	–	qwen3	qwen3
verification	prefix-ordered, cancel

Appendix

Table 17: vLLM server settings per benchmark (target / draft on the same H100). A dash marks a setting not used for that benchmark. On TaskBench at

k{=}5

(Table 10 ) a round issues up to six verification requests, so those beyond the fourth wait for a free slot.

benchmark	steps	tasks	$e$	$\sum_{j}q_{j}$	$c$	meas.	pred.
						speedup
OpenAGI	3	15	0.52	1.66	2.14	1.54 $\times$	1.35 $\times$
OpenAGI	4	21	0.31	1.46	2.15	1.52 $\times$	1.39 $\times$
OpenAGI	5	15	0.40	1.54	2.14	1.60 $\times$	1.57 $\times$
OpenAGI	$\geq 6$	9	0.30	1.10	1.80	1.39 $\times$	1.55 $\times$
TaskBench	$\leq 4$	27	0.38	1.29	1.91	1.48 $\times$	1.37 $\times$

Appendix

Table 18: Segment truncation by task length. Tasks of the main runs bucketed by their sequential length.

e

(Eq. 3 ), the hit-rate sum

\sum_{j}q_{j}

and

c

are measured per bucket; the prediction replays the bucket’s sequential traces with the run-wide hit rates truncated at the task end, so it carries the truncation effect only. Buckets hold 7–27 tasks, so individual ratios are noisy.

	evaluated on
draft trained on	OpenAGI	TaskBench	$\tau^{2}$ -bench	BFCL
OpenAGI	75.5	29.6	8.8	3.0
TaskBench	23.3	74.7	9.3	6.1
$\tau^{2}$ -bench	10.8	21.7	63.7	1.0
BFCL	12.0	23.1	8.0	66.7
untrained base	13.3	25.3	8.0	2.0

Appendix

Table 19: Transfer across benchmarks. Offline agreement (%) of each 0.6B draft (rows) on the held-out contexts of each benchmark (columns), scored as in Table 7 (BFCL: decision agreement on the 99 development contexts).

Benchmark	Offline	Online: retrain	Online: continual
OpenAGI	1.51 $\times$ [1.38, 1.64]	1.47 $\times$ [1.31, 1.63]	1.51 $\times$ [1.40, 1.64]
$\tau^{2}$ -bench	1.61 $\times$ [1.44, 1.79]	1.61 $\times$ [1.43, 1.81]	1.63 $\times$ [1.46, 1.83]

Appendix

Table 20: Final online adapters on the test split, Qwen3 target. Whole-task inference speedup and task-bootstrap 95% intervals, using trained 0.6B drafts with matched offline and online serving settings. Training time is reported separately.

version	1	2	3	4	5	6	7	8	9	10	update time
OpenAGI, retrain on buffer	53.0	58.8	67.0	69.6	73.3	74.7	73.4	72.8	74.1	74.0	27–73 s
OpenAGI, continual	48.7	61.6	65.7	69.8	63.5	69.2	70.8	72.5	75.9	74.9	22 s
$\tau^{2}$ -bench, retrain on buffer	33.6	57.5	58.4	58.4	64.6	66.4	–	–	–	–	3–18 min
$\tau^{2}$ -bench, continual	28.3	57.1	54.9	59.7	63.3	63.3	–	–	–	–	3–6 min

Appendix

Table 21: Online training: agreement of each adapter version (offline exact match on the held-out contexts, %). OpenAGI: 263 test contexts, one version per 25 training tasks;

\tau^{2}

-bench: 226 validation contexts, one version per 10 tasks, so its 60 training tasks give six versions. Offline drafts: 75.0 (OpenAGI) and 63.7 (

\tau^{2}

-bench). Update time is the time from the end of a batch to the loaded adapter (buffer build and training).

	Qwen3-32B target				Gemma-4-31B target
		speculative				speculative
benchmark	sequential	0.6B	1.7B	4B	sequential	0.6B	1.7B	4B
OpenAGI	126	126	126	126	63	63	63	63
TaskBench	116	114	115	114	60	60	60	60
$\tau^{2}$ -bench	22	23	24	21	39	40	40	40
BFCL multi-turn	34	34	36	34	48	47	47	48

Appendix

Table 22: Task outcomes per arm for the configurations of Table 9 . Counts denote normal termination on OpenAGI and TaskBench, and official task success on

\tau^{2}

-bench and BFCL. The sequential count is the same across the three draft-size comparisons.

LLM Agents

Large language model agents use language models to plan, use tools, and act across multi-step tasks and environments.

See all 685 papers

AgentSpec: Speculative Decoding for Batch Inference of LLM Agents
Aug 25, 2026Xin Wang, Ziming Miao, Yi Zhu +4Speculative Decoding LLM Inference Acceleration
The Ohio State University · Microsoft Research · University of Michigan
Speculate with Memory: Lossless Acceleration for LLM Agents
Jul 14, 2026Yu Li, Qinyuan Ye, Prafulla Kumar Choubey +2Next-Action Prediction LLM Agent Memory
Salesforce Research
Speculative Macro Commit for Faster Tool-Using Agents
Sep 3, 2026Zeyu Liu, Souvik Kundu, Peter A. BeerelTool-Augmented Language Model Agents LLM Agent Workflow Optimization
University of Southern California, Los Angeles, USA · Intel Labs, San Diego, USA

Aug 25, 2026J/K move · Enter open · S save

AgentSpec: Speculative Decoding for Batch Inference of LLM Agents

Xin Wang, Ziming Miao, Yi Zhu, Hui Shen+3

The Ohio State University · Microsoft Research · University of Michigan

Speculative Decoding LLM Inference Acceleration LLM Agent Serving

Open paper AskPDF arXiv

Large language model (LLM)-based agent applications often incur high response time. Speculative decoding is a promising solution to improve the inference efficiency of LLM agents without impacting generation quality. However, state-of-the-art speculative decoding algorithms exhibit substantial speed degradation under large batch sizes, limiting their effectiveness to deploy in real-world agent applications. In this work, we first present a systematic analysis of speculative decoding for LLM agents and identify two dominant factors of speedup degradation: high rejection rate of speculative tokens, and under-utilization of dynamic token budgets.B ased on these observations, we propose AgentSpec, a speculative decoding algorithm that addresses the limitations of existing methods for LLM agents. AgentSpec incorporates structure-isolated drafting that constrains speculation to semantically coherent segments of the agent workflow, reducing the drafts of irrelevant semantic paths and achieving an extremely low rejection rate. Moreover, AgentSpec adopts redundancy-aware budget allocation that exploits agent-level information to better utilize the dynamically-free token budget during the agent inference. We implement and evaluate AgentSpec on five different workloads and four different models from four different LLM families in vLLM. Our results demonstrate the superiority of AgentSpec over state-of-the-arts.

Abstract

LLM Agents

Explore similar work

AgentSpec: Speculative Decoding for Batch Inference of LLM Agents

Speculate with Memory: Lossless Acceleration for LLM Agents

Speculative Macro Commit for Faster Tool-Using Agents