ObsSpec: Accelerating Agentic Reinforcement Learning with Observation Speculation

Kyle Montgomery; Wang Zhang; Chenguang Wang

cs.LGOct 11, 2026

ObsSpec: Accelerating Agentic Reinforcement Learning with Observation Speculation

Authors: Kyle Montgomery, Wang Zhang, Chenguang Wang

Organizations: UC Santa Cruz · ByteDance Inc.

Abstract

Tool-using agents alternate between model generation and tool execution, waiting for each tool call to complete before starting the next generation. This sequential execution is costly during agentic reinforcement learning (RL), which requires large batches of multi-turn rollouts. We introduce ObsSpec, an observation speculation pipeline for agentic RL that uses a small world model to predict tool outputs while the environment executes, allowing the policy to begin its next generation early. ObsSpec retains speculative work only when the predicted and actual observations match exactly, making rollout collection lossless, while co-training keeps the world model up to date as the policy learns. On terminal-use tasks, predictions from a 0.6B world model exactly match actual observations in 66.4% of attempts and arrive roughly 16 times faster. Using these predictions, ObsSpec reduces agent latency by 9.3% and 14.2% for 8B and 32B policies. Across controlled experiments, we characterize the regimes in which ObsSpec reduces agent latency under shared policy resources. Finally, because ObsSpec overlaps policy generation with tool execution, it is architecturally compatible with existing techniques for accelerating RL. Code is available at https://github.com/kylemontgomery1/verl-recipe/tree/obsspec/obsspec.

Figures & tables

Panel of Figure 1 — Figure 1: Across policy sizes, ObsSpec reduces latency and increases asynchronous rollout throughput with similar rewards.

Figure 2: Comparison of the sequential and ObsSpec pipelines on a terminal task. In the sequential pipeline (top), policy generation and tool execution alternate, so generation of each action cannot begin until the previous action finishes. In our ObsSpec pipeline (bottom), a world model predicts each observation while the corresponding action executes, allowing the policy to generate the next action before the execution of the previous action finishes. The first predicted observation $\hat{o}_{1}$ matches the actual observation $o_{1}$ , whereas $\hat{o}_{2}$ differs from $o_{2}$ . Consequently, the in-progress generation of $a_{3}$ is canceled and restarted from the actual observation $o_{2}$ . By overlapping policy generation with tool execution, ObsSpec reduces agent latency by up to 14.2%.

Panel of Figure 3 — Figure 3: Observation prediction outcomes (top) and latency (bottom) for common commands.

Panel of Figure 4 — Figure 4: Effect of world model size with the 8B policy at $D=K=1$ . Larger models reduce rollbacks but take longer to predict and do not reduce agent latency.

Panel of Figure 5 — Figure 5: Depth sweep with the 8B policy and 0.6B world model. Greater depth increases realized speculation and discarded policy computation without reducing agent latency.

Panel of Figure 6 — Figure 6: Breadth sweep with the 8B policy and 0.6B world model. Greater breadth increases realized speculation and match rate, while reducing the policy decode rate and increasing agent latency.

Panel of Figure 7 — Figure 7: Policy improvement and the effects of co-training the world model on match rate and agent latency.

Panel of Figure 8 — Figure 8: Sensitivity to sandbox infrastructure with the 8B policy and 0.6B world model.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Data
Batch size	16 prompts
Group size	16 rollouts per prompt
Prompt budget	2,048 tokens
Response budget	16,384 tokens
Per-turn generation budget	2,048 tokens

Appendix

Table 1: Training and serving configuration for the policy and world model.

Policy	Pipeline	Reward (%) $\uparrow$	Agent (s) $\downarrow$	Batch (s) $\downarrow$	Rollouts/h $\uparrow$
Qwen3-8B	Sequential	$78.64\pm 0.64$	$41.98\pm 0.39$	$156.26\pm 2.94$	$13{,}762\pm 156$
Qwen3-8B	ObsSpec	$79.76\pm 0.63$	$38.06\pm 0.38$	$149.42\pm 3.06$	$14{,}571\pm 175$
Qwen3-32B	Sequential	$84.62\pm 0.56$	$42.59\pm 0.46$	$185.63\pm 2.40$	$14{,}066\pm 155$
Qwen3-32B	ObsSpec	$83.91\pm 0.57$	$36.53\pm 0.44$	$171.16\pm 2.81$	$15{,}689\pm 199$

Appendix

Table 2: Full results underlying Figure 1 .

Command	Count	Tool (s)	WM (s)	Match (%)	Rollback (%)	Cancel (%)
cat	19,544	$1.597\pm 0.006$	$0.113\pm 0.001$	$79.95\pm 0.29$	$16.75\pm 0.27$	$3.31\pm 0.13$
ls	10,681	$1.614\pm 0.008$	$0.121\pm 0.001$	$39.43\pm 0.47$	$59.54\pm 0.47$	$1.03\pm 0.10$
mkdir	2,943	$1.607\pm 0.016$	$0.036\pm 0.000$	$96.67\pm 0.33$	$3.33\pm 0.33$	$0.00\pm 0.00$
echo	1,169	$1.612\pm 0.026$	$0.038\pm 0.000$	$98.37\pm 0.37$	$1.63\pm 0.37$	$0.00\pm 0.00$
sha256sum	946	$1.625\pm 0.028$	$0.178\pm 0.002$	$35.20\pm 1.55$	$50.00\pm 1.63$	$14.80\pm 1.15$
sqlite3	668	$1.631\pm 0.033$	$0.102\pm 0.004$	$78.89\pm 1.58$	$15.57\pm 1.40$	$5.54\pm 0.89$

Appendix

Table 3: Full results underlying Figure 3 .

WM	WM latency (s)	Agent latency (s)	Match	Rollback	Cancel
0.6B	$0.105\pm 0.001$	$38.06\pm 0.38$	$66.45\pm 0.22$ %	$30.50\pm 0.21$ %	$3.05\pm 0.08$ %
1.7B	$0.137\pm 0.001$	$37.84\pm 0.37$	$68.50\pm 0.21$ %	$28.71\pm 0.21$ %	$2.80\pm 0.08$ %
4B	$0.239\pm 0.001$	$37.89\pm 0.36$	$68.74\pm 0.21$ %	$24.88\pm 0.20$ %	$6.38\pm 0.11$ %
8B	$0.350\pm 0.002$	$37.95\pm 0.37$	$66.17\pm 0.22$ %	$22.47\pm 0.19$ %	$11.36\pm 0.15$ %

Appendix

Table 4: Full results underlying Figure 4 .

Depth	Agent latency (s)	Realized depth	Policy tok/s	Discarded policy tok.
1	$38.06\pm 0.38$	$0.992\pm 0.001$	$145.5\pm 0.1$	$198.0\pm 3.4$
2	$38.27\pm 0.38$	$1.871\pm 0.006$	$143.0\pm 0.1$	$342.5\pm 5.8$
4	$38.02\pm 0.37$	$2.773\pm 0.017$	$140.5\pm 0.1$	$426.7\pm 7.5$
8	$38.47\pm 0.39$	$3.208\pm 0.029$	$139.5\pm 0.1$	$472.2\pm 9.8$

Appendix

Table 5: Full results underlying Figure 5 .

$K$	Agent latency (s)	Match	Realized breadth	Policy tok/s	Discarded policy tok.
1	$38.06\pm 0.38$	$66.45\pm 0.22$ %	1.0	$145.5\pm 0.1$	$198.0\pm 3.4$
2	$41.88\pm 0.40$	$67.36\pm 0.22$ %	$1.303\pm 0.002$	$126.2\pm 0.1$	$332.8\pm 5.8$
4	$44.60\pm 0.44$	$68.56\pm 0.21$ %	$1.940\pm 0.006$	$116.8\pm 0.1$	$574.9\pm 10.3$
8	$47.68\pm 0.46$	$69.01\pm 0.21$ %	$3.142\pm 0.013$	$107.5\pm 0.1$	$994.9\pm 19.0$

Appendix

Table 6: Full results underlying Figure 6 .

Policy step	Reward (%)	Updated match (%)	Frozen match (%)	Sequential agent (s)	Updated agent (s)	Frozen agent (s)
100	$58.95\pm 2.52$	$59.07\pm 0.23$	$59.07\pm 0.23$	$23.14\pm 0.42$	$21.35\pm 0.40$	$21.35\pm 0.40$
200	$68.68\pm 2.38$	$62.64\pm 0.23$	$57.88\pm 0.24$	$23.20\pm 0.27$	$21.44\pm 0.26$	$21.70\pm 0.30$
300	$73.95\pm 2.25$	$59.87\pm 0.22$	$54.19\pm 0.22$	$28.03\pm 0.33$	$24.99\pm 0.30$	$25.22\pm 0.28$
400	$76.84\pm 2.16$	$64.01\pm 0.21$	$55.96\pm 0.21$	$32.47\pm 0.31$	$28.92\pm 0.28$	$29.60\pm 0.29$
500	$80.26\pm 2.04$	$66.45\pm 0.22$	$57.35\pm 0.23$	$41.98\pm 0.39$	$38.06\pm 0.38$	$39.74\pm 0.39$

Appendix

Table 7: Full results underlying Figure 7 .

Sandbox	Tool latency (s)	Sequential agent (s)	ObsSpec agent (s)	Reduction
Default	$1.69\pm 0.01$	$41.98\pm 0.39$	$38.06\pm 0.38$	9.3%
Modal	$2.56\pm 0.01$	$52.17\pm 0.44$	$46.66\pm 0.51$	10.6%
Daytona	$2.99\pm 0.02$	$56.30\pm 0.59$	$49.94\pm 0.54$	11.3%

Appendix

Table 8: Full results underlying Figure 8 . All measured quantities report means and standard errors.

Method	Reward (%)	Agent (s)	WM (s)	Match (%)	Rollback (%)	Cancel (%)
Sequential	$78.25\pm 0.64$	$55.82\pm 0.83$	–	–	–	–
ObsSpec , 0.6B	$78.47\pm 0.64$	$56.42\pm 0.86$	$0.120\pm 0.003$	$54.96\pm 0.42$	$40.42\pm 0.41$	$4.62\pm 0.18$
ObsSpec , 8B	$78.39\pm 0.64$	$55.79\pm 0.82$	$0.367\pm 0.007$	$59.47\pm 0.41$	$31.72\pm 0.39$	$8.81\pm 0.24$

Appendix

Table 9: Full results for the TIR experiments in Section 3.8 .

Agentic RL

Agentic reinforcement learning trains AI agents to plan, use tools, and adapt through feedback from multi-step interactions.

See all 231 papers

Aug 1, 2026

AOSpec: Action and Observation Co-Speculation for Low-Latency Agent Serving

Large language model agents increasingly act through stateful tools, yet model generation and environment execution remain serialized at every step. As decoding accelerates, tool execution becomes a growing bottleneck. Existing action- or observation-only speculation leaves much of this latency exposed: value is concentrated in a few slow calls, some outcomes emerge only through execution, and longer lookahead typically requires an increasingly unlikely chain of action predictions. We present AOSpec, a lossless framework that co-speculates actions and observations across the full agent-environment loop. Expected Value Decoding (EVD) directs observation speculation toward outcomes with the greatest expected latency benefit, optimizing expected time hidden rather than hit rate. For outcomes only execution can reveal, AOSpec launches latency-critical target actions in isolated forks that contain their effects, while Joint Action-State Verification (JASV) verifies both the action and its origin state against committed execution before reuse. JASV recasts long-horizon action dependency from full-chain prediction into target action-state verification, breaking the lookahead--accuracy tradeoff and unlocking long-range overlap without sacrificing serial semantics. Across Terminal-Bench serving settings spanning four harnesses, five actor models, and five serving speeds, AOSpec outperforms every practical baseline, reducing mean end-to-end latency by 11.8-32.5% and p99 latency by up to 42.8%. Its gains increase as decoding accelerates, and its observation model transfers from Terminal-Bench to SWE-bench Verified without retraining.

Hao Mark Chen, Jinnan Guo, Wayne Luk +1

Imperial College London, UK

LLM Inference Acceleration Long-Horizon Agent Tasks

Ask PDF

Jul 28, 2026

Speculate While You Reason: Teaching Agents to Predict Their Next Tool Call via Joint Agent-Speculator RL

Large language model agents often spend substantial wall-clock time waiting for tool call results. Tool-call speculation can hide this latency by predicting and pre-executing an agent's next tool call if the prediction matches the agent's eventual tool call, but existing speculators are typically separate draft models or cached traces that are poorly aligned with the deployed agent's own behavior. We identify this speculator-agent gap and show that the target agent itself is a strong next-call speculator. This points to a simpler design: unifying the agent and speculator within the same model. In this paper, we introduce the self-speculating agent, a single model that both solves tasks in agent mode and predicts its next tool call from partial trajectories in speculator mode, fully reusing prefix KV cache. To enable this dual-mode agent without degrading performance, we propose a joint agent-speculator reinforcement learning method, which derives speculation targets from the agent's own rollouts and alternates agent and speculator updates. Across agentic search QA and conversational tool-use agentic tasks, our method improves average next tool-call Hit@1 from 44.1 to 61.2 for Qwen3-4B and from 48.9 to 66.3 for Qwen3.5-4B, while preserving agent task success.

Jiabao Ji, Yujian Liu, Li An +4

University of California, Santa Barbara · Linkedin Inc

Tool-Using Agents LLM Agent Training RL for Tool Use

Ask PDF

Jul 14, 2026

Speculate with Memory: Lossless Acceleration for LLM Agents

Speculative execution accelerates LLM agents by using a smaller, cheaper model to predict and pre-launch the next step while the environment is idle. However, existing speculators are stateless and discard all information between tasks, preventing prediction quality from improving with experience. We equip the speculator with three online memory systems that learn from past agent trajectories: a contrastive transition table tracking action-sequence statistics, an episodic memory retrieving contextually similar segments, and a confusion tracker suppressing recurring errors. We evaluate this approach on six benchmarks spanning three speculation types: action prediction, observation prediction, and chained prediction. Memory-augmented speculation yields a 19--39% relative accuracy improvement on action prediction and up to a

2.5\times

increase on observation prediction tasks with repetitive action spaces. These gains grow continuously as memory accumulates and generalize across speculator models of varying cost. All speculation is lossless because it runs during idle time at zero added wall-clock cost, and the actor's trajectory is identical to non-speculative execution.

Yu Li, Qinyuan Ye, Prafulla Kumar Choubey +2

Salesforce Research

Next-Action Prediction LLM Agent Memory LLM Inference Acceleration Memory-Augmented Agents

Ask PDF

Abstract

Agentic RL

Explore similar work

AOSpec: Action and Observation Co-Speculation for Low-Latency Agent Serving

Speculate While You Reason: Teaching Agents to Predict Their Next Tool Call via Joint Agent-Speculator RL

Speculate with Memory: Lossless Acceleration for LLM Agents