ObsSpec: Accelerating Agentic Reinforcement Learning with Observation Speculation

Kyle Montgomery; Wang Zhang; Chenguang Wang

cs.LGOct 11, 2026

ObsSpec: Accelerating Agentic Reinforcement Learning with Observation Speculation

Authors: Kyle Montgomery, Wang Zhang, Chenguang Wang

Organizations: UC Santa Cruz · ByteDance Inc.

Abstract

Tool-using agents alternate between model generation and tool execution, waiting for each tool call to complete before starting the next generation. This sequential execution is costly during agentic reinforcement learning (RL), which requires large batches of multi-turn rollouts. We introduce ObsSpec, an observation speculation pipeline for agentic RL that uses a small world model to predict tool outputs while the environment executes, allowing the policy to begin its next generation early. ObsSpec retains speculative work only when the predicted and actual observations match exactly, making rollout collection lossless, while co-training keeps the world model up to date as the policy learns. On terminal-use tasks, predictions from a 0.6B world model exactly match actual observations in 66.4% of attempts and arrive roughly 16 times faster. Using these predictions, ObsSpec reduces agent latency by 9.3% and 14.2% for 8B and 32B policies. Across controlled experiments, we characterize the regimes in which ObsSpec reduces agent latency under shared policy resources. Finally, because ObsSpec overlaps policy generation with tool execution, it is architecturally compatible with existing techniques for accelerating RL. Code is available at https://github.com/kylemontgomery1/verl-recipe/tree/obsspec/obsspec.

Figures & tables

Panel of Figure 1 — Figure 1: Across policy sizes, ObsSpec reduces latency and increases asynchronous rollout throughput with similar rewards.

Figure 2: Comparison of the sequential and ObsSpec pipelines on a terminal task. In the sequential pipeline (top), policy generation and tool execution alternate, so generation of each action cannot begin until the previous action finishes. In our ObsSpec pipeline (bottom), a world model predicts each observation while the corresponding action executes, allowing the policy to generate the next action before the execution of the previous action finishes. The first predicted observation $\hat{o}_{1}$ matches the actual observation $o_{1}$ , whereas $\hat{o}_{2}$ differs from $o_{2}$ . Consequently, the in-progress generation of $a_{3}$ is canceled and restarted from the actual observation $o_{2}$ . By overlapping policy generation with tool execution, ObsSpec reduces agent latency by up to 14.2%.

Panel of Figure 3 — Figure 3: Observation prediction outcomes (top) and latency (bottom) for common commands.

Panel of Figure 4 — Figure 4: Effect of world model size with the 8B policy at $D=K=1$ . Larger models reduce rollbacks but take longer to predict and do not reduce agent latency.

Panel of Figure 5 — Figure 5: Depth sweep with the 8B policy and 0.6B world model. Greater depth increases realized speculation and discarded policy computation without reducing agent latency.

Panel of Figure 6 — Figure 6: Breadth sweep with the 8B policy and 0.6B world model. Greater breadth increases realized speculation and match rate, while reducing the policy decode rate and increasing agent latency.

Panel of Figure 7 — Figure 7: Policy improvement and the effects of co-training the world model on match rate and agent latency.

Panel of Figure 8 — Figure 8: Sensitivity to sandbox infrastructure with the 8B policy and 0.6B world model.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Data
Batch size	16 prompts
Group size	16 rollouts per prompt
Prompt budget	2,048 tokens
Response budget	16,384 tokens
Per-turn generation budget	2,048 tokens

Appendix

Table 1: Training and serving configuration for the policy and world model.

Policy	Pipeline	Reward (%) $\uparrow$	Agent (s) $\downarrow$	Batch (s) $\downarrow$	Rollouts/h $\uparrow$
Qwen3-8B	Sequential	$78.64\pm 0.64$	$41.98\pm 0.39$	$156.26\pm 2.94$	$13{,}762\pm 156$
Qwen3-8B	ObsSpec	$79.76\pm 0.63$	$38.06\pm 0.38$	$149.42\pm 3.06$	$14{,}571\pm 175$
Qwen3-32B	Sequential	$84.62\pm 0.56$	$42.59\pm 0.46$	$185.63\pm 2.40$	$14{,}066\pm 155$
Qwen3-32B	ObsSpec	$83.91\pm 0.57$	$36.53\pm 0.44$	$171.16\pm 2.81$	$15{,}689\pm 199$

Appendix

Table 2: Full results underlying Figure 1 .

Command	Count	Tool (s)	WM (s)	Match (%)	Rollback (%)	Cancel (%)
cat	19,544	$1.597\pm 0.006$	$0.113\pm 0.001$	$79.95\pm 0.29$	$16.75\pm 0.27$	$3.31\pm 0.13$
ls	10,681	$1.614\pm 0.008$	$0.121\pm 0.001$	$39.43\pm 0.47$	$59.54\pm 0.47$	$1.03\pm 0.10$
mkdir	2,943	$1.607\pm 0.016$	$0.036\pm 0.000$	$96.67\pm 0.33$	$3.33\pm 0.33$	$0.00\pm 0.00$
echo	1,169	$1.612\pm 0.026$	$0.038\pm 0.000$	$98.37\pm 0.37$	$1.63\pm 0.37$	$0.00\pm 0.00$
sha256sum	946	$1.625\pm 0.028$	$0.178\pm 0.002$	$35.20\pm 1.55$	$50.00\pm 1.63$	$14.80\pm 1.15$
sqlite3	668	$1.631\pm 0.033$	$0.102\pm 0.004$	$78.89\pm 1.58$	$15.57\pm 1.40$	$5.54\pm 0.89$

Appendix

Table 3: Full results underlying Figure 3 .

WM	WM latency (s)	Agent latency (s)	Match	Rollback	Cancel
0.6B	$0.105\pm 0.001$	$38.06\pm 0.38$	$66.45\pm 0.22$ %	$30.50\pm 0.21$ %	$3.05\pm 0.08$ %
1.7B	$0.137\pm 0.001$	$37.84\pm 0.37$	$68.50\pm 0.21$ %	$28.71\pm 0.21$ %	$2.80\pm 0.08$ %
4B	$0.239\pm 0.001$	$37.89\pm 0.36$	$68.74\pm 0.21$ %	$24.88\pm 0.20$ %	$6.38\pm 0.11$ %
8B	$0.350\pm 0.002$	$37.95\pm 0.37$	$66.17\pm 0.22$ %	$22.47\pm 0.19$ %	$11.36\pm 0.15$ %

Appendix

Table 4: Full results underlying Figure 4 .

Depth	Agent latency (s)	Realized depth	Policy tok/s	Discarded policy tok.
1	$38.06\pm 0.38$	$0.992\pm 0.001$	$145.5\pm 0.1$	$198.0\pm 3.4$
2	$38.27\pm 0.38$	$1.871\pm 0.006$	$143.0\pm 0.1$	$342.5\pm 5.8$
4	$38.02\pm 0.37$	$2.773\pm 0.017$	$140.5\pm 0.1$	$426.7\pm 7.5$
8	$38.47\pm 0.39$	$3.208\pm 0.029$	$139.5\pm 0.1$	$472.2\pm 9.8$

Appendix

Table 5: Full results underlying Figure 5 .

$K$	Agent latency (s)	Match	Realized breadth	Policy tok/s	Discarded policy tok.
1	$38.06\pm 0.38$	$66.45\pm 0.22$ %	1.0	$145.5\pm 0.1$	$198.0\pm 3.4$
2	$41.88\pm 0.40$	$67.36\pm 0.22$ %	$1.303\pm 0.002$	$126.2\pm 0.1$	$332.8\pm 5.8$
4	$44.60\pm 0.44$	$68.56\pm 0.21$ %	$1.940\pm 0.006$	$116.8\pm 0.1$	$574.9\pm 10.3$
8	$47.68\pm 0.46$	$69.01\pm 0.21$ %	$3.142\pm 0.013$	$107.5\pm 0.1$	$994.9\pm 19.0$

Appendix

Table 6: Full results underlying Figure 6 .

Policy step	Reward (%)	Updated match (%)	Frozen match (%)	Sequential agent (s)	Updated agent (s)	Frozen agent (s)
100	$58.95\pm 2.52$	$59.07\pm 0.23$	$59.07\pm 0.23$	$23.14\pm 0.42$	$21.35\pm 0.40$	$21.35\pm 0.40$
200	$68.68\pm 2.38$	$62.64\pm 0.23$	$57.88\pm 0.24$	$23.20\pm 0.27$	$21.44\pm 0.26$	$21.70\pm 0.30$
300	$73.95\pm 2.25$	$59.87\pm 0.22$	$54.19\pm 0.22$	$28.03\pm 0.33$	$24.99\pm 0.30$	$25.22\pm 0.28$
400	$76.84\pm 2.16$	$64.01\pm 0.21$	$55.96\pm 0.21$	$32.47\pm 0.31$	$28.92\pm 0.28$	$29.60\pm 0.29$
500	$80.26\pm 2.04$	$66.45\pm 0.22$	$57.35\pm 0.23$	$41.98\pm 0.39$	$38.06\pm 0.38$	$39.74\pm 0.39$

Appendix

Table 7: Full results underlying Figure 7 .

Sandbox	Tool latency (s)	Sequential agent (s)	ObsSpec agent (s)	Reduction
Default	$1.69\pm 0.01$	$41.98\pm 0.39$	$38.06\pm 0.38$	9.3%
Modal	$2.56\pm 0.01$	$52.17\pm 0.44$	$46.66\pm 0.51$	10.6%
Daytona	$2.99\pm 0.02$	$56.30\pm 0.59$	$49.94\pm 0.54$	11.3%

Appendix

Table 8: Full results underlying Figure 8 . All measured quantities report means and standard errors.

Method	Reward (%)	Agent (s)	WM (s)	Match (%)	Rollback (%)	Cancel (%)
Sequential	$78.25\pm 0.64$	$55.82\pm 0.83$	–	–	–	–
ObsSpec , 0.6B	$78.47\pm 0.64$	$56.42\pm 0.86$	$0.120\pm 0.003$	$54.96\pm 0.42$	$40.42\pm 0.41$	$4.62\pm 0.18$
ObsSpec , 8B	$78.39\pm 0.64$	$55.79\pm 0.82$	$0.367\pm 0.007$	$59.47\pm 0.41$	$31.72\pm 0.39$	$8.81\pm 0.24$

Appendix

Table 9: Full results for the TIR experiments in Section 3.8 .

Agentic RL

Agentic reinforcement learning trains AI agents to plan, use tools, and adapt through feedback from multi-step interactions.

See all 231 papers

AOSpec: Action and Observation Co-Speculation for Low-Latency Agent Serving
Aug 1, 2026Hao Mark Chen, Jinnan Guo, Wayne Luk +1LLM Inference Acceleration Long-Horizon Agent Tasks
Imperial College London, UK
Speculate While You Reason: Teaching Agents to Predict Their Next Tool Call via Joint Agent-Speculator RL
Jul 28, 2026Jiabao Ji, Yujian Liu, Li An +4Tool-Using Agents LLM Agent Training
University of California, Santa Barbara · Linkedin Inc
Speculate with Memory: Lossless Acceleration for LLM Agents
Jul 14, 2026Yu Li, Qinyuan Ye, Prafulla Kumar Choubey +2Next-Action Prediction LLM Agent Memory
Salesforce Research

Aug 1, 2026J/K move · Enter open · S save

AOSpec: Action and Observation Co-Speculation for Low-Latency Agent Serving

Hao Mark Chen, Jinnan Guo, Wayne Luk, Hongxiang Fan

Imperial College London, UK

LLM Inference Acceleration Long-Horizon Agent Tasks

Open paper AskPDF arXiv

Large language model agents increasingly act through stateful tools, yet model generation and environment execution remain serialized at every step. As decoding accelerates, tool execution becomes a growing bottleneck. Existing action- or observation-only speculation leaves much of this latency exposed: value is concentrated in a few slow calls, some outcomes emerge only through execution, and longer lookahead typically requires an increasingly unlikely chain of action predictions. We present AOSpec, a lossless framework that co-speculates actions and observations across the full agent-environment loop. Expected Value Decoding (EVD) directs observation speculation toward outcomes with the greatest expected latency benefit, optimizing expected time hidden rather than hit rate. For outcomes only execution can reveal, AOSpec launches latency-critical target actions in isolated forks that contain their effects, while Joint Action-State Verification (JASV) verifies both the action and its origin state against committed execution before reuse. JASV recasts long-horizon action dependency from full-chain prediction into target action-state verification, breaking the lookahead--accuracy tradeoff and unlocking long-range overlap without sacrificing serial semantics. Across Terminal-Bench serving settings spanning four harnesses, five actor models, and five serving speeds, AOSpec outperforms every practical baseline, reducing mean end-to-end latency by 11.8-32.5% and p99 latency by up to 42.8%. Its gains increase as decoding accelerates, and its observation model transfers from Terminal-Bench to SWE-bench Verified without retraining.

Abstract

Agentic RL

Explore similar work

AOSpec: Action and Observation Co-Speculation for Low-Latency Agent Serving

Speculate While You Reason: Teaching Agents to Predict Their Next Tool Call via Joint Agent-Speculator RL

Speculate with Memory: Lossless Acceleration for LLM Agents