Large language model (LLM) agents are increasingly expected to interact with environments and improve their decisions during deployment. However, how anticipating future outcomes and drawing on accumulated interaction experience can jointly support decision-making remains underexplored. We propose a test-time framework, inspired by Model Predictive Path Integral (MPPI) control, that combines multi-branch predictive analysis with experience-based adaptation. At each step, the agent proposes multiple candidate actions and imagines their outcomes with a world model, so that alternatives can be compared without being executed in the environment. Since language environments provide neither explicit dynamics nor costs, the predicted outcomes are evaluated for task progress and risk, and MPPI-style weights over the resulting costs guide the final decision. A dual-branch experience memory enables self-improvement: the planner branch refines candidate generation and action selection, while the world model branch improves next-observation prediction. Experiments on ALFWorld and ScienceWorld across three backbones demonstrate the strong overall performance of our method and the effectiveness of experience-guided predictive control. Our code will be made publicly available.
Figures & tables
Figure 1: Overview of our method. At step t , the proposal policy takes the goal, history, and observation and proposes K candidate actions at(k) . The world model predicts the next observation o^t+1(k) of each candidate, the evaluator scores task progress and risk to give costs Ct(k) , and MPPI-style weights summarize these costs for the final controller, which decides the executed action at . After execution, the transition updates both experience branches. The planner branch labels it as success, failure, or ignore and summarizes the first two into Mtcand and Mtfinal , retrieved by the proposal policy and final controller in future steps; the world model branch updates its episodic and semantic memory Mtworld , retrieved by the world model in future steps.
Table 1: Overall success rates (%) on ALFWorld and ScienceWorld across three backbones, reported as SR@1 and SR@5. Bold and underlined success rates indicate the best and second-best results, respectively.
Figure 2: Executed actions by candidate rank under MPPI-style weighting with Gemma-4-26B-A4B. Rank 1 includes ties for the highest weight; Outside denotes actions not in the proposal set.
Figure 3: An example of how retrieved experience and predictive evaluation guide action selection. Our agent completes the focus subgoal before pickup and succeeds, while ReAct skips this subgoal and fails.
K
ALFWorld
ScienceWorld
Avg.
SR@1
SR@5
SR@1
SR@5
SR@1
SR@5
1
60.4
82.1
43.3
58.9
51.9
70.5
2
59.7
85.1
38.9
56.7
49.3
70.9
4
61.9
82.8
40.0
58.9
51.0
70.9
8
65.7
84.3
46.7
60.0
56.2
72.2
16
64.2
84.3
42.2
68.9
53.2
76.6
Table 2: Effect of the number of candidate actions K imagined per step, with Gemma-4-26B-A4B (%). The default setting is shaded.
λ
ALFWorld
ScienceWorld
Avg.
SR@1
SR@5
SR@1
SR@5
SR@1
SR@5
1
65.7
81.3
48.9
62.2
57.3
71.8
10
65.7
84.3
46.7
60.0
56.2
72.2
50
53.7
80.6
47.8
63.3
50.8
72.0
Table 3: Effect of weight temperature λ with Gemma-4-26B-A4B (%). Larger λ gives more uniform MPPI-style weights. The default setting is shaded.
Memory
ALFWorld
ScienceWorld
Avg.
Planner
WM
SR@1
SR@5
SR@1
SR@5
SR@1
SR@5
×
×
56.7
80.6
38.9
57.8
47.8
69.2
✓
×
49.3
73.9
44.4
60.0
46.9
67.0
×
✓
68.7
88.8
36.7
54.4
52.7
71.6
✓
✓
65.7
84.3
46.7
60.0
56.2
72.2
Table 4: Effect of planner and world model (WM) memory with Gemma-4-26B-A4B (%). ‘ ✓ ’ indicates an enabled branch. The default setting is shaded.
Figure 4: Cumulative SR@1 over executed tasks with and without dual-branch memory. Shaded areas show the gain in percentage points.
C
R
ALFWorld
ScienceWorld
SR@1
SR@5
SR@1
SR@5
4
4
62.7
85.8
36.7
60.0
12
4
64.9
85.8
44.4
58.9
24
4
65.7
84.3
46.7
60.0
36
4
63.4
84.3
42.2
64.4
24
1
64.2
89.6
46.7
63.3
Table 5: Effect of planner memory capacity C and retrieval count R with Gemma-4-26B-A4B (%). The upper block varies C at R=4 , the lower block varies R at C=24 . The default setting is shaded.
Method
Avg. sec. / step ↓
SR@1 ↑
SR@5 ↑
ReAct
0.51
20.1
24.6
ITP-I
1.94
32.8
54.5
WorldEvolver
1.33
16.4
20.9
Ours
6.72
65.7
84.3
Table 6: Per-step latency and success rates using Gemma-4-26B-A4B on 2 × NVIDIA A40 GPUs.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Table 7: Success rates (%) on ALFWorld by task type across three backbones. Each cell reports SR@1 / SR@5. Bold and underlined overall success rates indicate the best and second-best results, respectively.
Frontier black-box Large Language Model (LLM) agents are proficient at long-horizon reasoning and precise action generation. However, when it comes to a specific agentic task, they have no internalized knowledge of how that particular environment unfolds. A world model offers a natural remedy, providing the agent with environment specific knowledge by foreseeing how the environment will respond to the action it is about to take. Prior world models for LLM agents mainly predict next observations at each step, which is costly and often uninformative because individual observations provide limited task-relevant guidance and not every step requires call for foresight. This raises a key question: what a world model should predict and when should it be invoked for agentic tasks? Our pilot study suggests that world model guidance is most useful when it anticipates pitfalls within a bounded multi-step future. Motivated by this, we introduce Pitfall Anticipation World Model (PAWM), which selectively intervenes when a potential pitfall arises and predicts the impending failure to steer the agent away from it. PAWM is trained in two stages: Supervised Fine-Tuning (SFT) on synthetic data labeled with agent pitfalls, followed by GRPO co-training with a black-box LLM to further optimize it in the target environment. PAWM improves Claude Sonnet 4.5 performance by 24.3% on ScienceWorld and 7.7% on WebArena-Lite, outperforming a range of baselines. These results demonstrate that PAWM provides an effective form of world model guidance for black-box LLM agents.
World models offer a principled way to equip long-horizon LLM agents with foresight: predictions of action consequences before execution. However, unreliable foresight can be ignored, misused, or even degrade downstream decision-making. In this paper, we introduce WorldEvolver, a self-evolving world model framework that revises its deployment-time context while keeping the downstream agent and all model parameters frozen. WorldEvolver integrates three modules: (i) Episodic Memory, which exploits real action transitions through retrieval-based simulation; (ii) Semantic Memory, which extracts persistent heuristic rules from prediction-observation mismatches; and (iii) Selective Foresight, which filters low-confidence predictions before integrating them into agent reasoning context. We evaluate WorldEvolver on ALFWorld and ScienceWorld, measuring world model prediction accuracy on Word2World and downstream agent success rate on AgentBoard. Extensive experiments show that WorldEvolver achieves the highest prediction accuracy across three backbones and leads other world model baselines on downstream agent success rate, demonstrating that test-time memory revision enhances both predictive fidelity and planning performance.
Xuan Zhang, Wenxuan Zhang, See-Kiong Ng +1
National University of Singapore · Singapore University of Technology and Design · Singapore Management University
Recent studies on world modeling for Large Language Model (LLM) agents typically formulate the learning objective as next-observation prediction. However, this objective ties supervision to what a transition happens to reveal, which may omit the dynamics most relevant to the agent's current decision. To bridge this gap, we propose Agent-Authored World Modeling (AAWM), a training procedure that constructs supervision from the policy's own decision needs. Specifically, at each state, the agent identifies what it needs to understand about the environment before acting. These needs drive the retrieval of relevant transition evidence across trajectories, which is then synthesized into training targets that capture decision-oriented dynamics instead of reconstructing the next observation. This aligns the training objective with the dynamics the policy needs before acting, not with the contents of the next observation. Experimental results validate the effectiveness of AAWM across multiple environments and training settings. These results show that decision-aware world-model targets provide a more effective learning signal than next-observation prediction.