Self-evolving LLM agents typically improve how they act through accumulated experience, reflection, or skills. Yet even procedurally capable agents can fail when their beliefs about environment dynamics are incomplete, incorrect, or outdated. We study a complementary axis of self-evolution: evolving those beliefs during deployment. Such beliefs can support multiple tasks, but incorrect ones can also propagate across them. We introduce EvoBelief, a self-evolving world model that represents environment knowledge as falsifiable hypotheses. Unexpected outcomes and decision-relevant unknowns trigger hypothesis formation. Later observations and controlled experiments provide independent validation before a rule can guide predictions or plans within its supported scope. Counterexamples narrow or retire rules as the environment changes. The language model remains frozen, and each rule remains linked to its evidence and the decisions it changes. Across four benchmarks spanning static dynamics, hidden drift, and scientific discovery, EvoBelief improves task performance and reuses learned rules across tasks. On ALFWorld without actor-side transition memory, it improves success over the matched Frozen World Model by 23.9% and 31.3% with Claude Haiku 4.5 and GPT-5.4, respectively. On MiniGrid, it revises stale rules and recovers after unannounced dynamics changes. In DiscoveryWorld's Reactor Lab, it discovers and validates quantitative laws, completing 83.3% of Normal-tier and 58.3% of Challenge-tier instances, versus 0% for each evaluated baseline. These results demonstrate self-improvement through testable, revisable environment knowledge rather than model-parameter updates.
Figures & tables
Figure 1: Illustrative example. (a) Following its frozen prior, a capable agent heats compound X to 100 ∘ C and gets decomposition products. (b) The mismatch triggers a hypothesis, which controlled experiments test. (c) The verified belief is stored in the persistent world model W and changes later actions in the original task and in new ones, without any parameter update. Based on the updated belief, the agent avoids gas chromatography (GC), whose operating temperature exceeds the supported range, and correctly chooses to use liquid chromatography (LC) instead.
Figure 2: Verified belief revision closes the loop from experience to later behavior. EvoBelief uses the current zt and Wt to predict action consequences. A grounded prediction mismatch or a decision-critical unknown opens an evidence loop that forms falsifiable hypotheses, gathers evidence, and verifies their scope. Only validated, applicable rules guide predictions and plans.
ALFWorld
ScienceWorld
Setting
Method
Haiku 4.5
GPT-5.4
Haiku 4.5
GPT-5.4
No memory
ReAct
33.58
46.27
55.56
41.11
ITP-I ( Liu et al., 2026 )
29.85
39.55
42.22
36.67
WorldEvolver ( Zhang et al., 2026b )
35.07
61.94
52.22
65.56
Frozen WM
42.54
44.78
46.67
53.33
EvoBelief
66.42
76.12
64.44
71.11
Table 1: Success rate (%) with and without actor-side transition memory, on ALFWorld and ScienceWorld for both backbones. All methods are run through the same harness, interface, and budgets.
Figure 3: A rule-induced decision change on a real ALFWorld trace (Haiku 4.5, unseen task). Frozen WM predicts that cooling succeeds, executes the draft to no effect, and never recovers. EvoBelief uses a validated cooling rule, moves to the fridge first, and completes the task in ten decisions.
Figure 4: Learning over deployment and efficiency on ALFWorld. (a) Tasks solved so far over the 134 unseen tasks (Haiku 4.5, no transition memory). Shaded blocks are the six task types, and dotted and dashed lines mark when the generic and the clean, cool, and heat rules were committed. (b) Decisions saved by EvoBelief over Frozen WM and WorldEvolver. (c, d) Mean decisions and model calls per episode for both backbones, without and with memory (lower is better).
Figure 5: The two central MiniGrid drift families. Each episode is a two-room grid with a locked door, three keys, and a target box in the far room. Left: the drift shared by every family. Only the ringed key unlocks the door, blue for episodes 1–20 and green for 21–40. The switch is silent, and the red key that matches the door is a decoy. Right: the preconditions each family requires to complete the task of lifting the box. In MG-C the box lifts only with the door shut and hands empty. In MG-M the toggle only unlocks the door, and a second, empty-handed toggle opens it. Dark red marks steps the pretrained prior does not expect. Figure 8 in the appendix shows all three families.
MG-C
MG-M
MG-R
Method
Pre-drift
Post-drift
Pre-drift
Post-drift
Pre-drift
Post-drift
Frozen WM
0.00
0.00
1.67
0.00
48.33
50.00
WorldEvolver
0.00
0.00
6.67
16.67
20.00
26.67
EvoBelief
61.67
95.00
78.33
81.67
83.33
78.33
Table 2: MiniGrid task success (%) before and after the hidden key switch. Each cell pools 60 episodes from three 40-episode streams per family; MG-C and MG-M use development seeds and MG-R uses fresh control seeds. Haiku 4.5, without transition memory. Higher is better.
Figure 6: Adaptation over deployment on MiniGrid. (a) Success by episode. The dashed line marks the silent key switch after episode 20. (b) EvoBelief decisions per episode by source: pretrained prior, active experiment, or committed plan. Both panels average the six development streams from MG-C and MG-M.
Figure 7: Substrate ablation and held-out prediction. (a) Success on the ScienceWorld tasks of Table 1 with GPT-5.4 and no transition memory, by persistent substrate. Hatched bars store no explicit rules. (b) Structured state-change prediction on 1,500 held-out transitions per benchmark with Claude Haiku 4.5. State- Δ F1 measures how well the predicted state changes match the realized ones, Feasibility whether the action is correctly predicted to execute, and exact match (EM) and token F1 compare the predicted and realized state records. Exact prediction scores are in Table 15 .
Normal
Challenge
Method
Completion
Knowledge
Procedural
Completion
Knowledge
Procedural
Reactor Lab
ReAct
0.00
0.00
70.33
0.00
0.00
64.66
WorldEvolver
0.00
0.00
72.67
0.00
0.00
69.54
Hypothesizer
0.00
0.00
74.00
0.00
0.00
68.39
EvoBelief
83.33
79.17
89.00
58.33
62.50
84.20
Table 3: DiscoveryWorld results with GPT-5.4 on twelve fresh task instances per task family and difficulty tier. Completion, Knowledge, and Procedural are the benchmark’s task-completion, blind knowledge, and subgoal-progress scores, in percent.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Hypothesis source
Behavioral route
Experiment route
Rule-conditioned call
ALFWorld
Typed transitions; code induction
Rule-based planning; precondition veto
Passive evidence only
None
MiniGrid
Typed transitions; code induction
Validated plan steps replace the draft
Live experiments scheduled by code
None
ScienceWorld
Public observations and typed transitions; code induction
Rule-based control, observation obligations, and veto
Live experiments admitted by code
None
DiscoveryWorld
Laws fitted by code from measurements and observations
Table 4: Hypothesis source, behavioral route, experiment route, and use of the frozen engine’s rule-conditioned model call on each benchmark. Calls per decision by role are in Table 5 .
Benchmark
Actor
Alternatives
Anchor
Rule-conditioned
Analysis
ALFWorld
1
code
1 (batched)
none
code
MiniGrid
1
code
1 per candidate (5)
none
1
ScienceWorld
1
1
1+N
none
1
DiscoveryWorld
1
1
1+N
up to 1+N
1
Appendix
Table 5: Language-model calls per decision, by role. N is the number of W-blind alternative candidates (at most 4 on ScienceWorld, 2 on DiscoveryWorld). Per-episode auxiliary calls are a state seed at episode start, hypothesis proposal after a mismatch, trajectory distillation after a success, and role binding for the engine’s own experiments. Call counts are taken from one configuration per benchmark, Haiku 4.5 on ALFWorld and MiniGrid and GPT-5.4 on ScienceWorld and DiscoveryWorld; the roles and calls per decision do not depend on the backbone.
ALFWorld
MiniGrid
ScienceWorld
DW Reactor
DW Plant
Hypotheses formed (claims)
12
639
236
669
324
Passive supports (records)
230
2,156
4,031
–
–
Live experiments (runs)
0
458
54
96
51
Validated rules (distinct)
10
220
36
18
38
Counterexamples (records)
9
10,645
21
296
12
Retirements (rule versions)
2
845
70
–
15
Appendix
Table 6: What the belief machinery did, by benchmark. One setting per benchmark: ALFWorld, Haiku 4.5 without transition memory (134 tasks); MiniGrid, the six development streams (240 episodes); ScienceWorld, GPT-5.4 without transition memory (90 tasks); DiscoveryWorld Reactor Lab and Plant Nutrients, GPT-5.4 (24 task instances each). Each row counts the unit named in its label, and the rows are not stages of one funnel. ALFWorld: 10 validated rules over 43 validation events. MiniGrid: 220 distinct validated rules over 2,185 validated rule versions, and counterexamples are negative witness records. ScienceWorld: 54 experiments opened, 39 completed, yielding 573 support records, while the 4,031 passive supports come from ordinary task transitions. Supports are counted per occurrence, so a claim observed repeatedly in one context adds many records but little independent evidence, which is why validated rules are far fewer than support records. Confirmed: the predicted effect held in the next observation, over the decisions a rule changed on ALFWorld, all plan steps on MiniGrid, the scorable rule predictions on ScienceWorld, law-derived plans on Reactor Lab, and positive consequence verifications on Plant Nutrients. On DiscoveryWorld, the counterexample row counts rival laws falsified. ScienceWorld’s procedural safeguards for task milestones and irreversible steps, fixed before evaluation (§ 4.1 ), are not belief actions and are not counted. A dash means the implementation does not record the quantity.
Benchmark, setting
Baseline
n
Ours
Base
Δ
95% CI
Disc.
p
ALFWorld, Haiku, no memory
Frozen WM
134
66.42
42.54
23.88
16.42, 31.34
34 / 2
< 0.001
WorldEvolver
134
66.42
35.07
31.34
21.64, 41.04
51 / 9
< 0.001
ALFWorld, Haiku, memory
Frozen WM
134
68.66
57.46
11.19
2.99, 20.15
25 / 10
0.017
WorldEvolver
134
68.66
58.96
9.70
1.49, 17.91
24 / 11
0.041
ALFWorld, GPT-5.4, no memory
Frozen WM
134
76.12
44.78
31.34
23.13, 40.30
45 / 3
< 0.001
WorldEvolver
134
76.12
61.94
14.18
5.22, 23.13
29 / 10
0.003
Appendix
Table 7: Paired tests for the matched comparisons. Success rates in percent; Δ is EvoBelief minus the baseline in percentage points with its 95% percentile bootstrap interval over paired episodes; Ours and Base are the two success rates, Disc. counts the episodes only EvoBelief solved against those only the baseline solved; p is the exact two-sided McNemar test on those pairs. ALFWorld pairs the 134 unseen tasks, WorldEvolver being the authors’ release; ScienceWorld pairs its 90 tasks; MiniGrid pairs episodes by stream and position, 240 per seed set, and the post-versus-pre rows compare EvoBelief’s second half with its own first half, pairing episode i+20 with episode i of the same stream; MiniGrid episodes within a stream share an evolving belief state, and the episode-level tests and intervals do not account for this dependence; DiscoveryWorld pairs the 12 fresh task instances per tier of each theme, with the Hypothesizer at its raised output ceiling.
Setting
Rules
Rule-use decisions
Later task
Other family
Type unseen at formation
Changed
Confirmed
Haiku, no memory
10
707
99.86
82.46
88.12
324
81.79
Haiku, memory
8
753
99.87
85.79
90.17
250
78.40
GPT, no memory
8
774
99.87
85.66
91.34
304
83.22
GPT, memory
8
789
99.75
86.69
90.87
237
81.43
Appendix
Table 8: Where committed rules are used on ALFWorld. Rule-use decisions: decisions whose executed action a validated rule licensed. Later task: a task after the one in which the rule was first validated. Other family: the task family differs from that of the task in which the rule was validated. Type unseen at formation: the executed action binds at least one role to an object type absent from every binding that formed the rule (instance numbers ignored). Changed: decisions under a validated rule whose executed action differs from the controller’s draft after normalizing the surface forms of put. Confirmed: the transition was not rejected and every atom the rule predicted appears in the next observation. Percentages are of decisions under a validated rule, and of changed decisions for the last column.
Rule
Validation task
Tasks used
Families
Bindings
Changed
Confirmed
go to (contents)
1, look at in light
115
6
26
221
170
take
2, look at in light
109
6
83
20
20
put
7, look at in light
91
5
63
4
4
open
24, pick and place
63
5
8
8
8
go to (location)
21, pick and place
33
5
10
33
33
clean
44, clean then place
25
1
16
4
4
Appendix
Table 9: The rules of one setting, GPT-5.4 without transition memory. Validation task: the index of the task in the deployment order at which the rule first earned authority, with that task’s family. Tasks used: distinct later tasks with a decision taken under the rule. Bindings: distinct object and receptacle instances the rule was bound to. The two go-to rules differ in their effect: one predicts that reaching a receptacle reveals its contents, the other only the new location. Clean, heat, and cool rules can only be used in their own family.
Seeds
Method
Pre-switch
Post-switch
All
Held-out
Frozen WM
2.50
0.00
1.25
EvoBelief
63.33
75.83
69.58
Held-out, permuted labels
Frozen WM
10.00
4.17
7.08
EvoBelief
74.17
87.50
80.83
Appendix
Table 10: MiniGrid on held-out seeds (%). Haiku 4.5, actor without transition memory. Three seeds on both families, fixed before any run and never used while building the method, 240 episodes per method per block; the permuted block renames every color and room in the observations of both methods. Higher is better.
Seeds
Variant
Pre-switch
Post-switch
All
Development
Planner without learned rules
0.83
0.00
0.42
No active experiments
0.83
0.00
0.42
EvoBelief
70.00
88.33
79.17
Mechanism
Rules never executed (shadow)
50.00
64.17
57.08
No commitment gate
66.67
70.00
68.33
EvoBelief
61.67
77.50
69.58
Appendix
Table 11: What the MiniGrid gain needs (%). Each block compares variants of EvoBelief on one seed set, 240 episodes per row. The planner without learned rules never leaves the start state of its search and defers to the prior on every decision; without active experiments, rules never gather independent support; in the shadow variant rules are learned and validated but no plan step is executed; without the commitment gate a rule may act on its first supported instance. Higher is better.
Figure 8: What each MiniGrid family requires. One row per family, each frame a state of the two-room grid: the robot is the agent, a key drawn next to it is the key it carries, the box drawn next to it is the lifted target. A padlock marks a locked door, a plain door is shut but unlocked, and a door drawn aside is open. In every family the red key matches the door and never works, the blue key opens the door for the first twenty episodes, and after episode 20 the green key does instead; the ring marks the key that opens the door and the switch is never announced. In MG-R (R for rebind) one toggle with the working key unlocks and opens the door, and nothing more is asked. In MG-C (C for conjunctive) the door behaves the same, but the box lifts only with the door shut and the agent’s hands empty, so after entering the agent must drop the key and close the door it just opened. In MG-M (M for multi-stage) toggling with the working key only unlocks the door, which stays shut until a second toggle with empty hands. The switch is the only change within a stream; the family rules hold in both halves. The steps in dark red are the ones the pretrained prior does not expect.
Normal
Challenge
Hypotheses frozen
236
433
Experiments executed
48
48
Rivals falsified
157
139
Laws validated
10
8
Task instances with a validated law
10
8
Task instances whose law-derived plan passed
10
7
Appendix
Table 12: What EvoBelief did on the fresh task instances. Counts from the sealed science records of the twelve task instances per tier; the baselines have no such record by construction. A hypothesis is a candidate law with a prediction frozen before any experiment. One executed experiment settles every rival whose frozen prediction its realized outcome contradicts, so fewer experiments falsify more rivals, and a law is validated only by a held-out prediction the reactor confirms, never by fit quality alone. Model calls and environment actions are totals over the tier.
Seed
Validated law
x
Predicted
Dialed
Rivals
Plan
Knowledge
Normal
s9
f=104.99c5+107.25
30.50
3309.50
3310.00
18
pass
1.00
s10
f=99.01d+102.24
61.31
6172.69
6173.00
14
pass
1.00
s11
f=105.00t+95.96
17.81
1966.05
1966.05
18
pass
1.00
s12
f=105.01q+90.40
49.72
5311.60
5312.00
16
pass
1.00
s13
none (radiation seed)
7
–
0.00
Appendix
Table 13: The law validated on each fresh task instance. f is the reactor resonance in Hz; c5 , d , t , and q are the crystal’s spectrometer channel 5, density, temperature, and quantum gap. x is the held-out crystal’s reading, Predicted the prediction frozen before the reactor was dialed, and Dialed the frequency at which the reactor then activated. Rivals is the number of rival laws falsified on that task instance, Plan the outcome of the plan derived from the law for the target crystal (a passed plan completes the task), and Knowledge the judge’s score on that task instance. Normal seeds 13 and 18 and Challenge seeds 9, 14, and 19 hide the variable in a radiation reading with random background; on Challenge seed 20 no rival survived within budget, and on seed 17 the law was validated but its plan never settled.
Family
Tier
Sealed
Completion
Knowledge
Procedural
Reactor Lab
Normal
12 of 12
0.00
0.00
74.33
Reactor Lab
Challenge
2 of 12
0.00
0.00
74.14
Plant Nutrients
Normal
12 of 12
66.67
0.00
82.41
Plant Nutrients
Challenge
1 of 12
0.00
0.00
66.67
Appendix
Table 14: The Hypothesizer at its original 3,072-token output ceiling. Same metrics and task instances as Table 3 , in percent; the Challenge runs ended on output truncation, and their means are over the task instances sealed before the failure.
Model
State- Δ F1 ↑
Feasibility ↑
EM ↑
Token F1 ↑
ScienceWorld
Frozen WM (no evolving W )
0.7258
0.7547
0.6013
0.7866
WorldEvolver (semantic-only)
0.7061
0.7827
0.6027
0.7814
WorldEvolver (full)
0.6680
0.8753
0.5993
0.7820
EvoBelief
0.7530
0.8113
0.6407
0.8045
ALFWorld
Frozen WM (no evolving W )
0.7572
0.8453
0.6567
0.8080
WorldEvolver (semantic-only)
0.6862
0.8500
0.5733
0.7607
Appendix
Table 15: Held-out structured state-change prediction on ScienceWorld and ALFWorld, 1,500 transitions per benchmark, disjoint from belief formation. State- Δ F1 measures canonical state changes; Feasibility measures action executability; EM and Token F1 compare structured state. Haiku 4.5.
World models offer a principled way to equip long-horizon LLM agents with foresight: predictions of action consequences before execution. However, unreliable foresight can be ignored, misused, or even degrade downstream decision-making. In this paper, we introduce WorldEvolver, a self-evolving world model framework that revises its deployment-time context while keeping the downstream agent and all model parameters frozen. WorldEvolver integrates three modules: (i) Episodic Memory, which exploits real action transitions through retrieval-based simulation; (ii) Semantic Memory, which extracts persistent heuristic rules from prediction-observation mismatches; and (iii) Selective Foresight, which filters low-confidence predictions before integrating them into agent reasoning context. We evaluate WorldEvolver on ALFWorld and ScienceWorld, measuring world model prediction accuracy on Word2World and downstream agent success rate on AgentBoard. Extensive experiments show that WorldEvolver achieves the highest prediction accuracy across three backbones and leads other world model baselines on downstream agent success rate, demonstrating that test-time memory revision enhances both predictive fidelity and planning performance.
Xuan Zhang, Wenxuan Zhang, See-Kiong Ng +1
National University of Singapore · Singapore University of Technology and Design · Singapore Management University
Most agents today ``self-evolve'' by following rewards and rules defined by humans. However, this process remains fundamentally dependent on external supervision; without human guidance, the evolution stops. In this work, we train agents to possess an intrinsic meta-evolution capability to spontaneously learn about unseen environments prior to task execution. To instill this ability, we design an outcome-based reward mechanism that measures how much an agent's self-generated world knowledge improves its success rate on downstream tasks. This reward signal is used exclusively during the training phase to teach the model how to explore and summarize effectively. At inference time, the agent requires no external rewards or human instructions. It spontaneously performs native self-evolution to adapt to unknown environments using its internal parameters. When applied to Qwen3-30B and Seed-OSS-36B, this shift to native evolution yields a 20% performance increase on WebVoyager and WebWalker. Most strikingly, the generated world knowledge even enables a compact 14B Qwen3 model to outperform the unassisted Gemini-2.5-Flash, establishing a new paradigm for truly evolving agents.
Qifan Zhang, Dongyang Ma, Tianqing Fang +5
1Tencent · 2The Hong Kong University of Science and Technology (Guangzhou)
Self-evolving agents are expected to improve through interaction without external supervision, but this remains difficult in partially observable environments where agents must explore actively, learn from limited feedback, and decide when to trust prior experience. Existing LLM-agent methods often rely on memory or planning modules, yet they rarely close the loop between them to continually refine an internal understanding of environment dynamics. We introduce ProPlay, a procedural world model that supports procedure-level preplay, where agents can rehearse future procedural paths using the learned world knowledge. Rather than representing experience as isolated rules or low-level action constraints, ProPlay abstracts successful trajectories into procedures and organizes them in a procedure graph that captures causal transitions among task stages. Each transition is associated with a reliability record embedding to estimate its task-specific contribution from past outcomes. Before each episode, ProPlay simulates future procedural trajectories over known graph structures as structured soft guidance; after execution, it refines the graph using environment feedback. Experiments on public benchmarks show that ProPlay consistently improves environment understanding and self-evolution capability over strong baselines. Our code has been released in https://github.com/antman9914/proplay.
Yijun Ma, Zehong Wang, Yiyang Li +5
University of Notre Dame · University of Connecticut