EvoBelief: Falsifiable World Models for Self-Improving LLM Agents

Janet Wang; Jiang Guo; Yi Xiang; Jiaqing Yuan; Mingwen Dong; Hideo Kobayashi; Shuaichen Chang; Chandan K. Reddy

cs.AIOct 8, 2026

EvoBelief: Falsifiable World Models for Self-Improving LLM Agents

Authors: Janet Wang, Jiang Guo, Yi Xiang, Jiaqing Yuan, Mingwen Dong, Hideo Kobayashi, Shuaichen Chang, Chandan K. Reddy,

Organizations: Tulane University · Amazon

Abstract

Self-evolving LLM agents typically improve how they act through accumulated experience, reflection, or skills. Yet even procedurally capable agents can fail when their beliefs about environment dynamics are incomplete, incorrect, or outdated. We study a complementary axis of self-evolution: evolving those beliefs during deployment. Such beliefs can support multiple tasks, but incorrect ones can also propagate across them. We introduce EvoBelief, a self-evolving world model that represents environment knowledge as falsifiable hypotheses. Unexpected outcomes and decision-relevant unknowns trigger hypothesis formation. Later observations and controlled experiments provide independent validation before a rule can guide predictions or plans within its supported scope. Counterexamples narrow or retire rules as the environment changes. The language model remains frozen, and each rule remains linked to its evidence and the decisions it changes. Across four benchmarks spanning static dynamics, hidden drift, and scientific discovery, EvoBelief improves task performance and reuses learned rules across tasks. On ALFWorld without actor-side transition memory, it improves success over the matched Frozen World Model by 23.9% and 31.3% with Claude Haiku 4.5 and GPT-5.4, respectively. On MiniGrid, it revises stale rules and recovers after unannounced dynamics changes. In DiscoveryWorld's Reactor Lab, it discovers and validates quantitative laws, completing 83.3% of Normal-tier and 58.3% of Challenge-tier instances, versus 0% for each evaluated baseline. These results demonstrate self-improvement through testable, revisable environment knowledge rather than model-parameter updates.

Figures & tables

Figure 1: Illustrative example. (a) Following its frozen prior, a capable agent heats compound X to 100 ∘ C and gets decomposition products. (b) The mismatch triggers a hypothesis, which controlled experiments test. (c) The verified belief is stored in the persistent world model $W$ and changes later actions in the original task and in new ones, without any parameter update. Based on the updated belief, the agent avoids gas chromatography (GC), whose operating temperature exceeds the supported range, and correctly chooses to use liquid chromatography (LC) instead.

Figure 2: Verified belief revision closes the loop from experience to later behavior. EvoBelief uses the current

z_{t}

and

W_{t}

to predict action consequences. A grounded prediction mismatch or a decision-critical unknown opens an evidence loop that forms falsifiable hypotheses, gathers evidence, and verifies their scope. Only validated, applicable rules guide predictions and plans.

		ALFWorld		ScienceWorld
Setting	Method	Haiku 4.5	GPT-5.4	Haiku 4.5	GPT-5.4
No memory	ReAct	33.58	46.27	55.56	41.11
	ITP-I ( Liu et al., 2026 )	29.85	39.55	42.22	36.67
	WorldEvolver ( Zhang et al., 2026b )	35.07	61.94	52.22	65.56
	Frozen WM	42.54	44.78	46.67	53.33
	EvoBelief	66.42	76.12	64.44	71.11

Table 1: Success rate (%) with and without actor-side transition memory, on ALFWorld and ScienceWorld for both backbones. All methods are run through the same harness, interface, and budgets.

Figure 3: A rule-induced decision change on a real ALFWorld trace (Haiku 4.5, unseen task). Frozen WM predicts that cooling succeeds, executes the draft to no effect, and never recovers. EvoBelief uses a validated cooling rule, moves to the fridge first, and completes the task in ten decisions.

Figure 4: Learning over deployment and efficiency on ALFWorld. (a) Tasks solved so far over the 134 unseen tasks (Haiku 4.5, no transition memory). Shaded blocks are the six task types, and dotted and dashed lines mark when the generic and the clean, cool, and heat rules were committed. (b) Decisions saved by EvoBelief over Frozen WM and WorldEvolver. (c, d) Mean decisions and model calls per episode for both backbones, without and with memory (lower is better).

Figure 5: The two central MiniGrid drift families. Each episode is a two-room grid with a locked door, three keys, and a target box in the far room. Left: the drift shared by every family. Only the ringed key unlocks the door, blue for episodes 1–20 and green for 21–40. The switch is silent, and the red key that matches the door is a decoy. Right: the preconditions each family requires to complete the task of lifting the box. In MG-C the box lifts only with the door shut and hands empty. In MG-M the toggle only unlocks the door, and a second, empty-handed toggle opens it. Dark red marks steps the pretrained prior does not expect. Figure 8 in the appendix shows all three families.

	MG-C		MG-M		MG-R
Method	Pre-drift	Post-drift	Pre-drift	Post-drift	Pre-drift	Post-drift
Frozen WM	0.00	0.00	1.67	0.00	48.33	50.00
WorldEvolver	0.00	0.00	6.67	16.67	20.00	26.67
EvoBelief	61.67	95.00	78.33	81.67	83.33	78.33

Table 2: MiniGrid task success (%) before and after the hidden key switch. Each cell pools 60 episodes from three 40-episode streams per family; MG-C and MG-M use development seeds and MG-R uses fresh control seeds. Haiku 4.5, without transition memory. Higher is better.

Figure 6: Adaptation over deployment on MiniGrid. (a) Success by episode. The dashed line marks the silent key switch after episode 20. (b) EvoBelief decisions per episode by source: pretrained prior, active experiment, or committed plan. Both panels average the six development streams from MG-C and MG-M.

Panel of Figure 7 — Figure 7: Substrate ablation and held-out prediction. (a) Success on the ScienceWorld tasks of Table 1 with GPT-5.4 and no transition memory, by persistent substrate. Hatched bars store no explicit rules. (b) Structured state-change prediction on 1,500 held-out transitions per benchmark with Claude Haiku 4.5. State- $\Delta$ F1 measures how well the predicted state changes match the realized ones, Feasibility whether the action is correctly predicted to execute, and exact match (EM) and token F1 compare the predicted and realized state records. Exact prediction scores are in Table 15 .

	Normal			Challenge
Method	Completion	Knowledge	Procedural	Completion	Knowledge	Procedural
Reactor Lab
ReAct	0.00	0.00	70.33	0.00	0.00	64.66
WorldEvolver	0.00	0.00	72.67	0.00	0.00	69.54
Hypothesizer	0.00	0.00	74.00	0.00	0.00	68.39
EvoBelief	83.33	79.17	89.00	58.33	62.50	84.20

Table 3: DiscoveryWorld results with GPT-5.4 on twelve fresh task instances per task family and difficulty tier. Completion, Knowledge, and Procedural are the benchmark’s task-completion, blind knowledge, and subgoal-progress scores, in percent.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Hypothesis source	Behavioral route	Experiment route	Rule-conditioned call
ALFWorld	Typed transitions; code induction	Rule-based planning; precondition veto	Passive evidence only	None
MiniGrid	Typed transitions; code induction	Validated plan steps replace the draft	Live experiments scheduled by code	None
ScienceWorld	Public observations and typed transitions; code induction	Rule-based control, observation obligations, and veto	Live experiments admitted by code	None
DiscoveryWorld	Laws fitted by code from measurements and observations	Code-projected consequences; law-derived execution plans	Compiled live experiments	Conditional

Appendix

Table 4: Hypothesis source, behavioral route, experiment route, and use of the frozen engine’s rule-conditioned model call on each benchmark. Calls per decision by role are in Table 5 .

Benchmark	Actor	Alternatives	Anchor	Rule-conditioned	Analysis
ALFWorld	1	code	1 (batched)	none	code
MiniGrid	1	code	1 per candidate (5)	none	1
ScienceWorld	1	1	$1+N$	none	1
DiscoveryWorld	1	1	$1+N$	up to $1+N$	1

Appendix

Table 5: Language-model calls per decision, by role.

N

is the number of W-blind alternative candidates (at most 4 on ScienceWorld, 2 on DiscoveryWorld). Per-episode auxiliary calls are a state seed at episode start, hypothesis proposal after a mismatch, trajectory distillation after a success, and role binding for the engine’s own experiments. Call counts are taken from one configuration per benchmark, Haiku 4.5 on ALFWorld and MiniGrid and GPT-5.4 on ScienceWorld and DiscoveryWorld; the roles and calls per decision do not depend on the backbone.

	ALFWorld	MiniGrid	ScienceWorld	DW Reactor	DW Plant
Hypotheses formed (claims)	12	639	236	669	324
Passive supports (records)	230	2,156	4,031	–	–
Live experiments (runs)	0	458	54	96	51
Validated rules (distinct)	10	220	36	18	38
Counterexamples (records)	9	10,645	21	296	12
Retirements (rule versions)	2	845	70	–	15

Appendix

Table 6: What the belief machinery did, by benchmark. One setting per benchmark: ALFWorld, Haiku 4.5 without transition memory (134 tasks); MiniGrid, the six development streams (240 episodes); ScienceWorld, GPT-5.4 without transition memory (90 tasks); DiscoveryWorld Reactor Lab and Plant Nutrients, GPT-5.4 (24 task instances each). Each row counts the unit named in its label, and the rows are not stages of one funnel. ALFWorld: 10 validated rules over 43 validation events. MiniGrid: 220 distinct validated rules over 2,185 validated rule versions, and counterexamples are negative witness records. ScienceWorld: 54 experiments opened, 39 completed, yielding 573 support records, while the 4,031 passive supports come from ordinary task transitions. Supports are counted per occurrence, so a claim observed repeatedly in one context adds many records but little independent evidence, which is why validated rules are far fewer than support records. Confirmed: the predicted effect held in the next observation, over the decisions a rule changed on ALFWorld, all plan steps on MiniGrid, the scorable rule predictions on ScienceWorld, law-derived plans on Reactor Lab, and positive consequence verifications on Plant Nutrients. On DiscoveryWorld, the counterexample row counts rival laws falsified. ScienceWorld’s procedural safeguards for task milestones and irreversible steps, fixed before evaluation (§ 4.1 ), are not belief actions and are not counted. A dash means the implementation does not record the quantity.

Benchmark, setting	Baseline	$n$	Ours	Base	$\Delta$	95% CI	Disc.	$p$
ALFWorld, Haiku, no memory	Frozen WM	134	66.42	42.54	23.88	16.42, 31.34	34 / 2	$<$ 0.001
	WorldEvolver	134	66.42	35.07	31.34	21.64, 41.04	51 / 9	$<$ 0.001
ALFWorld, Haiku, memory	Frozen WM	134	68.66	57.46	11.19	2.99, 20.15	25 / 10	0.017
	WorldEvolver	134	68.66	58.96	9.70	1.49, 17.91	24 / 11	0.041
ALFWorld, GPT-5.4, no memory	Frozen WM	134	76.12	44.78	31.34	23.13, 40.30	45 / 3	$<$ 0.001
	WorldEvolver	134	76.12	61.94	14.18	5.22, 23.13	29 / 10	0.003

Appendix

Table 7: Paired tests for the matched comparisons. Success rates in percent;

\Delta

is EvoBelief minus the baseline in percentage points with its 95% percentile bootstrap interval over paired episodes; Ours and Base are the two success rates, Disc. counts the episodes only EvoBelief solved against those only the baseline solved;

p

is the exact two-sided McNemar test on those pairs. ALFWorld pairs the 134 unseen tasks, WorldEvolver being the authors’ release; ScienceWorld pairs its 90 tasks; MiniGrid pairs episodes by stream and position, 240 per seed set, and the post-versus-pre rows compare EvoBelief’s second half with its own first half, pairing episode

i+20

with episode

i

of the same stream; MiniGrid episodes within a stream share an evolving belief state, and the episode-level tests and intervals do not account for this dependence; DiscoveryWorld pairs the 12 fresh task instances per tier of each theme, with the Hypothesizer at its raised output ceiling.

Setting	Rules	Rule-use decisions	Later task	Other family	Type unseen at formation	Changed	Confirmed
Haiku, no memory	10	707	99.86	82.46	88.12	324	81.79
Haiku, memory	8	753	99.87	85.79	90.17	250	78.40
GPT, no memory	8	774	99.87	85.66	91.34	304	83.22
GPT, memory	8	789	99.75	86.69	90.87	237	81.43

Appendix

Table 8: Where committed rules are used on ALFWorld. Rule-use decisions: decisions whose executed action a validated rule licensed. Later task: a task after the one in which the rule was first validated. Other family: the task family differs from that of the task in which the rule was validated. Type unseen at formation: the executed action binds at least one role to an object type absent from every binding that formed the rule (instance numbers ignored). Changed: decisions under a validated rule whose executed action differs from the controller’s draft after normalizing the surface forms of put. Confirmed: the transition was not rejected and every atom the rule predicted appears in the next observation. Percentages are of decisions under a validated rule, and of changed decisions for the last column.

Rule	Validation task	Tasks used	Families	Bindings	Changed	Confirmed
go to (contents)	1, look at in light	115	6	26	221	170
take	2, look at in light	109	6	83	20	20
put	7, look at in light	91	5	63	4	4
open	24, pick and place	63	5	8	8	8
go to (location)	21, pick and place	33	5	10	33	33
clean	44, clean then place	25	1	16	4	4

Appendix

Table 9: The rules of one setting, GPT-5.4 without transition memory. Validation task: the index of the task in the deployment order at which the rule first earned authority, with that task’s family. Tasks used: distinct later tasks with a decision taken under the rule. Bindings: distinct object and receptacle instances the rule was bound to. The two go-to rules differ in their effect: one predicts that reaching a receptacle reveals its contents, the other only the new location. Clean, heat, and cool rules can only be used in their own family.

Seeds	Method	Pre-switch	Post-switch	All
Held-out	Frozen WM	2.50	0.00	1.25
	EvoBelief	63.33	75.83	69.58
Held-out, permuted labels	Frozen WM	10.00	4.17	7.08
	EvoBelief	74.17	87.50	80.83

Appendix

Table 10: MiniGrid on held-out seeds (%). Haiku 4.5, actor without transition memory. Three seeds on both families, fixed before any run and never used while building the method, 240 episodes per method per block; the permuted block renames every color and room in the observations of both methods. Higher is better.

Seeds	Variant	Pre-switch	Post-switch	All
Development	Planner without learned rules	0.83	0.00	0.42
	No active experiments	0.83	0.00	0.42
	EvoBelief	70.00	88.33	79.17
Mechanism	Rules never executed (shadow)	50.00	64.17	57.08
	No commitment gate	66.67	70.00	68.33
	EvoBelief	61.67	77.50	69.58

Appendix

Table 11: What the MiniGrid gain needs (%). Each block compares variants of EvoBelief on one seed set, 240 episodes per row. The planner without learned rules never leaves the start state of its search and defers to the prior on every decision; without active experiments, rules never gather independent support; in the shadow variant rules are learned and validated but no plan step is executed; without the commitment gate a rule may act on its first supported instance. Higher is better.

Figure 8: What each MiniGrid family requires. One row per family, each frame a state of the two-room grid: the robot is the agent, a key drawn next to it is the key it carries, the box drawn next to it is the lifted target. A padlock marks a locked door, a plain door is shut but unlocked, and a door drawn aside is open. In every family the red key matches the door and never works, the blue key opens the door for the first twenty episodes, and after episode 20 the green key does instead; the ring marks the key that opens the door and the switch is never announced. In MG-R (R for rebind) one toggle with the working key unlocks and opens the door, and nothing more is asked. In MG-C (C for conjunctive) the door behaves the same, but the box lifts only with the door shut and the agent’s hands empty, so after entering the agent must drop the key and close the door it just opened. In MG-M (M for multi-stage) toggling with the working key only unlocks the door, which stays shut until a second toggle with empty hands. The switch is the only change within a stream; the family rules hold in both halves. The steps in dark red are the ones the pretrained prior does not expect.

	Normal	Challenge
Hypotheses frozen	236	433
Experiments executed	48	48
Rivals falsified	157	139
Laws validated	10	8
Task instances with a validated law	10	8
Task instances whose law-derived plan passed	10	7

Appendix

Table 12: What EvoBelief did on the fresh task instances. Counts from the sealed science records of the twelve task instances per tier; the baselines have no such record by construction. A hypothesis is a candidate law with a prediction frozen before any experiment. One executed experiment settles every rival whose frozen prediction its realized outcome contradicts, so fewer experiments falsify more rivals, and a law is validated only by a held-out prediction the reactor confirms, never by fit quality alone. Model calls and environment actions are totals over the tier.

Seed	Validated law	$x$	Predicted	Dialed	Rivals	Plan	Knowledge
Normal
s9	$f=104.99\,c_{5}+107.25$	30.50	3309.50	3310.00	18	pass	1.00
s10	$f=99.01\,d+102.24$	61.31	6172.69	6173.00	14	pass	1.00
s11	$f=105.00\,t+95.96$	17.81	1966.05	1966.05	18	pass	1.00
s12	$f=105.01\,q+90.40$	49.72	5311.60	5312.00	16	pass	1.00
s13	none (radiation seed)				7	–	0.00

Appendix

Table 13: The law validated on each fresh task instance.

f

is the reactor resonance in Hz;

c_{5}

d

t

, and

q

are the crystal’s spectrometer channel 5, density, temperature, and quantum gap.

x

is the held-out crystal’s reading, Predicted the prediction frozen before the reactor was dialed, and Dialed the frequency at which the reactor then activated. Rivals is the number of rival laws falsified on that task instance, Plan the outcome of the plan derived from the law for the target crystal (a passed plan completes the task), and Knowledge the judge’s score on that task instance. Normal seeds 13 and 18 and Challenge seeds 9, 14, and 19 hide the variable in a radiation reading with random background; on Challenge seed 20 no rival survived within budget, and on seed 17 the law was validated but its plan never settled.

Family	Tier	Sealed	Completion	Knowledge	Procedural
Reactor Lab	Normal	12 of 12	0.00	0.00	74.33
Reactor Lab	Challenge	2 of 12	0.00	0.00	74.14
Plant Nutrients	Normal	12 of 12	66.67	0.00	82.41
Plant Nutrients	Challenge	1 of 12	0.00	0.00	66.67

Appendix

Table 14: The Hypothesizer at its original 3,072-token output ceiling. Same metrics and task instances as Table 3 , in percent; the Challenge runs ended on output truncation, and their means are over the task instances sealed before the failure.

	Model	State- $\Delta$ F1 $\uparrow$	Feasibility $\uparrow$	EM $\uparrow$	Token F1 $\uparrow$
ScienceWorld	Frozen WM (no evolving $W$ )	0.7258	0.7547	0.6013	0.7866
	WorldEvolver (semantic-only)	0.7061	0.7827	0.6027	0.7814
	WorldEvolver (full)	0.6680	0.8753	0.5993	0.7820
	EvoBelief	0.7530	0.8113	0.6407	0.8045
ALFWorld	Frozen WM (no evolving $W$ )	0.7572	0.8453	0.6567	0.8080
	WorldEvolver (semantic-only)	0.6862	0.8500	0.5733	0.7607

Appendix

Table 15: Held-out structured state-change prediction on ScienceWorld and ALFWorld, 1,500 transitions per benchmark, disjoint from belief formation. State-

\Delta

F1 measures canonical state changes; Feasibility measures action executability; EM and Token F1 compare structured state. Haiku 4.5.

World Models

World models represent and predict how environments change, helping agents plan, act, and reason about possible futures.

See all 366 papers

Self-Evolving World Models for LLM Agent Planning
Jun 29, 2026Xuan Zhang, Wenxuan Zhang, See-Kiong Ng +1Test-Time Adaptation World Model Learning
National University of Singapore · Singapore University of Technology and Design · Singapore Management University
Training LLM Agents for Spontaneous, Reward-Free Self-Evolution via World Knowledge Exploration
Apr 20, 2026Qifan Zhang, Dongyang Ma, Tianqing Fang +5Intrinsic Motivation LLM Agent Self-Improvement
1Tencent · 2The Hong Kong University of Science and Technology (Guangzhou)
ProPlay: Procedural World Models for Self-Evolving LLM Agents
Jun 11, 2026Yijun Ma, Zehong Wang, Yiyang Li +5World Models LLM Agent Planning
University of Notre Dame · University of Connecticut

Jun 29, 2026J/K move · Enter open · S save

Self-Evolving World Models for LLM Agent Planning

Xuan Zhang, Wenxuan Zhang, See-Kiong Ng, Yang Deng

National University of Singapore · Singapore University of Technology and Design · Singapore Management University

Test-Time Adaptation World Model Learning World Models World Model-Based Planning LLM Agent Planning Memory-Augmented Language Model Agents

Open paper AskPDF arXiv

World models offer a principled way to equip long-horizon LLM agents with foresight: predictions of action consequences before execution. However, unreliable foresight can be ignored, misused, or even degrade downstream decision-making. In this paper, we introduce WorldEvolver, a self-evolving world model framework that revises its deployment-time context while keeping the downstream agent and all model parameters frozen. WorldEvolver integrates three modules: (i) Episodic Memory, which exploits real action transitions through retrieval-based simulation; (ii) Semantic Memory, which extracts persistent heuristic rules from prediction-observation mismatches; and (iii) Selective Foresight, which filters low-confidence predictions before integrating them into agent reasoning context. We evaluate WorldEvolver on ALFWorld and ScienceWorld, measuring world model prediction accuracy on Word2World and downstream agent success rate on AgentBoard. Extensive experiments show that WorldEvolver achieves the highest prediction accuracy across three backbones and leads other world model baselines on downstream agent success rate, demonstrating that test-time memory revision enhances both predictive fidelity and planning performance.

Abstract

World Models

Explore similar work

Self-Evolving World Models for LLM Agent Planning

Training LLM Agents for Spontaneous, Reward-Free Self-Evolution via World Knowledge Exploration

ProPlay: Procedural World Models for Self-Evolving LLM Agents