Long-horizon language agents require compact memories that retain interaction evidence while preserving competing hypotheses about unobserved states. Although effective, many belief-state methods encode memory in discrete natural language, where repeated compression can omit evidence or prematurely discard plausible alternatives. We introduce LAMB, a recurrent framework for latent multi-hypothesis beliefs. A Common--Delta factorization combines a shared Common state for interaction evidence with weighted Delta components for competing hypotheses. Relative evidence from action-outcome predictions guides recurrent hypothesis updates without generating textual memory. A context-conditioned readout exposes hypotheses separately to the policy through a fixed-token belief prefix in the default configuration. The belief modules and policy are jointly trained using action imitation and auxiliary state, outcome, and observation-discrimination objectives. Across ALFWorld, BabyAI, ScienceWorld, and Crafter, LAMB matches or outperforms PABU, improving ALFWorld SR@30 and ScienceWorld success by 6.0 and 22.0 percentage points, respectively, and Crafter achievement progress by 26.37 points. On ALFWorld, it also reduces the redundant-action rate by 17.70 percentage points. These results support latent multi-hypothesis beliefs as compact memory for effective long-horizon decision making
Figures & tables
Figure 1: Existing text-based beliefs (left) prematurely drop or misjudge environmental observations, trapping the agent in invalid action loops. LAMB (right) factorizes beliefs into confirmed state facts and candidate spatial distributions, maintaining calibrated uncertainty to resolve the task efficiently in 5 steps.
Figure 2: Overview of LAMB. The recurrent updater incorporates the latest action and observation into a shared Common state and weighted Delta hypotheses. A context-conditioned readout produces a fixed 12-token prefix for action generation. Simulator facts supervise joint prediction during training only; subsequent interaction feedback enters the next belief update.
Model
ALFWorld SR ↑
ScienceWorld SR ↑
BabyAI Progress ↑
Crafter Progress ↑
Open-weight API Models zero-shot
Qwen3-8B-FH ( Yang et al., 2025 )
24.0
1.5
62.16
0.00
Qwen3-32B-FH ( Yang et al., 2025 )
57.5
17.0
66.73
12.73
GPT-OSS-120B-FH ( Agarwal et al., 2025 )
39.5
16.5
64.33
18.64
Proprietary API Models
GPT-5.6-luna ( OpenAI, 2026 )
65.50
60.00
73.39
20.00
Table 1: Overall performance across interactive environments. ALFWorld reports SR@30 (success within 30 actions). The three blocks contain open-model full-history (FH) baselines, proprietary API full-history baselines, and fine-tuned baselines. Best results within the benchmark-tuned block are in bold . Red annotations report percentage-point gains over the strongest benchmark-tuned baseline in each interactive environment.
Model
Avg. [-2pt] Steps ↓
Input [-2pt] Tokens ↓
Output [-2pt] Tokens ↓
Task Time [-2pt] (s/Task) ↓
API-based Models
Llama-3.2-3B-FH ( Meta, 2024 )
48.03
332,553
741
49.10
Qwen3-8B-FH ( Yang et al., 2025 )
41.24
297,154
657
60.20
Qwen3-32B-FH ( Yang et al., 2025 )
31.12
202,094
479
27.01
GPT-OSS-120B-FH ( Agarwal et al., 2025 )
37.25
244,216
3,927
198.52
DeepSeek-V4-Flash-FH ( DeepSeek-AI et al., 2026 )
18.64
81,475
13,439
132.64
Table 2: Inference cost on ALFWorld-200 with a 50-step limit. FH denotes full history. The two blocks contain API-based models and benchmark-tuned models; the shaded row highlights LAMB. Steps, token counts, and task times are averaged over all tasks, including failures. Best results are in bold.
Figure 3: Input context length on the 45 ALFWorld tasks successfully completed by four methods under a 50-step budget. Shaded bands denote ±1 SEM across active tasks at each step; the lower panel reports active task counts. The inset provides detailed scaling for PABU and LAMB.
Variant
SR@30 (%) ↑
SR@50 (%) ↑
Δ50
Avg. Steps ↓
Full Model
92.5
95.5
0.0
13.91
Unstructured Latent
46.5
51.5
−44.0
31.40
w/o Delta
86.0
90.5
−5.0
17.04
w/o Common
91.5
93.5
−2.0
15.38
w/o Belief Input
80.0
82.0
−13.5
19.13
w/o Lstate
86.5
92.0
−3.5
16.71
Table 3: Component ablations on 200 ALFWorld tasks under identical training and evaluation protocols. SR@30 and SR@50 measure success within 30 and 50 actions, respectively. Δ50 denotes the percentage-point change in SR@50 relative to the full model; average steps include successful and failed tasks under the 50-step limit.
Figure 4: Redundant action rates across three environments (%, lower is better), with absolute counts in parentheses, evaluated over all executed actions across successful and failed tasks.
Figure 5: ALFWorld redundant-action breakdown across five mutually exclusive categories (percentage of all executed actions, lower is better). Blue annotations mark LAMB’s rates.
Method
0 steps
1 step
2 steps
4 steps
8 steps
AgentEvol
78.13
66.67
62.50
52.60
42.19
PABU
25.00
25.00
25.52
25.00
25.00
Ours
99.48
99.48
98.96
98.96
98.44
Table 4: Observation identification (%, higher is better) across subsequent interaction lags on 48 ALFWorld trajectories, evaluated using vector-based 4-way classification probes.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Layer-wise top-10 token overlap across Delta particles, averaged over 28 particle pairs across 12 diagnostic states. Final-layer overlap drops to 30.68%.
Common Tokens
Delta Particles
Total Prefix
SR@30 (%) ↑
SR@50 (%) ↑
Avg. Steps ↓
2
4
6
87.5
92.0
16.39
4
8
12
89.0
94.0
15.78
8
16
24
88.0
93.5
16.09
Appendix
Table 5: ALFWorld performance under varying Common and Delta token budgets. All variants are trained on the unaugmented 7,200/800 source split. The green-shaded row denotes our default configuration. Bold marks the best result per column.
Hyperparameter
Value
Backbone & Adaptation
Policy model
Qwen3-8B (frozen)
LoRA rank r / α / dropout
32 / 64 / 0.05
Max sequence length
2,048 (+12 prefix)
Backbone precision
BF16
Belief Representation
Appendix
Table 6: Hyperparameter settings for the main experiments.
Figure 7: Joint vs. per-fact prediction (Eq. 5 ). With countertop 1 known to be empty, the mug is in either cabinet 1 or sink 1. Top: per-hypothesis predictions pϕ(yq∣ct,δti) for (a) uncertain and (b) competing hypotheses. Bottom: the resulting joint distributions. Both cases share the same per-fact marginals (0.5), but only (b) assigns zero probability to the mug being in both or neither location. The agreed fact (gray) factors out as a shared multiplier.
Method
Obs., lag 0
Obs., lag 8
Fact, lag 8
Full history
100.00
100.00
100.00
PABU
41.67
33.33
50.00
Ours
100.00
100.00
100.00
Appendix
Table 7: Candidate-assisted recovery (%) on 24 paired observation branches from 12 base trajectories. Obs.: identification among four complete observations. Fact: target-object presence derived from the selected observation. The shaded row highlights LAMB; best results (including ties) are in bold.
Figure 8: In this ALFWorld case-study, the Common component retains the lamp location on dresser 1, while Delta hypotheses shift away from drawers confirmed empty toward uninspected locations. Once the bowl is found on desk 1, the agent returns to the lamp and uses it to examine the bowl, ultimately completing the task successfully. Cross-hatched boxes denote latent belief components.
Large language model (LLM) agents can now undertake increasingly complex tasks, but the way they organize interaction history into memory does not ensure a coherent understanding of the current world. We introduce PoS, an inference-time framework that constructs and continually maintains explicit belief states as the agent's decision context. Each belief combines an estimate of the current world state with unresolved task requirements, making explicit what the agent still needs to learn and accomplish. To keep this belief reliable and actionable, PoS validates its consistency and monitors task progress to detect Belief Trapping, where the agent continues to act without making meaningful progress toward the goal. Recovery is then tailored to both the trapping pattern and the type of unresolved task requirement. Experiments on four benchmarks spanning execution and diagnosis show that PoS achieves the highest overall performance on every benchmark with all three LLM backbones. Ablations demonstrate the importance of consistency validation and recovery, while context-scaling experiments show resilience to context growth. Together, these results support belief construction and continual maintenance as a foundation for long-horizon context management beyond history retention and compression.
Yu Luo, Jiamin Jiang, Yimin Zuo +9
Nankai University · Alibaba Group · Tsinghua University
Large language models (LLMs) are increasingly deployed on long-horizon tasks in partially observable environments, where they must act while inferring and tracking a complex environment state over many steps. This leads to two challenges: partial observability requires maintaining uncertainty over unobserved world attributes, and long interaction history causes context to grow without bound, diluting task-relevant information. A principled solution to both challenges is a belief state: a posterior distribution over environment states given past observations and actions, which compactly encodes history for decision making regardless of episode length. In LLM agents, however, the open-ended nature of text makes it unclear how to represent such a distribution. Therefore, we introduce Agent-BRACE: Agent Belief state Representation via Abstraction and Confidence Estimation, a method that decouples an LLM agent into a belief state model and a policy model, jointly optimized via reinforcement learning. The belief state model produces a structured approximation of the belief distribution: a set of atomic natural language claims about the environment, each annotated with an ordinal verbalized certainty label ranging from certain to unknown. The policy model conditions on this compact, structured approximate belief rather than the full history, learning to select actions under explicit uncertainty. Across long-horizon, partially observable embodied language environments, Agent-BRACE achieves an average absolute improvement of +14.5% (Qwen2.5-3B-Instruct) and +5.3% (Qwen3-4B-Instruct), outperforming strong RL baselines while maintaining a near-constant context window independent of episode length. Further analysis shows that the learned belief becomes increasingly calibrated over the course of an episode as evidence accumulates.
Joykirat Singh, Zaid Khan, Archiki Prasad +5
UNC Chapel Hill · Microsoft Research · The University of Texas at Austin
Large language models (LLMs) are being used as policies for autonomous decision-making and planning in many domains. Despite their strong reasoning capabilities, LLMs struggle with long-horizon tasks, especially under partial observability. World models are a promising way to enhance policy performance, both during training and inference. During inference, agents currently use world models to simulate the consequences of candidate actions before choosing an action, which can improve decision-making. However, we argue that simulation alone is an incomplete interface for decision-making under partial observability: simulation does not adequately capture uncertainty about the current state, which agents may need for accurate decision-making. We address this limitation with Belief-Based World Models (BB-WMs), which maintain a belief that LLMs can query to access information on what is known and uncertain about the current state. Before developing methods to learn accurate BB-WMs, this paper focuses on a more fundamental question: does exposing a world model's belief directly to an LLM policy improve decision-making? Our results show that giving LLM agents access to beliefs improves task performance under partial observability, while remaining complementary to existing simulation-based world models. Code: https://github.com/skumar-ml/belief-world-models.