We investigate estimating a human's world belief state using a robot's observations in a dynamic, 3D, and partially observable environment. The methods are grounded in mental model theory, which posits that human decision making, contextual reasoning, situation awareness, and behavior planning draw from an internal simulation or world belief state. When in teams, the mental model also includes a team model of each teammate's beliefs and capabilities, enabling fluent teamwork without the need for constant and explicit communication. In this work we replicate a core component of the team model by inferring a teammate's belief state, or level one situation awareness, as a human-robot team navigates a household environment. We evaluate our methods in a realistic simulation, extend to a real-world robot platform, and demonstrate a downstream application of the belief state through an active assistance semantic reasoning task.
Figures & tables
Fig. 1: Overview of the predicted user belief state system. The robot observes a person in their environment and estimates their current belief state. The belief state can be applied to various downstream reasoning tasks.
Fig. 2: Overview of the predicted user belief state system. The ground truth world state information is filtered by the robot’s visibility ( Rrobot) and used to update the robot’s belief state, βrobot . The robot’s belief state is then filtered by the user’s visibility to update the predicted user belief state, βpred . The resulting belief state is what the robot thinks the user is aware of, i.e., a theory of mind that can inform downstream reasoning tasks.
∑o∈Ocd(o,f(o))
Algorithm 1 Updating a belief state from observed objects.
Fig. 3: Outline of the perception stack. The robot’s RGB camera and depth sensor are used to detect objects in 3D space and resolve the robot’s own world belief state βrobot . When a person is observed, their current belief state βhuman is inferred using their gaze region and estimated prior trajectory.
Fig. 4: Performance of the belief state inference in an episode of the “Parents are Out” scenario. The scenario shuffles the environment, and then the human and robot walk through each room. The primary figure of merit is the error of the inferred belief state (indicated with ⋆ ), or the SMCC difference between the inferred belief state and the human agent’s true belief state. Two trials are visualized – (left) uses the robot’s full perception stack (no privileged perception information) and (right) uses ground truth perception from the simulator. We find near-identical results, indicating an effective perception system.
Object Detection
Pose Detection
Trajectory Inference
Mean Inference Error [m]
OWLv2
RTM+MM
A*
0.356
GT
RTM+MM
A*
0.359
OWLv2
GT
A*
0.347
OWLv2
RTM+MM
GT
0.344
OWLv2
RTM+MM
None
0.345
GT
GT
GT
0.360
TABLE I: Ablation study on each perception module
Method
F1
Precision
Recall
LLM (List CoT)
0.68
0.56
0.86
LLM (Single CoT)
0.53
0.77
0.41
LLM (Single Binary)
0.46
0.76
0.33
deBERTa-v3
0.56
0.46
0.72
Random
0.14
0.08
0.50
TABLE II: Performance of semantic reasoning methods for the active assistance task
World models allow robots to anticipate action consequences before execution. This capability is especially valuable in excavation, where each scoop reshapes the terrain and affects subsequent actions. Local observations, however, cannot fully reveal the underlying support and material conditions. We present PileBelief, an interaction-driven persistent world model for partially observed excavation that retains physical evidence beyond the visible surface. It combines an observation-conditioned physical prior with world-addressed deformation memory and physical-response memory. Action-aligned reads and gated residual corrections refine terrain-change and outcome predictions. With deployment weights fixed, completed interactions update measured belief, while hypothetical actions advance a separate imagined state. Compared with a current-observation-only baseline, PileBelief reduces five-step joint prediction error by 10.8% and offline action-selection regret by 65.5%. Experiments on Newton/MPM and real excavation datasets further demonstrate improved terrain-change and bucket-volume prediction. Our method enables multi-step prediction and candidate-action ranking from local observations, even when the underlying soil state is unknown. These results identify persistent physical belief as a useful representation for world models of environments that robots continually reshape.
Hongyi Lin, Song Zhang, Haiquan Liu +3
Tsinghua University · Massachusetts Institute of Technology · Tsing-AI(Shanghai) Technology Co., Ltd
World models enable a predictive substrate for planning and action, yet existing formulations merely answer a physical question: what/where it is, and how will it evolve. Human behavior, however, is driven by hidden mental state (what a person believes, wants, intends, feels, and considers socially permissible), so a model that tracks the physical scene but not what each agent knows and believes about it predicts the wrong action for the right-looking scene. We formulate Mental World Modeling (MWM), a generic theoretical framework that makes mental variables core components of a world model rather than posthoc rationales: MWM aintains a coupled physical-mental world state, renders a target-specific partial observation, and simulates how candidate actions jointly update both components. We instantiate the framework in MENTIS, a training-free and fully inspectable baseline that decomposes the process into state parsing, target-observation generation, action decomposition, coupled physical and mental transition, and branch-level value evaluation. On a manually constructed, quality-controlled dataset of situated decision scenarios spanning text, image, and sounding-video stories, experiments with 8 modern LLM-based world models demonstrate that explicitly modeling the mental state is essential for predicting human decisions. Deeper analyses further expose the bottlenecks of current mental world modeling. We expect MWM as a next stage of world modeling, from simulating physical scenes to simulating the minds that act in them.
Hao Fei, Yiran Zhao
University of Oxford · National University of Singapore
Recent advances in visual generative models have highlighted the promise of learning generative world models. However, most existing approaches frame world modeling as novel-view synthesis or future-frame prediction, emphasizing visual realism rather than the structured uncertainty required by embodied agents acting under partial observability. In this work, we propose a different perspective: world modeling as embodied belief inference in 3D space. From this view, a world model should not merely render what may be seen, but maintain and update an agent's belief about the unobserved 3D world as new observations are acquired. We identify several key capabilities for such models, including spatially consistent scene memory, multi-hypothesis belief sampling, sequential belief updating, and semantically informed prediction of unseen regions. We instantiate these ideas in 3D-Belief, a generative 3D world model that infers explicit, actionable 3D beliefs from partial observations and updates them online over time. Unlike prior visual prediction models, 3D-Belief represents uncertainty directly in 3D, enabling embodied agents to imagine plausible scene completions and reason over partially observed environments. We evaluate 3D-Belief on 2D visual quality for scene memory and unobserved-scene imagination, object- and scene-level 3D imagination using our proposed 3D-CORE benchmark, and challenging object navigation tasks in both simulation and the real world. Experiments show that 3D-Belief improves 2D and 3D imagination quality and downstream embodied task performance compared to state-of-the-art methods.
Yifan Yin, Zehao Wen, Suyu Ye +10
Johns Hopkins University · Lambda · University of Cambridge