We investigate estimating a human's world belief state using a robot's observations in a dynamic, 3D, and partially observable environment. The methods are grounded in mental model theory, which posits that human decision making, contextual reasoning, situation awareness, and behavior planning draw from an internal simulation or world belief state. When in teams, the mental model also includes a team model of each teammate's beliefs and capabilities, enabling fluent teamwork without the need for constant and explicit communication. In this work we replicate a core component of the team model by inferring a teammate's belief state, or level one situation awareness, as a human-robot team navigates a household environment. We evaluate our methods in a realistic simulation, extend to a real-world robot platform, and demonstrate a downstream application of the belief state through an active assistance semantic reasoning task.
Figures & tables
Fig. 1: Overview of the predicted user belief state system. The robot observes a person in their environment and estimates their current belief state. The belief state can be applied to various downstream reasoning tasks.
Fig. 2: Overview of the predicted user belief state system. The ground truth world state information is filtered by the robot’s visibility ( Rrobot) and used to update the robot’s belief state, βrobot . The robot’s belief state is then filtered by the user’s visibility to update the predicted user belief state, βpred . The resulting belief state is what the robot thinks the user is aware of, i.e., a theory of mind that can inform downstream reasoning tasks.
∑o∈Ocd(o,f(o))
Algorithm 1 Updating a belief state from observed objects.
Fig. 3: Outline of the perception stack. The robot’s RGB camera and depth sensor are used to detect objects in 3D space and resolve the robot’s own world belief state βrobot . When a person is observed, their current belief state βhuman is inferred using their gaze region and estimated prior trajectory.
Fig. 4: Performance of the belief state inference in an episode of the “Parents are Out” scenario. The scenario shuffles the environment, and then the human and robot walk through each room. The primary figure of merit is the error of the inferred belief state (indicated with ⋆ ), or the SMCC difference between the inferred belief state and the human agent’s true belief state. Two trials are visualized – (left) uses the robot’s full perception stack (no privileged perception information) and (right) uses ground truth perception from the simulator. We find near-identical results, indicating an effective perception system.
Object Detection
Pose Detection
Trajectory Inference
Mean Inference Error [m]
OWLv2
RTM+MM
A*
0.356
GT
RTM+MM
A*
0.359
OWLv2
GT
A*
0.347
OWLv2
RTM+MM
GT
0.344
OWLv2
RTM+MM
None
0.345
GT
GT
GT
0.360
TABLE I: Ablation study on each perception module
Method
F1
Precision
Recall
LLM (List CoT)
0.68
0.56
0.86
LLM (Single CoT)
0.53
0.77
0.41
LLM (Single Binary)
0.46
0.76
0.33
deBERTa-v3
0.56
0.46
0.72
Random
0.14
0.08
0.50
TABLE II: Performance of semantic reasoning methods for the active assistance task