World models allow robots to anticipate action consequences before execution. This capability is especially valuable in excavation, where each scoop reshapes the terrain and affects subsequent actions. Local observations, however, cannot fully reveal the underlying support and material conditions. We present PileBelief, an interaction-driven persistent world model for partially observed excavation that retains physical evidence beyond the visible surface. It combines an observation-conditioned physical prior with world-addressed deformation memory and physical-response memory. Action-aligned reads and gated residual corrections refine terrain-change and outcome predictions. With deployment weights fixed, completed interactions update measured belief, while hypothetical actions advance a separate imagined state. Compared with a current-observation-only baseline, PileBelief reduces five-step joint prediction error by 10.8% and offline action-selection regret by 65.5%. Experiments on Newton/MPM and real excavation datasets further demonstrate improved terrain-change and bucket-volume prediction. Our method enables multi-step prediction and candidate-action ranking from local observations, even when the underlying soil state is unknown. These results identify persistent physical belief as a useful representation for world models of environments that robots continually reshape.
Figures & tables
Figure 1: Local sensing and physical ambiguity. The upper panel illustrates hidden influence beyond the visible surface. The controlled numerical pair below has identical current inputs but different outcomes and best actions.
Figure 2: PileBelief workflow. Completed interactions update spatial and response memories. Action-aligned reads provide gated corrections to a frozen local prior. Execution updates measured belief, while rollout advances a separate imagined state.
Figure 4: Prediction and decisions in the recorded controlled studies. (a) Free rollout. (b–d) Regret, Top-1 accuracy, and pairwise ordering with seven candidates per state.
Evidence
Gap 1 ↓
Gap 20 ↓
Zero memory
26.84±1.45
26.92±1.47
PileBelief (correct memory)
21.46±0.50
21.44±0.53
Table 3: Revisit height MAE (mm) across three seeds.
Condition
Height (mm) ↓
Bucket (L) ↓
Force (kN) ↓
Local (empty memory)
18.721
35.808
3.804
Fixed first three
15.228
47.602
4.064
PileBelief (recent three)
12.673
33.666
3.653
Table 4: Newton/MPM prediction on held-out material groups.
Figure 5: Indoor excavation settings (a) and a separately recorded terrain transition (b). Pre/post maps share a height scale. Blue denotes removal and orange deposition.
Condition
Height (mm) ↓
Bucket (m 3 ) ↓
Jreal↓
Local
114.74
0.2207
0.9164
Capacity-matched Local
112.65
0.2306
0.9324
Zero history input
112.83
0.2357
0.9451
Matched shuffled history
113.10
0.2120
0.8903
PileBelief
112.17
0.2056
0.8720
Table 5: Real excavation prediction on held-out segments.
Wheel-loader excavation is a sequential decision problem in which every scoop changes the terrain available to subsequent actions. A practical world model must predict action consequences accurately, rank candidates in real time, and operate inside the closed loop of a full-size machine. We present the World-Action Model (WAM), which proposes multiple scoops, rejects geometrically inadmissible candidates, jointly predicts signed terrain change and loaded volume, executes the candidate with the largest predicted load, and replans from the newly observed terrain. On 32 geometry-disjoint MinSlope test episodes, adding world-model ranking to matched diffusion proposals reduces the mean scoop count from 651.8 to 540.6 (17.1%), preserves 32/32 completion, and improves every paired episode. In a complete-system comparison, WAM completes 32/32 episodes versus 29/32 for an independently trained soft actor-critic policy. Comparisons of input representations, spatial support, and five architectures identify an accurate and efficient physics-structured predictor. We further evaluate the interface on event-disjoint full-size-loader data and deploy the complete perception-proposal-prediction-selection-execution loop for autonomous excavation. The ROS2/TensorRT implementation processes five candidates in 72.4 ms on a Jetson AGX Orin. The simulation results establish decision-level gains, while the physical experiments demonstrate real-world closed-loop feasibility.
In outdoor environments, varying terrain and contact conditions can cause identical action commands to produce different motion outcomes for quadruped robots. Predicting these outcomes is essential for reliable navigation and informed path selection. Existing video world models can generate visually plausible futures, yet often struggle to capture robot dynamics and terrain interactions, leading to overly optimistic predictions of action outcomes. We present DR-WM, a physically consistent world model that incorporates robot-terrain contact dynamics to predict the consequences of quadruped actions. Starting from an initial RGB image, our method reconstructs scene geometry and integrates dense, dimensionless priors over terrain contact properties, calibrated using real-world action-response data. The resulting terrain model supports the simulation of physics with spatially varying contact properties such as friction. Given the initial state of the robot and candidate action sequences, we simulate its future motion and use the resulting camera trajectories and rendered geometry to guide the training of a video world model. The generated future observations are then used to update the terrain reconstruction, enabling iterative model rollouts without additional real-world images. Extensive experiments demonstrate that DR-WM substantially outperforms geometry-only baselines and baselines with uniform physical parameters on key physical metrics characterizing action outcomes, while achieving the best performance among the evaluated baselines. These physically grounded predictions support more informed action selection: across 90 challenging scenarios, an 8B vision-language model augmented with DR-WM outperforms an unassisted 32B vision-language model on five of six evaluation metrics.
Ruo Chen, Jiahua Pang, Yingjie Niu +4
DEEP Robotics · Zhejiang University · Beijing Institute of Technology +3
We investigate estimating a human's world belief state using a robot's observations in a dynamic, 3D, and partially observable environment. The methods are grounded in mental model theory, which posits that human decision making, contextual reasoning, situation awareness, and behavior planning draw from an internal simulation or world belief state. When in teams, the mental model also includes a team model of each teammate's beliefs and capabilities, enabling fluent teamwork without the need for constant and explicit communication. In this work we replicate a core component of the team model by inferring a teammate's belief state, or level one situation awareness, as a human-robot team navigates a household environment. We evaluate our methods in a realistic simulation, extend to a real-world robot platform, and demonstrate a downstream application of the belief state through an active assistance semantic reasoning task.
Jack Kolb, Aditya Garg, Nikolai Warner +1
College of Engineering, Georgia Institute of Technology, North Avenue, Atlanta, GA 30332, USA