Motion Belief Action: From a Visual Scene Map to Physical Decisions
Abstract
A robot that works among people must judge what it cannot see: whether an object is still where it was last seen, and whether a pull on a door or drawer is worth its risk. Maps that track object change hold nothing about force, maps of physical properties answer a risk question at a fixed risk level, and vision-language-action models keep no explicit estimate of hidden physics. We propose Motion Belief Action (MBA), a decision layer on the scene map. From how often people move an object and how much force it needs, how much load it holds and whether it is blocked, MBA chooses under a force limit to look again, nudge, open, brace or decline, with its risk set by the task's costs. On real activity in 9 kitchens, it predicts whether an object is still in place better than FreMEn and Perpetua, and decides when to look again at lower cost than these and the persistence filter, because about a quarter of placements end in a pick-up within 30 s. On real pulls of doors and drawers in 17 locations, made by a person with a force-sensing gripper, the map's force estimates decide better than a part-type average. Setting the risk from the costs also beats a published risk rule at its fixed level. In simulation, with the force limit at the training parts' median force, MBA raises SmolVLA's and Octo's scores by declining pulls; most declined door pulls would fail, but on drawers more would have opened, and a part-type average gets about half of this gain. In both, a published risk rule set to the risk these costs imply is MBA's rule for a single pull and decides as well in the main settings. A link between opening and holding force, a probing nudge and an action prior for camera tracking do not help in the main settings, and over the real-pull sweep the nudge costs a little.
Figures & tables
| Question | Data and test set | Ground truth | Measures and settings | Published methods |
|---|---|---|---|---|
| Real pulls : do the map’s force estimates improve pulling decisions? | Hoi! [ 9 ] : pulls by a person with a force-sensing gripper ( Hz) on parts; the parts with both directions and a map entry, in locations | read from the recorded human pull; no robot acts. Open : peak force in the first half of the opening. Hold : peak of a closing pull (Hoi! has no holding trials). Nudge : did the part move before the force reached the nudge force (differs from the planner’s model on parts in the main setting). Blocked : none ( ). | score, unsafe; force limit: training locations’ median force (main), fixed N, sweep of settings | PhysVLMM’s risk query [ 11 ] (re-implemented) |
| Kitchen activity : is the object still in place? | HD-EPIC [ 28 ] : videos, objects, moves; queries in kitchens | annotated pick-ups: still in place if none starts within | Brier score, log-loss, re-look cost (extra cost over knowing the answer); s, min, min; look cost – | persistence filter [ 30 ] , FreMEn [ 17 ] , Perpetua [ 31 ] |
| Simulation : does MBA help vision-language-action policies at a safe force limit? | RoboCasa [ 26 ] in ManiSkill3 [ 37 ] , physics fitted to Hoi! [ 11 ] : kitchens; parts in kitchens | the executed episode: optional nudge, opening to half range within s, optional brace for s, by a force-limited controller at the handle (no arm or grasp) | score, opened, declined, damage (share of parts pushed above the safe limit for s); safe limit: training parts’ median force (main), robot limit N | SmolVLA [ 34 ] , Octo [ 27 ] , PhysVLMM’s risk query (re-implemented) |
| Wrist camera : does the action prior help tracking? | Hoi! gripper camera ( frames/s), one session in each of locations (the above and one more); frame pairs while holding a part | the recording device’s own camera trajectory | failed pair: rotation error or no pose; each SLAM system run three times | ORB-SLAM2 [ 25 ] , DynaSLAM [ 3 ] |
| Training locations’ median force | MBA method, other conditions | |||||
| Method | Score | Unsafe | MBA method | Simulated nudge | Fixed N | Sweep |
| MBA | 0.304 | 7.4 % | – | – | – | – |
| PhysVLMM risk query [ 11 ] , | 0.083 | 3.3 % | 258/10 | |||
| PhysVLMM risk query, | 0.322 | 7.4 % | 24/42 | |||
| Class-average prior | 25.6 % | 209/5 | ||||
| Never nudge | 0.322 | 7.4 % | 0/62 | |||
| Method | Brier | Log-loss | Re-look cost | Calib. | Re-look cost MBA |
| MBA | .0762 | .257 | .0472 | .026 | – |
| Persistence filter [ 30 ] | .0811 | .273 | .0522 | .011 | |
| FreMEn [ 17 ] | .0905 | .310 | .0603 | .021 | |
| Perpetua [ 31 ] | .0963 | .366 | .0705 | .069 | |
| MBA without quick pick-up | .1005 | .334 | .0576 | .097 | |
| MBA without object history | .0777 | .260 | .0488 | .022 |
| Method | Score | Opened | Declined | Damage | MBA method |
| SmolVLA + MBA | 0.165 | 33 % | 51 % | 0 a | – |
| SmolVLA [ 34 ] , safe limit | 0.101 | 55 % | 0 | 0 a | |
| SmolVLA, full N | 0.866 | 93 % | 0 | 75 % a | see Fig. 2 c |
| Octo + MBA | 0.188 | 34 % | 51 % | 0 | – |
| Octo [ 27 ] , safe limit | 0.144 | 57 % | 0 | 0 | |
| Octo, full N | 0.898 | 95 % | 0 | 63 % | see Fig. 2 c |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Share | Median err. ( ∘ ) | Fail ( ) | Err. | Flips |
|---|---|---|---|---|
| 0.0 | 0.23 | 16 % | 0 | 33 |
| 0.2 | 0.19 | 2 % | 0 | 5 |
| 0.3 | 0.19 | 3 % | 1 | 5 |
| 0.4 | 0.36 | 38 % | 69 | 1 |
| 0.45 | 2.95 | 77 % | 148 | 0 |
| 0.5 | 2.99 | 100 % | 194 | 0 |
| s | s | s | ||||
|---|---|---|---|---|---|---|
| Method | Brier | Re-look | Brier | Re-look | Brier | Re-look |
| MBA | .0271 | .0199 | .0659 | .0441 | .1357 | .0776 |
| Persistence filter | .0276 | .0207 | .0692 | .0478 | .1466 | .0881 |
| FreMEn | .0293 | .0233 | .0755 | .0561 | .1666 | .1016 |
| Perpetua | .0299 | .0232 | .0800 | .0636 | .1789 | .1247 |
| Method | Failed | Tracked | Method reference |
| MBA | 35.5 % | 100 % | – |
| Without the action prior (abl.) | 34.7 % | 100 % | |
| No part type (abl.) | 35.5 % | 100 % | |
| No image-position prior (abl.) | 35.5 % | 100 % | |
| Two-view odometry (ctrl.) | 38.3 % | 100 % | |
| Drop true co-moving (ref.) | 33.9 % | 99 % |
| Doors ( ) | Drawers ( ) | Microwaves ( ) | ||||
|---|---|---|---|---|---|---|
| Method | Score | Declined | Score | Declined | Score | Declined |
| MBA (scripted motion) | 0.174 | 42 % | 0.166 | 69 % | 0.381 | 28 % |
| SmolVLA + MBA | 0.154 | 42 % | 0.170 | 69 % | 0.381 | 28 % |
| Octo + MBA | 0.190 | 42 % | 0.170 | 69 % | 0.381 | 28 % |
| Class-average prior only (abl.) | 0.216 | 63 % | 0.000 | 100 % | 0.389 | 0 % |
| Scripted, safe limit (ctrl.) | 0.046 | 0 % | 0.226 | 0 % | 0.389 | 0 % |