PhysVLMM: Interaction-Conditioned Physical Belief Maps for Articulated Objects in Robot Scene Maps

Dibyendu Ghosh

cs.ROOct 29, 2026

PhysVLMM: Interaction-Conditioned Physical Belief Maps for Articulated Objects in Robot Scene Maps

Authors: Dibyendu Ghosh

Abstract

Mobile robots must open doors and drawers, yet their scene maps record what each object is and whether it has moved, not how it will respond to a pull. This paper proposes PhysVLMM, a method that maintains a physical belief for every articulated part of a Vision-Language-Motion Map and answers a planner's queries about it, with two algorithms: a map update and a query. Watching a part move provably cannot identify its dynamics, and a measured pull identifies only the pulled part, up to one direction. PhysVLMM therefore (i) keeps that direction as a posterior instead of a point estimate, (ii) passes the evidence of each pull in closed form to the untouched look-alike parts of the map, and (iii) answers whether an action will succeed at a stated risk, or abstains. On real doors and drawers in 18 homes, it identifies the joint type in 86% of held-out pulls (48% for a learned single-view method) and predicts the response to a pull with 41% lower error than differentiable simulation (CRPS 0.323 vs. 0.544), four orders of magnitude faster (13 ms vs. 230 s per case). The posterior lowers the final-position error from 0.460 to 0.270, sharing lowers the force error of untouched parts from 56% to 42%, and at every stated risk from 0.02 to 0.30 the false-safe answers stay within it. Pooling all measured parts of a home does as well as the look-alike groups, and simpler variants predict where a pulled part stops at least as well.

Figures & tables

Panel of Figure 1 — Figure 1: Architecture of PhysVLMM on one Hoi! scene: four look-alike drawers (blue), one pulled (orange). (a) Modules M1–M6, from sensors to the answer. (b) Look-alike prior of M4: a pull updates the group effort, and every untouched member receives the predictive effort ( 11 ).

Figure 2: Predicted effort after one measured pull (bars: 80 % interval; $\times$ : measured effort, held out). (a) Per-object: only the pulled drawer D3 is known. (b) PhysVLMM: the pull also informs its look-alikes (blue); the oven O, in another group, keeps the prior.

Figure 3: One recorded pull (wardrobe drawer); shaded: grasped phase.

Simulation (120 RoboCasa kitchens, 4327 parts, 12320 pulls)
	joint type	axis, drawer /	hinge	response	time
method	$\uparrow$	door ( ∘ ) $\downarrow$	(cm) $\downarrow$	$\downarrow$	$\downarrow$
FlowBot3D [ 3 ]	66 %	15.4 / 33.8	14.7	–
GAPartNet [ 7 ]	35 % a	70.6 / –	–	–
RigPI [ 11 ]	–	–	–	0.310
PhysVLMM	99 %	0.5 / 0.6	0.2	0.062

Table 1: Comparison with published methods (official code), same cases per block. Joint type: % correct; axis, hinge: median error; response: CRPS of the final position (RigPI minus PhysVLMM, 95 % CI: +0.249 [+0.239, +0.261] in simulation, +0.221 [+0.126, +0.312] on real data); time: median per case on one thread of the same CPU, measured on real data. –: not predicted. a Answers 42 % / 45 % of the parts / pulls. b On its 106 drawer pulls (PhysVLMM: 2.8 ∘ ). c GPU only. d Joint, physical belief and response. ∗ Code not released; ‡ released code lacks the articulation stage; † no joint response.

Figure 4: (a) Axis error on the held-out homes, PhysVLMM vs. FlowBot3D. (b) Final-position error, split by whether the pull reached the stop.

variant	description	physics fit	distribution	category prior	look-alike prior	force $\downarrow$	query $\downarrow$	final position $\downarrow$ ( $\Delta$ to PhysVLMM)
B1	joint only, no physics	✗	✗	✗	✗	–	–	0.164 ( $-$ 0.151 [ $-$ 0.236, $-$ 0.086])
B0	category prior only	✗	✓	✓	✗	56 %	0.184	0.253 ( $-$ 0.062 [ $-$ 0.124, $-$ 0.011])
B2 ${}_{\text{pt}}$	physics fit, point estimate	✓	✗	✗	✗	–	–	0.460 (+0.145 [+0.101, +0.186])
B2	physics fit, posterior	✓	✓	✗	✗	–	–	0.270 ( $-$ 0.045 [ $-$ 0.102, +0.004])
B3	B2 + category prior	✓	✓	✓	✗	56 %	0.184	0.285 ( $-$ 0.030 [ $-$ 0.059, $-$ 0.006])
PhysVLMM	B3 + look-alike prior	✓	✓	✓	✓	42 %	0.145	0.315

Table 2: Ablations (same inputs). –: cannot answer. Force, query: 140 untouched parts (17 homes); final position: CRPS of the final joint position of 253 held-out pulls (18 homes).

\Delta

: variant minus PhysVLMM (positive: PhysVLMM better). Control: mean of the home’s measured parts of the type.

variant	description	force $\downarrow$	query Brier $\downarrow$	final position $\downarrow$ ( $\Delta$ to PhysVLMM)
B1	joint only, no physics	–	–	0.743 (+0.682 [+0.675, +0.688])
B0	category prior only	58 %	0.183	0.131 (+0.070 [+0.066, +0.074])
B2 ${}_{\text{pt}}$	physics fit, point estimate	–	–	0.208 (+0.146 [+0.145, +0.149])
B2	physics fit, posterior	–	–	0.169 (+0.107 [+0.103, +0.110])
B3	B2 + category prior	58 %	0.183	0.064 (+0.003 [+0.002, +0.003])
PhysVLMM	B3 + look-alike prior	50 % (41 %)	0.165 (0.133)	0.062

Table 3: Simulation ablations, same inputs and parts. Final position: CRPS of the final joint position under fixed 3–12 N pulls (12320 pulls, the RigPI set),

\Delta

= variant minus PhysVLMM (positive: PhysVLMM better) with 95 % CI over layouts. Force and query: look-alike rule of the paper on the rendered part points; true groups in brackets (diagnostic).

Figure 5: Force error of untouched parts per home, per-object prior vs. PhysVLMM; labels: improvement factor.

Figure 6: Query “will

F

N keep it moving?” on untouched parts, swept over the stated risk

\delta

. (a) Share of “yes” answers that were false-safe (band: 95 % CI over homes for PhysVLMM; dotted:

\delta

). (b) Share of queries answered.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Worked door model (plan view). The door link rotates about the hinge axis by the joint coordinate $q$ ( 3 ); a wrench $w$ at the handle drives it through ( 4 ). The swept volume of the link is tested against the scanned scene; the first contact opening $q_{\mathrm{free}}$ bounds the collision-free motion.

applied torque profile	rank	condition no.
constant force, opening only	3/5	$\infty$
ramped force, opening only	4/5	$\infty$
pull, then release (closer returns the door)	5/5	17
two force levels, then release	5/5	13

Appendix

Table 4: Identifiability of the door dynamics

\theta=(I,b,\tau_{c},k,\tau_{0})

from a measured applied torque (simulated single revolute joint; rank of the Fisher information out of 5). Without a measured torque the rank is 0 ( Proposition 1 ).

home	parts	groups	pulls	per-object	PhysVLMM	factor
H2	2	1	4	14	13	1.1 $\times$
H3	8	1	16	69	27	2.6 $\times$
H4	11	2	21	79	68	1.2 $\times$
H5	5	2	9	49	25	1.9 $\times$
H6	5	1	10	63	71	0.9 $\times$
H7	6	1	12	26	15	1.7 $\times$

Appendix

Table 5: Test homes (Hoi! dataset) and the typical error (%) of the force predicted for a part before it is touched : per-object model (category prior) vs. the PhysVLMM map belief. Parts: parts with at least one look-alike; groups: look-alike groups of at least two parts; pulls: valid pulls of these parts; homes are numbered, not named.

predictor	trajectory RMSE $\downarrow$
B1: mechanism only (runs to the stop)	–
B0: category prior	0.543
B2 ${}_{\text{pt}}$ : physics fit, point estimate	0.468
B2: physics fit, posterior	0.608
B3: B2 + category prior	0.502
PhysVLMM : B3 + look-alike prior	0.488

Appendix

Table 6: Whole-trajectory error of the held-out response (RMSE as a fraction of the stroke; the same 253 pulls as Tab. 2 , whose motion column gives the final-position CRPS). –: B1 predicts no trajectory.

joint axis	held-out homes	doors	drawers
estimated	0.23	0.20	0.25
annotated (oracle)	0.19	0.16	0.22

Appendix

Table 7: Swept-volume collision check: share of parts for which a collision is predicted before 90 % of the opening a person actually reached (false alarm; lower is better). Development homes were used to set the edge tolerance.

force value	data	$\omega^{2}/s^{2}$ [CI]	look-alike pairs	other pairs
open + close	all windows	0.60 [0.41, 0.83]	0.35	0.46
opening	all windows	0.77 [0.57, 0.99]	0.55	0.58
closing	all windows	0.49 [0.28, 0.76]	0.41	0.64
open + close	suspect windows removed	0.60 [0.41, 0.81]	0.37	0.58
opening	suspect windows removed	0.82 [0.62, 1.02]	0.59	0.58
closing	suspect windows removed	0.44 [0.23, 0.73]	0.38	0.68

Appendix

Table 8: How alike are look-alikes? Within-group variance

\omega^{2}

over the variance of all parts

s^{2}

(log force; 0 = look-alikes identical, 1 = no more alike than random parts), with 95 % bootstrap CI over groups, and the mean squared log-force difference of look-alike pairs vs. other pairs in the same home. Fixed rule: GO if

\omega^{2}/s^{2}\leq 0.25

, STOP if

\geq 0.5

	running force				speed-norm.
predictor	NLL	MAE	$\times$	cov.	NLL	MAE
category prior (per-object)	0.852	0.446	1.56	0.79	1.184	0.636
mean of the same home	0.661	0.349	1.42	0.80	0.974	0.477
mean of the look-alikes	0.695	0.355	1.43	0.81	1.073	0.512
hierarchical pooling	0.662	0.352	1.42	0.81	0.980	0.495
outlier-aware pooling	0.635	0.342	1.41	0.81	1.015	0.503

Appendix

Table 9: Predicting an untouched part’s force from the measured forces of the other parts in its home (leave-one-part-out; hyper-parameters fitted on the other homes). NLL and MAE of log force (lower is better);

\times

= typical multiplicative error; cov. = coverage of the 80 % interval. Left: running force; right: speed-normalised force.

predictor (touch order)	running force	speed-norm.
no sharing	0.771	0.808
hierarchical (random)	0.673	0.765
hierarchical (greedy)	0.670	0.747
outlier-aware (random)	0.675	0.768
outlier-aware (greedy)	0.695	0.753

Appendix

Table 10: Touch one, know many: share of a home’s parts the robot must touch before 80 % of them are known within a factor of 1.25 (touched, or predicted within that factor). Mean over homes; random = 20 random touch orders; greedy = largest summed variance reduction.

		per-object prior			PhysVLMM map belief
grouping	$F$	Brier	answered	false-safe	Brier	answered	false-safe
geometric	3 N	0.188	0.00	–	0.135	0.19	–
geometric	5 N	0.247	0.00	–	0.192	0.01	0.00
geometric	8 N	0.117	0.09	0.50	0.108	0.47	0.09
semantic	3 N	0.204	0.00	–	0.131	0.32	0.00
semantic	5 N	0.218	0.00	–	0.137	0.24	0.00

Appendix

Table 11: Query “will

F

N keep part

j

moving?” before touching

j

(

\delta=0.1

). False-safe: “yes” although more force is needed.

$\rho$	running force	speed-normalised
0.0	$-$ 0.04 [ $-$ 0.16, +0.08]	+0.05 [ $-$ 0.00, +0.14]
0.1	$-$ 0.01 [ $-$ 0.13, +0.12]	+0.05 [+0.00, +0.11]
0.2	$-$ 0.05 [ $-$ 0.15, +0.06]	+0.03 [+0.01, +0.06]
0.3	$-$ 0.01 [ $-$ 0.12, +0.11]	+0.02 [+0.00, +0.05]
0.5	+0.01 [ $-$ 0.09, +0.12]	+0.04 [+0.01, +0.09]

Appendix

Table 12: Grouping mistakes: each part is moved to a wrong look-alike group with probability

\rho

(5 seeds). NLL difference outlier-aware

-

hierarchical pooling with 95 % CI over homes (negative = outlier-aware better).

home	type	parts	running force per part (N)
H3	revolute	8	3.8, 1.7, 2.9, 2.3, 2.2, 3.0, 3.7, 3.2
H4	revolute	3	1.5, 2.7, 1.5
H4	prismatic	8	6.8, 10.1, 12.3, 4.1, 2.7, 3.7, 3.5, 3.0
H5	prismatic	3	7.3, 5.5, 6.9
H6	prismatic	5	6.5, 6.1, 1.9, 3.7, 2.2
H7	revolute	6	3.3, 3.6, 3.5, 3.6, 4.8, 3.3

Appendix

Table 13: Running force (N) of every part in look-alike groups with at least 3 parts; homes are numbered, not named.

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,704 papers

Vision-Language-Motion Maps: An Open-Vocabulary, Uncertainty-Aware, Queryable Motion Attribute for 3D Scene Maps
Jul 17, 2026Dibyendu Ghosh, Ayushi ShakyaUncertainty Quantification 3D Scene Representation
Rekise Marine, Bangalore, India
Open-World Law-to-Action: Vision-Language Priors and Touch for Parts with Hidden Physics
Oct 24, 2026Mohamed A. MabrokLanguage-Conditioned Robot Manipulation Robot Dynamics Learning
Qatar University
PhysMem: Scaling Test-Time Memory for Embodied Physical Reasoning
Feb 23, 2026Haoyang Li, Yang You, Hao Su +1Memory-Augmented Agents Embodied Reasoning
UC San Diego · Stanford University

Jul 17, 2026J/K move · Enter open · S save

Vision-Language-Motion Maps: An Open-Vocabulary, Uncertainty-Aware, Queryable Motion Attribute for 3D Scene Maps

Dibyendu Ghosh, Ayushi Shakya

Rekise Marine, Bangalore, India

Uncertainty Quantification 3D Scene Representation Robotic Perception 3D Mapping 3D Semantic Mapping

Open paper AskPDF arXiv

Open-vocabulary 3D maps let robots answer language queries about what and where, but they assume a static world and cannot answer queries about how scene elements behave. We introduce Vision-Language-Motion Maps (VLMM), an open-vocabulary, language-queryable 3D map - queried through a rule-based intent router over open-vocabulary object nouns, not a general natural-language interface - in which each element carries a fused motion attribute: a VLM/LLM semantic movability prior combined with geometrically observed cross-frame motion, together with a per-element uncertainty. Queries reduce to attribute filters that distinguish what has been seen to move, what could move but has not, and what stays still. On a controlled simulator benchmark with exact ground truth (AI2-THOR, three scene types) we show through ablation that the schema fields are non-substitutable: a semantic-only baseline fails motion queries even with strong features, and neither motion field substitutes for the other (the prior cannot answer "what is moving," observed motion cannot answer "what could move"). On real dynamic RGB-D (TUM and Bonn, six sequences) we show the uncertainty channel - our key difference from prior fused-motion work - consistently improves moving-vs-static average precision and reduces false motion flags, and that it is robust to estimated (noisy) poses. The raw confidence is not calibrated, but post-hoc isotonic calibration reaches an expected calibration error of 0.10. VLMM is a representation contribution: the closest prior maps each lack at least one of the four properties - open-vocabulary, language-queryable, fused prior-and-observed motion, and per-element uncertainty - that our combination provides.

Abstract

Robotic Manipulation

Explore similar work

Vision-Language-Motion Maps: An Open-Vocabulary, Uncertainty-Aware, Queryable Motion Attribute for 3D Scene Maps

Open-World Law-to-Action: Vision-Language Priors and Touch for Parts with Hidden Physics

PhysMem: Scaling Test-Time Memory for Embodied Physical Reasoning