PhyVGGT: Feed-Forward Articulated Object Reconstruction from Monocular Videos
Authors: Qi Shao, Zhihao Cao, Shuhao Zhai, Anh Nguyen, Hong Liang, Baoru Huang, Liang Li
Organizations: Harbin Engineering University, Harbin, China · University of Liverpool, Liverpool, U.K. · ETH Zurich, Zurich, Switzerland · National University of Singapore, Singapore · University of Macau, Macau, China · Imperial College London, London, U.K.
Reconstructing an articulated object with its 3D movable parts, joint parameters and unseen articulation states from an uncalibrated monocular video is essential for robot manipulation and interactive digital twins. Existing per-scene optimisation methods require minutes to hours and external depth or camera poses, whereas feed-forward 3D foundation models reconstruct dense point maps in one pass without segmenting movable parts or inferring joints. Even on these monocular point maps, regressing or fitting continuous joints directly fails on casual videos, since cross-frame attention blurs moving part boundaries in deep layers, underestimates motion along the viewing ray and leaves continuous axes ill-conditioned under small opening angles. To address these challenges, we propose PhyVGGT, a single-pass framework on a frozen 3D foundation model that anchors moving-part matching and closed-form joint estimation to the undistorted geometry of the stationary base. First, our Part and Motion Decoder fuses intermediate per-frame tokens with global tokens and references part queries to a static-base query to segment 3D parts, align point maps on the static base and predict learned Bingham priors. Second, our Kinematic Inference Layer fits an orthogonal Manhattan frame to static surface normals, selects the joint axis in closed form under a Bingham posterior combining pairwise motion scatter with the learned priors, and solves a linear system for the hinge pivot to drive unseen states. Trained solely on synthetic data, PhyVGGT achieves state-of-the-art monocular performance across five benchmarks, outperforming FAMOS and monocular ArtiPoint, surpassing posed RGB-D ArtiPoint in median error, and running in 6.8 s per 48-frame video against hours of per-scene optimisation in Articulat3D. Project page: https://phyvggt.github.io.
Figures & tables
Fig. 1: Overview of single-pass 3D articulated object reconstruction from uncalibrated monocular RGB videos. Given casual RGB frames ( left ), PhyVGGT segments 3D movable parts (teal), estimates joint axes (orange) and hinge pivots (red) across observed articulation states, and drives each part to unseen states in 1.2 s per clip ( centre ), generalising zero-shot from synthetic training to diverse real-world scenes ( right ).
Fig. 2: Architecture overview of PhyVGGT. Given an uncalibrated monocular video, a frozen VGGT backbone extracts patch, per-frame and global tokens alongside 3D point maps ( DPT ) and tracking features ( Track ). First, our trainable Part and Motion Decoder fuses these streams and updates learnable base and part queries across N attention blocks to output part masks ( Mask Head ), correspondences and Bingham priors ( Joint Head ). Second, our Kinematic Inference Layer fits a Manhattan Frame to static normals, scores its three axes via a three-way softmax ( Axis Selection ) and solves for joint type, Hinge Pivot and Unseen States via Forward Kinematics .
Fig. 3: Mechanism of the Kinematic Inference Layer on real point maps. Dense static-base normals {ni} form an orthogonal Manhattan frame G=[g1,g2,g3] ( Candidates ). Combining pairwise motion scatter Sk with the learned Bingham prior Mk defines an antipodal Bingham posterior p(a∣Bk) ( Criterion ), selecting ak to solve for the hinge pivot ok and states {qkt} ( Selected ).
Fig. 4: Qualitative comparison of 3D joint axis estimation on Arti4D test videos. Blue shows ArtiPoint with RGB-D input and ground-truth poses, purple shows FAMOS on PhyVGGT point maps, orange shows RGB-only PhyVGGT, and green dashed lines denote ground truth. Arrows indicate prismatic joints while lines indicate revolute joints, with ∗ marking wrong joint types.
Arti4D (329)
D3D-HOI (170)
Method
≤5∘
≤10∘
Med.
≤5∘
≤10∘
Med.
ArtiPoint, RGB-D ∗
0.422
0.614
6.30
–
–
–
Direct regression
0.006
0.020
57.07
0.022
0.063
37.15
OPDMulti
0.146
0.292
64.15
0.147
0.418
11.55
ArtiPoint, π3
0.198
0.389
15.90
–
–
–
FAMOS
0.241
0.369
23.24
0.108
0.145
68.93
TABLE I: Comparison on real-world Arti4D and D3D-HOI. Joint accuracy and median axis error in degrees. ∗ Sensor depth and poses.
Fig. 5: Qualitative 3D part reconstructions and unseen articulation states on Arti4D test videos. For each scene, we show two input RGB frames alongside PhyVGGT’s 3D point map near the rest state, in which the segmented movable part (teal) is driven by the predicted joint axis (orange) and hinge pivot (red) to the start and end of the observed motion and then to an unseen articulation state via forward kinematics. Drawers show the volume swept by the part along the predicted axis, with axis angular errors listed above.
Method
≤5∘
≤10∘
Med.
Rev.
Pris.
ArtiPoint, RGB-D ∗
0.148
0.246
51.3
0.253
0.229
Direct regression
0.015
0.046
48.7
0.041
0.059
OPDMulti
0.050
0.129
44.7
0.126
0.135
ArtiPoint, π3
0.034
0.098
64.2
0.119
0.042
FAMOS
0.045
0.119
68.9
0.153
0.024
Motion estimate
0.063
0.185
29.2
0.206
0.128
TABLE II: Comparison on real-world RBO. Joint accuracy, median axis error in degrees, and revolute/prismatic accuracy at 10∘ . ∗ Sensor depth and poses.
Benchmark
Method
≤5∘
≤10∘
Med.
Syn-10
Direct regression
0.061
0.198
20.60
(290)
OPDMulti ‡
0.028
0.072
75.03
FAMOS
0.426
0.529
7.08
Motion estimate
0.366
0.543
7.96
PhyVGGT (Ours)
0.762
0.778
1.53
Syn-38
Direct regression
0.042
0.128
33.40
TABLE III: Comparison on synthetic PartNet-Mobility. Joint accuracy and median axis error on 10 trained categories in Syn-10 and 38 categories with 28 unseen in Syn-38. ‡ Pretrained on openable furniture.
Trained
All joints (48)
Unseen
Method
Med.
≤5∘
≤10∘
≤10∘
Time
Articulat3D ∗
2.63
0.479
0.521
0.316
2.6–11.9 h
Direct regression
21.67
0.035
0.139
0.000
6.8 s
Motion estimate
11.79
0.215
0.347
0.158
6.8 s
PhyVGGT (Ours)
2.20
0.431
0.486
0.246
6.8 s
TABLE IV: Comparison on 48-frame Articulat3D-Sim. ∗ Per-scene optimisation with GT depth, poses, masks and kinematic trees.
3D Part & Surface Geometry
Syn-10-val
Syn-38-val
Pt. err.
Final layer only ( l=23 )
0.877 / 0.663
0.590 / 0.542
0.032
w/o Tracking features
0.888 / 0.694
0.608 / 0.564
0.032
PhyVGGT (Ours)
0.891 / 0.701
0.617 / 0.576
0.032
3D Hinge Pivot (Med. cm / ≤ 10 cm)
Syn-10 (290)
Syn-38 (510)
RBO (357)
Direct regression
6.49 / 0.64
5.70 / 0.68
9.34 / 0.53
FAMOS [ 10 ]
4.35 / 0.69
4.96 / 0.67
11.14 / 0.46
TABLE V: 3D part, surface and hinge pivot reconstruction. Upper rows report 3D movable / part IoU and surface error, and lower rows report held-out hinge median error in cm and accuracy within 10 cm.
Configuration
Arti4D-dev
D3D-dev
Syn-10-val
Syn-38-val
Direct regression
0.063
0.058
0.188
0.125
+ Part and Motion Decoder
0.259
0.198
0.390
0.303
+ Motion scatter Sk only
0.482
0.628
0.497
0.386
+ Bingham prior Mk on G only
0.522
0.620
0.778
0.601
+ Full Kinematic Inference Layer (Ours)
0.725
0.744
0.803
0.636
TABLE VI: Additive ablation study on the four development splits. Joint accuracy at 10∘ as each module and cue is introduced.
Method
Backbone
Arti4D-dev
D3D-dev
Syn-10-val
Direct regression
VGGT-1B
0.063
0.058
0.188
ArtiPoint [ 19 ]
VGGT-1B
0.271
–
–
FAMOS [ 10 ]
VGGT-1B
0.376
0.116
0.502
Motion estimate
VGGT-1B
0.482
0.628
0.497
PhyVGGT (Ours)
VGGT-1B
0.725
0.744
0.803
PhyVGGT (Ours)
StreamVGGT † [ 29 ]
0.753
0.783
0.743
TABLE VII: Controlled VGGT-1B comparison and zero-shot backbone transfer. Upper rows share VGGT-1B point maps and masks at 10∘ , and lower rows ( † ) swap unseen backbones zero-shot.
Reconstructing articulated 3D objects from a single image requires jointly inferring object geometry, part structure, and motion parameters from limited visual evidence. A key difficulty lies in the entanglement between motion cues and object structure, which makes direct articulation regression unstable. Existing methods address this challenge through multi-view supervision, retrieval-based assembly, or auxiliary video generation, often sacrificing scalability or efficiency. We present MonoArt, a unified framework grounded in progressive structural reasoning. Rather than predicting articulation directly from image features, MonoArt progressively transforms visual observations into canonical geometry, structured part representations, and motion-aware embeddings within a single architecture. This structured reasoning process enables stable and interpretable articulation inference without external motion templates or multi-stage pipelines. Extensive experiments on PartNet-Mobility demonstrate that OM achieves state-of-the-art performance in both reconstruction accuracy and inference speed. The framework further generalizes to robotic manipulation and articulated scene reconstruction.
Human videos contain rich causal evidence for robot manipulation: they reveal how hand motion induces object motion and produces task-relevant changes in object state. In this work, we study articulated objects such as doors, drawers, cabinets, laptops, ovens, and hinged containers that are ubiquitous in daily life and present unique challenges for embodied interaction. These objects cannot be represented by a single pose; their motion depends on the underlying parts and joints. We introduce a real-to-sim framework that reconstructs a simulation-ready articulated object and hand-object interaction from a casual monocular RGB video, without RGB-D or multi-view input, prior scans, manually specified joints, or robot demonstrations. Our key insight is that dense 3D point tracks provide an embodiment-agnostic articulation cue: points on the fixed link remain approximately stationary, while points on the moving link follow coherent revolute or prismatic motion. Our method segments the links, estimates the joint and its state trajectory, reconstructs an articulated asset, and aligns the recovered 3D hand motion with the object. Central to our approach is a modular recipe that repurposes powerful pretrained models for single-image 3D reconstruction, mesh segmentation, and 3D scene flow, connecting their predictions through explicit geometric reasoning to infer articulation. We use the reconstructed articulated object and the human hand trajectory to replay interactions through contact in MuJoCo. The framework shows how pretrained vision models and explicit motion reasoning can turn casual human videos into articulated object models suitable for downstream embodied interactions. https://track-articulate-act.github.io/
Jiaming Zhang, Homanga Bharadhwaj
Department of Computer Science, Johns Hopkins University
Understanding articulated objects is fundamental for robotic interaction, requiring accurate rigid-part discovery and the recovery of their kinematic relations. Existing approaches often treat articulation as a by-product of reconstructed geometry or recover it through per-instance optimization. We instead build on the hypothesis that articulation is directly observable from persistent motion: points on the same rigid part move coherently, while relative motion between parts reveals their kinematic constraints. We present Track2Art, a motion-centric framework for recovering structured articulated objects from RGB-D interaction videos. Track2Art lifts tracked image points into persistent 3D trajectories and combines pretrained tracking features, visual descriptors, and explicit trajectory geometry. These representations are grouped into a variable number of rigid-part hypotheses and subsequently used to recover directed kinematic relations, joint types, and joint geometry through rotation-equivariant learned--analytic reasoning. On the aligned 20-object PartNet-Mobility suite, Track2Art achieves 0.695 Point IoU and 0.410 end-to-end J@20, while requiring neither ground-truth part counts nor test-time optimization.
Xiaotong Li, Yixiong Jing, Junsheng Ding +4
Department of Engineering, University of Cambridge, Cambridge CB2 1PZ, U.K. · School of Engineering and Design, Technical University of Munich · Munich Center for Machine Learning (MCML), Munich, Germany.