PAWS: Perception of Articulation in the Wild at Scale from Egocentric Videos
Authors: Yihao Wang, Yang Miao, Wenshuai Zhao, Wenyan Yang, Zihan Wang, Joni Pajarinen, Luc Van Gool, Danda Pani Paudel, Juho Kannala, Xi Wang, Arno Solin
Organizations: Aalto University · ELLIS Institute Finland · INSAIT, Sofia University · ETH Zurich · University of Oulu · TU Munich · MCML
Articulation perception aims to recover the motion and structure of articulated objects (e.g., drawers and cupboards), and is fundamental to 3D scene understanding in robotics, simulation, and animation. Existing learning-based methods rely heavily on supervised training with high-quality 3D data and manual annotations, limiting scalability and diversity. To address this limitation, we propose PAWS, a method that directly extracts object articulations from hand-object interactions in large-scale in-the-wild egocentric videos. We evaluate our method on the public data sets, including HD-EPIC and Arti4D data sets, achieving significant improvements over baselines. We further demonstrate that the extracted articulations benefit downstream tasks, including fine-tuning 3D articulation prediction models and enabling robot manipulation. See the project website at https://aaltoml.github.io/PAWS/.
Figures & tables
Figure 1: PAWS : Articulation perception and localization from in-the-wild egocentric videos. (a) From raw videos of human interactions, (b) our method reconstructs the 3D scene and object articulations using hand cues, geometric recovery, and VLM reasoning. (c) These serve as annotations to improve downstream articulation prediction models via finetuning, while also providing 3D priors for real-world robotic manipulation.
Method
Input
Supervision
Output
Articulation3D [ 67 ]
RGBs
SL, 3D anno.
3D planes
OPDMulti [ 73 ]
RGB
SL, 2D masks
2D masks
USDNet [ 23 ]
PC
SL, 3D anno.
PC
RSRD [ 34 ]
RGBs
SSL, RGBs
Meshes
Articulate-Anything [ 40 ]
RGBs
Transfer Learning
Meshes
iTACO [ 63 ]
RGB-Ds
Transfer Learning
Meshes
Table 1: Comparison to prior art. We summarize related articulation perception methods by input modality, articulation perception strategy (SL: supervised learning; SSL: self-supervised learning), supervision format, and output format. PC denotes point clouds, which correspond to high-fidelity scanned meshes in [ 23 ] .
Figure 2: Overall pipeline. Given a full in-the-wild egocentric video and a language description as input, our pipeline consists of four parts: (1) Dynamic Interaction Perception: We first segment the video based on the language description and extract interactive frames (referred to as "local views"), 3D hand trajectories, motion types, and coarse object localizations. (2) Geometric Structure Recovery: Based on the object’s location, we select "global views" from the full video. Depending on the motion type, we recover the scene geometry using different flows. (3) VLM-guided Reasoning: The VLM first infers the motion type to provide a prior for global view selection, and then identifies plausible articulation axes during the geometry recovery stage. (4) Joint Articulation Inference: We integrate 3D hand trajectories and the recovered geometry to infer the final articulations.
Algorithm 1 Static Geometry Recovery Pipeline
Figure 3: Illustration of VLM Reasoning. (a) Temporal Motion Type Classification. (b) Spatial Axis Grounding via Set-of-Marks VQA.
HD-EPIC [ 64 ]
Arti4D [ 82 ]
Methods
Match ( % )
M
MA
MAO
M †
MA †
MAO †
Match ( % )
M
MA
MAO
M †
MA †
MAO †
Articulation3D ∗ [ 67 ]
22.68
0.24
0.12
0.08
0.95
0.48
0.35
45.07
0.32
0.03
0.02
0.70
0.07
0.05
Articulation3D ∗∗ [ 67 ]
42.22
0.34
0.15
0.11
0.81
0.37
0.27
84.29
0.53
0.06
0.03
0.62
0.07
0.04
ArtiPoint [ 82 ]
70.66
0.47
0.06
0.00
0.72
0.09
0.01
85.02
0.63
0.52
0.47
0.75
0.61
0.56
PAWS
55.38
0.52
0.36
0.20
0.96
0.66
0.35
48.02
0.47
0.37
0.34
1.00
0.80
0.75
PAWS-s
71.43
0.71
0.46
0.36
0.98
0.63
0.48
–
–
–
–
–
–
–
Table 2: Quantitative results on articulation estimation accuracy. Our method achieves higher accuracy on the HD-Epic [ 64 ] and Arti4D [ 82 ] data sets, outperforming the RGB-D–based articulation extraction baseline. Note that in the HD-Epic data set, since not every articulable fixture involves hand interaction, we evaluate only the actually interactive fixtures. The evaluated pairs means the actual evaluated articulation. We denote PAWS-s as aggregated articulation estimates at the scene level. Since the scene configuration varies across videos in the Arti4D data set, we do not report aggregated results for this data set.
Ablation Type
Model / Configuration Variant
M
MA
MAO
Architecture Components
w/o Filter, w/o GPT-4o (Heuristic)
0.46
0.28
0.13
w/ Filter, w/o GPT-4o
0.45
0.30
0.15
w/o Filter, w/ GPT-4o
0.47
0.32
0.18
Full PAWS (Filter + GPT-4o)
0.46
0.33
0.18
Table 3: Ablation study. Component-wise ablation demonstrating the necessity of the Manhattan Geometric Filter and the GPT-4o reasoning engine.
HD-Epic [ 64 ]
Arti4D [ 82 ]
Methods
M
MS
MSA
MSAO
M
MA
MAO
Articulate3D (Zero-shot)
0.46
0.02
0.02
0.02
0.41
0.14
0.12
Articulate3D (Finetuned)
0.58
0.13
0.08
0.05
0.48
0.22
0.17
Table 4: Quantitative results on articulation prediction using the EgoArti benchmark. The results before and after fine-tuning on our data set demonstrate the effectiveness of our benchmark. MS refers to the average accuracy of predictions with correct motion type and correct object mask prediction (IoU larger than 50 % ), similar cases for MSA and MSAO . We evaluate different fine-tuning strategies across multiple data sets.
Figure 4: PAWS for robot manipulation. Left: Spot closes the cupboard. Right: Spot opens the drawer. Insets show egocentric videos of hand-object interactions and the reconstructed 3D articulations.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Notation
Definition
System Input & Temporal Sets
V
Egocentric RGB video ( ∈RT×H×W×3 ) with T frames
L,lj,M
Set of M action descriptions; lj is a single action
vj
Temporal clip for action lj
K
Frames sparsely sampled from vj
K′
Frames with visible hands for MANO estimation
Appendix
Table A5 : Summary of key mathematical notations used in PAWS.
Figure A5 : Illustration of the hand filtering pipeline. Starting from noisy MANO fingertip observations zt , we apply contact-based trimming, forward Kalman filtering with χ2 outlier rejection and RTS smoothing to obtain the refined trajectories {p^t} used for articulation parameter estimation.
Figure A6 : Prompt for dynamic interaction perception.
Figure A7 : Prompt for dynamic interaction perception (cont’d).
Figure A8 : Prompt for selecting plausible rotation axes.
Module
Parameter
Value
Hand Trajectory Smoothing
Length scale ℓ
10.0 m
Observation noise σobs
0.05 m
Process noise std q
0.01 m
χ2 gating p -value ( df=3 )
0.05
Scene Reconstruction & Localization
Voxel size
0.05 m
Min. views for high-confidence voxel
4
Appendix
Table A6: Key hyperparameters of the PAWS pipeline.
Figure A9 : Visualization of annotated scenes in extended HD-EPIC [ 64 ] data set.
Figure A10 : An example of the benchmark list.
Figure A11 : Scene-level aggregation. Multiple articulated object instances (highlighted by bounding boxes) are localized and annotated within the same reconstructed 3D scene, enabling scene-level aggregation of articulation estimates across repeated interactions.
Figure A12 : Qualitative comparison of articulation prediction. We compare our method against Articulate-Anything [ 40 ] and ArtiPoint [ 82 ] across four articulation tasks. The yellow arrows denote the predicted articulation. Where applicable, we also visualize the object and hand trajectories.
Figure A13 : Failure Case Examples. (a, b) A fridge may be misclassified as a “cupboard” due to its visually ambiguous exterior, causing both VLM reasoning and grounding to fail. (c, d) Examples where the interacting hand does not appear in the frame throughout the entire sequence, leading to tracking failures.
Human videos contain rich causal evidence for robot manipulation: they reveal how hand motion induces object motion and produces task-relevant changes in object state. In this work, we study articulated objects such as doors, drawers, cabinets, laptops, ovens, and hinged containers that are ubiquitous in daily life and present unique challenges for embodied interaction. These objects cannot be represented by a single pose; their motion depends on the underlying parts and joints. We introduce a real-to-sim framework that reconstructs a simulation-ready articulated object and hand-object interaction from a casual monocular RGB video, without RGB-D or multi-view input, prior scans, manually specified joints, or robot demonstrations. Our key insight is that dense 3D point tracks provide an embodiment-agnostic articulation cue: points on the fixed link remain approximately stationary, while points on the moving link follow coherent revolute or prismatic motion. Our method segments the links, estimates the joint and its state trajectory, reconstructs an articulated asset, and aligns the recovered 3D hand motion with the object. Central to our approach is a modular recipe that repurposes powerful pretrained models for single-image 3D reconstruction, mesh segmentation, and 3D scene flow, connecting their predictions through explicit geometric reasoning to infer articulation. We use the reconstructed articulated object and the human hand trajectory to replay interactions through contact in MuJoCo. The framework shows how pretrained vision models and explicit motion reasoning can turn casual human videos into articulated object models suitable for downstream embodied interactions. https://track-articulate-act.github.io/
Jiaming Zhang, Homanga Bharadhwaj
Department of Computer Science, Johns Hopkins University
Articulation modeling enables robots to learn joint parameters of articulated objects for effective manipulation which can then be used downstream for skill learning or planning. Existing approaches often rely on prior knowledge about the objects, such as the number or type of joints. Some of these approaches also fail to recover occluded joints that are only revealed during interaction. Others require large numbers of multi-view images for every object, which is impractical in real-world settings. Furthermore, prior works neglect the order of manipulations, which is essential for many multi-DoF objects where one joint must be operated before another, such as a dishwasher. We introduce PokeNet, an end-to-end framework that estimates articulation models from a single human demonstration without prior object knowledge. Given a sequence of point cloud observations of a human manipulating an unknown object, PokeNet predicts joint parameters, infers manipulation order, and tracks joint states over time. PokeNet outperforms existing state-of-the-art methods, improving joint axis and state estimation accuracy by an average of over 27% across diverse objects, including novel and unseen categories. We demonstrate these gains in both simulation and real-world environments.
Anmol Gupta, Weiwei Gu, Omkar Patil +2
School of Computation and AI, ASU, Tempe · AI Institute, Seoul National University, Seoul, South Korea
Retrieving the 3D kinematics of articulated objects from monocular video is a fundamental challenge in computer vision. Existing methods rely on complex video setups or cues such as long-term point tracking or wide-baseline matching, but are frequently brittle under severe occlusions, rapid camera ego-motion, or weak local features. Learning-based methods, meanwhile, struggle to generalize beyond their training categories. We propose a category-agnostic optimization framework that treats articulated object understanding as a primitive-fitting problem. Geometric primitives serve as a proxy representation that avoids the pitfalls of unstable point tracks; a novel mechanism organizes them into coherent parts constrained by revolute and prismatic joints. Our formulation jointly optimizes part segmentation and joint parameters, recovering complex kinematics from a single casually captured video. A visibility-aware procedure handles partial observations and occlusions inherent to real-world data. We also propose the AiP-synth and AiP-real benchmarks, featuring significant camera motion and heavy occlusions, and outperform existing methods. Project page: https://aartykov.github.io/Articulation-in-Prime/
Arslan Artykov, Tom Ravaud, Nicolás Violante-Grezzi +1
LIGM, CNRS, Univ Gustave Eiffel, ENPC, Institut Polytechnique de Paris, France