PAWS: Perception of Articulation in the Wild at Scale from Egocentric Videos

Yihao Wang; Yang Miao; Wenshuai Zhao; Wenyan Yang; Zihan Wang; Joni Pajarinen; Luc Van Gool; Danda Pani Paudel

cs.CVMar 26, 2026

PAWS: Perception of Articulation in the Wild at Scale from Egocentric Videos

Authors: Yihao Wang, Yang Miao, Wenshuai Zhao, Wenyan Yang, Zihan Wang, Joni Pajarinen, Luc Van Gool, Danda Pani Paudel,

Organizations: Aalto University · ELLIS Institute Finland · INSAIT, Sofia University · ETH Zurich ·

Abstract

Articulation perception aims to recover the motion and structure of articulated objects (e.g., drawers and cupboards), and is fundamental to 3D scene understanding in robotics, simulation, and animation. Existing learning-based methods rely heavily on supervised training with high-quality 3D data and manual annotations, limiting scalability and diversity. To address this limitation, we propose PAWS, a method that directly extracts object articulations from hand-object interactions in large-scale in-the-wild egocentric videos. We evaluate our method on the public data sets, including HD-EPIC and Arti4D data sets, achieving significant improvements over baselines. We further demonstrate that the extracted articulations benefit downstream tasks, including fine-tuning 3D articulation prediction models and enabling robot manipulation. See the project website at https://aaltoml.github.io/PAWS/.

Figures & tables

Panel of Figure 1 — Figure 1: PAWS : Articulation perception and localization from in-the-wild egocentric videos. (a) From raw videos of human interactions, (b) our method reconstructs the 3D scene and object articulations using hand cues, geometric recovery, and VLM reasoning. (c) These serve as annotations to improve downstream articulation prediction models via finetuning, while also providing 3D priors for real-world robotic manipulation.

Method	Input	Supervision	Output
Articulation3D [ 67 ]	RGBs	SL, 3D anno.	3D planes
OPDMulti [ 73 ]	RGB	SL, 2D masks	2D masks
USDNet [ 23 ]	PC	SL, 3D anno.	PC
RSRD [ 34 ]	RGBs	SSL, RGBs	Meshes
Articulate-Anything [ 40 ]	RGBs	Transfer Learning	Meshes
iTACO [ 63 ]	RGB-Ds	Transfer Learning	Meshes

Table 1: Comparison to prior art. We summarize related articulation perception methods by input modality, articulation perception strategy (SL: supervised learning; SSL: self-supervised learning), supervision format, and output format. PC denotes point clouds, which correspond to high-fidelity scanned meshes in [ 23 ] .

Figure 2: Overall pipeline. Given a full in-the-wild egocentric video and a language description as input, our pipeline consists of four parts: (1) Dynamic Interaction Perception: We first segment the video based on the language description and extract interactive frames (referred to as "local views"), 3D hand trajectories, motion types, and coarse object localizations. (2) Geometric Structure Recovery: Based on the object’s location, we select "global views" from the full video. Depending on the motion type, we recover the scene geometry using different flows. (3) VLM-guided Reasoning: The VLM first infers the motion type to provide a prior for global view selection, and then identifies plausible articulation axes during the geometry recovery stage. (4) Joint Articulation Inference: We integrate 3D hand trajectories and the recovered geometry to infer the final articulations.

Panel of Figure 4 — Algorithm 1 Static Geometry Recovery Pipeline

Figure 3: Illustration of VLM Reasoning. (a) Temporal Motion Type Classification. (b) Spatial Axis Grounding via Set-of-Marks VQA.

	HD-EPIC [ 64 ]							Arti4D [ 82 ]
Methods	Match ( $\%$ )	M	MA	MAO	M †	MA †	MAO †	Match ( $\%$ )	M	MA	MAO	M †	MA †	MAO †
Articulation3D ∗ [ 67 ]	22.68	0.24	0.12	0.08	0.95	0.48	0.35	45.07	0.32	0.03	0.02	0.70	0.07	0.05
Articulation3D ∗∗ [ 67 ]	42.22	0.34	0.15	0.11	0.81	0.37	0.27	84.29	0.53	0.06	0.03	0.62	0.07	0.04
ArtiPoint [ 82 ]	70.66	0.47	0.06	0.00	0.72	0.09	0.01	85.02	0.63	0.52	0.47	0.75	0.61	0.56
PAWS	55.38	0.52	0.36	0.20	0.96	0.66	0.35	48.02	0.47	0.37	0.34	1.00	0.80	0.75
PAWS-s	71.43	0.71	0.46	0.36	0.98	0.63	0.48	–	–	–	–	–	–	–

Table 2: Quantitative results on articulation estimation accuracy. Our method achieves higher accuracy on the HD-Epic [ 64 ] and Arti4D [ 82 ] data sets, outperforming the RGB-D–based articulation extraction baseline. Note that in the HD-Epic data set, since not every articulable fixture involves hand interaction, we evaluate only the actually interactive fixtures. The evaluated pairs means the actual evaluated articulation. We denote PAWS-s as aggregated articulation estimates at the scene level. Since the scene configuration varies across videos in the Arti4D data set, we do not report aggregated results for this data set.

Ablation Type	Model / Configuration Variant	M	MA	MAO
Architecture Components	w/o Filter, w/o GPT-4o (Heuristic)	0.46	0.28	0.13
	w/ Filter, w/o GPT-4o	0.45	0.30	0.15
	w/o Filter, w/ GPT-4o	0.47	0.32	0.18
	Full PAWS (Filter + GPT-4o)	0.46	0.33	0.18

Table 3: Ablation study. Component-wise ablation demonstrating the necessity of the Manhattan Geometric Filter and the GPT-4o reasoning engine.

	HD-Epic [ 64 ]				Arti4D [ 82 ]
Methods	M	MS	MSA	MSAO	M	MA	MAO
Articulate3D (Zero-shot)	0.46	0.02	0.02	0.02	0.41	0.14	0.12
Articulate3D (Finetuned)	0.58	0.13	0.08	0.05	0.48	0.22	0.17

Table 4: Quantitative results on articulation prediction using the EgoArti benchmark. The results before and after fine-tuning on our data set demonstrate the effectiveness of our benchmark. MS refers to the average accuracy of predictions with correct motion type and correct object mask prediction (IoU larger than 50

\%

), similar cases for MSA and MSAO . We evaluate different fine-tuning strategies across multiple data sets.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

System Input & Temporal Sets
Notation	Definition
$\mathbf{V}$	Egocentric RGB video ( $\in\mathbb{R}^{T\times H\times W\times 3}$ ) with $T$ frames
$\mathcal{L},l_{j},M$	Set of $M$ action descriptions; $l_{j}$ is a single action
$\mathbf{v}_{j}$	Temporal clip for action $l_{j}$
$K$	Frames sparsely sampled from $\mathbf{v}_{j}$
$K^{\prime}$	Frames with visible hands for MANO estimation

Appendix

Table A5 : Summary of key mathematical notations used in PAWS.

Figure A5 : Illustration of the hand filtering pipeline. Starting from noisy MANO fingertip observations $\mathbf{z}_{t}$ , we apply contact-based trimming, forward Kalman filtering with $\chi^{2}$ outlier rejection and RTS smoothing to obtain the refined trajectories $\{\hat{\mathbf{p}}_{t}\}$ used for articulation parameter estimation.

Panel of Figure 12 — Figure A6 : Prompt for dynamic interaction perception.

Panel of Figure 13 — Figure A7 : Prompt for dynamic interaction perception (cont’d).

Panel of Figure 14 — Figure A8 : Prompt for selecting plausible rotation axes.

Module	Parameter	Value
Hand Trajectory Smoothing	Length scale $\ell$	$10.0$ m
	Observation noise $\sigma_{\mathrm{obs}}$	$0.05$ m
	Process noise std $q$	$0.01$ m
	$\chi^{2}$ gating $p$ -value ( $\mathrm{df}{=}3$ )	$0.05$
Scene Reconstruction & Localization	Voxel size	$0.05$ m
	Min. views for high-confidence voxel	$4$

Appendix

Table A6: Key hyperparameters of the PAWS pipeline.

Panel of Figure 16 — Figure A9 : Visualization of annotated scenes in extended HD-EPIC [ 64 ] data set.

Panel of Figure 17 — Figure A10 : An example of the benchmark list.

Figure A11 : Scene-level aggregation. Multiple articulated object instances (highlighted by bounding boxes) are localized and annotated within the same reconstructed 3D scene, enabling scene-level aggregation of articulation estimates across repeated interactions.

Panel of Figure 19 — Figure A12 : Qualitative comparison of articulation prediction. We compare our method against Articulate-Anything [ 40 ] and ArtiPoint [ 82 ] across four articulation tasks. The yellow arrows denote the predicted articulation. Where applicable, we also visualize the object and hand trajectories.

Panel of Figure 20 — Figure A13 : Failure Case Examples. (a, b) A fridge may be misclassified as a “cupboard” due to its visually ambiguous exterior, causing both VLM reasoning and grounding to fail. (c, d) Examples where the interacting hand does not appear in the frame throughout the entire sequence, leading to tracking failures.

Articulated Object Reconstruction

Articulated object reconstruction recovers 3D part geometry, joints, and motion from images, video, or depth for digital twins and robot interaction.

See all 60 papers

Track, Articulate, Act: Generating Articulation from Casual Human Videos
Sep 16, 2026Jiaming Zhang, Homanga BharadhwajArticulated Object Reconstruction Articulated Object Manipulation
Department of Computer Science, Johns Hopkins University
PokeNet: Learning Kinematic Models of Articulated Objects from Human Observations
Feb 2, 2026Anmol Gupta, Weiwei Gu, Omkar Patil +2Articulated Object Manipulation Robotic Perception
School of Computation and AI, ASU, Tempe · AI Institute, Seoul National University, Seoul, South Korea
Articulation in Prime: Primitive-Based Articulated Object Understanding from a Single Casual Video
May 18, 2026Arslan Artykov, Tom Ravaud, Nicolás Violante-Grezzi +1Articulated Object Reconstruction 3D Reconstruction
LIGM, CNRS, Univ Gustave Eiffel, ENPC, Institut Polytechnique de Paris, France

Sep 16, 2026J/K move · Enter open · S save

Track, Articulate, Act: Generating Articulation from Casual Human Videos

Jiaming Zhang, Homanga Bharadhwaj

Department of Computer Science, Johns Hopkins University

Articulated Object Reconstruction Articulated Object Manipulation Hand-Object Interaction Reconstruction Hand-Object Interaction

Open paper AskPDF arXiv

Human videos contain rich causal evidence for robot manipulation: they reveal how hand motion induces object motion and produces task-relevant changes in object state. In this work, we study articulated objects such as doors, drawers, cabinets, laptops, ovens, and hinged containers that are ubiquitous in daily life and present unique challenges for embodied interaction. These objects cannot be represented by a single pose; their motion depends on the underlying parts and joints. We introduce a real-to-sim framework that reconstructs a simulation-ready articulated object and hand-object interaction from a casual monocular RGB video, without RGB-D or multi-view input, prior scans, manually specified joints, or robot demonstrations. Our key insight is that dense 3D point tracks provide an embodiment-agnostic articulation cue: points on the fixed link remain approximately stationary, while points on the moving link follow coherent revolute or prismatic motion. Our method segments the links, estimates the joint and its state trajectory, reconstructs an articulated asset, and aligns the recovered 3D hand motion with the object. Central to our approach is a modular recipe that repurposes powerful pretrained models for single-image 3D reconstruction, mesh segmentation, and 3D scene flow, connecting their predictions through explicit geometric reasoning to infer articulation. We use the reconstructed articulated object and the human hand trajectory to replay interactions through contact in MuJoCo. The framework shows how pretrained vision models and explicit motion reasoning can turn casual human videos into articulated object models suitable for downstream embodied interactions. https://track-articulate-act.github.io/

Abstract

Articulated Object Reconstruction

Explore similar work

Track, Articulate, Act: Generating Articulation from Casual Human Videos

PokeNet: Learning Kinematic Models of Articulated Objects from Human Observations

Articulation in Prime: Primitive-Based Articulated Object Understanding from a Single Casual Video