cs.CVSep 16, 2026

Track, Articulate, Act: Generating Articulation from Casual Human Videos

Authors: Jiaming Zhang, Homanga Bharadhwaj

Organizations: Department of Computer Science, Johns Hopkins University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Human videos contain rich causal evidence for robot manipulation: they reveal how hand motion induces object motion and produces task-relevant changes in object state. In this work, we study articulated objects such as doors, drawers, cabinets, laptops, ovens, and hinged containers that are ubiquitous in daily life and present unique challenges for embodied interaction. These objects cannot be represented by a single pose; their motion depends on the underlying parts and joints. We introduce a real-to-sim framework that reconstructs a simulation-ready articulated object and hand-object interaction from a casual monocular RGB video, without RGB-D or multi-view input, prior scans, manually specified joints, or robot demonstrations. Our key insight is that dense 3D point tracks provide an embodiment-agnostic articulation cue: points on the fixed link remain approximately stationary, while points on the moving link follow coherent revolute or prismatic motion. Our method segments the links, estimates the joint and its state trajectory, reconstructs an articulated asset, and aligns the recovered 3D hand motion with the object. Central to our approach is a modular recipe that repurposes powerful pretrained models for single-image 3D reconstruction, mesh segmentation, and 3D scene flow, connecting their predictions through explicit geometric reasoning to infer articulation. We use the reconstructed articulated object and the human hand trajectory to replay interactions through contact in MuJoCo. The framework shows how pretrained vision models and explicit motion reasoning can turn casual human videos into articulated object models suitable for downstream embodied interactions. https://track-articulate-act.github.io/

Figures & tables

Figure 1: Overview . Starting from only a casual RGB human video, our method extracts motion cues, predicts articulation, constructs an approximate simulation-ready articulated object model, and replays the recovered hand-object interaction in a physics simulator. Results show that our framework is able to infer articulation for different types of joints including revolute and prismatic across diverse real-world objects.
Figure 2: Method. Given a casual human manipulation video, we select frames that expose different articulation states, reconstruct and register part-level geometry, and fit joint hypotheses to metrically grounded 3D tracks. We refine the joint through track and silhouette reprojection, then align and replay the recovered 3D hand motion with the articulated object in MuJoCo.
Figure 3: Articulation recovery from real human videos. From left to right, we show the RGB video input, 3D point tracks overlaid on an intermediate frame, and the reconstructed articulated asset. Our method handles both revolute and prismatic objects without any sensor depth or a prior object model.
 MethodType acc. ↑\uparrowAxis/dir. err. ( ∘ ) ↓\downarrowAxis loc. err. (mm) ↓\downarrow
w/ OrbitVideoArtGS [ 35 ]0.931.81±0.741.81\pm 0.742.44±3.862.44\pm 3.86
Articulat3D [ 14 ]0.911.53±1.171.53\pm 1.172.58±0.992.58\pm 0.99
w/o OrbitVideoArtGSn/an/an/a
Articulat3D0.856.15±9.036.15\pm 9.037.82±8.167.82\pm 8.16
Articulate-Anything [ 25 ]0.6630.66±25.9230.66\pm 25.9279.30±125.5879.30\pm 125.58
Ours0.971.96±0.45\mathbf{1.96\pm 0.45}3.93±1.013.93\pm 1.01
Table 1: Articulated joint estimation on simulation videos. This evaluation uses simulator-derived depth, tracks, and masks, and analyzes the articulation optimization instead of the full pipeline. (n/a) means the method fails to produce any result at given condition.
Figure 4: Qualitative results for articulated joint estimation on real data for our approach compared with Articulat3D. Each column shows the inferred joint for different objects.
InteractionJoint progress (%) ↑\uparrowFinal qq err. (%) ↓\downarrow
Laptop6335
Monitor3959
Stapler924
Puncher971
Mean7325
Table 2: Contact-driven interaction after re-targeting. We test whether the aligned and re-targeted hand can reproduce the demonstrated joint-state change in simulation, which is produced through contact and normalized against the demonstrated joint range.
Figure 5: Interactive hand re-targeting in MuJoCo. We align the reconstructed human hand with the articulated asset and re-target its motion to a simulated robot hand. Results show that the re-targeted hand, when actuated, is able to plausibly manipulate the object.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 1: Synthetic dataset for benchmarking the joint estimation method. There are two different camera configurations: orbit camera and static camera. The actual articulation sequences are interpolated between the given initial and final states of the moving part.
More in this topic

Articulated Object Reconstruction

Articulated object reconstruction recovers 3D part geometry, joints, and motion from images, video, or depth for digital twins and robot interaction.

See all 59 papers

Explore similar work

CardsList
Aug 5, 2026

RORA: Realistic Object Reconstruction with Articulation

Replicating real-world environments into simulation by realistic visual representation like NeRF and 3D Gaussian Splatting (3DGS) has emerged as an effective strategy to reduce the sim-to-real gap in robot learning. However, implementing object articulation during the real-to-sim process is still a challenging task. Existing motion tracking or learning based articulation methods shows low success rates on complex kinematic structures having multiple joints. Furthermore, those methods require scan of dynamic motion of objects, which makes reconstruction process much complicated. In this work, we propose the first end-to-end pipeline that reconstructs simulation-ready assets with accurate articulation from a single static object video input through suggestion based human-in-the-loop process. Our approach exports a hybrid representation combining 3DGS for photorealistic rendering and mesh-based geometry for physical interaction. In the reconstruction process, our pipeline performs convex decomposition followed by user grouping for intuitive part segmentation, subsequently binding 3D Gaussians to the corresponding mesh parts. An Automatic Joint Suggestion Algorithm then calculates candidate joint axes from local boundary geometries and presents them to users for efficient articulated asset reconstruction. We have shown that our method achieves precise articulation results on partnet-mobility-v0 dataset and real objects. Additionally we presented a potential usage of our framework on robot learning, deploying the reconstructed assets in Unreal Engine and NVIDIA Isaac Sim, demonstrating real-time dexterous hand manipulation tasks.

Hyesung Lee, Youngseon Lee, Kyutae Lee +2

Department of Mechanical Engineering, Seoul National University, Seoul, Korea · Department of Robotics and Mechatronics Engineering, DGIST, Daegu, Korea

Articulated Object Reconstruction3D Asset GenerationRobotics Simulation
AskPDF
Sep 23, 2026

Track2Art: Articulated Object Model Recovery with Visual-Geometric Track Representations

Understanding articulated objects is fundamental for robotic interaction, requiring accurate rigid-part discovery and the recovery of their kinematic relations. Existing approaches often treat articulation as a by-product of reconstructed geometry or recover it through per-instance optimization. We instead build on the hypothesis that articulation is directly observable from persistent motion: points on the same rigid part move coherently, while relative motion between parts reveals their kinematic constraints. We present Track2Art, a motion-centric framework for recovering structured articulated objects from RGB-D interaction videos. Track2Art lifts tracked image points into persistent 3D trajectories and combines pretrained tracking features, visual descriptors, and explicit trajectory geometry. These representations are grouped into a variable number of rigid-part hypotheses and subsequently used to recover directed kinematic relations, joint types, and joint geometry through rotation-equivariant learned--analytic reasoning. On the aligned 20-object PartNet-Mobility suite, Track2Art achieves 0.695 Point IoU and 0.410 end-to-end J@20, while requiring neither ground-truth part counts nor test-time optimization.

Xiaotong Li, Yixiong Jing, Junsheng Ding +4

Department of Engineering, University of Cambridge, Cambridge CB2 1PZ, U.K. · School of Engineering and Design, Technical University of Munich · Munich Center for Machine Learning (MCML), Munich, Germany.

Articulated Object Reconstruction3D Part Segmentation
AskPDF
Jun 29, 2026

UnfoldArt: Zero-Shot Recovery of Full Articulated 3D Objects from Text or Image

Articulated 3D objects are essential for interactive environments in embodied AI, robotics, and virtual reality, but reconstructing their structure and motion from sparse observations remains challenging. Existing approaches remain largely constrained by lack of supervised data or lack the priors needed to reliably recover articulation, hidden geometry, and internal object structure. We present the first debate-driven agentic approach to articulated 3D object reconstruction from text or image inputs that both grounds articulation reasoning in concrete motion and exposes the occluded geometry revealed under articulation. High-level agents reason about object semantics and motion using knowledge from vision-language and video models, while low-level agents estimate articulation parameters and interaction points; together, they engage in a two-round structured debate that first exploits global--local disagreement and then grounds the agents in freely generated video. The same video prior, conditioned on the agreed articulation, then drives each part through its motion to expose occluded interiors and geometry that cannot be inferred from a single static view. By combining agentic reasoning with a video generative prior, our approach jointly infers articulation and reconstructs complete 3D articulated objects, producing high-fidelity geometry, internal structure, and motion-consistent states beyond directly observed surfaces.

Mohamed el Amine Boudjoghra, Ivan Laptev, Angela Dai

1Technical University of Munich · 2Mohamed Bin Zayed University of Artificial Intelligence

Articulated Object Reconstruction3D Reconstruction
AskPDF