Articulated objects are ubiquitous in daily life. Our goal is to achieve a high-quality reconstruction, segmentation of independent moving parts, and analysis of articulation. Recent methods analyse two different articulation states and perform per-point part segmentation, optimising per-part articulation using cross-state correspondences, given a priori knowledge of the number of parts. Such assumptions greatly limit their applications and performance. Their robustness is reduced when objects cannot be clearly visible in both states. To address these issues, in this paper, we present a new framework, Articulation in Motion (AiM). We infer part-level decomposition, articulation kinematics, and reconstruct an interactive 3D digital replica from a user-object interaction video and a start-state scan. We propose a dual-Gaussian scene representation that is learned from an initial 3DGS scan of the object and a video that shows the movement of separate parts. It uses motion cues to segment the object into parts and assign articulation joints. Subsequently, a robust, sequential RANSAC is employed to achieve part mobility analysis without any part-level structural priors, which clusters moving primitives into rigid parts and estimates kinematics while automatically determining the number of parts. The proposed approach separates the object into parts, each represented as a 3D Gaussian set, enabling high-quality rendering. Our approach yields higher quality part segmentation than previous methods, without prior knowledge. Extensive experimental analysis on both simple and complex objects validates the effectiveness and strong generalisation ability of our approach. Project page: https://haoai-1997.github.io/AiM/.
Figures & tables
Figure 1: Left : Prior two-state methods often degrade on the sequences from closed-start to open-end. Right : Results of the proposed AiM , compared to ground truth (GT) geometry.
Figure 2: Left: DTA and ArtGS fail to recover from an incorrect input number of parts (4 here) and result in over-segmentations; Right : Visual results of DTA and ArtGS with closed-start and open-end states. The static part is gray and the moving part is green. In contrast, Ours requires no geometric priors and recovers accurate part-level segmentation from the continuous closed-start → open-end interaction process.
Figure 3: Overview of the first two stages: I) 3DGS start-state {GS} reconstruction from a multi-view RGB scan. II) A deformable 3DGS {GM,t} tracks motion video, while joint optimisation prunes moving components from {GS} . Pruned static Gaussian set {GpS} encodes the static base. An SDMD module handles newly revealed but static Gaussians. Together, these yield two separated Gaussian sets ( {GpS} and {GM,t} ) for the articulation analysis (Fig. 5 ).
Figure 4: Renderings of the start (left), and end (middle) states. Without SDMD detection, some newly revealed static parts are wrongly associated with the moving Gaussian set (right).
Figure 5: Stage III : Motion-based part segmentation and articulation analysis . As the clean {GM,t} provides time-varying trajectories, Sequential RANSAC groups trajectories into rigid parts (multi-part supported) without priors or optimisation, and directly outputs per-part articulation parameters. The green (top) and purple (bottom) points are our predicted moving Gaussians.
Table 1: Part segmentation performance on articulated objects. (a) Two-part; (b) Three-part; (c) Complex objects. For two-part objects, 3D IoU(%) is reported as mean ± std over 10 trials, while for three-part and complex objects, we report mean 3D IoU(%) over 10 trials. Davg represents the average over all movable parts. F denotes failure. Higher is better, with the best highlighted in bold .; Gray means two-state methods.
Table 2: Mesh reconstruction comparison. (a) Two-part objects; (b) Three-part objects; (c) Complex objects. For two-part objects, we report CD distance (mm) as mean ±std across 10 trials. For three-part and complex objects, we only report the mean value, while we report average CD for movable parts. Lower ( ↓ ) is better.
Table 3: Quantitative evaluation of articulation estimation. (a) Two-part; (b) Three-part; (c) Complex objects. For complex objects, we report the average of all moving parts. Due to the different magnitudes of part motion for revolute and prismatic joints, we report both of them. F denotes failure. WT denotes that more than 6 out of 10 trials result in an incorrect joint-type prediction. − indicates prismatic joints w/o rotation axis.
Figure 6: Qualitative results of part segmentation and articulation estimation on two two-part objects (fridge, left; oven, middle) and a complex multi-part object (Storage-47648, right). For complex object, each predicted joint axis is visualised using the same colour as its corresponding part segmentation mask. Across the two-part objects, DTA and ArtGS often struggle with mis-segmentation and inaccurate joint-axis/type predictions. In contrast, our method produces clean part segmentation and consistent joint-axis estimation across all objects.
Ang avg ↓
Pose avg ↓
Motion avgr↓
Motion avgp↓
CD-m avg ↓
3D IoU avgD↑
ArtGS
12.78
3.34
81.93
0.18
71.17
52.23
w/o start state scan
1.57
1.49
20.61
0.05
97.65
37.60
Storage
w/o SDMD
2.62
1.41
14.72
0.06
91.52
77.45
47648
w/o dual-GS
2.95
1.77
15.36
0.08
17.43
67.66
w/o RANSAC
3.80
1.41
12.62
0.38
78.54
67.06
Full
0.58
1.31
10.56
0.02
8.36
79.34
Table 4: Ablation studies on complex objects. We report the average metrics of dynamic parts. And we calculate the mean across three trials.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Scene
Blade 103706
Fridge 10905
Oven 101917
Scissor 11100
Stapler 103111
Storage 45135
USB 100109
Washer 103776
Motion type
Translate
Rotate
Rotate
Rotate
Rotate
Translate
Rotate
Rotate
Motion
0→0.5
−110∘→0∘
0∘→90∘
45∘→−45∘
0∘→−80∘
0→0.5
0∘→−90∘
0∘→−60∘
Appendix
Table A1: Motion type and motion records across ten scenes on the Two-part objects dataset
Scene
Part ID
Motion Type
Range
Part ID
Motion Type
Range
Fridge 11304
0
Rotate
0→−180∘
1
Rotate
0∘→−90∘
Storage 47024
0
Rotate
0∘→90∘
1
Translate
0→0.7
Appendix
Table A2: Motion types and ranges extracted from the two scenes of the Three-part objects dataset.
Scene
Part ID
Motion Type
Range
Part ID
Motion Type
Range
Storage 47648
0
Rotate
0→120∘
1
Rotate
0→−120∘
2
Rotate
0→−60∘
3
Rotate
0→60∘
4
Translate
0→0.1
5
Translate
0→0.16
Table 31249
0
Translate
0→0.38
1
Translate
0.35→0
3
Rotate
0→−90∘
4
Rotate
0→90∘
Appendix
Table A3: Motion types and ranges extracted from two additional scenes of the Complex objects dataset.
Figure A1: Examples from the Two objects dataset
Figure A2: Examples from the Three objects dataset
Figure A3: Examples from the Complex objects dataset
Figure A4: Rendering of our dual-Gaussian representation ( left : static result {RS} , right : moving result {RM,t=1} ). All objects ( categoryinstance ) are from PartNet-Mobility dataset ( Mo et al., 2019 ) .
Figure A5: Rendering results based on articulation estimation parameters for the start state of two-part objects: Top (fridge): From the visualisation, it can be observed that the newly seen interior content severely influences the performance of ArtGS’s articulation estimation, causing the door located inside the body. Bottom (oven): Similarly, during opening the oven, the newly seen content could not be well aligned between the two states, leading to wrong axis estimation and joint type estimation. The handle moves into the oven.
Figure A6: Qualitative comparison between DTA, ArtGS and ours, w.r.t. GT.
Method
Two-part
Three-part
Fridge
Storage
USB
Washer
Fridge (Joint0)
Fridge (Joint1)
Axis
Video2Articulation
3.80 ±0.00
6.53 ±0.00
1.89 ±0.00
Fail (100%)
2.17
1.35
Ang ( ∘ )
Ours
2.70 ±1.73
1.52 ±0.88
0.59 ±0.30
1.63 ±0.90
1.67
0.68
Axis
Video2Articulation
0.95 ±0.00
—
0.12 ±0.00
Fail (100%)
0.71
1.92
Pose (0.1m)
Ours
0.86 ±0.34
—
1.45 ±0.71
1.12 ±0.29
0.68
3.58
CD-m
Video2Articulation
8.06 ±0.16
141.95 ±12.02
24.68 ±0.49
Fail (100%)
2.88
41.38
Appendix
Table A4: Comparison results with Video2Articulation ( Peng et al., 2025 ) . We report four metrics, i.e. Axis Ang ( ∘ ), Axis Pose (0.1m), CD-m (mm) and CD-s (mm). Especially, for two-part objects, we report the metrics as mean ±std across 10 trials. For a three-part object, we report the mean value on different moving parts. Fail represents that Video2Articulation fails to detect the correct part segmentation masks, and ( % ) shows the failure rate. Bold means better performance.
Figure A7: Visual results of Video2Articulation Peng et al. (2025) (Left) and ours (Right) (Notably, as the expensive pre-processing process of Video2Articulation, we directly use their released dataset to reproduce Video2Articulation, there is some colour difference). Left : From the visualisations, we observe that the pre-trained-segmentation–driven Video2Articulation method fails to predict correct moving parts when the underlying pre-trained segmentation models cannot provide reliable masks. Typical failure cases include mis-segmenting the drawer and its cabinet, or losing track of the inside of the refrigerator door once it opens. Right: In contrast, our approach performs part mobility analysis by directly exploiting motion cues from the interaction video. After achieving clean static–dynamic separation, we apply multi-model fitting based on the inferred motion patterns, which leads to consistently more robust and accurate results.
Table A5: Quantitative evaluation of articulation estimation under the open-start and open-end conditions. (a) Two-part; (b) Three-part; (c) Complex objects. For complex objects, we report the average of all moving parts. Due to the different magnitudes of part motion for revolute and prismatic joints, we report both of them. − indicates prismatic joints w/o rotation axis.
Table A6: Mesh reconstruction comparison under the open-start and open-end condition. (a) Two-part objects; (b) Three-part objects; (c) Complex objects. For two-part objects, we report CD distance (mm) as mean ±std across 5 trials. For three-part and complex objects, we only report the mean value, while we report the average CD for movable parts. Lower ( ↓ ) is better.
Figure A8: The Meta Project Aria Glasses.
Figure A9: The captured interaction videos (The video could be found in the Supplementary Material). In particular, we use the hand as an indicator to automatically detect the motion start ( hand in ) and end times ( hand out ), enabling a fully automated data-processing pipeline.
Figure A10: The estimated pose of real-world video via using COLMAP Schonberger & Frahm (2016) .
Figure A11: Qualitative results of our AiM on the real-world data of the oven. Left: Comparison between the ground-truth views and rendered views. Besides, we provide the rendered masks based on our dual-Gaussian representation (via directly changing the spherical harmonics of Gaussians) . Due to the strong specular reflections on the oven’s glass door, the appearance of the moving part undergoes frequent and significant changes during interaction. Despite this challenge, our dual-Gaussian representation still achieves clean dynamic–static disentanglement by relying on stable motion cues. Moreover, the SDMD module reliably reassigns the newly revealed static interior regions back to the static base as the motion unfolds, further improving the quality of disentanglement and reconstruction. Right: Our prior-free part mobility analysis. Based on our dual-Gaussian representation, we can easily infer the trajectories of moving Gaussians, and obtain the articulation parameters based on the optimisation-free and robust sequential RANSAC without any prior structural knowledge.
Figure A12: Qualitative results of our AiM on the real-world data of storage. Left: Comparison between the ground-truth views and rendered views. Besides, we provide the rendered masks based on our dual-Gaussian representation (via directly changing the spherical harmonics of Gaussians). Although large regions of the storage are occluded by the oven and the hand during the interaction video (see Fig. A9 ), AiM remains robust and accurately performs dynamic–static disentanglement, enabling clean separation of the static base and the moving part purely from motion cues. Right: Our prior-free part mobility analysis. Based on our dual-Gaussian representation, we can easily infer the trajectories of moving Gaussians, and obtain the articulation parameters based on the optimisation-free and robust sequential RANSAC without any prior structural knowledge. (Notably, for prismatic joint, we do not consider the axis pose).
Figure A13: Qualitative comparisons for the ablation studies.
Figure A14: Part segmentation results based on Deformable 3DGS and our dual-Gaussian representation. We cluster Gaussians using the same sequential RANSAC settings; colours denote groups. Static noise in Deformable 3DGS noticeably degrades the segmentation.
Reconstructing articulated objects is essential for building digital twins of interactive environments. However, prior methods typically decouple geometry and motion by first reconstructing object shape in distinct states and then estimating articulation through post-hoc alignment. This separation complicates the reconstruction pipeline and restricts scalability, especially for objects with complex, multi-part articulation. We introduce a unified representation that jointly models geometry and motion using articulated 3D Gaussians. This formulation improves robustness in motion decomposition and supports articulated objects with up to 20 parts, significantly outperforming prior approaches that often struggle beyond 2--3 parts due to brittle initialization. To systematically assess scalability and generalization, we propose MPArt-90, a new benchmark consisting of 90 articulated objects across 20 categories, each with diverse part counts and motion configurations. Extensive experiments show that our method consistently achieves superior accuracy in part-level geometry reconstruction and motion estimation across a broad range of object types. We further demonstrate applicability to downstream tasks such as robotic simulation and human-scene interaction modeling, highlighting the potential of unified articulated representations in scalable physical modeling.
Understanding articulated objects is fundamental for robotic interaction, requiring accurate rigid-part discovery and the recovery of their kinematic relations. Existing approaches often treat articulation as a by-product of reconstructed geometry or recover it through per-instance optimization. We instead build on the hypothesis that articulation is directly observable from persistent motion: points on the same rigid part move coherently, while relative motion between parts reveals their kinematic constraints. We present Track2Art, a motion-centric framework for recovering structured articulated objects from RGB-D interaction videos. Track2Art lifts tracked image points into persistent 3D trajectories and combines pretrained tracking features, visual descriptors, and explicit trajectory geometry. These representations are grouped into a variable number of rigid-part hypotheses and subsequently used to recover directed kinematic relations, joint types, and joint geometry through rotation-equivariant learned--analytic reasoning. On the aligned 20-object PartNet-Mobility suite, Track2Art achieves 0.695 Point IoU and 0.410 end-to-end J@20, while requiring neither ground-truth part counts nor test-time optimization.
Xiaotong Li, Yixiong Jing, Junsheng Ding +4
Department of Engineering, University of Cambridge, Cambridge CB2 1PZ, U.K. · School of Engineering and Design, Technical University of Munich · Munich Center for Machine Learning (MCML), Munich, Germany.
Building interactive digital twins requires recovering both 3D geometry and the kinematic structures that govern how objects articulate. Yet existing methods for articulated object reconstruction require explicitly observable motion from multiple articulation states. We introduce a rest-state formulation that reconstructs articulated objects from a single closed configuration, an inherently ill-posed setting where geometry, semantics, and motion priors compensate for the absence of motion cues. Our framework adopts an explicit mesh as an intermediate representation for cross-model verification and fusion, reconciling noisy outputs from vision-language and segmentation models into spatially consistent part structures. To estimate joint parameters without observed motion, we use a video diffusion model to synthesize articulation hypotheses and validate them through geometric consistency. Our approach achieves accurate part decomposition and physically plausible articulation, performing competitively with motion-observing reconstruction-based, generation-based, and modular pretrained-model baselines.