Structured 3D representations such as keypoints and meshes offer compact, expressive descriptions of deformable objects, jointly capturing geometric and topological information useful for downstream tasks such as dynamics modeling and motion planning. However, robustly extracting such representations remains challenging, as current perception methods struggle to handle complex deformations. Moreover, large-scale 3D data collection remains a bottleneck: existing approaches either require prohibitive data collection efforts, such as labor-intensive annotation or expensive motion capture setups, or rely on simplifying assumptions that break down in unstructured environments. As a result, large-scale 3D datasets and benchmarks for deformable objects remain scarce. To address these challenges, this paper presents an affordable and autonomous framework for collecting 3D datasets of deformable objects using only RGB-D cameras. The proposed method identifies 3D keypoints and robustly tracks their trajectories, incorporating motion consistency constraints to produce temporally smooth and geometrically coherent data. TrackDeform3D is evaluated against several state-of-the-art tracking methods across diverse object categories and demonstrates consistent improvements in both geometric and tracking accuracy. Using this framework, this paper presents a high-quality, large-scale dataset consisting of 6 deformable objects, totaling 110 minutes of trajectory data. Project page: https://roahmlab.github.io/trackDeform3D-core-tracking/
Figures & tables
Fig. 1: This paper presents TrackDeform3D, a system for keypoint initialization, tracking, and dataset generation for 1D and 2D deformable objects during dual-arm robotic manipulation. For each object category, the figure illustrates tracked keypoint trajectories during manipulation and the resulting 3D trajectory data. Color encodes spatial position across keypoints, while transparency encodes temporal progression, with earlier frames appearing more transparent and later frames more opaque.
Fig. 2: Overview of TrackDeform3D. Given an RGB-D video, TrackDeform3D first lifts depth images to point clouds and segments the deformable object via point cloud differencing (§ III-C ). From the first segmented frame, the object type is classified and 3D keypoints are initialized by detecting anchor points, generating warm-start positions, inferring object topology, and solving a constrained geometric optimization (§ III-B , § III-D ). During tracking, anchor points are re-detected at each frame and the same optimization is solved recursively, warm-started from the previous solution. A temporal moving-average filter is applied to suppress high-frequency jitter, producing smooth and temporally consistent 3D keypoint trajectories (§ III-E ).
Fig. 3: An illustration of anchor point placement on different deformable object categories: DLO, BDLO, fabric, and T-shirt.
Fig. 5: Upper: Qualitative comparison of keypoint tracking results between TrackDeform3D and all baseline methods on a cloth sequence at the start and end of manipulation. While CDCPD2, the best-performing baseline, maintains track of the deformable object, it exhibits inconsistent edge lengths and distorted mesh topology over time. SpatialTracker and CoTracker both show significant keypoint drift by the end of the sequence. Bottom: Edge length RMSE and Chamfer distance over time for all methods.
Method
Rope
Branched Rope
Fabric
Abs. (mm)
E<5 (%)
CD (mm)
F<10 (%)
Abs. (mm)
E<5 (%)
CD (mm)
F<10 (%)
Abs. (mm)
E<5 (%)
CD (mm)
F<10 (%)
CoTracker3 [ 19 ]
42.31
1.0
8.54
74.9
20.35
24.7
42.87
36.4
25.09
61.3
20.60
13.2
CDCPD2 [ 9 ]
3.89
25.3
10.73
63.7
5.08
42.8
8.02
80.3
13.28
20.4
13.97
47.7
SpatialTrackerV2 [ 21 ]
40.21
53.6
121.15
0.3
182.9
20.8
253.8
0.5
22.42
12.2
88.58
8.74
TrackDeform3D (Ours)
3.19
63.3
8.04
74.1
3.83
87.8
6.70
87.3
2.99
97.6
7.52
77.8
TABLE I: Baseline comparison. Abs. denotes absolute edge length error. CD denotes Chamfer distance. E<5 denotes the percentage of frames with Abs. below 5 mm. F<10 denotes the fraction of frames with CD below 10 mm over the entire trajectory.
Method
Rope
Branched Rope
Fabric
Abs. (mm)
E<5 (%)
CD (mm)
F<10 (%)
Abs. (mm)
E<5 (%)
CD (mm)
F<10 (%)
Abs. (mm)
E<5 (%)
CD (mm)
F<10 (%)
w/o Anchor
3.19
63.3
8.04
74.1
9.56
62.7
15.00
60.0
6.44
56.7
22.72
58.0
w/o Repulsion
74.34
6.8
14.49
59.9
52.35
9.9
8.43
77.6
48.21
70.0
10.46
65.4
w/o Projection
1.49
96.0
120.83
3.9
3.51
91.6
27.44
24.69
2.81
99.6
17.11
44.5
TrackDeform3D (Ours)
3.19
63.3
8.04
74.1
3.83
87.8
6.70
87.3
2.99
97.6
7.52
77.8
TABLE II: Ablation study. Metrics are defined in Table I .
Variant
Branched Rope
Fabric
Abs. (mm)
CD (mm)
F<10 (%)
Abs. (mm)
CD (mm)
F<10 (%)
w/o Anchor
2.45
3.16
95.3
3.89
9.94
61.9
w/o Repulsion
15.08
3.08
96.4
22.28
5.50
86.2
w/o Projection
1.81
4.58
91.2
2.87
8.70
66.2
Full
1.62
3.06
96.1
2.58
6.66
79.0
TABLE III: Initialization consistency ablation. Metrics are defined in Table I .
Fig. 6: Demonstration of keypoint initialization consistency across four data collection sessions, showing that the full formulation produces stable and repeatable keypoint layouts.
DOT [ 5 ]
Mocap [ 4 ]
Ours
DLO objects
1
0
2
BDLO objects
0
0
2
2D objects
2
4
2
Total time
20 min
9.8 min
110 min
Motion sequences
220
120
1320
TABLE IV: Dataset comparison with existing deformable object tracking benchmarks.
Fig. 7: Example 3D trajectory visualizations from the collected dataset for a BDLO and cloth object. Color encodes spatial position across keypoints, while transparency encodes temporal progression, with earlier frames appearing more transparent and later frames more opaque. Each trajectory preserves the keypoint topology throughout the manipulation sequence.
Predicting object dynamics (i.e., world modeling) is a fundamental challenge for robotic manipulation, and modeling deformable objects presents a particularly difficult case due to their high-dimensional state spaces and complex material properties. While current world models approach this through two distinct paradigms: learning the dynamics over the 2D pixel space or more explicit 3D geometric space. A systematic understanding of their relative strengths and limitations remains elusive due to the lack of diverse, large-scale real-world data. To address this, we present Deform360, a large-scale visuotactile dataset featuring 198 daily-life objects, 1,980 interaction sequences, and over 215 hours of observations from 41 surround-view cameras and bimanual tactile grippers to capture both global motion and contact-induced local deformations. Leveraging a novel markerless visuotactile 3D tracking pipeline to extract dense geometry and motion, we systematically evaluate current state-of-the-art world models, comparing 2D video models against 3D particle models. Finally, we provide a preliminary demonstration indicating the real-world applicability of our dataset by performing robot planning tasks on deformable objects. Our analysis reveals key insights into the trade-offs between structural priors and scalability, providing a solid benchmark for future research in generalizable deformable object-centric world modeling. Project website: https://deform360.lhy.xyz
Hongyu Li, Wanjia Fu, Xiaoyan Cong +11
Brown University, Providence, RI · Columbia University, New York, NY · Massachusetts Institute of Technology, Cambridge, MA
We present DeSOPE, a large-scale dataset for 6DoF deformed objects. Most 6D object pose methods assume rigid or articulated objects, an assumption that fails in practice as objects deviate from their canonical shapes due to wear, impact, or deformation. To model this, we introduce the DeSOPE dataset, which features high-fidelity 3D scans of 26 common object categories, each captured in one canonical state and three deformed configurations, with accurate 3D registration to the canonical mesh. Additionally, it features an RGB-D dataset with 133K frames across diverse scenarios and 665K pose annotations produced via a semi-automatic pipeline. We begin by annotating 2D masks for each instance, then compute initial poses using an object pose method, refine them through an object-level SLAM system, and finally perform manual verification to produce the final annotations. We evaluate several object pose methods and find that performance drops sharply with increasing deformation, suggesting that robust handling of such deformations is critical for practical applications.
Zhiqiang Liu, Rui Song, Duanmu Chuangqi +3
State Key Laboratory of ISN, Xidian University · MagicLeap
Monocular 3D object detection is crucial for scalable perception across fields like autonomous driving, robotics, and surveillance. However, progress is hindered by limited 3D annotations and the inherent ambiguity of single-image geometry. Existing methods often rely on strong geometric assumptions or carefully curated datasets, which limit their applicability to real-world scenarios. In this paper, we present PLOT (Pseudo-Labeling via Object Tracking), a framework that generates 3D annotations from monocular videos without auxiliary sensors or model retraining. PLOT tracks object and background trajectories to estimate camera motion and perform object association in pose-unknown settings. These trajectories provide point correspondences that align frame-wise pseudo-LiDARs, which are then fused via simple optimization into a unified object shape robust to occlusion and viewpoint shifts. Recognizing temporal coherence as a fundamental requirement for reliable shape fusion and video perception, we design a global object memory that preserves consistent object identities across frames. PLOT achieves robust annotation quality and strong generalization on both M3OD video benchmarks and in-the-wild videos, proving its effectiveness across diverse and unconstrained domains. Project page: https://plot-eccv.github.io.
Seokyeong Lee, Sithu Aung, Junyong Choi +3
KAIST AI · VRG, FEE, Czech Technical University in Prague · Samsung Research +3