AuRA: From Atoms to Bits through Autonomous Robot Interaction

Xinchen Yu; Yoonkyo Jung; Haoran Chen; Zikui Cai; Aayush Talreja; Seungjae Lee; Hyunjin Kim; Furong Huang

cs.ROOct 25, 2026

AuRA: From Atoms to Bits through Autonomous Robot Interaction

Authors: Xinchen Yu, Yoonkyo Jung, Haoran Chen, Zikui Cai, Aayush Talreja, Seungjae Lee, Hyunjin Kim, Furong Huang

Organizations: University of Maryland, College Park · All Purpose AI

Abstract

Simulation enables robot-learning experiments at scale, but acquiring physics-informed digital twins often requires human manipulation and manual annotation of recorded interactions. To reduce this reliance on human effort, we present AuRA, a system that automates robot interaction and data preparation for digital-twin reconstruction. AuRA combines execution-aware grasp selection, gripper-motion measurements derived from robot states, and registered pre-interaction geometry and appearance priors. These choices enable autonomous collection, avoid reliance on visual gripper tracking, and recover object information obscured by the robot. Across 13 deformable objects, AuRA reduces active human labor and wall-clock collection time per successfully reconstructed asset by 88.5% and 55.4%, respectively. Relative to the human-collected PhysTwin baseline, AuRA improves seven of nine metrics for future prediction under known gripper motion, including a 28.1% reduction in one-way $L_1$ Chamfer distance. It also achieves similar mean silhouette IoU and Chamfer distance under the evaluated unseen interactions. These results highlight the potential of autonomous robot interaction to support more efficient digital-twin acquisition with comparable reconstruction quality.

Figures & tables

Fig. 2: AuRA’s autonomous physical-to-digital acquisition pipeline. Multi-view RGB-D capture supports grasp generation and execution-feasibility filtering. The robot executes the selected grasp and records the interaction with synchronized robot states. During processing, visual tracking provides object trajectories, while robot poses and calibrated gripper geometry provide controller trajectories. AuRA also constructs a static point cloud and trains a Gaussian appearance model from its pre-interaction captures, then aligns both with the first interaction frame to supplement robot-occluded regions. These inputs support spring–mass optimization and Gaussian rendering through the PhysTwin reconstruction backend.

Parameter	Default value
Valid depth measurement range	$0.2<d<1.5$ m
Reference table height $z_{\mathrm{table}}$	$0$ m
Below-table tolerance $\epsilon$	$0.01$ m
Downsampling voxel size	$0.003$ m
Neighbors for statistical filtering	$35$
Statistical rejection threshold	$1.2$ standard deviations

TABLE I: Point-cloud preprocessing parameters for grasp pose generation.

Fig. 3: Data collection efficiency and reconstruction reliability. Human (Overall) pools the three groups below the dotted line. (a) End-to-end reconstruction rate: AuRA fails during acquisition, whereas human recordings fail during data processing. (b) Wall-clock time (

\circ

) and human labor (

\bullet

) per reconstructed asset (log scale): the two are equal for human collection. (c) Mean failure detection time: AuRA detects failures after 88.3 s on average, compared with 133.0 s for human collection overall and 89.2 s for experts. Collection costs in (b) exclude data processing, reconstruction training, and untimed backup trials.

Metric	Resimulation		Future Prediction
Metric	PhysTwin	AuRA	PhysTwin	AuRA
PSNR (dB) $\uparrow$	12.289	11.549	10.325	10.590
SSIM $\uparrow$	0.9856	0.9852	0.9805	0.9797
LPIPS $\downarrow$	0.0126	0.0148	0.0233	0.0230
IoU $\uparrow$	0.6675	0.6090	0.4607	0.5001
Precision $\uparrow$	0.7521	0.7021	0.5899	0.6132

TABLE II: Resimulation and future-prediction quality. Trajectory-level metrics are averaged within each object and then equally across 13 objects. PhysTwin uses 29 human trajectories, with shape priors when available; AuRA uses 26 robot trajectories processed by its standard pipeline. PhysTwin (CoTracker3) and AuRA (MV-TAP) retain an average of 1,493.76 and 615.96 tracking points per trajectory, respectively. All available views with usable object masks are used in evaluation.

First-Frame Alignment
Metric	PhysTwin	AuRA
Metric	PhysTwin	Static GS	First-Frame GS
Mask IoU $\uparrow$	0.5096	0.4939	0.4095
Mask Precision $\uparrow$	0.6228	0.5817	0.5285
Mask Recall $\uparrow$	0.7146	0.7650	0.6752
Unseen Interaction

TABLE III: First-frame alignment and unseen-interaction quality. The upper block reports silhouette metrics at the aligned first frame; the lower block evaluates the subsequent interaction. Bold indicates the best value in each row.

Panel of Figure 6 — Fig. 4: Object appearance at the first frame of an unseen interaction. From left to right: the original data-collection frame 0, the rendered object at unseen interaction frame 0 after manual asset alignment, its overlay with the unseen-interaction ground truth (GT), and that GT image.

Resimulation
Metric	CoTracker3-m	CoTracker3-p	MVTracker	MV-TAP
PSNR (dB) $\uparrow$	11.341 / 11.629	11.245 / 11.520	11.445 / 11.543	11.468 / 11.549
SSIM $\uparrow$	0.9843 / 0.9852	0.9843 / 0.9851	0.9847 / 0.9851	0.9848 / 0.9852
LPIPS $\downarrow$	0.0173 / 0.0148	0.0172 / 0.0149	0.0162 / 0.0149	0.0159 / 0.0148
Mask IoU $\uparrow$	0.5651 / 0.6131	0.5564 / 0.6093	0.5826 / 0.6067	0.5875 / 0.6090
Mask Precision $\uparrow$	0.6577 / 0.7040	0.6539 / 0.7015	0.6726 / 0.6966	0.6825 / 0.7021

TABLE IV: Object-tracker and pre-interaction static geometry-prior ablations. All configurations use the same 26 robot-collected trajectories and static Gaussians. Each cell reports results without / with the geometry prior. CoTracker3-m refers to mask-based initialization and CoTracker3-p refers to point-cloud-based initialization, respectively. Bold indicates the best score across all methods within each prior setting, and underlining marks the better CoTracker3 initialization. Both use unrounded values.

Robotic Data Collection

Robotic data collection develops ways to gather, label, and validate robot and human-generated experience for learning, evaluation, and sensing.

See all 178 papers

ArtiTwinSplat: Interactable Digital Twin Reconstruction via Gaussian Splatting from RGB-D videos
Jun 23, 2026Pranjal Mishra, René Zurbrügg, Max Wilder-Smith +4Articulated Object Reconstruction 3D Gaussian Splatting
1ETH Zürich, Zürich, Switzerland. · 2Microsoft, Zürich, Switzerland. · University of Bonn, Bonn, Germany.
ForceTwin: Physics-informed Digital Twins for Robotic Manipulation from Instrumented Human Interaction
Sep 18, 2026Tim Engelbracht, René Zurbrügg, Mayank Mittal +4Digital Twins Robotic Manipulation
ETH Zurich · NVIDIA · Microsoft +1
AutoRSR: Automated Real-to-Sim-to-Real Data Generation for Scalable Robot Learning
Oct 14, 2026Jiawei Zhang, Chang Liu, Litao Deng +2Synthetic Data Generation for Robotics Robotic Data Collection
Renmin University of China

Jun 23, 2026J/K move · Enter open · S save

ArtiTwinSplat: Interactable Digital Twin Reconstruction via Gaussian Splatting from RGB-D videos

Pranjal Mishra, René Zurbrügg, Max Wilder-Smith, Marco Hutter+3

1ETH Zürich, Zürich, Switzerland. · 2Microsoft, Zürich, Switzerland. · University of Bonn, Bonn, Germany.

Articulated Object Reconstruction 3D Gaussian Splatting 3D Reconstruction Digital Twins

Open paper AskPDF arXiv

Deploying robots in unstructured real-world environments needs accurate, interactive models of the objects. Constructing these models at scale remains a critical bottleneck for robotic system integration. We present ArtiTwinSplat, a framework that automatically constructs articulated, photo-realistic digital twins of objects directly from RGB-D videos, requiring no CAD models, simulation assets, or manual annotations. Our method is built on 3D Gaussian Splatting that preserve geometric fidelity and photometric realism, coupled with an unsupervised articulation discovery pipeline that recovers part structure and joint kinematics from observed motion alone. With tracking and optimization stages our method provides stable, queryable digital twins that support real-time rendering, viewpoint control, and interactive manipulation. Unlike prior methods confined to simulation, ArtiTwinSplat operates directly on real-world observations and produces twins that are immediately usable by downstream robot planning and learning systems. This method offers a practical, scalable pathway toward digital twin construction, lowering the integration barrier for articulated object manipulation in embodied AI and human-robot collaboration contexts.

Abstract

Robotic Data Collection

Explore similar work

ArtiTwinSplat: Interactable Digital Twin Reconstruction via Gaussian Splatting from RGB-D videos

ForceTwin: Physics-informed Digital Twins for Robotic Manipulation from Instrumented Human Interaction

AutoRSR: Automated Real-to-Sim-to-Real Data Generation for Scalable Robot Learning