Real2Sem2Real: 3D Semantic Augmentations for Robot Learning from a Single Human RGB Demonstration

Matthew Strong; Ankush K. Dhawan; Julien Poffet; Giuse Pham; Jinho So; Monroe Kennedy

cs.ROOct 20, 2026

Real2Sem2Real: 3D Semantic Augmentations for Robot Learning from a Single Human RGB Demonstration

Authors: Matthew Strong, Ankush K. Dhawan, Julien Poffet, Giuse Pham, Jinho So, Monroe Kennedy

Organizations: Department of Computer Science, Stanford University, Stanford, CA, USA · Department of Mechanical Engineering, Stanford University, Stanford, CA, USA · D-MAVT, ETH Zurich, Switzerland; work done while visiting Stanford University

Abstract

Human video scales easily and does not need robot hardware, but existing methods require depth sensing, overlay the robot onto 2D video, or generate data only for the demonstrated objects. We present Real2Sem2Real (R2Sem2R), a real-to-semantic-to-real pipeline that turns one monocular RGB video of a human demonstration into thousands of robot demonstrations across the semantic object instances of a category. R2Sem2R tracks the SE(3) pose of each object with DINOv3-distilled Gaussian splats, outperforming existing methods and FoundationPose with estimated metric depth. R2Sem2R then aligns digital cousins, each generated from one or a few photographs, to the demonstrated objects with a semantic alignment. This alignment matches whole objects and the functional regions of target objects, such as a coffee machine's tray. R2Sem2R maps the human hand to a feasible robot grasp that extends to the cousins. Robot demonstrations are generated either in IsaacSim, for photorealistic rendering, or MuJoCo, for contact physics, and R2Sem2R can generate over 2,000 episodes in about two hours. On two real robots and four tasks, we train a vision-language-action policy on data from one human video per task, and policies trained on our data match or exceed policies trained on teleoperation across diverse object instances. The project page is available at https://real2sem2real.github.io/.

Figures & tables

Fig. 1: Real2Sem2Real at a glance. One human RGB video and a phone scan are the only inputs. DINOv3 part matches align demonstrated objects to digital cousins, including second objects such as coffee machines, and transferred motion generates data for many embodiments. A VLA fine-tuned on it runs on a real xArm7 and Franka Panda, comparable to teleoperation.

Fig. 2: Failure modes. Prior trackers lose objects, prior grasps collide with the ground, and prior meshes have poor textures. R2Sem2R tracks the mug, recovers collision-free grasps, and renders realistic objects (GT: ground truth/real scene).

Method	Demos	No teleop	RGB only	Two- obj.	Cross- obj.	Any arm
MimicGen [ 5 ]	10	$\times$	$\times$	$\times$	$\times$	✓
DemoGen [ 6 ]	1	$\times$	$\times$	$\times$	$\times$	$\times$
Phantom [ 13 ]	250+	✓	$\times$	$\times$	$\times$	✓
WARPED [ 24 ]	30	✓	✓	$\times$	$\times$	✓
R2R2R [ 14 ]	1	✓	✓	$\times$	$\times$	✓
SimFoundry [ 16 ]	10–15	$\times$	✓	$\times$	✓	✓

TABLE I: Demonstration generation methods. RGB only : no depth or privileged state. Two-obj. : carries a demonstrated two-object relation, such as a mug resting on a machine, to new instances of both. Cross-obj. : transfer across a category. Demos : source demos per task. Any arm : any parallel-jaw arm.

Fig. 3: Render-and-compare tracking. Top: frozen DINOv3, SAM2, and Depth Anything turn a frame into RGB, features, a hand-free mask, and monocular depth; the splat is rendered at $T_{t}$ into the same channels and the losses update the pose. Bottom: the tracked mug and per-frame mask IoU for ours, WARPED, FoundationPose with predicted metric depth, and RSRD [ 30 ] .

Fig. 4: Semantic alignment across tasks. Source objects at the task pose (left, blue) and cousins carried there by semantic alignment (right). Pink marks the region anchoring the task-weighted alignment: the machine’s tray, the poured mug’s rim, the mug handle and its branch, and the drill’s handle.

Fig. 5: Grasp transfer. The grasp retargeted from the human hand on the scanned mug (left) is carried to each cousin by its semantic alignment (right), closing on the handle of every one.

Fig. 6: Scene construction and real setup. Top: the Viser layout is converted into IsaacSim (photorealistic rendering, kinematic carry) and MuJoCo (contact physics) scenes. Bottom: the real (a) xArm7 and (b) Franka Panda, each with one external RGB camera as the policy’s only image input.

Fig. 7: Tasks. A single RGB human demonstration (left) generates training data for both robots. Real rollouts conducted on the xArm7 and Franka Panda (right). The manipulated object is outlined in orange with its motion.

Our videos: mask IoU
				FoundationPose [ 17 ]
Object	Ours	WARPED	RSRD	per-video	raw
Mug (to machine)	0.949	0.890	0.197	0.897	0.648
Coffee machine	0.916	0.894	0.861	0.872	0.680
Mug (to tree)	0.904	0.000	0.097	0.864	0.750
Tree stand	0.840	0.712	0.505	0.757	0.601

TABLE II: Tracking accuracy. Top: mask IoU on our objects. Bottom: AUC (%) on YCBInEOAT, starting from the ground-truth pose, with the per-video depth scale set on the first frame. FoundationPose with depth gets 95.8 ADD-S and 93.5 ADD.

Fig. 8: Real-robot success rate on the xArm7 (top) and Franka Panda (bottom). Whiskers: Wilson 95% interval.

Fig. 9: Object diversity versus data source on placement. One pair : 150 teleop demos or 2,000 R2R2R+ episodes on demonstrated pair. Several objects : teleop on 4x4 pairs, or R2Sem2R. Whiskers: Wilson 95% interval.

Fig. 10: Success versus training-set size on xArm7 placement. Solid: several objects; dashed: demonstrated pair; bands: Wilson 95% interval.

Fig. 11: Demonstrations against wall-clock time (log-log) for 1 and 10 teleoperators or GPUs; R2R2R+ coincides with R2Sem2R and is omitted. Strips show where each method’s time goes, shaded as human or GPU time. Robot setup, shared by all methods, is not shown.

Robot Learning from Video

See all 49 papers

Real2Gym: Building Gyms from Videos, Bringing Skills to Robots
Sep 29, 2026Kerui Ren, Yingxiang Xu, Kaiwen Song +4Robot Skill Learning Robotics Simulation
Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University · Zhejiang University +3
Human Demonstrations as Post-Training Data for Any Robot Body: Retargeting the Object, Not the Hand
Oct 14, 2026Nagasrikanth KallakuriCross-Embodiment Robot Learning Robotic Manipulation
Dxtr AI Co.
ARSTAG: An Agentic Real2Sim2Real System for Task-Specific Robot Data Generation
Sep 21, 2026Bowei Li, Yuner Zhang, Changliu LiuRobotics Simulation Sim-to-Real Transfer
Carnegie Mellon University, Pittsburgh, PA 15213, USA

Sep 29, 2026J/K move · Enter open · S save

Real2Gym: Building Gyms from Videos, Bringing Skills to Robots

Kerui Ren, Yingxiang Xu, Kaiwen Song, Linning Xu+3

Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University · Zhejiang University+3

Robot Skill Learning Robotics Simulation Sim-to-Real Transfer Robotic Manipulation Robot Learning from Video Robot Learning

Open paper AskPDF arXiv

Real-world videos provide rich demonstrations of manipulation, but turning them into reusable robot skills requires visually aligned environments, executable physical interactions, and mechanisms for learning from experience. We introduce Real2Gym, an agentic Real2Sim2Real framework that turns human and robot demonstrations into interactive simulation gyms and brings skills acquired in simulation to physical robots. The Real2Sim module reconstructs editable scenes, aligns objects and cameras with the input, validates demonstrated or retargeted actions through native physics execution, and generates task-conditioned variations with action-feasibility checks. Within these environments, the agent generates executable code for manipulation stages, observes their outcomes, and distills successful attempts and failures into reusable task procedures, object-relative motions, and recovery strategies. Through a shared perception-and-control interface, these skills guide subsequent execution in simulation and on real robots, with motions adapted to current observations and no updates to the underlying model weights. Extensive evaluations demonstrate that Real2Gym enables high-fidelity simulation environment reconstruction, outperforming GPT-6 Astra Direct Mode by 16.7% in success rate with approximately 74.9% fewer policy-execution tokens across these environments, while exceeding it by 33.3% in physical robot execution success rate across four tasks on a real Franka robot.

Abstract

Robot Learning from Video

Explore similar work

Real2Gym: Building Gyms from Videos, Bringing Skills to Robots

Human Demonstrations as Post-Training Data for Any Robot Body: Retargeting the Object, Not the Hand

ARSTAG: An Agentic Real2Sim2Real System for Task-Specific Robot Data Generation