Robotic Scene Cloning:Advancing Zero-Shot Robotic Scene Adaptation in Manipulation via Visual Prompt Editing

Binyuan Huang; Yuqing Wen; Yucheng Zhao; Yaosi Hu; Tiancai Wang; Chang Wen Chen; Haoqiang Fan; Zhenzhong Chen

cs.ROMar 10, 2026

Robotic Scene Cloning:Advancing Zero-Shot Robotic Scene Adaptation in Manipulation via Visual Prompt Editing

Authors: Binyuan Huang, Yuqing Wen, Yucheng Zhao, Yaosi Hu, Tiancai Wang, Chang Wen Chen, Haoqiang Fan, Zhenzhong Chen

Organizations: Binyuan Huang and Zhenzhong Chen are with Wuhan University. · Yuqing Wen is with University of Science and Technology of China. · Yucheng Zhao, Tiancai Wang and Haoqiang Fan are with Dexmal. · Yaosi Hu and Chang Wen Chen are with The Hong Kong Polytechnic University.

Abstract

Modern robots can perform a wide range of simple tasks and adapt to diverse scenarios in the well-trained environment. However, deploying pre-trained robot models in real-world user scenarios remains challenging due to their limited zero-shot capabilities, often necessitating extensive on-site data collection. To address this issue, we propose Robotic Scene Cloning (RSC), a novel method designed for scene-specific adaptation by editing existing robot operation trajectories. RSC achieves accurate and scene-consistent sample generation by leveraging a visual prompting mechanism and a carefully tuned condition injection module. Not only transferring textures but also performing moderate shape adaptations in response to the visual prompts, RSC demonstrates reliable task performance across a variety of object types. Experiments across various simulated and real-world environments demonstrate that RSC significantly enhances policy generalization in target environments.

Figures & tables

Fig. 1: Existing robotic policies face challenges when migrating from training to deployment environments, particularly in handling novel products. (a) Recollect Data: Collecting deployment-specific data enables fine-tuning for high accuracy but is labor-intensive and moderately data-efficient. (b) Existing Embodied Augmentation Methods: Augmentation using text prompts reduces labor but achieves limited accuracy and low data efficiency. (c) Robotic Scene Cloning: Cloning scenes with visual cues from the deployment environment achieves high accuracy with better data efficiency and lower labor cost. The comparison highlights the trade-offs in accuracy, labor intensity, and data efficiency for each method. The two rightmost bar charts show the success rates of grasping Monster Energy Drink (V1), Monster Energy Drink (V2), and Disinfectant Bottle in the Simpler environment under distractor conditions.

Fig. 2: Overview for Robotic Scene Cloning . (a) RSC pipeline follows a two-stage process. Robotic Condition Generator prepares scene-specific conditions from training trajectories and a new product. Visual Prompt Editor then generates visually cloned trajectories, which fine-tune robotic models for better adaptation to novel products. (b) Robotic Condition Generator prepares three conditions: First, the Grounding Resampler combines the new product’s visual, textual, and positional encodings to generate position-bound visual conditions. Second, Grounding-DINO + SAM2 extract the coordinates and masks of existing products. Third, DepthAnythingV2 and ControlNet capture the pose conditions. (c) Visual Prompt Editor applies the three conditions—visual, pose, and layout—generated by the Robotic Condition Generator. Visual and pose conditions guide the DDIM processes, while the layout condition is used in the progressive masked fusion.

Fig. 3: Editing effect comparison between RSC and GreenAug in the SIMPLER Benchmark.

Fig. 4: The visualization results of our real-world validation task on the WidowX robot.

Fig. 5: Each pair of rows shows the original video frames (Raw) and the corresponding frames edited by our method (RSC) after replacing the target object.

Method	MonsterEnergy Drink V1	MonsterEnergy Drink V2	Disinfectant Bottle	Spray Paint Can	Avg
OpenVLA [ 1 ]	0.0	0.0	0.0	0.0	0.0
CogACT [ 9 ]	0.0	0.0	45.0	10.0	13.8
CogACT [ 9 ] +GAug [ 12 ]	5.0	10.0	45.0	25.0	21.3
CogACT [ 9 ] +RSC	70.0	50.0	60.0	45.0	56.3

TABLE I: Performance of state-of-the-art VLA models on four novel deployment tasks of the SIMPLER benchmark, highlighting the impact of GreenAug (GAug) and Robotic Scene Cloning.

Method	No-Real-Demo Task				Real-Demo Task
Method	Multi-Object	Single Object			Multi-Object	Single Object
	Pepper & Cube	Cube	Glue stick	Pepper	Banana & Strawberry	Banana
Baseline [ 35 ]	20%	40%	40%	40%	50%	70%
Baseline [ 35 ] +RSC	50%	80%	60%	70%	50%	70%

TABLE II: We evaluate two real-demo tasks and four no-real-demo tasks . Real-demo tasks include a single-object task (banana) and a multi-object task (banana & strawberry), both of which provide real-world robot demonstrations used for fine-tuning. No-real-demo tasks involve novel objects or object combinations (cube, glue stick, pepper, and pepper & cube) and provide no real-world robot demonstrations; they are used solely for evaluation. All results report success rates over 10 trials per task.

Fig. 6: Comparison of different data augmentation strategies on the CALVIN long-horizon task. AvgLen represents the average task execution length. FG denotes foreground augmentation, and BG denotes background augmentation.

Robot Policy Adaptation

Robot policy adaptation changes or steers learned robot behaviors to handle new tasks, environments, hardware, and user preferences.

See all 173 papers

SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation
Jun 26, 2026Nadun Ranawaka, Josiah Wong, Wei-Lin Pai +15Robotics Simulation Sim-to-Real Transfer
NVIDIA · Stanford University · The University of Texas at Austin +2
Function-Preserving Data Generation for Zero-Shot Real-to-Sim-to-Real Manipulation
Sep 16, 2026Tianyi Xiang, Xupeng Xie, Jiahang Cao +3Synthetic-to-Real Domain Adaptation Long-Horizon Robotic Manipulation
Robotics and Autonomous Systems Thrust, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, China · Institute of Data Science, The University of Hong Kong, Hong Kong SAR, China
Agentic Scene Policies
Sep 23, 2025Sacha Morin, Kumaraditya Gupta, Mahtab Sandhu +4Robot Policy Generalization Robot Learning
Université de Montréal · Mila - Quebec AI Institute · Sapienza University of Rome

Jun 26, 2026J/K move · Enter open · S save

SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation

Nadun Ranawaka, Josiah Wong, Wei-Lin Pai, Wei-Teng Chu+14

NVIDIA · Stanford University · The University of Texas at Austin+2

Robotics Simulation Sim-to-Real Transfer Robotic Policy Evaluation Synthetic Data Generation for Robotics Digital Twins

Open paper AskPDF arXiv

Training and evaluating robot policies in the real world is costly and difficult to scale. We introduce SimFoundry, a modular and automated system for zero-shot real-to-sim scene construction from a video. SimFoundry generates sim-ready digital twins and supports object, scene, and task editing, enabling the automated generation of diverse digital cousins: affordance-preserving variations of reconstructed real-world scenes. Policies trained on SimFoundry data transfer zero-shot to challenging real tasks involving multi-step manipulation, articulated object interaction, and bimanual interaction, and its digital cousins (variations of the original scene, objects, and tasks) facilitate generalization to new real-world conditions. Across 7 manipulation tasks and 5 policy architectures, SimFoundry simulation evaluations strongly predict real-world performance, with mean Pearson correlation 0.911 and mean maximum ranking violation 0.018. When evaluating sim-trained policies zero-shot in the real world, policies trained with object, scene, and task cousins in simulation show average task success rate improvements of 17%, 21%, and 40%, respectively. Additional details at https://research.nvidia.com/labs/gear/simfoundry/ .

Abstract

Robot Policy Adaptation

Explore similar work

SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation

Function-Preserving Data Generation for Zero-Shot Real-to-Sim-to-Real Manipulation

Agentic Scene Policies