Organizations: Binyuan Huang and Zhenzhong Chen are with Wuhan University. · Yuqing Wen is with University of Science and Technology of China. · Yucheng Zhao, Tiancai Wang and Haoqiang Fan are with Dexmal. · Yaosi Hu and Chang Wen Chen are with The Hong Kong Polytechnic University.
Modern robots can perform a wide range of simple tasks and adapt to diverse scenarios in the well-trained environment. However, deploying pre-trained robot models in real-world user scenarios remains challenging due to their limited zero-shot capabilities, often necessitating extensive on-site data collection. To address this issue, we propose Robotic Scene Cloning (RSC), a novel method designed for scene-specific adaptation by editing existing robot operation trajectories. RSC achieves accurate and scene-consistent sample generation by leveraging a visual prompting mechanism and a carefully tuned condition injection module. Not only transferring textures but also performing moderate shape adaptations in response to the visual prompts, RSC demonstrates reliable task performance across a variety of object types. Experiments across various simulated and real-world environments demonstrate that RSC significantly enhances policy generalization in target environments.
Figures & tables
Fig. 1: Existing robotic policies face challenges when migrating from training to deployment environments, particularly in handling novel products. (a) Recollect Data: Collecting deployment-specific data enables fine-tuning for high accuracy but is labor-intensive and moderately data-efficient. (b) Existing Embodied Augmentation Methods: Augmentation using text prompts reduces labor but achieves limited accuracy and low data efficiency. (c) Robotic Scene Cloning: Cloning scenes with visual cues from the deployment environment achieves high accuracy with better data efficiency and lower labor cost. The comparison highlights the trade-offs in accuracy, labor intensity, and data efficiency for each method. The two rightmost bar charts show the success rates of grasping Monster Energy Drink (V1), Monster Energy Drink (V2), and Disinfectant Bottle in the Simpler environment under distractor conditions.
Fig. 2: Overview for Robotic Scene Cloning . (a) RSC pipeline follows a two-stage process. Robotic Condition Generator prepares scene-specific conditions from training trajectories and a new product. Visual Prompt Editor then generates visually cloned trajectories, which fine-tune robotic models for better adaptation to novel products. (b) Robotic Condition Generator prepares three conditions: First, the Grounding Resampler combines the new product’s visual, textual, and positional encodings to generate position-bound visual conditions. Second, Grounding-DINO + SAM2 extract the coordinates and masks of existing products. Third, DepthAnythingV2 and ControlNet capture the pose conditions. (c) Visual Prompt Editor applies the three conditions—visual, pose, and layout—generated by the Robotic Condition Generator. Visual and pose conditions guide the DDIM processes, while the layout condition is used in the progressive masked fusion.
Fig. 3: Editing effect comparison between RSC and GreenAug in the SIMPLER Benchmark.
Fig. 4: The visualization results of our real-world validation task on the WidowX robot.
Fig. 5: Each pair of rows shows the original video frames (Raw) and the corresponding frames edited by our method (RSC) after replacing the target object.
Method
MonsterEnergy Drink V1
MonsterEnergy Drink V2
Disinfectant Bottle
Spray Paint Can
Avg
OpenVLA [ 1 ]
0.0
0.0
0.0
0.0
0.0
CogACT [ 9 ]
0.0
0.0
45.0
10.0
13.8
CogACT [ 9 ] +GAug [ 12 ]
5.0
10.0
45.0
25.0
21.3
CogACT [ 9 ] +RSC
70.0
50.0
60.0
45.0
56.3
TABLE I: Performance of state-of-the-art VLA models on four novel deployment tasks of the SIMPLER benchmark, highlighting the impact of GreenAug (GAug) and Robotic Scene Cloning.
Method
No-Real-Demo Task
Real-Demo Task
Multi-Object
Single Object
Multi-Object
Single Object
Pepper & Cube
Cube
Glue stick
Pepper
Banana & Strawberry
Banana
Baseline [ 35 ]
20%
40%
40%
40%
50%
70%
Baseline [ 35 ] +RSC
50%
80%
60%
70%
50%
70%
TABLE II: We evaluate two real-demo tasks and four no-real-demo tasks . Real-demo tasks include a single-object task (banana) and a multi-object task (banana & strawberry), both of which provide real-world robot demonstrations used for fine-tuning. No-real-demo tasks involve novel objects or object combinations (cube, glue stick, pepper, and pepper & cube) and provide no real-world robot demonstrations; they are used solely for evaluation. All results report success rates over 10 trials per task.
Fig. 6: Comparison of different data augmentation strategies on the CALVIN long-horizon task. AvgLen represents the average task execution length. FG denotes foreground augmentation, and BG denotes background augmentation.
Training and evaluating robot policies in the real world is costly and difficult to scale. We introduce SimFoundry, a modular and automated system for zero-shot real-to-sim scene construction from a video. SimFoundry generates sim-ready digital twins and supports object, scene, and task editing, enabling the automated generation of diverse digital cousins: affordance-preserving variations of reconstructed real-world scenes. Policies trained on SimFoundry data transfer zero-shot to challenging real tasks involving multi-step manipulation, articulated object interaction, and bimanual interaction, and its digital cousins (variations of the original scene, objects, and tasks) facilitate generalization to new real-world conditions. Across 7 manipulation tasks and 5 policy architectures, SimFoundry simulation evaluations strongly predict real-world performance, with mean Pearson correlation 0.911 and mean maximum ranking violation 0.018. When evaluating sim-trained policies zero-shot in the real world, policies trained with object, scene, and task cousins in simulation show average task success rate improvements of 17%, 21%, and 40%, respectively. Additional details at https://research.nvidia.com/labs/gear/simfoundry/ .
Nadun Ranawaka, Josiah Wong, Wei-Lin Pai +15
NVIDIA · Stanford University · The University of Texas at Austin +2
Robotic data generation is a promising paradigm for scaling robot learning without collecting large-scale real-world data. However, generating geometrically diverse yet physically valid data for contact-rich tasks remains challenging, especially when success depends on precise geometric interfaces. Standard shape augmentation methods often distort task-critical interfaces, resulting in invalid contact relationships, e.g., fit mismatches or interpenetration, rendering downstream interactions infeasible. To address these limitations, we propose a function-preserving Real-to-Sim-to-Real framework that generates synthetic demonstrations from reconstructed assets without teleoperated source trajectories. Our method augments task-relevant object geometries through constraint-guided mesh deformation, together with physically consistent transfer of task poses and collision proxies. Visual domain randomization is further applied during simulation rollouts, enabling robust zero-shot policy deployment without real-world fine-tuning. Extensive experiments in both real-world and simulation settings demonstrate that our method enables robust generalization across unseen object geometries and diverse visual conditions in contact-rich and long-horizon tasks. Our method provides a practical path toward scalable robot learning for contact-rich tasks via shape deformation.
Tianyi Xiang, Xupeng Xie, Jiahang Cao +3
Robotics and Autonomous Systems Thrust, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, China · Institute of Data Science, The University of Hong Kong, Hong Kong SAR, China
Designing or learning robot policies that generalize zero-shot across a range of language instructions and objects is a core problem in robotics. Vision-Language-Action models (VLAs) learn such policies end-to-end by repurposing existing Vision-Language Models (VLMs), but generalization to new instructions and objects remains challenging. An alternative is to implement a modular policy by leveraging an explicit VLM-based 3D scene representation and motion planning. While modular policies show strong zero-shot potential, they typically retrieve objects based on semantics without explicit spatial reasoning, severely restricting their overall grounding capabilities. They also interact with objects using basic grasping and navigation skills. In this work, we address these limitations by unifying grounding capabilities and robot skills in a single agentic action space through a scene-agent tool interface. By leveraging part-level affordances, our skills generalize across diverse objects and enable zero-shot interactions such as unplugging chargers and opening drawers. We name the resulting framework Agentic Scene Policies (ASP). Through extensive real-world experiments, we show how ASP consistently outperforms leading VLAs in the zero-shot setting. We also demonstrate the extensibility of our framework by introducing a mobile version of ASP to tackle room-level queries. See our project page (https://montrealrobotics.ca/agentic-scene-policies.github.io/) for more results.
Sacha Morin, Kumaraditya Gupta, Mahtab Sandhu +4
Université de Montréal · Mila - Quebec AI Institute · Sapienza University of Rome