Transferring robotic skills from simulation to reality requires task knowledge that remains usable across differences in perception, dynamics, and embodiment. We introduce Skill2Real, an agentic policy framework that learns executable skills through a shared application programming interface (API). A Proposer-Verifier-Governor (PVG) loop uses privileged simulation evidence to diagnose outcomes and validate updates, while keeping learned skills grounded in public observations and API semantics. The Cerebellum first acquires local manipulation skills; the Brain then learns task-level composition with the Cerebellum frozen. Both memories transfer to the real robot without task-policy fine-tuning or skill-memory updates. As GPT-5.6 Sol learns skills on LIBERO-90, evaluating each frozen checkpoint with GPT-6 Astra raises LIBERO-Pro Long success from 2.0% to 56.3%, without training on Pro Long. Independent Robosuite training reaches 85.1% and 89.4% mean success with Sol and Opus 5 across seven tasks, respectively. Frozen Sol-trained LIBERO-90 skills achieve 78.75% mean completion across four real-world manipulation tasks with Astra. Removing the Verifier or Governor during LIBERO-90 training lowers final Pro Long success by 17.3 and 13.3 percentage points, respectively. These results support learning and transferring a hierarchy of executable skills through a common robot interface.
Figures & tables
Figure 1: Zero-shot real-world transfer of simulation-learned skills. Skill2Real learns transferable, skill-based knowledge across diverse simulation environments. The resulting frozen skills are evaluated on real-world tasks without additional learning, demonstrating robustness to variations in lighting, materials, and dynamics.
Figure 2: Skill2Real pipeline. All in-context skill learning occurs in simulated scenes. Stage I learns reusable Cerebellum skills for local manipulation; Stage II freezes the Cerebellum and learns Brain skills that compositionally organize the local library into task programs. In both stages, the Proposer generates programs from public observations, the Verifier converts privileged execution evidence into publicly grounded feedback, and the Governor admits only updates supported by validation rollouts.
Figure 3
Figure 5: Experimental setups. (a) Representative LIBERO-Pro Long scene. (b) Representative Robosuite scene. (c) real front view. (d) real top-down view.
Figure 6: Skill2Real provides consistent hierarchical gains across foundation VLMs. With GPT-6 Astra, GPT-5.6 Sol, and Opus 5, success rises through Cerebellum training and continues through Brain training in both settings: (a) transfer of LIBERO-90 checkpoints to unseen LIBERO-Pro Long tasks; and (b) independently learned Robosuite libraries. This shared trend demonstrates that the framework’s learning gains are orthogonal to the underlying VLM.
Figure 7: LIBERO-Pro Long comparison. Skill2Real with Astra achieves the highest Overall and instruction-perturbation (Task) success; Zetta leads under position perturbations (Pos). Overall averages Task and Pos. Skill2Real evaluates a fixed Sol-trained C3/B3 hierarchy with Astra or Opus 4.6. Comparisons include CaP-Agent0 ( Fu et al., 2026 ) , ASPIRE ( N=90 ), π0 , and π0.5 ( Lu et al., 2026 ) , and Zetta ( Ding et al., 2026 ) .
Method
R1
R2
R3
R4
Mean
CaP-Agent0 (CaP-X)
15
20
10
0
11.25
Astra, no learned skills
40
35
35
0
27.50
Astra, Brain only
50
60
40
0
37.50
Astra, Cerebellum only
80
80
65
0
56.25
Astra, full Skill2Real
95
85
85
50
78.75
Table 1: Real-world task completion (%; 20 trials per method/task). R1: pick-and-place; R2: sorting; R3: equation assembly; R4: drawer manipulation. Mean is the unweighted average across R1–R4.
Figure 8: Both PVG roles improve transfer. Removing either role lowers Pro Long success throughout hierarchical learning; labels give B4 Overall success (%).
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
ID
Task
Instruction template
Completion condition
R1
Pick-and-place
Place the specified object in the specified container.
The target object is inside the target container.
R2
Attribute-based sorting
Sort objects by color or semantic category.
Each object is placed at the destination assigned to its category.
R3
Equation assembly
Move cubes to form the requested equation.
The cubes form the requested equation in the correct order.
R4
Drawer manipulation
Open the target drawer, insert the specified object, then close the drawer.
The target object is inside the correct drawer, which is closed.
Appendix
Table A.1: Real-world task suite. R1–R4 are the task identifiers used in Table 1 ; object instances are specified per trial.
Figure A.1: Representative real-world task scenes. One recorded image illustrates each task in Table A.1 : (a) picking an object for placement on a plate; (b) sorting objects into color-labeled boxes; (c) arranging answer cubes for 23+45=68 ; and (d) drawer manipulation.
Figure A.2: Tabletop and lighting variation in real deployment. Recorded scenes span white, floral, and red-gingham surfaces and dim side lighting. All images retain their recorded exposure; the examples are qualitative.
Family
Manipulation scope and representative operations
Full mark
Articulated storage
Opening or closing cabinet drawers and microwave doors; placing objects inside drawers.
95
Stacking
Placing one bowl on another, requiring alignment and stable support after release.
20
Stove manipulation
Switching a stove on or off; placing frying pans or moka pots on the stove.
40
Caddy storage
Placing objects into specified caddy compartments, requiring compartment selection and constrained placement.
60
Basket / tray storage
Transferring household objects into open baskets or trays.
85
Surface placement
Placing objects on plates, shelves, or cabinet tops, or at specified positions relative to other objects.
150
Appendix
Table B.1: Six task families in the LIBERO-90 analysis. Full mark is the score denominator used to normalize each family’s learning curve, not its task count. The six denominators sum to 450.
Figure B.1: Source-suite skill learning on LIBERO-90. Sol and Opus train separate libraries and are evaluated on the same 450 held-out trials after each iteration. C0 is the no-skill baseline; C1–C3 and B1–B4 denote Cerebellum and Brain training. The Cerebellum remains fixed after C3, and both memories are frozen during evaluation.
Figure B.2: Learning across six LIBERO-90 families. Each family is normalized by its full mark and uses its own vertical scale. C0 is the no-skill baseline; C1–C3 track cerebellum learning, and B1–B4 track subsequent brain learning. The background regions meet at C3.
Figure B.3: Learning across seven Robosuite tasks with Sol and Opus. Task success uses a shared vertical scale. C0 is the no-skill baseline; C1–C3 track cerebellum learning, and B1–B4 track brain learning with the C3 cerebellum fixed. Background regions meet at C3.
Skill2Real (ours)
ASPIRE
Zetta
Task
GPT-6 Astra
Claude Opus 4.6
Claude Opus 4.6
π0.5
Stove + moka pot
100.0
100.0
100.0
95.0
Bowl in drawer
20.0
0.0
0.0
75.0
Mug in microwave
15.0
0.0
0.0
5.0
Both mokas on stove
5.0
0.0
0.0
0.0
Soup + cream cheese
0.0
0.0
0.0
0.0
Appendix
Table B.2: Per-task LIBERO-Pro Long success under position perturbations (Pos). Success rates (%) for the fixed C3/B3 comparison. Skill2Real uses 20 trials per task. Baselines: ASPIRE ( N=90 ) ( Lu et al., 2026 ) ; Zetta (S) ( Ding et al., 2026 ) . π0.5 is Zetta’s base policy.
Skill2Real (ours)
ASPIRE
Zetta
Task
GPT-6 Astra
Claude Opus 4.6
Claude Opus 4.6
π0.5
Stove + moka pot
100.0
100.0
26.0
100.0
Bowl in drawer
55.0
0.0
0.0
0.0
Mug in microwave
15.0
0.0
0.0
0.0
Both mokas on stove
100.0
100.0
100.0
100.0
Soup + cream cheese
95.0
50.0
70.0
80.0
Appendix
Table B.3: Per-task LIBERO-Pro Long success under instruction perturbations (Task). Success rates (%) for the fixed C3/B3 comparison. Skill2Real uses 20 trials per task. Baselines: ASPIRE ( N=90 ) ( Lu et al., 2026 ) ; Zetta (T) ( Ding et al., 2026 ) . π0.5 is Zetta’s base policy.
Figure B.4: PVG learning dynamics on LIBERO-90 with GPT-5.6 Sol. Success is normalized by 450. Background shading separates cerebellum and brain learning at C3; colors, markers, and line styles distinguish the three training conditions.
Operation
Public result and interpretation
get_scene_image
Path to the current front scene image.
get_topdown_image
Path to the current top-down image; target pixels are selected in an observation frame.
get_wrist_view
Wrist image for the selected arm, with available depth preview and camera calibration.
get_topdown_stereo_view
Stereo RGB pair with intrinsics and extrinsics for both cameras.
get_foundation_stereo_depth
Metric depth computed from a calibrated image pair, with observation_ref and depth_ref identifying the source observation and its depth/calibration binding. Missing calibration causes failure.
get_sam_mask_from_box
Segmentation mask for a supplied image and bounding box.
Appendix
Table C.1: Public perception operations (8 of 18). Image retrieval and geometric measurement supply evidence for the Proposer’s decisions. They do not certify manipulation outcomes.
Operation
Public result or commanded effect
preprocess
Refresh public observations used by perception operations.
get_observation_geometry
Observation-frame geometry for a supplied public description, providing an alternative source-localization path.
get_previous_proposer_state
Previous decision-stage public state and the latest still-valid candidate scan. After an executed compiled grasp, while the gripper remains closed, this can include held_object_state for place compilation.
grasp_center_to_tool0_pose
Tool-frame pose corresponding to a selected grasp center and orientation.
compile_pose
In grasp mode, construct poses from Proposer-supplied grasp_center_xyz and closing_axis_world . In place mode, construct carry, release, and retreat poses from a target pixel, bound depth, and held-object geometry.
get_end_effector_pose
Current end-effector pose for the selected arm.
Appendix
Table C.2: State, compilation, and control operations (10 of 18). Pose construction does not move the robot. The final four operations issue arm or gripper commands.
Candidate
Decision
Recorded basis
Relative referent
Admit
Selects the requested left bowl from fresh image evidence. The executed program implements the rule, and the entry states when the ordering must be reconsidered.
Contact binding
Reject
The proposed contact-point and local-geometry rule belongs to the frozen Cerebellum rather than Brain memory.
Destination geometry
Reject
Computing tray geometry and release clearance is a local placement procedure beyond the permitted Brain update.
Progress checking
Reject
The entry includes an object–hand co-motion test that was not implemented or tested.
Appendix
Table D.1: Governor decisions in the recorded bowl-transfer case. Descriptions condense the supplied candidate and decision records.
Real-world videos provide rich demonstrations of manipulation, but turning them into reusable robot skills requires visually aligned environments, executable physical interactions, and mechanisms for learning from experience. We introduce Real2Gym, an agentic Real2Sim2Real framework that turns human and robot demonstrations into interactive simulation gyms and brings skills acquired in simulation to physical robots. The Real2Sim module reconstructs editable scenes, aligns objects and cameras with the input, validates demonstrated or retargeted actions through native physics execution, and generates task-conditioned variations with action-feasibility checks. Within these environments, the agent generates executable code for manipulation stages, observes their outcomes, and distills successful attempts and failures into reusable task procedures, object-relative motions, and recovery strategies. Through a shared perception-and-control interface, these skills guide subsequent execution in simulation and on real robots, with motions adapted to current observations and no updates to the underlying model weights. Extensive evaluations demonstrate that Real2Gym enables high-fidelity simulation environment reconstruction, outperforming GPT-6 Astra Direct Mode by 16.7% in success rate with approximately 74.9% fewer policy-execution tokens across these environments, while exceeding it by 33.3% in physical robot execution success rate across four tasks on a real Franka robot.
Kerui Ren, Yingxiang Xu, Kaiwen Song +4
Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University · Zhejiang University +3
Traditional robot programming is challenging: it requires orchestrating multimodal perception, managing physical contact dynamics, and handling diverse configurations and execution failures. We introduce ASPIRE (Agentic Skill Programming through Iterative Robot Exploration), a continual learning system that autonomously writes and refines robot control programs in a code-as-policy paradigm while compounding experience into a reusable skill library. ASPIRE discovers skills that persist across tasks, simulation and real-world settings, and embodiments. It operates in an open-ended loop with three components: (1) a closed-loop robot execution engine that exposes fine-grained multimodal traces, enabling autonomous failure diagnosis, repair synthesis, and validation; (2) a continually expanding skill library that distills validated fixes into reusable, transferable knowledge; and (3) evolutionary search that generates diverse task sequences and control programs to explore beyond single-trajectory refinement. ASPIRE surpasses prior methods by up to 77% on LIBERO-Pro manipulation under perturbation, 72% on Robosuite bimanual handover, and 32% on BEHAVIOR-1K long-horizon household tasks. Its accumulated library also enables zero-shot generalization to unseen long-horizon tasks: on LIBERO-Pro Long, ASPIRE achieves 31% success versus 4% for prior methods despite their use of test-time reasoning and retries. Finally, simulation-discovered skills provide initial evidence of sim-to-real transfer, substantially reducing real-robot programming effort across different embodiments and robot APIs.
A key challenge in bringing embodied intelligence into the real world is transferring capabilities from simulation to reality and enabling agents to continually adapt after deployment. End-to-end vision-language-action policies provide strong manipulation capabilities, but their transfer to physical environments typically relies on calibrating simulated visual and dynamical conditions, collecting additional target-domain demonstrations, and optimizing the policy through further training. Tool-using embodied agents offer flexible task orchestration, yet existing systems primarily emphasize task execution and experience reuse within a given environment, with limited support for transferring procedural knowledge and continuously adapting it across simulation and reality. We propose RoboBridge, a framework that treats sim-to-real transfer as the continued adaptation of executable task skills. The agent represents task knowledge as procedures connecting task intent, observations, tool operations, and outcome verification. Interaction feedback is used to generate candidate skill revisions, which are evaluated before being persisted or rejected. A pretrained vision-language-action policy is exposed as a reusable action tool and enhanced with inference-time guidance, enabling fine-grained execution without retraining the underlying policy. RoboBridge grounds transferable skills in task semantics and interaction interfaces shared across simulation and reality. This representation preserves reusable task structure while allowing environment-dependent operations to be selectively revised through real-world execution feedback. We evaluate the framework on LIBERO-PRO and corresponding physical tasks, studying both skill evolution and post-transfer adaptation. Our framework provides a route from one-shot policy deployment to continual procedural learning across environments.
Chenxi Li, Zhangrui Zhao, Rui Li +8
Shanghai Artificial Intelligence Laboratory · Zhejiang University · Beihang University +3