cs.ROOct 2, 2026
SaveSkill2Real: Agentic Skill Learning for Zero-Shot Sim-to-Real Robot Manipulation
Organizations: University of California, Los Angeles · Shanghai Jiao Tong University · University of Utah · Envora
Abstract
Transferring robotic skills from simulation to reality requires task knowledge that remains usable across differences in perception, dynamics, and embodiment. We introduce Skill2Real, an agentic policy framework that learns executable skills through a shared application programming interface (API). A Proposer-Verifier-Governor (PVG) loop uses privileged simulation evidence to diagnose outcomes and validate updates, while keeping learned skills grounded in public observations and API semantics. The Cerebellum first acquires local manipulation skills; the Brain then learns task-level composition with the Cerebellum frozen. Both memories transfer to the real robot without task-policy fine-tuning or skill-memory updates. As GPT-5.6 Sol learns skills on LIBERO-90, evaluating each frozen checkpoint with GPT-6 Astra raises LIBERO-Pro Long success from 2.0% to 56.3%, without training on Pro Long. Independent Robosuite training reaches 85.1% and 89.4% mean success with Sol and Opus 5 across seven tasks, respectively. Frozen Sol-trained LIBERO-90 skills achieve 78.75% mean completion across four real-world manipulation tasks with Astra. Removing the Verifier or Governor during LIBERO-90 training lowers final Pro Long success by 17.3 and 13.3 percentage points, respectively. These results support learning and transferring a hierarchy of executable skills through a common robot interface.
Figures & tables
Figure 1: Zero-shot real-world transfer of simulation-learned skills. Skill2Real learns transferable, skill-based knowledge across diverse simulation environments. The resulting frozen skills are evaluated on real-world tasks without additional learning, demonstrating robustness to variations in lighting, materials, and dynamics.
Figure 2: Skill2Real pipeline. All in-context skill learning occurs in simulated scenes. Stage I learns reusable Cerebellum skills for local manipulation; Stage II freezes the Cerebellum and learns Brain skills that compositionally organize the local library into task programs. In both stages, the Proposer generates programs from public observations, the Verifier converts privileged execution evidence into publicly grounded feedback, and the Governor admits only updates supported by validation rollouts.
Figure 5: Experimental setups. (a) Representative LIBERO-Pro Long scene. (b) Representative Robosuite scene. (c) real front view. (d) real top-down view.
Figure 6: Skill2Real provides consistent hierarchical gains across foundation VLMs. With GPT-6 Astra, GPT-5.6 Sol, and Opus 5, success rises through Cerebellum training and continues through Brain training in both settings: (a) transfer of LIBERO-90 checkpoints to unseen LIBERO-Pro Long tasks; and (b) independently learned Robosuite libraries. This shared trend demonstrates that the framework’s learning gains are orthogonal to the underlying VLM.
Figure 7: LIBERO-Pro Long comparison. Skill2Real with Astra achieves the highest Overall and instruction-perturbation (Task) success; Zetta leads under position perturbations (Pos). Overall averages Task and Pos. Skill2Real evaluates a fixed Sol-trained C3/B3 hierarchy with Astra or Opus 4.6. Comparisons include CaP-Agent0 ( Fu et al., 2026 ) , ASPIRE ( ), , and ( Lu et al., 2026 ) , and Zetta ( Ding et al., 2026 ) .
| Method | R1 | R2 | R3 | R4 | Mean |
|---|---|---|---|---|---|
| CaP-Agent0 (CaP-X) | 15 | 20 | 10 | 0 | 11.25 |
| Astra, no learned skills | 40 | 35 | 35 | 0 | 27.50 |
| Astra, Brain only | 50 | 60 | 40 | 0 | 37.50 |
| Astra, Cerebellum only | 80 | 80 | 65 | 0 | 56.25 |
| Astra, full Skill2Real | 95 | 85 | 85 | 50 | 78.75 |
Table 1: Real-world task completion (%; 20 trials per method/task). R1: pick-and-place; R2: sorting; R3: equation assembly; R4: drawer manipulation. Mean is the unweighted average across R1–R4.
Figure 8: Both PVG roles improve transfer. Removing either role lowers Pro Long success throughout hierarchical learning; labels give B4 Overall success (%).
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| ID | Task | Instruction template | Completion condition |
|---|---|---|---|
| R1 | Pick-and-place | Place the specified object in the specified container. | The target object is inside the target container. |
| R2 | Attribute-based sorting | Sort objects by color or semantic category. | Each object is placed at the destination assigned to its category. |
| R3 | Equation assembly | Move cubes to form the requested equation. | The cubes form the requested equation in the correct order. |
| R4 | Drawer manipulation | Open the target drawer, insert the specified object, then close the drawer. | The target object is inside the correct drawer, which is closed. |
Table A.1: Real-world task suite. R1–R4 are the task identifiers used in Table 1 ; object instances are specified per trial.
Figure A.1: Representative real-world task scenes. One recorded image illustrates each task in Table A.1 : (a) picking an object for placement on a plate; (b) sorting objects into color-labeled boxes; (c) arranging answer cubes for ; and (d) drawer manipulation.
Figure A.2: Tabletop and lighting variation in real deployment. Recorded scenes span white, floral, and red-gingham surfaces and dim side lighting. All images retain their recorded exposure; the examples are qualitative.
| Family | Manipulation scope and representative operations | Full mark |
|---|---|---|
| Articulated storage | Opening or closing cabinet drawers and microwave doors; placing objects inside drawers. | 95 |
| Stacking | Placing one bowl on another, requiring alignment and stable support after release. | 20 |
| Stove manipulation | Switching a stove on or off; placing frying pans or moka pots on the stove. | 40 |
| Caddy storage | Placing objects into specified caddy compartments, requiring compartment selection and constrained placement. | 60 |
| Basket / tray storage | Transferring household objects into open baskets or trays. | 85 |
| Surface placement | Placing objects on plates, shelves, or cabinet tops, or at specified positions relative to other objects. | 150 |
Table B.1: Six task families in the LIBERO-90 analysis. Full mark is the score denominator used to normalize each family’s learning curve, not its task count. The six denominators sum to 450.
Figure B.1: Source-suite skill learning on LIBERO-90. Sol and Opus train separate libraries and are evaluated on the same 450 held-out trials after each iteration. C0 is the no-skill baseline; C1–C3 and B1–B4 denote Cerebellum and Brain training. The Cerebellum remains fixed after C3, and both memories are frozen during evaluation.
Figure B.2: Learning across six LIBERO-90 families. Each family is normalized by its full mark and uses its own vertical scale. C0 is the no-skill baseline; C1–C3 track cerebellum learning, and B1–B4 track subsequent brain learning. The background regions meet at C3.
Figure B.3: Learning across seven Robosuite tasks with Sol and Opus. Task success uses a shared vertical scale. C0 is the no-skill baseline; C1–C3 track cerebellum learning, and B1–B4 track brain learning with the C3 cerebellum fixed. Background regions meet at C3.
| Skill2Real (ours) | ASPIRE | Zetta | ||
|---|---|---|---|---|
| Task | GPT-6 Astra | Claude Opus 4.6 | Claude Opus 4.6 | |
| Stove + moka pot | 100.0 | 100.0 | 100.0 | 95.0 |
| Bowl in drawer | 20.0 | 0.0 | 0.0 | 75.0 |
| Mug in microwave | 15.0 | 0.0 | 0.0 | 5.0 |
| Both mokas on stove | 5.0 | 0.0 | 0.0 | 0.0 |
| Soup + cream cheese | 0.0 | 0.0 | 0.0 | 0.0 |
Table B.2: Per-task LIBERO-Pro Long success under position perturbations (Pos). Success rates (%) for the fixed C3/B3 comparison. Skill2Real uses 20 trials per task. Baselines: ASPIRE ( ) ( Lu et al., 2026 ) ; Zetta (S) ( Ding et al., 2026 ) . is Zetta’s base policy.
| Skill2Real (ours) | ASPIRE | Zetta | ||
|---|---|---|---|---|
| Task | GPT-6 Astra | Claude Opus 4.6 | Claude Opus 4.6 | |
| Stove + moka pot | 100.0 | 100.0 | 26.0 | 100.0 |
| Bowl in drawer | 55.0 | 0.0 | 0.0 | 0.0 |
| Mug in microwave | 15.0 | 0.0 | 0.0 | 0.0 |
| Both mokas on stove | 100.0 | 100.0 | 100.0 | 100.0 |
| Soup + cream cheese | 95.0 | 50.0 | 70.0 | 80.0 |
Table B.3: Per-task LIBERO-Pro Long success under instruction perturbations (Task). Success rates (%) for the fixed C3/B3 comparison. Skill2Real uses 20 trials per task. Baselines: ASPIRE ( ) ( Lu et al., 2026 ) ; Zetta (T) ( Ding et al., 2026 ) . is Zetta’s base policy.
Figure B.4: PVG learning dynamics on LIBERO-90 with GPT-5.6 Sol. Success is normalized by 450. Background shading separates cerebellum and brain learning at C3; colors, markers, and line styles distinguish the three training conditions.
| Operation | Public result and interpretation |
|---|---|
| get_scene_image | Path to the current front scene image. |
| get_topdown_image | Path to the current top-down image; target pixels are selected in an observation frame. |
| get_wrist_view | Wrist image for the selected arm, with available depth preview and camera calibration. |
| get_topdown_stereo_view | Stereo RGB pair with intrinsics and extrinsics for both cameras. |
| get_foundation_stereo_depth | Metric depth computed from a calibrated image pair, with observation_ref and depth_ref identifying the source observation and its depth/calibration binding. Missing calibration causes failure. |
| get_sam_mask_from_box | Segmentation mask for a supplied image and bounding box. |
Table C.1: Public perception operations (8 of 18). Image retrieval and geometric measurement supply evidence for the Proposer’s decisions. They do not certify manipulation outcomes.
| Operation | Public result or commanded effect |
|---|---|
| preprocess | Refresh public observations used by perception operations. |
| get_observation_geometry | Observation-frame geometry for a supplied public description, providing an alternative source-localization path. |
| get_previous_proposer_state | Previous decision-stage public state and the latest still-valid candidate scan. After an executed compiled grasp, while the gripper remains closed, this can include held_object_state for place compilation. |
| grasp_center_to_tool0_pose | Tool-frame pose corresponding to a selected grasp center and orientation. |
| compile_pose | In grasp mode, construct poses from Proposer-supplied grasp_center_xyz and closing_axis_world . In place mode, construct carry, release, and retreat poses from a target pixel, bound depth, and held-object geometry. |
| get_end_effector_pose | Current end-effector pose for the selected arm. |
Table C.2: State, compilation, and control operations (10 of 18). Pose construction does not move the robot. The final four operations issue arm or gripper commands.
| Candidate | Decision | Recorded basis |
|---|---|---|
| Relative referent | Admit | Selects the requested left bowl from fresh image evidence. The executed program implements the rule, and the entry states when the ordering must be reconsidered. |
| Contact binding | Reject | The proposed contact-point and local-geometry rule belongs to the frozen Cerebellum rather than Brain memory. |
| Destination geometry | Reject | Computing tray geometry and release clearance is a local placement procedure beyond the permitted Brain update. |
| Progress checking | Reject | The entry includes an object–hand co-motion test that was not implemented or tested. |
Table D.1: Governor decisions in the recorded bowl-transfer case. Descriptions condense the supplied candidate and decision records.