cs.ROOct 29, 2026

CaPtain: A Pipeline for Post-Training Code-as-Policy Robot Agents

Authors: Yijia Fan, Xilin He, Yilei Jiang, Dongming Wu, Jiaxu Wang, Xiangyu Yue

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Across coding, web, and tool-use tasks, language-model agents are now routinely trained with the same recipe: supervised fine-tuning (SFT) to learn the interface, then multi-turn reinforcement learning (RL) to master it. Code-as-policy (CaP) casts robot manipulation in the same form, as the policy writes code against robot APIs and reads back the results, so the same recipe should apply. Yet existing CaP agents sidestep training: they either prompt closed frontier models or delegate manipulation to pre-trained vision-language-action models (VLAs) and high-level skills, while small open vision-language models (VLMs) fail outright. We present CaPtain, an open post-training pipeline that turns a small open VLM into a capable CaP agent, with no VLA, frontier model, or privileged state at test time. CaPtain trains and deploys the policy inside CaPtain-Harness, which exposes low-level perception, geometry, and guarded-motion primitives across simulation and real hardware. We first align the base model via SFT on synthetic trajectories generated with GPT-6 Astra, then apply multi-turn RL. Multi-turn RL must credit tens of agent requests, whose physical effects often surface turns later, from a single end-of-episode outcome. We therefore introduce a physical critic that reads privileged simulator state, never seen by the policy, and credits each request by how much it changes the predicted chance of success. Starting from Qwen3.5-4B (0% zero-shot), CaPtain-4B reaches 55.4% held-out LIBERO success after SFT and 73.8% after RL. On LIBERO-PRO, it reaches 78.5%, surpassing several frontier-model agents, and its RL gains persist across three perturbation suites (+17.6 to +21.2 points). Without target-domain training, it attains 83.3% on SimplerEnv WidowX, exceeds frontier models on CaP-Bench, and runs on a real robot after simulation-only adaptation. We release all code, data, and model weights.

Figures & tables

Figure 1: Overview of CaPtain . Top: inference, one episode in CaPtain-Harness : at each request the policy writes a Python cell against a low-level API and receives text receipts and new images, for up to 60 requests. Bottom: training and test: SFT on synthetic agent trajectories, then RL in which a simulator critic turns privileged physical state into per-request credit; the critic is removed at test time.
  Host Success (%)Strict Success (%)
BenchmarknnBaseSFTCaPtain-4BΔ\DeltaSFTCaPtain-4BΔ\Delta
LIBERO2400.055.473.8+18.352.167.5+15.4
LIBERO-PRO (3 suites)6000.359.578.5+19.056.771.5+14.8
LIBERO-PRO (All 4 suites)8000.249.268.6+19.445.158.9+13.8
LIBERO-Plus7000.048.666.1+17.644.659.0+14.4
LIBERO-PRO (4 categories)1600.048.868.8+20.048.160.6+12.5
Table 1: SFT vs. RL across the LIBERO benchmark family. We report host and strict success rates (%) and the paired RL gain Δ\Delta (in percentage points) over CaPtain-4B -SFT across nn evaluation episodes per suite. Exact McNemar pp -values are in Appendix K ( p<10−5p<10^{-5} on all primary suites).
Figure 2: RL gains survive perturbation. SFT vs. RL host and strict success on held-out LIBERO and on four perturbation suites.
Figure 3: Breakdown , CaPtain-4B -SFT vs. CaPtain-4B . (a) LIBERO strict success by suite. (b) LIBERO-Plus host success by perturbation dimension. (c) LIBERO-PRO host success by suite and perturbation type.
  FrontierVLAIn-DomainLIBERO-PRO (%)
SystemPolicy ModelAPISkillData3 SuitesAll 4 Suites
End-to-end VLAs & VLA-assisted agents
Harness VLA [ 94 ]Claude Opus 4.7 + π0.5\pi_{0.5}✓✓✓87.782.4
AgenticRoboRL [ 48 ]Qwen3.5-9B + RL-tuned π0.5\pi_{0.5}✗✓✓81.371.0
RLinf π0.5\pi_{0.5} [ 94 ]π0.5\pi_{0.5} (RL fine-tuned)✗✓✓56.250.0
π0.5\pi_{0.5} [ 97 ]π0.5\pi_{0.5} (SFT)✗✓✓12.810.8
Table 2: Comparison with published systems on LIBERO-PRO (success %). Systems are grouped by paradigm. Frontier API : queries a closed frontier model at test time; VLA Skill : invokes a policy trained on LIBERO demonstrations; In-Domain Data : uses LIBERO demonstrations or LIBERO-PRO development states. Protocol notes are in Appendix K .
MethodTypeAll1-Arm
PyRUA-Lean Coder-4B [ 67 ]CaP+VLA52.4–
OpenWAM- α\alpha [ 76 ]WAM48.7–
π0.5\pi_{0.5} [ 55 ]VLA46.045.8
X-VLA [ 96 ]VLA20.9–
Xiaomi-Robotics-0 [ 5 ]VLA18.2–
π0\pi_{0} [ 2 ]VLA16.318.3
Table 3: Zero-shot cross-benchmark transfer. Trained exclusively on LIBERO, CaPtain-4B is evaluated zero-shot across four unseen benchmarks and compared against in-domain VLAs/WAMs (trained on target-benchmark demonstrations) and CaP agents. RoboDojo reports official leaderboard Score and success rate (SR %); all other benchmarks report success rate (%).
Figure 4: Real-robot episodes. Two tasks with new objects and layouts; ten frames per episode, with time stamps. (b) requires reasoning about the instruction: the agent must infer that the cube with the color of the sky is the light-blue one and that the container that is not round is the rectangular tray.
Figure 5: The twin simulation used to adapt the policy to the XR1 arm (left) and the real scene at test time (right).
Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Design of CaPtain-Harness . The policy writes Python into a persistent session ① that exposes only low-level primitives ②; motions are executed with IK and guarded contact ③, every call returns a receipt ④, a completion gate checks the agent’s own measurements ⑤, and environment-specific backends sit below the API ⑥. Receipt values are illustrative.
CapabilitySemantics
observehead + wrist RGB-D ( 5122512^{2} ), calibration, robot state
segmentSAM 3 text segmentation (Molmo point fallback) →\to depth-backed detections
select_detectionbind a chosen detection to identity evidence (read-only)
pointMolmo pointing →\to deprojected, point-segmented evidence
mark_pointvalidate and back-project a model-chosen pixel →\to detection id
track_targetoptical-flow tracking of a detection into a newer observation
Appendix
Table 4: The 22 capabilities plus finish() . Physical actions (counted against the action budget) are marked ∙\bullet .
SourceRowsTokensShare
Broad-coverage trajectories (code-normalized, filtered)7,903348.9M40.5%
Hard-task targeted trajectories (Round 2)5,090238.4M27.7%
Hard-task targeted trajectories (Round 1)2,13899.3M11.5%
Hard-task trajectories (60-request cap)92543.4M5.0%
Self-generated RL rollouts (successful episodes)1,386 (267 episodes)52.3M6.1%
Visual QA (loss weight 0.5)8,51978.8M9.2%
Appendix
Table 5: CaPtain-4B -SFT data mix (25,961 rows, 861M tokens). Generating agents: GPT-6 Astra (654M tokens), GPT-6.1 Sol (63M), GPT-6 Sol (14M); RL policy (52M).
OptimizationEffect
FlashInfer attention and a chunked-prefill budget of 8,192 tokens in vLLM+17.6% policy throughput (30.4 →\to 35.7 requests per minute per GPU); log-probabilities within the kernel noise floor
Sensor artifacts in shared memoryobserve latency 2.2 s →\to 0.16 s
Harness fast loop and lazy rendering−27%-27\% episode wall time, bit-identical physics and responses
Critic-based early stopping (5% control slice never stopped)24–26% of episodes cut early; no successful episode in the control slice would have been cut
KL reference evaluated every 8 stepsunbiased KL estimate at 1/81/8 of the reference cost
Pack-level rank assignmentbalanced trainer ranks; identical log-probabilities, gradient cosine 0.9999
Appendix
Table 6: RL infrastructure optimizations that passed the numerical gates below.
EstimatorBnoiseB_{\text{noise}}
GRPO (0/1 outcome)9.0k
simulator critic, λ=0\lambda{=}012.6k
simulator critic, λ=0.9\lambda{=}0.96.7k
simulator critic, λ=1\lambda{=}16.5k
Appendix
Table 7: Gradient-noise scale BnoiseB_{\text{noise}} (episodes) of advantage estimators on 109 groups ×\times 16 SFT rollouts (offline; last four layers sketched). Lower is better.
Figure 7: Finish collapse. Evaluation of saved checkpoints during RL: 40 official seed-11 episodes plus 40 wide-layout episodes per checkpoint; dots are raw values and lines a 5-checkpoint moving average. After about round 340, host success stays flat while strict success falls.
Failure CategoryRoboTwin ( n=437n=437 )RoboDojo ( n=46n=46 )
Bimanual coordination required1671
Contact / force guard triggered1494
Reach / inverse kinematics (IK)2221
Perception / viewpoint occlusion348
Request cap / looping19–
Grasp failure (empty / slip)76
Appendix
Table 8: First blocking failure of failed transfer episodes (counts). Checkpoints: a checkpoint from an earlier RL run and CaPtain-4B -SFT, pooled; RoboTwin (dual Franka) and RoboDojo with earlier adapter versions.
ModelHostStrictReq. CapFalse FinishMissed FinishSpatialObjectGoalLong
CaPtain-4B -SFT1331259617847/4646/4525/2115/13
RL, round 12817415968101554/5459/5735/3126/17
RL, round 2921771568042156/5657/5138/3526/14
CaPtain-4B (RL, round 300)1771627351558/5855/5241/3623/16
Appendix
Table 9: LIBERO Held-Out (240 Episodes) Breakdown. Episode counts out of 240 overall, and per-suite Host/Strict counts out of 60.
BenchmarknnHost pp -valueStrict pp -value
LIBERO2404×10−84{\times}10^{-8}7×10−67{\times}10^{-6}
LIBERO-PRO (3 suites)6001×10−201{\times}10^{-20}2×10−142{\times}10^{-14}
LIBERO-PRO (All 4 suites)8008×10−288{\times}10^{-28}1×10−161{\times}10^{-16}
LIBERO-Plus7008×10−218{\times}10^{-21}5×10−155{\times}10^{-15}
LIBERO-PRO (4 categories)1604×10−74{\times}10^{-7}3×10−33{\times}10^{-3}
LIBERO-MAX Dynamic802×10−32{\times}10^{-3}0.150.15
Appendix
Table 10: Exact two-sided McNemar pp -values for paired comparisons between CaPtain-4B and CaPtain-4B -SFT across all LIBERO benchmark suites (corresponding to Table 1 ).
BenchmarkChangeScope
RoboTwin 2.0Optional arm argument; a call without it goes to the arm nearer the targetGeneric
 Contact detection recalibrated for the Franka gripperGeneric
 Tilted re-approach when IK fails; gripper closes on contact; grasp check reported in the receiptGeneric
 Automatic hand-over for tasks that pass an object between armsSpecific
RoboDojoSuccess check that does not end the episode (the official check is final)Generic
 Simulated depth and camera calibration as perception inputsSpecific
Appendix
Table 11: Backend changes per transfer benchmark ( Scope : Generic applies to any benchmark; Benchmark-specific applies to a single benchmark or embodiment).
RequestTime (s)ActionOutcome
17.7observe() , robot_geometry()Head and wrist images; cube lies just outside the tray’s right edge
211.0–15.5mark_point on cube and tray in head image; measure_object_geometryCube mask leaks into background (height 0.7 mm); not used
3–519.0–27.2segment("yellow cube") , segment("wooden tray") ; select_detection bound to clicked points; re-measureFirst binding rejected (point outside mask); after a new click, cube 3.0 cm tall and tray interior clear
631.7inspect_contact_geometry , preview_motionTop-down grasp with cube between the jaws
734.9–43.2move_to 11 cm above the cube; observetarget_reached ; wrist view shows cube below gripper
845.9–52.2move_to down at 2.5 cm/s; observetarget_reached , no contact stop
Appendix
Table 12: Real-robot trace (a). Instruction in TASK.md : “put the yellow cube in the wooden tray”. Times are seconds from episode start in the log and include model inference; the video in Figure 4 is shortened, so its time stamps differ.
RequestTime (s)ActionOutcome
17.8observe() , robot_geometry()Head and wrist images
210.4–14.0Reasoning: the pale-blue cube matches the sky; the brown tray is rectangular and the white bowl is round, so it is excluded. mark_point on both; measure_object_geometryCube 2.7 cm tall; tray center 11 cm away
318.1–18.2Re-measure with shape="box" ; inspect_contact_geometryCube footprint 3.4 ×\times 3.5 cm; centered top-down grasp
422.4–35.7move_to 12 cm above cube; guarded descent at 2.5 cm/s; observeApproach target_reached ; descent stops ( motion_stalled ) about 4 mm above target with no force; cube between open jaws
5–739.8–56.6Re-select cube (stale detection rejected; wrist camera has no depth, so head camera); inspect_contact_geometry ; close; lift 7 cmOpen fraction 1.00 →\to 0.33; cube follows gripper
8–1360.6–86.3Measure held cube and tray; raise 10 cm moreOccluded masks of the held cube rejected; cube still held
Appendix
Table 13: Real-robot trace (b). Instruction in TASK.md : “Put the cube that has the color of the sky into the container that is not round.” Times as in Table 12 .
Figure 8: RoboTwin 2.0 episode. The scene starts with distractors around the cup and coaster. CaPtain-4B grasps the cup by its rim, carries it by the measured cup-to-coaster offset, and releases it on the coaster.
Figure 9: RoboDojo episode (head camera, cropped to the right half of the table). CaPtain-4B lifts the purple block, puts it on the yellow block, then picks up the dark gray block and puts it on top.
MethodFamilyFrontierLIBEROLIBEROLIBERO-PROLIBERO-LIBERO-MAXPlus /
 at testdemos3 Suites4 SuitesPlusBaseDyn.LIBERO
CaPtain-4B (ours)CaP✗✗73.878.568.666.170.061.290
CaPtain-4B -SFT (ours)CaP✗✗55.459.549.248.651.240.088
Qwen3.5-4B (base)CaP✗✗0.00.30.20.00.00.0–
Vision-language-action models
OpenVLA [ 30 ]VLA✗✓76.50.00.015.6––20
Appendix
Table 14: LIBERO-family comparison (success %). Frontier : a frontier model is queried while the robot acts. Demos : trained on LIBERO demonstrations, directly or through a VLA skill. Bold: best among methods that use neither a frontier model at test time nor LIBERO demonstrations. Plus / LIBERO : LIBERO-Plus success as a percentage of standard LIBERO success, i.e. how much performance is retained under perturbation. Protocol differences are listed below the table.
More in this topic

RL for Robotics

Reinforcement learning for robotics trains robots to act through interaction, from locomotion and navigation to dexterous manipulation.

See all 260 papers

Explore similar work

CardsList
Mar 23, 2026

CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation

"Code-as-Policy" considers how executable code can complement data-intensive Vision-Language-Action (VLA) methods, yet their effectiveness as autonomous controllers for embodied manipulation remains underexplored. We present CaP-X, an open-access framework for systematically studying Code-as-Policy agents in robot manipulation. At its core is CaP-Gym, an interactive environment in which agents control robots by synthesizing and executing programs that compose perception and control primitives. Building on this foundation, CaP-Bench evaluates frontier language and vision-language models across varying levels of abstraction, interaction, and perceptual grounding. Across 12 models, CaP-Bench reveals a consistent trend: performance improves with human-crafted abstractions but degrades as these priors are removed, exposing a dependence on designer scaffolding. At the same time, we observe that this gap can be mitigated through scaling agentic test-time computation--through multi-turn interaction, structured execution feedback, visual differencing, automatic skill synthesis, and ensembled reasoning--substantially improves robustness even when agents operate over low-level primitives. These findings allow us to derive CaP-Agent0, a training-free framework that recovers human-level reliability on several manipulation tasks in simulation and on real embodiments. We further introduce CaP-RL, showing reinforcement learning with verifiable rewards improves success rates and transfers from sim2real with minimal gap. Together, CaP-X provides a principled, open-access platform for advancing embodied coding agents.

Letian Fu, Justin Yu, Karim El-Refai +13

1NVIDIA · 2UC Berkeley · 3Stanford University +1

Robotic ManipulationRobot Manipulation Benchmarks
AskPDF
Jun 15, 2026

RHO: Your Coding Agent is Secretly a Roboticist

Code-as-Policies (CaP) has shown that large language models (LLMs) can write code to solve robotics tasks by composing perception, planning, and control primitives. Recent CaP systems, however, rely on multi-turn code-generation loops at test time, which is often infeasible for real-time robot control. We introduce Robotics Harness Optimization (RHO), a novel paradigm in which tool-enabled coding agents, at training time, propose and search for interpretable, neurosymbolic multi-file policy repositories (Repositories-as-Policies) that compose these primitives rather than a single prompt, function, or file. RHO searches with reflective feedback from environment reward and execution rather than teleoperation demonstrations. It generalizes to perturbed pick-and-place settings like LIBERO-PRO, where OpenVLA scores 0.0% and π0.5π_{0.5} averages 12.83%. Using the same low-level primitives, RHO reaches a 45.0% success rate, 2.5x higher than the strongest multi-turn agentic system, and 3.5x higher than π0.5π_{0.5}. On Robosuite, RHO sets a new state-of-the-art of 70.0%, exceeding the prior multi-turn record of 68.29% using single-turn execution with no corrective LLM code edits at deployment. When an LLM is used in the control loop, as on RAI's O3DE benchmark, RHO optimizes the deployed agent's multi-file harness of prompts, tools, and control code, improving held-out success from 23.5% to 44.3% with 20% less wall-clock time and 27% fewer tool calls.

Karim Elmaaroufi, Justin Svegliato, Sarunas Kalade +3

University of California, Berkeley · Embodied Science · AMD

LLM Agent HarnessesAI ControlRobotic ManipulationRobot Policy LearningAI Coding AgentsLarge Language Model-Based Robot Planning+5
AskPDF
Sep 27, 2026

CodeActionBench: Evaluating Agentic Code-as-Policy for Embodied Manipulation

How well can general-purpose multimodal models turn visual understanding and reasoning into embodied manipulation via executable code? We introduce CodeActionBench, a benchmark of 25 manipulation tasks that evaluates this capability through agentic Code-as-Policy. Without task-specific fine-tuning, demonstrations, external specialist perception or grasp modules, privileged scene state, or predefined task policies, agents should select visual evidence, form task-relevant 3D estimates, construct manipulation targets, and iteratively execute and revise their policies. A shared robot API provides RGB observations, calibrated geometric operations, robot feedback, and bounded motion, leaving task-dependent decisions to the evaluated agent. Fixed task instances, resource budgets, and a hidden physical-outcome verifier support controlled comparisons across models and harness configurations. Extensive evaluations across nine configurations and 675 attempts achieve success rates ranging from 2.7% to 73.3%. The strongest configuration, GPT-6 Astra with Codex CLI, solves 22 of 25 tasks at least once in three attempts, demonstrating the best performance while still leaving substantial room for improvement. Trajectory analyses reveal difficulties in spatial alignment, object retention, and completion judgment, including task failures despite successfully completed motions. CodeActionBench provides a controlled testbed for measuring how general-purpose models translate their capabilities into manipulation behavior and for examining typical failure scenarios in that process.

Yiheng Lyu, Xueying Jiang, Wenhao Li +2

Nanyang Technological University, Singapore · Independent Researcher

Robotic Policy EvaluationRobotic ManipulationRobot Manipulation BenchmarksVLMs for RoboticsEmbodied Agent Benchmarks
AskPDF