CaPtain: A Pipeline for Post-Training Code-as-Policy Robot Agents

Yijia Fan; Xilin He; Yilei Jiang; Dongming Wu; Jiaxu Wang; Xiangyu Yue

cs.ROOct 29, 2026

CaPtain: A Pipeline for Post-Training Code-as-Policy Robot Agents

Authors: Yijia Fan, Xilin He, Yilei Jiang, Dongming Wu, Jiaxu Wang, Xiangyu Yue

Abstract

Across coding, web, and tool-use tasks, language-model agents are now routinely trained with the same recipe: supervised fine-tuning (SFT) to learn the interface, then multi-turn reinforcement learning (RL) to master it. Code-as-policy (CaP) casts robot manipulation in the same form, as the policy writes code against robot APIs and reads back the results, so the same recipe should apply. Yet existing CaP agents sidestep training: they either prompt closed frontier models or delegate manipulation to pre-trained vision-language-action models (VLAs) and high-level skills, while small open vision-language models (VLMs) fail outright. We present CaPtain, an open post-training pipeline that turns a small open VLM into a capable CaP agent, with no VLA, frontier model, or privileged state at test time. CaPtain trains and deploys the policy inside CaPtain-Harness, which exposes low-level perception, geometry, and guarded-motion primitives across simulation and real hardware. We first align the base model via SFT on synthetic trajectories generated with GPT-6 Astra, then apply multi-turn RL. Multi-turn RL must credit tens of agent requests, whose physical effects often surface turns later, from a single end-of-episode outcome. We therefore introduce a physical critic that reads privileged simulator state, never seen by the policy, and credits each request by how much it changes the predicted chance of success. Starting from Qwen3.5-4B (0% zero-shot), CaPtain-4B reaches 55.4% held-out LIBERO success after SFT and 73.8% after RL. On LIBERO-PRO, it reaches 78.5%, surpassing several frontier-model agents, and its RL gains persist across three perturbation suites (+17.6 to +21.2 points). Without target-domain training, it attains 83.3% on SimplerEnv WidowX, exceeds frontier models on CaP-Bench, and runs on a real robot after simulation-only adaptation. We release all code, data, and model weights.

Figures & tables

Figure 1: Overview of CaPtain . Top: inference, one episode in CaPtain-Harness : at each request the policy writes a Python cell against a low-level API and receives text receipts and new images, for up to 60 requests. Bottom: training and test: SFT on synthetic agent trajectories, then RL in which a simulator critic turns privileged physical state into per-request credit; the critic is removed at test time.

		Host Success (%)				Strict Success (%)
Benchmark	$n$	Base	SFT	CaPtain-4B	$\Delta$	SFT	CaPtain-4B	$\Delta$
LIBERO	240	0.0	55.4	73.8	+18.3	52.1	67.5	+15.4
LIBERO-PRO (3 suites)	600	0.3	59.5	78.5	+19.0	56.7	71.5	+14.8
LIBERO-PRO (All 4 suites)	800	0.2	49.2	68.6	+19.4	45.1	58.9	+13.8
LIBERO-Plus	700	0.0	48.6	66.1	+17.6	44.6	59.0	+14.4
LIBERO-PRO (4 categories)	160	0.0	48.8	68.8	+20.0	48.1	60.6	+12.5

Table 1: SFT vs. RL across the LIBERO benchmark family. We report host and strict success rates (%) and the paired RL gain

\Delta

(in percentage points) over CaPtain-4B -SFT across

n

evaluation episodes per suite. Exact McNemar

p

-values are in Appendix K (

p<10^{-5}

on all primary suites).

Figure 2: RL gains survive perturbation. SFT vs. RL host and strict success on held-out LIBERO and on four perturbation suites.

Figure 3: Breakdown , CaPtain-4B -SFT vs. CaPtain-4B . (a) LIBERO strict success by suite. (b) LIBERO-Plus host success by perturbation dimension. (c) LIBERO-PRO host success by suite and perturbation type.

End-to-end VLAs & VLA-assisted agents
		Frontier	VLA	In-Domain	LIBERO-PRO (%)
System	Policy Model	API	Skill	Data	3 Suites	All 4 Suites
Harness VLA [ 94 ]	Claude Opus 4.7 + $\pi_{0.5}$	✓	✓	✓	87.7	82.4
AgenticRoboRL [ 48 ]	Qwen3.5-9B + RL-tuned $\pi_{0.5}$	✗	✓	✓	81.3	71.0
RLinf $\pi_{0.5}$ [ 94 ]	$\pi_{0.5}$ (RL fine-tuned)	✗	✓	✓	56.2	50.0
$\pi_{0.5}$ [ 97 ]	$\pi_{0.5}$ (SFT)	✗	✓	✓	12.8	10.8

Table 2: Comparison with published systems on LIBERO-PRO (success %). Systems are grouped by paradigm. Frontier API : queries a closed frontier model at test time; VLA Skill : invokes a policy trained on LIBERO demonstrations; In-Domain Data : uses LIBERO demonstrations or LIBERO-PRO development states. Protocol notes are in Appendix K .

Method	Type	All	1-Arm
PyRUA-Lean Coder-4B [ 67 ]	CaP+VLA	52.4	–
OpenWAM- $\alpha$ [ 76 ]	WAM	48.7	–
$\pi_{0.5}$ [ 55 ]	VLA	46.0	45.8
X-VLA [ 96 ]	VLA	20.9	–
Xiaomi-Robotics-0 [ 5 ]	VLA	18.2	–
$\pi_{0}$ [ 2 ]	VLA	16.3	18.3

Table 3: Zero-shot cross-benchmark transfer. Trained exclusively on LIBERO, CaPtain-4B is evaluated zero-shot across four unseen benchmarks and compared against in-domain VLAs/WAMs (trained on target-benchmark demonstrations) and CaP agents. RoboDojo reports official leaderboard Score and success rate (SR %); all other benchmarks report success rate (%).

Figure 4: Real-robot episodes. Two tasks with new objects and layouts; ten frames per episode, with time stamps. (b) requires reasoning about the instruction: the agent must infer that the cube with the color of the sky is the light-blue one and that the container that is not round is the rectangular tray.

Figure 5: The twin simulation used to adapt the policy to the XR1 arm (left) and the real scene at test time (right).

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Design of CaPtain-Harness . The policy writes Python into a persistent session ① that exposes only low-level primitives ②; motions are executed with IK and guarded contact ③, every call returns a receipt ④, a completion gate checks the agent’s own measurements ⑤, and environment-specific backends sit below the API ⑥. Receipt values are illustrative.

Capability	Semantics
observe	head + wrist RGB-D ( $512^{2}$ ), calibration, robot state
segment	SAM 3 text segmentation (Molmo point fallback) $\to$ depth-backed detections
select_detection	bind a chosen detection to identity evidence (read-only)
point	Molmo pointing $\to$ deprojected, point-segmented evidence
mark_point	validate and back-project a model-chosen pixel $\to$ detection id
track_target	optical-flow tracking of a detection into a newer observation

Appendix

Table 4: The 22 capabilities plus finish() . Physical actions (counted against the action budget) are marked

\bullet

Source	Rows	Tokens	Share
Broad-coverage trajectories (code-normalized, filtered)	7,903	348.9M	40.5%
Hard-task targeted trajectories (Round 2)	5,090	238.4M	27.7%
Hard-task targeted trajectories (Round 1)	2,138	99.3M	11.5%
Hard-task trajectories (60-request cap)	925	43.4M	5.0%
Self-generated RL rollouts (successful episodes)	1,386 (267 episodes)	52.3M	6.1%
Visual QA (loss weight 0.5)	8,519	78.8M	9.2%

Appendix

Table 5: CaPtain-4B -SFT data mix (25,961 rows, 861M tokens). Generating agents: GPT-6 Astra (654M tokens), GPT-6.1 Sol (63M), GPT-6 Sol (14M); RL policy (52M).

Optimization	Effect
FlashInfer attention and a chunked-prefill budget of 8,192 tokens in vLLM	+17.6% policy throughput (30.4 $\to$ 35.7 requests per minute per GPU); log-probabilities within the kernel noise floor
Sensor artifacts in shared memory	observe latency 2.2 s $\to$ 0.16 s
Harness fast loop and lazy rendering	$-27\%$ episode wall time, bit-identical physics and responses
Critic-based early stopping (5% control slice never stopped)	24–26% of episodes cut early; no successful episode in the control slice would have been cut
KL reference evaluated every 8 steps	unbiased KL estimate at $1/8$ of the reference cost
Pack-level rank assignment	balanced trainer ranks; identical log-probabilities, gradient cosine 0.9999

Appendix

Table 6: RL infrastructure optimizations that passed the numerical gates below.

Estimator	$B_{\text{noise}}$
GRPO (0/1 outcome)	9.0k
simulator critic, $\lambda{=}0$	12.6k
simulator critic, $\lambda{=}0.9$	6.7k
simulator critic, $\lambda{=}1$	6.5k

Appendix

Table 7: Gradient-noise scale

B_{\text{noise}}

(episodes) of advantage estimators on 109 groups

\times

16 SFT rollouts (offline; last four layers sketched). Lower is better.

Figure 7: Finish collapse. Evaluation of saved checkpoints during RL: 40 official seed-11 episodes plus 40 wide-layout episodes per checkpoint; dots are raw values and lines a 5-checkpoint moving average. After about round 340, host success stays flat while strict success falls.

Failure Category	RoboTwin ( $n=437$ )	RoboDojo ( $n=46$ )
Bimanual coordination required	167	1
Contact / force guard triggered	149	4
Reach / inverse kinematics (IK)	22	21
Perception / viewpoint occlusion	34	8
Request cap / looping	19	–
Grasp failure (empty / slip)	7	6

Appendix

Table 8: First blocking failure of failed transfer episodes (counts). Checkpoints: a checkpoint from an earlier RL run and CaPtain-4B -SFT, pooled; RoboTwin (dual Franka) and RoboDojo with earlier adapter versions.

Model	Host	Strict	Req. Cap	False Finish	Missed Finish	Spatial	Object	Goal	Long
CaPtain-4B -SFT	133	125	96	17	8	47/46	46/45	25/21	15/13
RL, round 128	174	159	68	10	15	54/54	59/57	35/31	26/17
RL, round 292	177	156	80	4	21	56/56	57/51	38/35	26/14
CaPtain-4B (RL, round 300)	177	162	73	5	15	58/58	55/52	41/36	23/16

Appendix

Table 9: LIBERO Held-Out (240 Episodes) Breakdown. Episode counts out of 240 overall, and per-suite Host/Strict counts out of 60.

Benchmark	$n$	Host $p$ -value	Strict $p$ -value
LIBERO	240	$4{\times}10^{-8}$	$7{\times}10^{-6}$
LIBERO-PRO (3 suites)	600	$1{\times}10^{-20}$	$2{\times}10^{-14}$
LIBERO-PRO (All 4 suites)	800	$8{\times}10^{-28}$	$1{\times}10^{-16}$
LIBERO-Plus	700	$8{\times}10^{-21}$	$5{\times}10^{-15}$
LIBERO-PRO (4 categories)	160	$4{\times}10^{-7}$	$3{\times}10^{-3}$
LIBERO-MAX Dynamic	80	$2{\times}10^{-3}$	$0.15$

Appendix

Table 10: Exact two-sided McNemar

p

-values for paired comparisons between CaPtain-4B and CaPtain-4B -SFT across all LIBERO benchmark suites (corresponding to Table 1 ).

Benchmark	Change	Scope
RoboTwin 2.0	Optional arm argument; a call without it goes to the arm nearer the target	Generic
	Contact detection recalibrated for the Franka gripper	Generic
	Tilted re-approach when IK fails; gripper closes on contact; grasp check reported in the receipt	Generic
	Automatic hand-over for tasks that pass an object between arms	Specific
RoboDojo	Success check that does not end the episode (the official check is final)	Generic
	Simulated depth and camera calibration as perception inputs	Specific

Appendix

Table 11: Backend changes per transfer benchmark ( Scope : Generic applies to any benchmark; Benchmark-specific applies to a single benchmark or embodiment).

Request	Time (s)	Action	Outcome
1	7.7	observe() , robot_geometry()	Head and wrist images; cube lies just outside the tray’s right edge
2	11.0–15.5	mark_point on cube and tray in head image; measure_object_geometry	Cube mask leaks into background (height 0.7 mm); not used
3–5	19.0–27.2	segment("yellow cube") , segment("wooden tray") ; select_detection bound to clicked points; re-measure	First binding rejected (point outside mask); after a new click, cube 3.0 cm tall and tray interior clear
6	31.7	inspect_contact_geometry , preview_motion	Top-down grasp with cube between the jaws
7	34.9–43.2	move_to 11 cm above the cube; observe	target_reached ; wrist view shows cube below gripper
8	45.9–52.2	move_to down at 2.5 cm/s; observe	target_reached , no contact stop

Appendix

Table 12: Real-robot trace (a). Instruction in TASK.md : “put the yellow cube in the wooden tray”. Times are seconds from episode start in the log and include model inference; the video in Figure 4 is shortened, so its time stamps differ.

Request	Time (s)	Action	Outcome
1	7.8	observe() , robot_geometry()	Head and wrist images
2	10.4–14.0	Reasoning: the pale-blue cube matches the sky; the brown tray is rectangular and the white bowl is round, so it is excluded. mark_point on both; measure_object_geometry	Cube 2.7 cm tall; tray center 11 cm away
3	18.1–18.2	Re-measure with shape="box" ; inspect_contact_geometry	Cube footprint 3.4 $\times$ 3.5 cm; centered top-down grasp
4	22.4–35.7	move_to 12 cm above cube; guarded descent at 2.5 cm/s; observe	Approach target_reached ; descent stops ( motion_stalled ) about 4 mm above target with no force; cube between open jaws
5–7	39.8–56.6	Re-select cube (stale detection rejected; wrist camera has no depth, so head camera); inspect_contact_geometry ; close; lift 7 cm	Open fraction 1.00 $\to$ 0.33; cube follows gripper
8–13	60.6–86.3	Measure held cube and tray; raise 10 cm more	Occluded masks of the held cube rejected; cube still held

Appendix

Table 13: Real-robot trace (b). Instruction in TASK.md : “Put the cube that has the color of the sky into the container that is not round.” Times as in Table 12 .

Figure 8: RoboTwin 2.0 episode. The scene starts with distractors around the cup and coaster. CaPtain-4B grasps the cup by its rim, carries it by the measured cup-to-coaster offset, and releases it on the coaster.

Figure 9: RoboDojo episode (head camera, cropped to the right half of the table). CaPtain-4B lifts the purple block, puts it on the yellow block, then picks up the dark gray block and puts it on top.

Method	Family	Frontier	LIBERO	LIBERO	LIBERO-PRO		LIBERO-	LIBERO-MAX		Plus /
	Family	at test	demos	LIBERO	3 Suites	4 Suites	Plus	Base	Dyn.	LIBERO
CaPtain-4B (ours)	CaP	✗	✗	73.8	78.5	68.6	66.1	70.0	61.2	90
CaPtain-4B -SFT (ours)	CaP	✗	✗	55.4	59.5	49.2	48.6	51.2	40.0	88
Qwen3.5-4B (base)	CaP	✗	✗	0.0	0.3	0.2	0.0	0.0	0.0	–
Vision-language-action models
OpenVLA [ 30 ]	VLA	✗	✓	76.5	0.0	0.0	15.6	–	–	20

Appendix

Table 14: LIBERO-family comparison (success %). Frontier : a frontier model is queried while the robot acts. Demos : trained on LIBERO demonstrations, directly or through a VLA skill. Bold: best among methods that use neither a frontier model at test time nor LIBERO demonstrations. Plus / LIBERO : LIBERO-Plus success as a percentage of standard LIBERO success, i.e. how much performance is retained under perturbation. Protocol differences are listed below the table.

RL for Robotics

Reinforcement learning for robotics trains robots to act through interaction, from locomotion and navigation to dexterous manipulation.

See all 260 papers

CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation
Mar 23, 2026Letian Fu, Justin Yu, Karim El-Refai +13Robotic Manipulation Robot Manipulation Benchmarks
1NVIDIA · 2UC Berkeley · 3Stanford University +1
RHO: Your Coding Agent is Secretly a Roboticist
Jun 15, 2026Karim Elmaaroufi, Justin Svegliato, Sarunas Kalade +3LLM Agent Harnesses AI Control
University of California, Berkeley · Embodied Science · AMD
CodeActionBench: Evaluating Agentic Code-as-Policy for Embodied Manipulation
Sep 27, 2026Yiheng Lyu, Xueying Jiang, Wenhao Li +2Robotic Policy Evaluation Robotic Manipulation
Nanyang Technological University, Singapore · Independent Researcher

Mar 23, 2026J/K move · Enter open · S save

CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation

Letian Fu, Justin Yu, Karim El-Refai, Ethan Kou+12

1NVIDIA · 2UC Berkeley · 3Stanford University+1

Robotic Manipulation Robot Manipulation Benchmarks

Open paper AskPDF arXiv

"Code-as-Policy" considers how executable code can complement data-intensive Vision-Language-Action (VLA) methods, yet their effectiveness as autonomous controllers for embodied manipulation remains underexplored. We present CaP-X, an open-access framework for systematically studying Code-as-Policy agents in robot manipulation. At its core is CaP-Gym, an interactive environment in which agents control robots by synthesizing and executing programs that compose perception and control primitives. Building on this foundation, CaP-Bench evaluates frontier language and vision-language models across varying levels of abstraction, interaction, and perceptual grounding. Across 12 models, CaP-Bench reveals a consistent trend: performance improves with human-crafted abstractions but degrades as these priors are removed, exposing a dependence on designer scaffolding. At the same time, we observe that this gap can be mitigated through scaling agentic test-time computation--through multi-turn interaction, structured execution feedback, visual differencing, automatic skill synthesis, and ensembled reasoning--substantially improves robustness even when agents operate over low-level primitives. These findings allow us to derive CaP-Agent0, a training-free framework that recovers human-level reliability on several manipulation tasks in simulation and on real embodiments. We further introduce CaP-RL, showing reinforcement learning with verifiable rewards improves success rates and transfers from sim2real with minimal gap. Together, CaP-X provides a principled, open-access platform for advancing embodied coding agents.

Abstract

RL for Robotics

Explore similar work

CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation

RHO: Your Coding Agent is Secretly a Roboticist

CodeActionBench: Evaluating Agentic Code-as-Policy for Embodied Manipulation