Organizations: University of Monastir · St. Mildred’s-Lightbourn School · Cornell University · Carnegie Mellon University · Silverstream AI · Mila – Quebec AI Institute
Turning a frontier vision-language model into a robot policy usually means fine-tuning it to emit an action representation it never saw in pretraining, which throws away much of the reasoning that made the model worth reaching for. We go the other way and keep the VLM frozen. It writes the policy as a short Python control function, with no demonstrations and no fine-tuning. Writing that code once is open-loop, though. Existing closed-loop methods react at the wrong level: they retry a fixed policy or pick a different subtask, but never rewrite the code that failed. VLCP closes the loop where the failure actually lives, on the control code, within a single episode. Every K steps the VLM re-observes the scene from multi-view RGB, proprioceptive state, and a state delta, then rewrites the control function from what it just saw, so a failure is caught before it compounds. We evaluate on a 57-task MuJoCo/RoboVerse sweep. This training-free policy reaches 35.1% pooled success, against 3.5% for the identical system queried once per episode. That tenfold gap holds with non-overlapping confidence intervals in every scene family. The gain traces to a 27.3% within-episode recovery rate on failed grasps: a miss an open-loop controller would carry to the end of the episode gets re-observed and fixed at the next replan. And the loop stays cheap. A median 84% of input tokens hit cache, an episode needs only about 10 compact queries, and control blocks written during any replan persist to a cross-episode skill library reused in later prompts.
Figures & tables
Figure 1: VLCP closes the loop at the level of the control code. A frozen VLM observes synchronized multi-view RGB, numeric state, and an explicit state delta ( observe ), then writes a short-horizon Python control(obs, J) function ( code ) that runs open-loop for K steps ( execute ). At each block boundary the VLM is re-queried and rewrites the function from what it just saw, so a failure is caught and corrected within the same episode (block n+1 ). The prompt is structured for cache reuse (A docs/API/examples, B skill library, C goal pin), and helper modules written during any replan are saved to a cross-episode skill library and re-injected into later prompts.
Method
Code
Runtime
Edits
Within-
Train-
policy
re-obs.
code
ep.
free
CaP / ProgPrompt / ELLMER [ 8 , 10 , 9 ]
✓
✗
✗
✗
✓
Robotic Programmer [ 12 ]
✓
✗
✗
✗
✓
Code-as-Monitor / DoReMi [ 13 , 4 ]
✓
✓
✗
✓
✓
Inner Monologue [ 5 ]
✗
✓
✗
✓
✓
VLA-0 / OpenVLA / π0 [ 3 , 6 , 1 ]
✗
✓
✗
✓
✗
Table 1: Where prior families sit along the axes VLCP unifies. Code policy : the executed controller is human-readable code. Runtime re-obs. : the method senses the scene during the episode. Edits code : the revision mechanism rewrites the control program that failed, instead of retrying it, swapping in a library entry, or escalating to a planner. Within-ep. : revision happens mid-episode, not only between trials. Train-free : no demonstration fine-tuning. Closing the loop on the control code within an episode (all five columns) is a very recent frontier. To our knowledge, VLCP and the concurrent CaP-Agent0 [ 2 ] are among the few methods that occupy it, and of the two it is VLCP that contributes the controlled ablation isolating the loop (Section 4.3 ) and a direct measurement of within-episode recovery (Section 4.5 ).
Figure 2: Within-episode replanning is the dominant factor. The same system with the replan cadence set to K=T (open-loop, grey) collapses to near-zero success in every scene family, while VLCP’s K=50 closed-loop replanning (orange) recovers an order of magnitude more task completions. Bars are pooled point estimates, whiskers are Wilson 95% CIs, and the dashed separator marks the pooled Overall group.
Figure 3: Success rate under instruction reframing, per task and pooled overall (right, dashed separator). Bars are point estimates and whiskers the Wilson 95% confidence interval; colour encodes the method (blue: VLA-0, orange: VLCP). Solid bars: original instruction. Hatched bars: reframed instruction ( “put { object } in the basket” ). VLA-0’s response is uneven: near-ceiling and unchanged on chocolate_pudding and ketchup , but collapsing outright on butter , for example (non-overlapping intervals). VLCP (orange) shows no comparable phrasing dependence, and its pooled success is unchanged under reframing.
Figure 4: Closed-loop recovery within a single episode ( chocolate_pudding , front camera), one instance from the recovery population of Table 2 . (a) The first control block reaches for the object and attempts a grasp. (b) The gripper closes short and lifts empty ; the object stays on the table. An open-loop controller would carry this failure to the end of the episode. (c) At the next replan boundary VLCP re-observes the miss and rewrites the control function, re-approaching the object. (d) The corrected code grasps and lifts the pudding. (e) It goes into the basket and the task succeeds. Recovery works because the loop closes on the control code itself, not just on the observations fed to a fixed policy.
Quantity
Value
Pick-and-place episodes (closed-loop)
75
with ≥1 failed grasp
55
that recover and lift the object
15
Recovery rate (recovered / failed)
27.3%
Median replans to recover
3
Open-loop ( K=T ) recovery rate
0% (by construction)
Table 2: Within-episode recovery on the 75 closed-loop pick-and-place episodes, computed directly from the logged per-block object height and gripper state ( paper/analysis/recovery_rate.py ). A failed grasp is a block with the hand at the object and the gripper closed but the object not lifted; an episode recovers if a later block secures and lifts it. Open-loop ( K=T ) recovery is 0 by construction, since it never re-observes.
Quantity
Median
Mean
Replan (VLM) calls per episode
10
8.9
Static prompt prefix (tokens)
7.9k
8.2k
Input tokens per replan
43k
43k
Cached-prefix fraction per replan (%)
83.8
79.4
Output tokens per replan
349
421
Latency per replan (s)
18.8
25.9
Table 3: Measured operating cost of VLCP, aggregated over the closed-loop runs ( 81 episodes, ≈721 replan calls total). Latency is wall-clock per replan call, recovered from the logged request/response timestamps. The cached-prefix fraction is the share of each replan’s input tokens lying in the static, cache-eligible prompt prefix, measured over the 640 post-first calls that can hit cache.
Vision-Language-Action (VLA) models remain brittle in long-horizon, contact-rich manipulation because success-only imitation provides little supervision for execution drift, while failed rollouts are often discarded. We introduce RePO-VLA, a recovery-driven policy optimization framework that assigns distinct roles to success, recovery, and failure trajectories. RePO-VLA first applies Recovery-Aware Initialization (RAI), slicing recovery segments and resetting history so corrective actions depend on the current adverse state rather than the preceding failure. It then learns a Progress-Aware Semantic Value Function (PAS-VF), aligning spatiotemporal trajectory features with instructions and successful references. The resulting labels salvage useful failure prefixes via reliability decay, while low-value labels mark drift and terminal breakdowns, teaching differences among nominal, failed, and corrective actions. The data engine turns adverse states into planner-generated or human-collected corrective rollouts, teaching recovery to the success manifold. Value-Conditioned Refinement (VCR) trains the policy to prefer high-progress actions. At deployment, a fixed high value (v=1.0) biases actions toward the learned success manifold without online failure detectors or heuristic retries. We introduce FRBench, with standardized error injection and recovery-focused evaluation. Across simulated and real-world bimanual tasks, RePO-VLA improves robustness, raising adversarial success from 20% to 75% on average and up to 80% in scaled real-world trials.
Weijia Liufu, Xiaoyu Guo, Ruiyi Chen +16
1Sun Yat-sen University · 2South China University of Technology · 3Peng Cheng Laboratory +3
Vision-language-action (VLA) policies provide strong priors for language-conditioned manipulation, but remain brittle in off-nominal states requiring targeted recovery. We propose ReCoVLA -- a failure-conditioned residual recovery framework that keeps a pretrained VLA policy frozen, uses an external vision-language model (VLM) to infer the failure mode and recovery stage, and compiles a structured reward from task-relevant components. Rather than using the VLM to generate actions or rewards directly, ReCoVLA uses it as a semantic reward selector: it predicts a recovery descriptor and reward mask for in-simulation residual-policy training, followed by zero-shot sim-to-real deployment of the trained recovery policies. This decouples high-level failure understanding from low-level corrective control to support different VLAs. Experiments across short-horizon, long-horizon, and contact-rich manipulation tasks show that ReCoVLA outperforms the tested baselines on average. In simulation, our reward compiler improves average success from 36.7% for the fine-tuned π0.5 baseline to 66.7%. In physical zero-shot sim-to-real experiments, ReCoVLA achieves the best average performance, with 61.7% success.
Haodi Hu, Chung-Ta Huang, Jing Liu +4
University of Southern California, USA · Mitsubishi Electric Research Laboratories (MERL), USA · Harvard University, USA
Vision-language-action (VLA) policies commonly place a multi-billion-parameter model inside the feedback loop, repeatedly recomputing task context whenever they predict a new action chunk. Yet task understanding changes far more slowly than the robot state. We introduce Fast-VLA, which invokes a frozen 4B VLM once per episode to contextualize the instruction and initial views as fixed, multi-layer key-value task memory. During execution, a lightweight online actor fuses learned time- and arm-indexed queries with current visual and proprioceptive feedback to form action queries. These queries retrieve task context from the fixed memory and predict action chunks without re-running the foundation model. Fast-VLA achieves 92.46% and 92.40% success on RoboTwin 2.0 Clean and Randomized, and 95.95% and 79.11% on LIBERO and LIBERO-Plus evaluation. Against three continually conditioned baselines on the same hardware, it reduces RoboTwin amortized policy inference time by 10.9-17.1x and requires 1.22 GiB of post-prefill online resident memory. Task-memory swaps further show that the actor relies on episode-matched task memory. These results show that task context can be established once while closed-loop control remains responsive throughout execution. Project page: https://garfield-kh.github.io/FastVLA
Kehong Gong, Guanli Hou, Xiaoxu Zheng +6
Huawei Technologies Co., Ltd · Huawei Central Research Institute