OGPO: Offline Goal-conditioned Policy Optimization for Recoverable Vision-Language-Action Models
Authors: Xule Gao, Xi Wang, Zehua Zang, Rui Wang, Changwen Zheng, Chuxiong Sun
Organizations: National Key Laboratory of Space Integrated Information System · Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Wangxuan Institute of Computer Technology, Peking University
Vision-language-action (VLA) models have shown strong promise in robotic manipulation, yet their post-training still predominantly relies on trajectory-level supervised imitation over robot demonstrations. While effective for learning feasible actions, this paradigm supervises policies to reproduce expert actions at each step, biasing them toward demonstration-specific action paths and providing limited guidance for recovering toward task-progress states once execution deviates from demonstrated trajectories. To address this limitation, we propose OGPO, an Offline Goal-conditioned Policy Optimization framework for recoverable VLA post-training. Instead of treating offline demonstrations merely as step-wise action labels, OGPO adaptively relabels trajectories with semantic key states and constructs goal-conditioned process rewards to optimize the policy for reaching these task-progress states. In this way, OGPO shifts VLA post-training from trajectory-level action imitation to key-state reachability learning, enabling policies to acquire a more robust ability to complete goal states from feasible off-demonstration states without requiring additional online interaction. Experiments on LIBERO and MetaWorld show that OGPO consistently improves VLA policy performance over standard supervised fine-tuning. Moreover, evaluations on LIBERO-Plus and our proposed LIBERO-ReAct, a mid-execution perturbation benchmark, further demonstrate that OGPO achieves stronger robustness and goal-directed recoverability under distribution shifts and recoverable off-demonstration states.
Figures & tables
Figure 1: We design two representative perturbation cases to evaluate the robustness and goal-directed recoverability of offline-trained VLA policies under deviations from demonstration trajectories. In case (a), the gripper is forcibly opened after grasping the object, while position perturbation is applied when the end effector approaches the drawer. The results show that π0.5 -SFT fails to complete the task, whereas π0.5 -OGPO successfully recovers toward the goal state and finishes the task, demonstrating stronger recoverability under execution-time deviations.
Table 1: Performance comparison on the LIBERO benchmark. Policy input notation: T (third-view image), I (instruction), W (wrist image), P (proprioception). OGPO is built upon the few-shot SFT policy, with ↑ indicating performance gains over the few-shot baseline.
Model
Input
MetaWorld-MT50
Easy
Medium
Hard
Very Hard
Avg.
Diffusion Policy [ 8 ]
T+C
23.1
10.7
1.9
6.1
10.5
TinyVLA [ 32 ]
T+P+C
77.6
21.5
11.4
15.8
31.6
SmolVLA [ 29 ]
T+P+C
87.1
51.8
70.0
64.0
68.2
π0 -PPO (RLinf) [ 5 ]
T+P+I
91.1
81.8
78.3
92.0
85.8
π0 -SFT
T+P+C
77.9
51.8
53.3
20.0
50.8
Table 2: Performance comparison on the MetaWorld-MT50 benchmark. Policy input notation: T (third-view image), P (proprioception), C (task conditioning). C denotes the task information used to distinguish MetaWorld tasks, and may be instantiated as a language instruction, task ID, or task embedding depending on the model.
Model
Perturbation Dimensions
Total
Camera
Robot-Init
Noise
Layout
Action
Gripper
End-effector
π0 [ 2 ]
13.8
6.0
79.0
68.9
81.5
9.3
62.4
45.8
π0 +fast [ 27 ]
65.1
21.6
74.4
68.8
75.2
19.8
63.8
55.5
OpenVLA [ 17 ]
0.8
3.5
15.2
28.5
19.7
24.3
31.5
17.6
OpenVLA-OFT [ 16 ]
56.4
31.9
75.8
74.2
85.2
84.4
87.6
70.8
π0.5 -SFT [ 1 ]
62.8
69.8
69.5
85.4
86.6
76.8
86.0
76.7
Table 3: Robustness Evaluation on the LIBERO-Plus Benchmark and Our Extended Perturbation Settings. The red-highlighted columns indicate the additional mid-episode perturbations introduced by us. All results are evaluated without perturbation-specific fine-tuning.
Figure 3: Training efficiency comparison between Ours and SFT using the π0 backbone.
Figure 4: Ablation studies using the π0 backbone trained for 3,000 steps on LIBERO-Goal and LIBERO-Spatial.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Prompt template for VLM-based annotation of failed trajectory segments. We use Qwen2.5-VL-72B-Instruct as the VLM annotator.
Figure 6: Annotated example of a successful trajectory.
Figure 7: Annotated example of a failed trajectory.
Figure 8: Visualization of the learned goal-conditioned value along execution trajectories. Local high-value frames are aligned with semantically meaningful task-progress key states, such as placing the bowl on the plate, opening the drawer, or moving the plate toward the stove. This qualitative evidence suggests that the learned value provides dense progress signals beyond final task success.
λSFT
λRL
LIBERO-Goal
0.1
0.9
79.6
0.2
0.8
83.2
0.3
0.7
86.8
0.4
0.6
92.3
0.5
0.5
89.2
0.6
0.4
82.5
Appendix
Table 4: Sensitivity analysis of the objective weights. The default setting is highlighted in gray.
Hyperparameter
LIBERO
MetaWorld
Optimizer
AdamW
AdamW
Policy learning rate
2.5×10−5
2.5×10−5
Value learning rate
1.55×10−4
1.55×10−4
Adam β1
0.9
0.9
Adam β2
0.95
0.95
Adam ϵ
1.0×10−8
1.0×10−8
Appendix
Table 5: Optimization hyperparameters.
Setting
LIBERO
MetaWorld
Value loss
TD
TD
Value training epochs
5
5
Steps per epoch
600
1000
Input images
2
1
Value feature
After VLM
After VLM
Pooling mode
Mean token
Mean token
Appendix
Table 6: Value model training settings.
Hyperparameter
LIBERO
MetaWorld
AWAC temperature λ
1.0
1.0
Maximum advantage weight
20.0
20.0
Normalize advantage
Yes
Yes
Advantage clipping
3.0
3.0
External reward model
No
No
Online critic model
No
No
Appendix
Table 7: Offline objective hyperparameters.
Method
Online Rollouts
Hardware
Task Suites
Wall-clock Time
RLinf-GRPO
Yes
8 × A100
1 LIBERO suite
∼ 175 hours
Ours
No
4 × RTX 4090 48GB
4 LIBERO suites
∼ 68 hours
Appendix
Table 8: Computational cost comparison on LIBERO. GRPO denotes the online RL training recipe in RLinf. The GRPO cost is reported for one LIBERO task suite, while our method is reported for all four LIBERO suites.
Vision-Language-Action (VLA) models remain brittle in long-horizon, contact-rich manipulation because success-only imitation provides little supervision for execution drift, while failed rollouts are often discarded. We introduce RePO-VLA, a recovery-driven policy optimization framework that assigns distinct roles to success, recovery, and failure trajectories. RePO-VLA first applies Recovery-Aware Initialization (RAI), slicing recovery segments and resetting history so corrective actions depend on the current adverse state rather than the preceding failure. It then learns a Progress-Aware Semantic Value Function (PAS-VF), aligning spatiotemporal trajectory features with instructions and successful references. The resulting labels salvage useful failure prefixes via reliability decay, while low-value labels mark drift and terminal breakdowns, teaching differences among nominal, failed, and corrective actions. The data engine turns adverse states into planner-generated or human-collected corrective rollouts, teaching recovery to the success manifold. Value-Conditioned Refinement (VCR) trains the policy to prefer high-progress actions. At deployment, a fixed high value (v=1.0) biases actions toward the learned success manifold without online failure detectors or heuristic retries. We introduce FRBench, with standardized error injection and recovery-focused evaluation. Across simulated and real-world bimanual tasks, RePO-VLA improves robustness, raising adversarial success from 20% to 75% on average and up to 80% in scaled real-world trials.
Weijia Liufu, Xiaoyu Guo, Ruiyi Chen +16
1Sun Yat-sen University · 2South China University of Technology · 3Peng Cheng Laboratory +3
Although pre-trained Vision-Language-Action (VLA) models exhibit impressive generalization in robotic manipulation, post-training remains crucial to ensure reliable performance during deployment. However, standard offline Supervised Fine-Tuning (SFT) suffers from distribution shifts and catastrophic forgetting of pre-trained capabilities, while online Reinforcement Learning (RL) struggles with sparse rewards and poor sample efficiency. In this paper, we propose On-Policy VLA Distillation (VLA-OPD), a framework bridging the efficiency of SFT with the robustness of RL. Instead of relying on sparse environmental rewards, VLA-OPD leverages an expert teacher to provide dense, token-level supervision on the student's self-generated trajectories. This enables active error correction on policy-induced states while preserving pre-trained general capabilities through gentle alignment. Crucially, we formulate VLA-OPD via a Reverse-KL objective. Unlike standard Forward-KL that induces mode-covering entropy explosion, or Hard-CE that causes premature entropy collapse, our bounded mode-seeking objective ensures stable policy learning by filtering out the teacher's epistemic uncertainty while maintaining action diversity. Experiments on LIBERO and RoboTwin2.0 benchmarks demonstrate that VLA-OPD significantly improves sample efficiency over RL and robustness over SFT, while effectively mitigating catastrophic forgetting during post-training.
Vision-language-action (VLA) policies provide strong priors for language-conditioned manipulation, but remain brittle in off-nominal states requiring targeted recovery. We propose ReCoVLA -- a failure-conditioned residual recovery framework that keeps a pretrained VLA policy frozen, uses an external vision-language model (VLM) to infer the failure mode and recovery stage, and compiles a structured reward from task-relevant components. Rather than using the VLM to generate actions or rewards directly, ReCoVLA uses it as a semantic reward selector: it predicts a recovery descriptor and reward mask for in-simulation residual-policy training, followed by zero-shot sim-to-real deployment of the trained recovery policies. This decouples high-level failure understanding from low-level corrective control to support different VLAs. Experiments across short-horizon, long-horizon, and contact-rich manipulation tasks show that ReCoVLA outperforms the tested baselines on average. In simulation, our reward compiler improves average success from 36.7% for the fine-tuned π0.5 baseline to 66.7%. In physical zero-shot sim-to-real experiments, ReCoVLA achieves the best average performance, with 61.7% success.
Haodi Hu, Chung-Ta Huang, Jing Liu +4
University of Southern California, USA · Mitsubishi Electric Research Laboratories (MERL), USA · Harvard University, USA