OGPO: Offline Goal-conditioned Policy Optimization for Recoverable Vision-Language-Action Models

Xule Gao; Xi Wang; Zehua Zang; Rui Wang; Changwen Zheng; Chuxiong Sun

cs.AIOct 9, 2026

OGPO: Offline Goal-conditioned Policy Optimization for Recoverable Vision-Language-Action Models

Authors: Xule Gao, Xi Wang, Zehua Zang, Rui Wang, Changwen Zheng, Chuxiong Sun

Organizations: National Key Laboratory of Space Integrated Information System · Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Wangxuan Institute of Computer Technology, Peking University

Abstract

Vision-language-action (VLA) models have shown strong promise in robotic manipulation, yet their post-training still predominantly relies on trajectory-level supervised imitation over robot demonstrations. While effective for learning feasible actions, this paradigm supervises policies to reproduce expert actions at each step, biasing them toward demonstration-specific action paths and providing limited guidance for recovering toward task-progress states once execution deviates from demonstrated trajectories. To address this limitation, we propose OGPO, an Offline Goal-conditioned Policy Optimization framework for recoverable VLA post-training. Instead of treating offline demonstrations merely as step-wise action labels, OGPO adaptively relabels trajectories with semantic key states and constructs goal-conditioned process rewards to optimize the policy for reaching these task-progress states. In this way, OGPO shifts VLA post-training from trajectory-level action imitation to key-state reachability learning, enabling policies to acquire a more robust ability to complete goal states from feasible off-demonstration states without requiring additional online interaction. Experiments on LIBERO and MetaWorld show that OGPO consistently improves VLA policy performance over standard supervised fine-tuning. Moreover, evaluations on LIBERO-Plus and our proposed LIBERO-ReAct, a mid-execution perturbation benchmark, further demonstrate that OGPO achieves stronger robustness and goal-directed recoverability under distribution shifts and recoverable off-demonstration states.

Figures & tables

Panel of Figure 1 — Figure 1: We design two representative perturbation cases to evaluate the robustness and goal-directed recoverability of offline-trained VLA policies under deviations from demonstration trajectories. In case (a), the gripper is forcibly opened after grasping the object, while position perturbation is applied when the end effector approaches the drawer. The results show that $\pi_{0.5}$ -SFT fails to complete the task, whereas $\pi_{0.5}$ -OGPO successfully recovers toward the goal state and finishes the task, demonstrating stronger recoverability under execution-time deviations.

Figure 2: Offline Goal-conditioned Policy Optimization( OGPO ) framework.

Model	Input	LIBERO
Model	Input	Spatial	Object	Goal	Long	Avg.
OpenVLA [ 17 ]	T+I	84.7	88.4	79.2	53.7	76.5
$\pi_{0}$ -fast [ 27 ]	T+W+P+I	96.4	96.8	88.6	60.2	85.5
$\pi_{0}$ [ 2 ]	T+W+P+I	96.8	98.8	95.8	85.2	94.2
SmolVLA [ 29 ]	T+W+P+I	93.0	94.0	91.0	77.0	88.8
WorldVLA [ 4 ]	T+I	85.6	89.0	82.6	59.0	79.1

Table 1: Performance comparison on the LIBERO benchmark. Policy input notation: T (third-view image), I (instruction), W (wrist image), P (proprioception). OGPO is built upon the few-shot SFT policy, with

\uparrow

indicating performance gains over the few-shot baseline.

Model	Input	MetaWorld-MT50
Model	Input	Easy	Medium	Hard	Very Hard	Avg.
Diffusion Policy [ 8 ]	T+C	23.1	10.7	1.9	6.1	10.5
TinyVLA [ 32 ]	T+P+C	77.6	21.5	11.4	15.8	31.6
SmolVLA [ 29 ]	T+P+C	87.1	51.8	70.0	64.0	68.2
$\pi_{0}$ -PPO (RLinf) [ 5 ]	T+P+I	91.1	81.8	78.3	92.0	85.8
$\pi_{0}$ -SFT	T+P+C	77.9	51.8	53.3	20.0	50.8

Table 2: Performance comparison on the MetaWorld-MT50 benchmark. Policy input notation: T (third-view image), P (proprioception), C (task conditioning). C denotes the task information used to distinguish MetaWorld tasks, and may be instantiated as a language instruction, task ID, or task embedding depending on the model.

Model	Perturbation Dimensions							Total
Model	Camera	Robot-Init	Noise	Layout	Action	Gripper	End-effector	Total
$\pi_{0}$ [ 2 ]	13.8	6.0	79.0	68.9	81.5	9.3	62.4	45.8
$\pi_{0}$ +fast [ 27 ]	65.1	21.6	74.4	68.8	75.2	19.8	63.8	55.5
OpenVLA [ 17 ]	0.8	3.5	15.2	28.5	19.7	24.3	31.5	17.6
OpenVLA-OFT [ 16 ]	56.4	31.9	75.8	74.2	85.2	84.4	87.6	70.8
$\pi_{0.5}$ -SFT [ 1 ]	62.8	69.8	69.5	85.4	86.6	76.8	86.0	76.7

Table 3: Robustness Evaluation on the LIBERO-Plus Benchmark and Our Extended Perturbation Settings. The red-highlighted columns indicate the additional mid-episode perturbations introduced by us. All results are evaluated without perturbation-specific fine-tuning.

Panel of Figure 3 — Figure 3: Training efficiency comparison between Ours and SFT using the $\pi_{0}$ backbone.

Panel of Figure 4 — Figure 4: Ablation studies using the $\pi_{0}$ backbone trained for 3,000 steps on LIBERO-Goal and LIBERO-Spatial.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Prompt template for VLM-based annotation of failed trajectory segments. We use Qwen2.5-VL-72B-Instruct as the VLM annotator.

Figure 6: Annotated example of a successful trajectory.

Figure 7: Annotated example of a failed trajectory.

Panel of Figure 8 — Figure 8: Visualization of the learned goal-conditioned value along execution trajectories. Local high-value frames are aligned with semantically meaningful task-progress key states, such as placing the bowl on the plate, opening the drawer, or moving the plate toward the stove. This qualitative evidence suggests that the learned value provides dense progress signals beyond final task success.

$\lambda_{\mathrm{SFT}}$	$\lambda_{\mathrm{RL}}$	LIBERO-Goal
0.1	0.9	79.6
0.2	0.8	83.2
0.3	0.7	86.8
0.4	0.6	92.3
0.5	0.5	89.2
0.6	0.4	82.5

Appendix

Table 4: Sensitivity analysis of the objective weights. The default setting is highlighted in gray.

Hyperparameter	LIBERO	MetaWorld
Optimizer	AdamW	AdamW
Policy learning rate	$2.5\times 10^{-5}$	$2.5\times 10^{-5}$
Value learning rate	$1.55\times 10^{-4}$	$1.55\times 10^{-4}$
Adam $\beta_{1}$	0.9	0.9
Adam $\beta_{2}$	0.95	0.95
Adam $\epsilon$	$1.0\times 10^{-8}$	$1.0\times 10^{-8}$

Appendix

Table 5: Optimization hyperparameters.

Setting	LIBERO	MetaWorld
Value loss	TD	TD
Value training epochs	5	5
Steps per epoch	600	1000
Input images	2	1
Value feature	After VLM	After VLM
Pooling mode	Mean token	Mean token

Appendix

Table 6: Value model training settings.

Hyperparameter	LIBERO	MetaWorld
AWAC temperature $\lambda$	1.0	1.0
Maximum advantage weight	20.0	20.0
Normalize advantage	Yes	Yes
Advantage clipping	3.0	3.0
External reward model	No	No
Online critic model	No	No

Appendix

Table 7: Offline objective hyperparameters.

Method	Online Rollouts	Hardware	Task Suites	Wall-clock Time
RLinf-GRPO	Yes	8 $\times$ A100	1 LIBERO suite	$\sim$ 175 hours
Ours	No	4 $\times$ RTX 4090 48GB	4 LIBERO suites	$\sim$ 68 hours

Appendix

Table 8: Computational cost comparison on LIBERO. GRPO denotes the online RL training recipe in RLinf. The GRPO cost is reported for one LIBERO task suite, while our method is reported for all four LIBERO suites.

Explore similar work

RePO-VLA: Recovery-Driven Policy Optimization for Vision-Language-Action Models
May 10, 2026Weijia Liufu, Xiaoyu Guo, Ruiyi Chen +16VLM Robustness Robot Failure Recovery
1Sun Yat-sen University · 2South China University of Technology · 3Peng Cheng Laboratory +3
VLA-OPD: Bridging Offline SFT and Online RL for Vision-Language-Action Models via On-Policy Distillation
Mar 27, 2026Zhide Zhong, Haodong Yan, Junfeng Li +3Vision-Language-Action Models On-Policy Distillation
HKUST (GZ)
ReCoVLA: VLM-Guided Reward Compilation for Failure Recovery in Vision-Language-Action Policies
Jun 8, 2026Haodi Hu, Chung-Ta Huang, Jing Liu +4Robot Failure Recovery Vision-Language-Action Models
University of Southern California, USA · Mitsubishi Electric Research Laboratories (MERL), USA · Harvard University, USA

May 10, 2026J/K move · Enter open · S save

RePO-VLA: Recovery-Driven Policy Optimization for Vision-Language-Action Models

Weijia Liufu, Xiaoyu Guo, Ruiyi Chen, Jingzhi Liu+15

1Sun Yat-sen University · 2South China University of Technology · 3Peng Cheng Laboratory+3

VLM Robustness Robot Failure Recovery Robotics Vision-Language-Action Models Robustness of VLA Models

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models remain brittle in long-horizon, contact-rich manipulation because success-only imitation provides little supervision for execution drift, while failed rollouts are often discarded. We introduce RePO-VLA, a recovery-driven policy optimization framework that assigns distinct roles to success, recovery, and failure trajectories. RePO-VLA first applies Recovery-Aware Initialization (RAI), slicing recovery segments and resetting history so corrective actions depend on the current adverse state rather than the preceding failure. It then learns a Progress-Aware Semantic Value Function (PAS-VF), aligning spatiotemporal trajectory features with instructions and successful references. The resulting labels salvage useful failure prefixes via reliability decay, while low-value labels mark drift and terminal breakdowns, teaching differences among nominal, failed, and corrective actions. The data engine turns adverse states into planner-generated or human-collected corrective rollouts, teaching recovery to the success manifold. Value-Conditioned Refinement (VCR) trains the policy to prefer high-progress actions. At deployment, a fixed high value (

v=1.0

) biases actions toward the learned success manifold without online failure detectors or heuristic retries. We introduce FRBench, with standardized error injection and recovery-focused evaluation. Across simulated and real-world bimanual tasks, RePO-VLA improves robustness, raising adversarial success from 20% to 75% on average and up to 80% in scaled real-world trials.

Abstract

Explore similar work

RePO-VLA: Recovery-Driven Policy Optimization for Vision-Language-Action Models

VLA-OPD: Bridging Offline SFT and Online RL for Vision-Language-Action Models via On-Policy Distillation

ReCoVLA: VLM-Guided Reward Compilation for Failure Recovery in Vision-Language-Action Policies