Advantage-guided reinforcement learning provides a practical way to post-train vision-language-action (VLA) policies using limited robot data. However, its performance depends on several coupled choices, including how critic-derived advantages are constructed, calibrated, and used for policy training. Existing recipes often combine these choices into a single end-to-end procedure, making their individual effects difficult to identify. In this work, we dissect advantage-guided VLA post-training through a controlled empirical study that separates these design choices while accounting for their distinct estimands. We develop stage-specific offline evaluation methods to screen alternative choices efficiently, without requiring extensive real-robot policy evaluations for every possible combination. The staged evaluation identifies a modular recipe that combines temporal-difference advantage construction, group-wise calibration, and continuous advantage weighting. Across four real-world bimanual tasks, the resulting recipe improves mean task progress and success over the SFT initialization by 0.42 and 0.63, respectively. Moreover, the proposed evaluation diagnostics show an overall alignment with downstream real-world performance, supporting their use for interpreting empirical outcomes and selecting advantage-guided post-training designs in practice.
Figures & tables
Fig. 1: Representative scenes from the four real-world manipulation tasks in our evaluation. (a) Pack Phone Easy, (b) Pack Phone Hard, (c) Pack Box, and (d) Refill Printer Paper.
Fig. 2: Overview of our three-stage evaluation of advantage-guided VLA post-training. Stage I compares different advantage construction schemes, Stage II varies the grouping rule in advantage calibration, and Stage III examines three sample-weighting strategies in real-robot policy training.
Fig. 3: Representative intervention and progress pairs across four real-world tasks. Each strip contains three frames in temporal order. The intervention examples (left) show failed policy behavior ( red ) followed by recovery after human takeover ( green ), whereas the progress examples (right) contrast stalled behavior ( red ) with advancing behavior ( green ) within the same sub-task. Both diagnostics assess whether the green examples receive higher estimated advantages than their red counterparts. Images use the base-camera view except for the PhoneH and Printer intervention examples, which use the left- and right-wrist views, respectively.
Fig. 4: Advantage-construction diagnostics computed across four real-world tasks. The top row reports intervention ordering Rint and the bottom row reports progress ordering Rprog . For each task, the intervention and progress diagnostics use approximately 200 and 400 paired comparisons, respectively, after applying their diagnostic-specific filtering rules. Point markers show the ordering diagnostics (higher is better), and horizontal bars show 95% confidence intervals. The dashed line marks chance ordering at 0.5 . Bold numeric labels with heavier marker outlines identify the within-panel maxima.
Calibration
PhoneE
PhoneH
Box
Printer
Mean
Raw
0.072
0.063
0.106
0.127
0.092
Global
0.057
0.051
0.090
0.120
0.080
Temporal
0.071
0.023
0.053
0.028
0.044
Value
0.053
0.027
0.013
0.010
0.026
TABLE I: Stage-II offline calibration results measured by η2 , the proportion of total advantage variance explained by sub-task identity. Lower η2 indicates less sub-task-dependent variation; Mean is the average across the four tasks.
Condition
Task progress
Task success
PhoneE
PhoneH
Box
Printer
Mean
PhoneE
PhoneH
Box
Printer
Mean
SFT Init
0.50
0.32
0.69
0.24
0.44
0.30
0.00
0.15
0.00
0.11
DAgger
0.65
0.62
0.91
0.66
0.71
0.35
0.30
0.70
0.45
0.45
Weight
0.88
0.81
0.95
0.79
0.86
0.80
0.70
0.80
0.65
0.74
Filter
0.78
0.69
0.91
0.60
0.75
0.70
0.40
0.55
0.35
0.50
TABLE II: Real-world policy evaluation for RQ3. Values in the task columns are averages over 20 real-robot trials per condition; Mean is the average across the four tasks. Higher is better; bold denotes the highest in each column.
Fig. 5: Ablations of advantage construction and calibration under fixed Weight utilization, with corresponding offline diagnostics overlaid (higher is better). The first four conditions vary Stage-I construction and the last three vary Stage-II calibration; the shared fourth condition is the reference configuration reported in Table II .
Reinforcement learning post-training adapts pretrained models with reward or feedback signals to improve reasoning, alignment, and interactive behavior.
Vision-Language-Action (VLA) models that couple pretrained Vision-Language Models (VLMs) with continuous action experts have achieved strong manipulation performance, yet generalization to out-of-distribution (OOD) language instructions remains poor. A known challenge is the structural imbalance in VLA data, where language is far less diverse than visual and action content, making policies prone to visual shortcuts. While discrete-action methods mitigate this through vision-language co-training, continuous action experts lack such protection: they start from random initialization and learn entirely from imbalanced data, producing noisy gradients that corrupt the VLM and fail to exploit its language capability. We address this from a Bayesian perspective, factorizing the policy into a language-agnostic Vision-Action (VA) prior and a language-conditioned VLA likelihood, and propose APT, a two-stage training method emphasizing Action expert PreTraining. In Stage 1, the action expert is pretrained as a VA prior on vision-action pairs from a frozen VLM, bypassing the language imbalance. In Stage 2, language tokens are injected through a gated fusion mechanism that integrates VLM features while preserving the learned visuomotor prior. APT applies to mainstream VLA architectures, including the π and GR00T-style architectures. Comprehensive experiments validate that APT achieves consistent gains on unseen instructions and compositional tasks. Project Page: https://xukechun.github.io/papers/APT/
Kechun Xu, Zhenjie Zhu, Anzhe Chen +2
Zhejiang University · Zhejiang Humanoid Robot Innovation Center
Although pre-trained Vision-Language-Action (VLA) models exhibit impressive generalization in robotic manipulation, post-training remains crucial to ensure reliable performance during deployment. However, standard offline Supervised Fine-Tuning (SFT) suffers from distribution shifts and catastrophic forgetting of pre-trained capabilities, while online Reinforcement Learning (RL) struggles with sparse rewards and poor sample efficiency. In this paper, we propose On-Policy VLA Distillation (VLA-OPD), a framework bridging the efficiency of SFT with the robustness of RL. Instead of relying on sparse environmental rewards, VLA-OPD leverages an expert teacher to provide dense, token-level supervision on the student's self-generated trajectories. This enables active error correction on policy-induced states while preserving pre-trained general capabilities through gentle alignment. Crucially, we formulate VLA-OPD via a Reverse-KL objective. Unlike standard Forward-KL that induces mode-covering entropy explosion, or Hard-CE that causes premature entropy collapse, our bounded mode-seeking objective ensures stable policy learning by filtering out the teacher's epistemic uncertainty while maintaining action diversity. Experiments on LIBERO and RoboTwin2.0 benchmarks demonstrate that VLA-OPD significantly improves sample efficiency over RL and robustness over SFT, while effectively mitigating catastrophic forgetting during post-training.
Online post-training of vision-language-action (VLA) models requires efficient use of robot interaction and reliable policy improvement from continually collected experience. We propose asynchronous Replay-Anchored Policy improvement (RAPolicy), a framework that performs rollout and learning concurrently while grounding both critic and actor updates in replayed behavior. The critic learns chunk-level values from recorded actions and constructs Bellman targets without predicting next actions, reducing computation and dependence on action-value estimates outside replay coverage. The one-step flow actor reuses the initial noise stored during rollout and learns through advantage-weighted conditional likelihood, directly supervising the action mapping used for execution. We evaluate RAPolicy across four single-task settings and one joint five-task setting in the real world, with online training budgets of only 1--2 hours. Starting from policies fine-tuned on just 10 demonstrations per task, RAPolicy rapidly adapts to new single tasks and achieves an average 86.3% success rate. In the joint multi-task setting, RAPolicy improves overall success rate from 52% to 88% while preserving performance on already reliable tasks and improving weaker capabilities. Overall, RAPolicy substantially outperforms the baselines in aggregate task success while requiring fewer human interventions, demonstrating stable policy improvement and high online training efficiency. Project page: https://flyfaerss.github.io/RAPolicy.
Jiarui Yang, Jiajin Zhang, Bin Zhu +2
Institute of Trustworthy Embodied AI, Fudan University · Shanghai Key Laboratory of Multimodal Embodied AI · Singapore Management University