Dissecting Advantage-Guided Post-Training for Vision-Language-Action Policies

Jiahang Cao; Hanye Zhao; Hang Lai; Shenyu Zhang; Xiaoshen Han; Xinghang Li; Futeng Liu; Wanli Peng

cs.ROSep 23, 2026

Dissecting Advantage-Guided Post-Training for Vision-Language-Action Policies

Authors: Jiahang Cao, Hanye Zhao, Hang Lai, Shenyu Zhang, Xiaoshen Han, Xinghang Li, Futeng Liu, Wanli Peng,

Organizations: School of Computer Science, Shanghai Jiao Tong University. · Xiaomi Robotics.

Abstract

Advantage-guided reinforcement learning provides a practical way to post-train vision-language-action (VLA) policies using limited robot data. However, its performance depends on several coupled choices, including how critic-derived advantages are constructed, calibrated, and used for policy training. Existing recipes often combine these choices into a single end-to-end procedure, making their individual effects difficult to identify. In this work, we dissect advantage-guided VLA post-training through a controlled empirical study that separates these design choices while accounting for their distinct estimands. We develop stage-specific offline evaluation methods to screen alternative choices efficiently, without requiring extensive real-robot policy evaluations for every possible combination. The staged evaluation identifies a modular recipe that combines temporal-difference advantage construction, group-wise calibration, and continuous advantage weighting. Across four real-world bimanual tasks, the resulting recipe improves mean task progress and success over the SFT initialization by 0.42 and 0.63, respectively. Moreover, the proposed evaluation diagnostics show an overall alignment with downstream real-world performance, supporting their use for interpreting empirical outcomes and selecting advantage-guided post-training designs in practice.

Figures & tables

Fig. 1: Representative scenes from the four real-world manipulation tasks in our evaluation. (a) Pack Phone Easy, (b) Pack Phone Hard, (c) Pack Box, and (d) Refill Printer Paper.

Fig. 2: Overview of our three-stage evaluation of advantage-guided VLA post-training. Stage I compares different advantage construction schemes, Stage II varies the grouping rule in advantage calibration, and Stage III examines three sample-weighting strategies in real-robot policy training.

Fig. 3: Representative intervention and progress pairs across four real-world tasks. Each strip contains three frames in temporal order. The intervention examples (left) show failed policy behavior ( red ) followed by recovery after human takeover ( green ), whereas the progress examples (right) contrast stalled behavior ( red ) with advancing behavior ( green ) within the same sub-task. Both diagnostics assess whether the green examples receive higher estimated advantages than their red counterparts. Images use the base-camera view except for the PhoneH and Printer intervention examples, which use the left- and right-wrist views, respectively.

Fig. 4: Advantage-construction diagnostics computed across four real-world tasks. The top row reports intervention ordering

R_{\mathrm{int}}

and the bottom row reports progress ordering

R_{\mathrm{prog}}

. For each task, the intervention and progress diagnostics use approximately

200

and

400

paired comparisons, respectively, after applying their diagnostic-specific filtering rules. Point markers show the ordering diagnostics (higher is better), and horizontal bars show

95\%

confidence intervals. The dashed line marks chance ordering at

0.5

. Bold numeric labels with heavier marker outlines identify the within-panel maxima.

Calibration	PhoneE	PhoneH	Box	Printer	Mean
Raw	0.072	0.063	0.106	0.127	0.092
Global	0.057	0.051	0.090	0.120	0.080
Temporal	0.071	0.023	0.053	0.028	0.044
Value	0.053	0.027	0.013	0.010	0.026

TABLE I: Stage-II offline calibration results measured by

\eta^{2}

, the proportion of total advantage variance explained by sub-task identity. Lower

\eta^{2}

indicates less sub-task-dependent variation; Mean is the average across the four tasks.

Condition	Task progress					Task success
	PhoneE	PhoneH	Box	Printer	Mean	PhoneE	PhoneH	Box	Printer	Mean
SFT Init	0.50	0.32	0.69	0.24	0.44	0.30	0.00	0.15	0.00	0.11
DAgger	0.65	0.62	0.91	0.66	0.71	0.35	0.30	0.70	0.45	0.45
Weight	0.88	0.81	0.95	0.79	0.86	0.80	0.70	0.80	0.65	0.74
Filter	0.78	0.69	0.91	0.60	0.75	0.70	0.40	0.55	0.35	0.50

TABLE II: Real-world policy evaluation for RQ3. Values in the task columns are averages over

20

real-robot trials per condition; Mean is the average across the four tasks. Higher is better; bold denotes the highest in each column.

Fig. 5: Ablations of advantage construction and calibration under fixed Weight utilization, with corresponding offline diagnostics overlaid (higher is better). The first four conditions vary Stage-I construction and the last three vary Stage-II calibration; the shared fourth condition is the reference configuration reported in Table II .

RL Post-Training

Reinforcement learning post-training adapts pretrained models with reward or feedback signals to improve reasoning, alignment, and interactive behavior.

See all 426 papers

APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies
Jun 10, 2026Kechun Xu, Zhenjie Zhu, Anzhe Chen +2Visuomotor Policy Learning Vision-Language-Action Models
Zhejiang University · Zhejiang Humanoid Robot Innovation Center
VLA-OPD: Bridging Offline SFT and Online RL for Vision-Language-Action Models via On-Policy Distillation
Mar 27, 2026Zhide Zhong, Haodong Yan, Junfeng Li +3Vision-Language-Action Models On-Policy Distillation
HKUST (GZ)
Stable and Efficient Real-World Online VLA Post-Training via Asynchronous Replay-Anchored Policy Improvement
Sep 19, 2026Jiarui Yang, Jiajin Zhang, Bin Zhu +2Robot Policy Learning RL Post-Training
Institute of Trustworthy Embodied AI, Fudan University · Shanghai Key Laboratory of Multimodal Embodied AI · Singapore Management University

Jun 10, 2026J/K move · Enter open · S save

APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies

Kechun Xu, Zhenjie Zhu, Anzhe Chen, Rong Xiong+1

Zhejiang University · Zhejiang Humanoid Robot Innovation Center

Visuomotor Policy Learning Vision-Language-Action Models Instruction Generalization in VLA Models VLA Model Training

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models that couple pretrained Vision-Language Models (VLMs) with continuous action experts have achieved strong manipulation performance, yet generalization to out-of-distribution (OOD) language instructions remains poor. A known challenge is the structural imbalance in VLA data, where language is far less diverse than visual and action content, making policies prone to visual shortcuts. While discrete-action methods mitigate this through vision-language co-training, continuous action experts lack such protection: they start from random initialization and learn entirely from imbalanced data, producing noisy gradients that corrupt the VLM and fail to exploit its language capability. We address this from a Bayesian perspective, factorizing the policy into a language-agnostic Vision-Action (VA) prior and a language-conditioned VLA likelihood, and propose APT, a two-stage training method emphasizing Action expert PreTraining. In Stage 1, the action expert is pretrained as a VA prior on vision-action pairs from a frozen VLM, bypassing the language imbalance. In Stage 2, language tokens are injected through a gated fusion mechanism that integrates VLM features while preserving the learned visuomotor prior. APT applies to mainstream VLA architectures, including the

π

and GR00T-style architectures. Comprehensive experiments validate that APT achieves consistent gains on unseen instructions and compositional tasks. Project Page: https://xukechun.github.io/papers/APT/

Abstract

RL Post-Training

Explore similar work

APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies

VLA-OPD: Bridging Offline SFT and Online RL for Vision-Language-Action Models via On-Policy Distillation

Stable and Efficient Real-World Online VLA Post-Training via Asynchronous Replay-Anchored Policy Improvement