Trust-Calibrated VLA Policy Post-Training with Evolving Imagination

Wenzhan Li; Yiran Qin; Heng Zhou; Huirui Wang; Yulan Guo; Ruimao Zhang

cs.ROOct 12, 2026

Trust-Calibrated VLA Policy Post-Training with Evolving Imagination

Authors: Wenzhan Li, Yiran Qin, Heng Zhou, Huirui Wang, Yulan Guo, Ruimao Zhang

Organizations: Sun Yat-sen University · Shenzhen Loop Area Institute · The Chinese University of Hong Kong, Shenzhen · Shanghai AI Laboratory ·

Abstract

Vision-Language-Action (VLA) models are often pretrained via imitation learning, making reinforcement learning (RL) post-training important when deployment moves beyond demonstration coverage. Existing refinement paradigms expose a reliability--efficiency tension: model-free refinement avoids simulator bias but is interaction-intensive, whereas model-based refinement can exploit unreliable imagination even as its simulator adapts. We propose Model-Evolving Reinforcement Learning (MERL), a trust-calibrated framework that couples policy optimization with an online-updated simulator comprising a world model and a reward proxy head. Across stages, simulator reliability controls the imagined-data ratio and rollout horizon; within rollouts, chunk-level trust controls replay selection and optimization weight. For recursive imagination without exact grounded continuations, a residual predictor estimates trust without querying future ground truth. On perturbed LIBERO-PRO tasks, MERL reaches 79.7% average success, improves high-threshold learning speed by 10.9% over the matched model-free control, and outperforms same-protocol external baselines across three training seeds. Fixed-data adaptation from 50 demonstrations per task achieves 80.0% average success on two physical-robot tasks.

Figures & tables

Figure 1: Paradigm evolution toward trust calibration. The columns illustrate design patterns represented by our matched refinement controls, from imitation and model-free RL to static and online-updated model-based RL. MeRL uses simulator reliability to schedule, select, and weight rollouts before the subsequent mixed real–imagined policy optimization.

Figure 2: Overview of the MeRL pipeline. At each online stage $s$ , the policy collects grounded rollouts and updates an evolving simulator composed of a world model $\mathcal{W}_{\phi}$ and a reward proxy head $\mathcal{Q}_{\psi}$ . Calibration reuses stored grounded continuations with the same anchors and recorded actions; stage-level trust maps their measured errors to $\rho_{s}$ and $h_{s}$ . A residual predictor refreshed after simulator update supplies no-oracle scores for unpaired recursive chunks and is frozen during imagination and policy optimization. Chunk-level trust then evaluates generated imagined chunks, assigns replay probabilities $p_{s,i}$ and optimization weights $w_{s,i}$ , and combines trusted imagined chunks with grounded rollouts for mixed policy optimization.

	Suite SR (%) $\uparrow$				Overall
Method	Spatial	Object	Goal	Long	Avg. SR $\uparrow$	AUC $\uparrow$	S2T-H $\downarrow$
SFT	38.6	42.4	23.8	15.9	30.2	–	–
MFRL	80.4	83.1	79.4	66.0	77.2	67.9	103.4
Static-MBRL	76.8	79.4	76.3	60.7	73.3	59.8	168.9
Static-MBRL+Trust	78.2	80.8	78.4	62.8	75.1	63.5	145.8
Online-Updated MBRL	79.5	82.0	77.0	66.1	76.2	65.4	139.6

Table 1: Matched-control results on LIBERO-PRO. SR and AUC are on a percentage scale; S2T-H counts refinement stages. See Tab. 2 for three-seed comparisons.

Method	Paradigm	Avg. SR $\uparrow$	AUC $\uparrow$	S2T-H $\downarrow$
SimpleVLA-RL	Model-free	$77.1\pm 1.1$	$67.8\pm 1.0$	$104.1\pm 7.0$
WMPO †	Static model-based	$75.4\pm 1.2$	$63.2\pm 1.3$	$149.6\pm 10.2$
WoVR †	Online-updated	$77.8\pm 0.9$	$68.4\pm 0.9$	$101.3\pm 6.2$
MeRL	Trust-calibrated	$\mathbf{79.7\pm 0.7}$	$\mathbf{70.6\pm 0.8}$	$\mathbf{92.1\pm 4.8}$

Table 2: Same-protocol external comparisons on LIBERO-PRO: mean

\pm

standard deviation over three training seeds.

\dagger

denotes a port changing benchmark/model assets while retaining the defining algorithm; each WoVR seed has an independent simulator.

Figure 3: Learning dynamics, two-level trust, and real-robot task setups. Left: SR and S2T on Goal and Long. Right: trust diagnostics and AgileX PiPER X setups for Block-to-Box and Lid-to-Pot. All training axes count outer refinement stages. Panel (e) displays the horizon in action-chunk units $h_{s}/C\in[1,4]$ , with $C=8$ ; panel (f) displays replay priority and trust weight against chunk error.

Method	Block-to-Box	Lid-to-Pot	Avg. SR (%) [95% CI] $\uparrow$
SFT	8/20	6/20	$35.0\,[22.1,50.5]$
MFRL	12/20	10/20	$55.0\,[39.8,69.3]$
Online-Updated MBRL	13/20	11/20	$60.0\,[44.6,73.7]$
MeRL -Visual	16/20	15/20	$77.5\,[62.5,87.7]$
MeRL (full)	17/20	15/20	$\mathbf{80.0\,[65.2,89.5]}$

Table 3: AgileX PiPER X closed-loop results: 20 trials/task after offline refinement from 50 demonstrations/task without extra physical training rollouts. Average-SR CIs are 95% intervals.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Comparison	Metric	Difference [95% CI]
MeRL – WoVR	$\Delta$ AUC $\uparrow$	$2.2\,[1.2,3.3]$
MeRL – WoVR	$\Delta$ S2T-H $\downarrow$	$-9.2\,[-15.4,-3.1]$
MeRL – matched Online-Updated MBRL	$\Delta$ AUC $\uparrow$	$5.0\,[3.8,6.2]$

Appendix

Table 4: Paired 95% CIs over three seeds. AUC: points; S2T-H: refinement stages.

Variant	Avg. SR $\uparrow$	AUC $\uparrow$	S2T-H $\downarrow$
MFRL	77.2	67.9	103.4
Static-MBRL	73.3	59.8	168.9
Online-Updated MBRL	76.2	65.4	139.6
Online-Updated+StageTrust	77.9	68.4	113.2
Online-Updated+ChunkTrust	77.5	67.6	121.5
MeRL (Ours)	79.7	70.6	92.1

Appendix

Table 5: Component ablation on simulator evolution and two-level trust. Values preserve the original matched-control evaluation; three-seed external comparisons are reported separately in Tab. 2 .

Simulator	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FVD $\downarrow$	Proxy MAE $\downarrow$	Proxy MSE $\downarrow$
Frozen simulator	22.4	0.684	0.238	286.0	0.183	0.057
Evolving simulator	25.9	0.759	0.171	198.0	0.126	0.031

Appendix

Table 6: Simulator diagnostics on held-out policy-induced trajectories. The frozen simulator corresponds to Static-MBRL, while the evolving simulator is updated online during MeRL training.

Figure 4: Qualitative comparison of simulator predictions. Rows show ground-truth future frames, predictions from the frozen simulator, and predictions from the evolving simulator under the same rollout condition. The evolving simulator produces visual continuations that remain closer to the ground truth over the prediction horizon.

Estimator	Pearson $r$ $\uparrow$	Rank-AUC $\uparrow$	ECE $\downarrow$	Top30 Prec. $\uparrow$	AUC $\uparrow$
Random / uniform	0.02	0.50	0.176	0.31	66.1
WM latent error only	0.55	0.72	0.114	0.66	67.0
Proxy residual only	0.49	0.68	0.126	0.61	66.6
Residual predictor	0.71	0.82	0.063	0.79	69.4
Combined MeRL estimator	0.79	0.87	0.044	0.86	70.6

Appendix

Table 7: Trust-estimator validity and downstream policy performance.

Diagnostic link	Statistic	Value
Simulator error vs. imagined ratio $\rho_{s}$	Spearman	$-0.967$
Simulator error vs. rollout horizon $h_{s}$	Spearman	$-0.963$
Chunk error vs. replay probability $p_{s,i}$	Spearman	$-0.750$
Chunk error vs. trust weight $w_{s,i}$	Spearman	$-0.950$
High-error quartile replay share / uniform share	Ratio	$0.180$
KL standard deviation reduction vs. Online-Updated MBRL	Relative	$0.430$

Appendix

Table 8: Trust-chain diagnostics. Negative correlations indicate that lower estimated error leads to more imagination, longer rollout horizons, higher replay probability, or larger optimization weight.

Setting	Pearson $r$ $\uparrow$	Rank-AUC $\uparrow$	ECE $\downarrow$	Top30 Prec. $\uparrow$
Residual predictor	0.71	0.82	0.063	0.79
Combined MeRL	0.79	0.87	0.044	0.86
Depth 2 ( $N=600$ )	0.72 [0.68,0.76]	0.82 [0.79,0.85]	0.061 [0.052,0.070]	0.80 [0.76,0.84]
Depths 3–4 ( $N=900$ )	0.64 [0.60,0.68]	0.77 [0.74,0.80]	0.074 [0.065,0.083]	0.74 [0.70,0.78]
All 2–4 ( $N=1500$ )	0.68 [0.65,0.71]	0.80 [0.78,0.82]	0.067 [0.060,0.074]	0.77 [0.74,0.80]

Appendix

Table 9: No-oracle trust estimation with matched future observations withheld from scoring. The first two rows give estimator reference results; subsequent rows report recursive-depth estimates with the corresponding trajectory-bootstrap 95% confidence intervals.

Trust bin	Top	Mid	Bottom
Policy AUC $\uparrow$	$69.9\pm 0.8$	$68.3\pm 0.9$	$66.7\pm 1.0$

Appendix

Table 10: Matched-count trust-bin policy AUC: mean

\pm

SD over three seeds.

Method	Avg. SR $\uparrow$	Worst SR $\uparrow$	Std. $\downarrow$
MFRL	77.2	66.0	6.61
Static-MBRL	73.3	60.7	7.36
Online-Updated MBRL	76.2	66.1	6.08
MeRL (Ours)	79.7	68.8	6.37

Appendix

Table 11: Cross-suite robustness across perturbed LIBERO-PRO suites. Avg. SR measures overall performance, Worst SR measures the weakest suite, and Std. measures sensitivity across suites.

Panel of Figure 5 — Figure 5: AgileX PiPER X task setups for fixed-data adaptation and closed-loop physical evaluation.

Figure 6: MERL real-robot deployment and simulator visualization on AgileX PiPER X. From top to bottom, rows show real Block-to-Box rollout, simulator-generated Block-to-Box continuation, real Lid-to-Pot rollout, and simulator-generated Lid-to-Pot continuation.

RL Post-Training

Reinforcement learning post-training adapts pretrained models with reward or feedback signals to improve reasoning, alignment, and interactive behavior.

See all 425 papers

Trust Your Instincts: Confidence-Driven Test-Time RL for Vision-Language-Action Models
Jun 29, 2026Siyao Chen, Jiakang Yuan, Jiaxin Wang +1Intrinsic Reward Methods Reinforcement Learning
College of Future Information Technology, Fudan University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China · MoShen Intelligence, Shanghai, China
WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RL
Feb 15, 2026Zhennan Jiang, Shangqing Zhou, Yutong Jiang +11World Models for Robotics World Models
University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Zhongguancun Academy +2
VLRLA: Deployment-Time Adaptation of Vision-Language-Action Models through In-Context Reinforcement Learning
Oct 14, 2026Diyuan Shi, Yiqi Tang, Hongyin Zhang +2Test-Time Adaptation of VLA Models In-Context RL
Zhejiang University · Westlake University

Jun 29, 2026J/K move · Enter open · S save

Trust Your Instincts: Confidence-Driven Test-Time RL for Vision-Language-Action Models

Siyao Chen, Jiakang Yuan, Jiaxin Wang, Tao Chen

College of Future Information Technology, Fudan University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China · MoShen Intelligence, Shanghai, China

Intrinsic Reward Methods Reinforcement Learning Test-Time RL Robotic Manipulation Vision-Language-Action Models Test-Time Adaptation of VLA Models

Open paper AskPDF arXiv

Reinforcement learning (RL) has become indispensable for pushing Vision-Language-Action Models (VLAs) beyond static imitation learning. However, existing RL methods typically require external environmental feedback, relying on predefined success signals to guide policy updates. In this work, we show that VLA models possess useful internal evaluative capabilities: in discrete-action VLAs, trajectories with higher generation confidence are significantly more likely to succeed. Based on this observation, we introduce T^2VLA (Test-time VLA), an architecture-agnostic test-time RL framework that enables VLA models to achieve self-bootstrapping policy improvement. Instead of relying on external rewards, T^2VLA leverages trajectory-level similarity to high-confidence expert demonstrations as an intrinsic reward signal. In addition, we propose a Confidence-Driven Dual Expert Bootstrapping mechanism, which dynamically balances a Local Pseudo-Expert for exploration and a Global Expert Pool for training stability. Extensive experiments on the LIBERO and RoboTwin benchmarks show that T^2VLA consistently outperforms supervised baselines and approaches oracle RL performance with ground-truth rewards, achieving effective improvement without external reward feedback. Furthermore, T^2VLA adapts to distinct VLA paradigms, including both OpenVLA-OFT and the pi series.

Abstract

RL Post-Training

Explore similar work

Trust Your Instincts: Confidence-Driven Test-Time RL for Vision-Language-Action Models

WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RL

VLRLA: Deployment-Time Adaptation of Vision-Language-Action Models through In-Context Reinforcement Learning