Trusting Imagined Futures: Auditing Imagined Rewards for VLA Post-Training

Zicheng Kong; Zhenbo Xu; Haoyan Zhang; Jinghan Yang; Zhenzhen Li; Yiwei Ru; Liuyu Xiang; WanTing Xu

cs.RONov 3, 2026

Trusting Imagined Futures: Auditing Imagined Rewards for VLA Post-Training

Authors: Zicheng Kong, Zhenbo Xu, Haoyan Zhang, Jinghan Yang, Zhenzhen Li, Yiwei Ru, Liuyu Xiang, WanTing Xu,

Organizations: Beijing University of Posts and Telecommunications, Beijing, P.R. China · ShiFang Technology Inc., Hangzhou, P.R. China · Institute of Automation, Chinese Academy of Sciences, Beijing, P.R. China · Wuhan Electric Power Technical College, Wuhan, P.R. China

Abstract

Video world models enable candidate actions to be evaluated through imagined outcomes, but high predicted rewards can favor trajectories with poor execution quality. We introduce Trusting Imagined Futures (TIF), an offline supervision selection method for vision-language-action (VLA) post-training that audits predicted rewards using execution feedback. A multidimensional reward model predicts a utility score for candidate ranking and five dimensions of execution quality. An auditor uses reward predictions and uncertainty to identify overestimation patterns learned from paired imagined videos and real executions. TIF accepts the candidate with the highest predicted utility when its advantage over the existing trajectory exceeds a margin and its audit risk passes a threshold; otherwise, it retains the existing supervision. The retained real trajectories then supervise behavior cloning. Across four LIBERO suites and three training seeds, TIF achieves a mean success rate of 76.25%, improving over training on the existing trajectories by 3.00 percentage points with the same trajectory count and training budget. Adding audit gating yields 1.47 percentage points with the reward model and advantage margin held fixed. The gains persist against advantage ranking and logistic regression risk selection at matched replacement counts. On recorded videos from three real robot tasks, TIF increases clean completion among selected trajectories from 58.33% to 70.00% across 60 candidate pools.

Figures & tables

Figure 1: A candidate may receive a higher predicted utility than Source yet exhibit poorer execution quality. TIF audits evaluation reliability before accepting a replacement.

Figure 2: Overview of TIF. Matched executions supervise reward learning and reliability auditing. During curation, predicted utility and audit risk determine whether to replace Source. The selected real trajectories supervise VLA post-training across all valid steps.

Model	Pairwise $\uparrow$	Top-1 $\uparrow$	Utility $\uparrow$
Scalar	56.88	30.99	.4574
2D	57.45	32.55	.4598
5D	57.79	33.07	.4603
5D w/o pairwise	57.42	30.99	.4598
5D shuffled M/S/E	56.80	29.17	.4550

Table 1: Candidate ranking with different reward supervision. Pairwise and Top-1 accuracies are reported in %. Top-1 measures whether the selected candidate has maximal execution reference utility within its candidate set. Utility is the mean execution reference utility of selected candidates. The final two rows independently ablate 5D.

(a) Default selection rules
Method	Replaced	Better	Worse	Utility $\uparrow$
Source	0	0	0	.4444
Reward-only	310	91	219	.4603
Reward+Margin	186	65	121	.4587
TIF	111	52	59	.4602

Table 2: Supervision selection with reliability auditing. Better/Worse count replacements with higher/lower execution reference utility than Source. Utility averages the retained trajectories over all 384 states. (a) Default selection rules. (b) Each rule accepts 111 of the same 310 non-Source reward winners without an advantage margin and retains Source elsewhere. Random reports expected values.

Method	Spatial	Object	Goal	Long	Average
Original OpenVLA	74.80	86.20	77.20	37.80	69.00
Source	$80.20\pm 0.92$	$86.07\pm 1.03$	$\mathbf{83.73\pm 0.31}$	$43.00\pm 1.06$	$73.25\pm 0.68$
Greedy	$78.00\pm 1.11$	$84.53\pm 0.76$	$81.67\pm 0.42$	$41.93\pm 1.33$	$71.53\pm 0.51$
Random	$79.07\pm 1.21$	$85.33\pm 0.42$	$82.67\pm 0.42$	$41.93\pm 1.21$	$72.25\pm 0.48$
Reward-only	$81.60\pm 0.72$	$87.40\pm 0.72$	$82.93\pm 0.64$	$45.00\pm 1.20$	$74.23\pm 0.40$
Reward+Margin	$82.13\pm 1.01$	$88.27\pm 0.50$	$83.33\pm 0.50$	$45.40\pm 1.39$	$74.78\pm 0.63$

Table 3: Policy success on LIBERO (%). Post-training methods share the same policy initialization, trajectory count, and training budget, using all valid time steps. Values are means

\pm

sample standard deviations across three training seeds. Original OpenVLA is evaluated before post-training. Bold indicates the highest mean in each column.

(a) Audit training target
Target	Offline selection		Policy success $\uparrow$
	Worse $\downarrow$	Utility $\uparrow$
Signed error regression	56	.4624	$75.60\pm 0.46$
Below-Source classification	50	.4629	$75.85\pm 0.59$
High-score mismatch classification	53	.4636	$\mathbf{76.25\pm 0.58}$

Table 4: Audit targets and supervision ablations. Policy success is averaged over four suites (%, mean

\pm

sample SD across three training seeds). (a) Auditors share inputs, training data, and training budget. Offline selection accepts 111 of 310 test proposals without a margin; Worse counts replacements below Source, and Utility averages all 384 test states including Source fallback. Policy training separately uses 230 replacements from training candidates that pass the margin, with counts matched by suite. (b) Each reward model has its own auditor trained under the same protocol. (c) Selected trajectories and training budgets are fixed within each selection method.

Selection rule	Success $\uparrow$
Random-K	$74.15\pm 0.63$
Advantage Top-K	$75.55\pm 0.50$
Logistic risk-K	$75.65\pm 0.52$
Low-entropy-K	$75.03\pm 0.51$
Consistency-K	$75.60\pm 0.52$
TIF	$\mathbf{76.25\pm 0.58}$

Table 5: Policy success at matched replacement counts. All rules select from the same 348 training candidates that pass the advantage margin, accepting 57/61/54/58 replacements in Spatial/Object/Goal/Long. Each training set contains 230 replacements and 570 Source trajectories. Selections remain unchanged across training seeds. Success is averaged over all four suites (%, mean

\pm

sample SD). TIF uses the result from Tab. 3 .

Figure 3: Real-robot candidate selection. (a) Tasks and setup. (b) Scores and selections, with advantage relative to Source C1: TIF rejects C2 and retains C1. (c,d) Recorded executions: both reach the goal, but only C2 collides with the obstacle.

	Task completion		Clean completion
Task	Reward	TIF	Reward	TIF
Push	15/20	16/20	11/20	14/20
One cup	14/20	15/20	11/20	13/20
Relocation	16/20	16/20	13/20	15/20
Total	45/60	47/60	35/60	42/60

Table 6: Selection outcomes on recorded robot videos. Counts refer to selected trajectories, with one selection per candidate pool. Reward denotes Reward-only.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Initial states	Candidates	Pairs	Role
Component	8,000	4	32,000	Reward model and auditor training and evaluation
Natural training	800	5	4,000	Supervision selection for behavior cloning
Natural validation	400	5	2,000	Validation and configuration selection
Natural testing	400	5	2,000	Offline candidate evaluation and selection analysis

Appendix

Table S1 : Dataset sizes and roles. Candidate counts are per initial state. Each pair consists of a recorded execution trajectory and its corresponding imagined video.

Dimension	Evaluation focus	Execution signals
Task progress ( $P$ )	Advancement toward task completion	Changes in success status, environment rewards, approach to the target, and relevant object state changes
Goal correctness ( $G$ )	Agreement with the object and goal specified by the instruction	Target object interaction, approach to the goal location, and satisfaction of task conditions
Manipulation quality ( $M$ )	Effectiveness of grasping, transport, and placement	Gripper engagement, object motion, end-effector motion, and feedback on task-relevant operations
Safety ( $S$ )	Risks associated with abnormal contact and motion	Contact anomalies, excessive actions, action saturation, and abrupt state changes
Execution stability ( $E$ )	Smoothness of actions and state changes	Action smoothness, action reversals, state jitter, and gripper switching

Appendix

Table S2 : Execution quality dimensions and the signals used to construct their labels.

Metric	Definition
Macro-F1	For each quality dimension, the unweighted mean of the F1 scores for the low, medium, and high levels.
Ordinal MAE	Mean absolute error between predicted and reference levels encoded as $0$ , $1$ , and $2$ .
Pairwise accuracy	The fraction of candidate comparisons for which predicted utility agrees with the ordering given by execution reference utility. Comparisons are made within the same initial state.
Top-1 accuracy	The fraction of initial states for which the candidate selected by predicted utility has the highest execution reference utility in its pool.
Selected utility	Mean execution reference utility of the final retained candidates across initial states, including states that retain Source.
Better / Worse	Counts of replacements whose execution reference utility is higher or lower than that of Source, respectively.

Appendix

Table S3 : Evaluation metrics and their aggregation. Execution reference utility is used for candidate evaluation; task success is used for downstream policy evaluation.

Category	Fields	Description
Availability and decoding	feature_available , decode_ok	Feature availability and successful video decoding.
Video format	width , height , fps , frame_count_meta , frame_count_decoded	Width, height, frame rate, metadata frame count, and actual decoded frame count.
Brightness and contrast	mean_brightness , mean_contrast	Image brightness and contrast statistics.
Sharpness	mean_laplacian_var , median_laplacian_var	Mean and median Laplacian variance.
Temporal changes	mean_frame_absdiff , max_frame_absdiff , first_last_absdiff	Mean and maximum absolute differences between adjacent frames, and the absolute difference between the first and last frames.
Diagnostic flags	is_low_motion , is_very_low_motion , is_low_contrast , is_dark_or_blank , is_blurry_proxy , is_bad_decode	Low motion, very low motion, low contrast, dark or blank images, a blur proxy, and decoding failure.

Appendix

Table S4 : Imagined video diagnostic fields. The shared prefix vq__ is omitted.

Item	Setting
Policy	OpenVLA
Updated parameters	LoRA parameters in attention and MLP modules
Training data	One retained actual execution segment per initial state in the natural training set
Training segments	200 per suite; 800 in total
Supervision steps	All valid execution steps
Sampling	Uniform over segments, then uniform over valid time steps within the selected segment

Appendix

Table S5 : Policy post-training and evaluation configuration.

Metric	P	G	M	S	E
Macro-F1 $\uparrow$	.748	.740	.732	.745	.758
Ordinal MAE $\downarrow$	.248	.247	.222	.216	.208

Appendix

Table S6 : Prediction of individual quality dimensions. P, G, M, S, and E denote task progress, goal correctness, manipulation quality, safety, and execution stability, respectively.

Reward model input	Pairwise $\uparrow$	Top-1 $\uparrow$	Utility $\uparrow$	Below Source $\downarrow$
Initial image + instruction + actions	55.91	29.95	.4527	233
Same inputs + imagined video	57.79	33.07	.4603	219
Same inputs + recorded execution video	64.32	50.52	.4735	165

Appendix

Table S7 : Candidate selection with different reward inputs. Pairwise and Top-1 accuracies are percentages. Utility is the mean execution reference utility of selected candidates across all 384 states. Below Source counts selections with lower execution reference utility than Source from the same initial state.

(a) Event discrimination
Event and population	Positives / Candidates	AUROC $\uparrow$	AP $\uparrow$
Below Source, all proposals	219/310	.774	.872
High-score mismatch, all proposals	23/310	.77	.30
High-score mismatch, high-score subset	23/142	.73	.38

Appendix

Table S8 : Event discrimination and grouping by audit risk. (a) The high-score subset contains candidates with predicted utility of at least

0.58

. (b) Candidates are ranked by audit risk. Percentages in parentheses are event rates within each group.

Selection rule	Better $\uparrow$	Worse $\downarrow$	Utility $\uparrow$
Random (expected)	32.58	78.42	.4501
Predicted advantage	47	64	.4589
Simple score risk	46	65	.4602
Logistic regression risk	52	59	.4621
Auditor w/o utility	51	60	.4621
Full auditor	58	53	.4636

Appendix

Table S9 : Selection with 111 replacements. Better and Worse count replacements with higher and lower execution reference utility than Source, respectively. Utility is averaged over all 384 test states.

Training target	Mismatch AUROC $\uparrow$	Mismatch AP $\uparrow$	Below Source AUROC $\uparrow$
Signed-error regression	.73	.25	.75
Below-Source classification	.64	.16	.79
High-score mismatch classification	.77	.30	.77

Appendix

Table S10 : Event discrimination with different audit training targets.

Reference configurations and reward supervision
Configuration	Seed 7	Seed 11	Seed 23	Mean success (%)
Source	1464	1479	1452	$73.25\pm 0.68$
2D complete pipeline	1507	1519	1500	$75.43\pm 0.48$
TIF (5D)	1521	1538	1516	$76.25\pm 0.58$
Selection rules at matched replacement counts
Random-K	1484	1495	1470	$74.15\pm 0.63$

Appendix

Table S11 : Policy success counts for individual training seeds. Each seed is evaluated over 2,000 episodes. Mean success rates are reported in percent with sample standard deviations. The TIF row also supplies the results for matched replacement comparisons and the high-score mismatch target.

Supervision	Long	Average
Scalar	$43.33\pm 0.92$	$74.57\pm 0.33$
2D	$46.13\pm 1.30$	$75.43\pm 0.48$
5D	$\mathbf{48.73\pm 0.83}$	$\mathbf{76.25\pm 0.58}$

Appendix

Table S12 : Policy success on Long and averaged across all four suites with different reward supervision. Values are percentages, reported as mean

\pm

sample standard deviation across three training seeds.

Task	Goal and completion criteria	Execution quality
Push across line	Push the green block completely beyond the blue target line and leave it in a stable state.	Whether the block fully crosses the line and whether execution collides with or displaces the obstacle.
Exactly-one-cup placement	Place exactly one cup upright and fully supported on the target platform beyond the blue line.	Compliance with the single-cup constraint, interaction with additional cups, and placement stability.
Specified-cup relocation	Move the specified cup from the green platform to the yellow platform and place it upright.	Selection of the specified cup, attempts to manipulate the wrong cup, and final placement stability.

Appendix

Table S13 : Real-robot tasks, completion criteria, and aspects of execution quality.

Candidate	Pred. utility	Audit risk	Advantage
C1: Source	.62	.14	$0.00$
C2	.83	.72	$+0.21$
C3	.72	.23	$+0.10$
C4	.56	.18	$-0.06$
C5	.66	.31	$+0.04$

Appendix

Table S14 : Scores for the five candidates in the push example shown in Fig. 3 . Predicted advantage is relative to Source C1.

VLMs for Robotics

Vision-language models for robotics connect visual and linguistic understanding to grounded navigation, manipulation, and robot control.

See all 683 papers

Imagine-RL: Residual-Confidence-Guided Cross-Attention for World-Model-Augmented VLA Reinforcement Learning
Sep 21, 2026Kejia Hu, Wentong Zhai, Bo Zhao +1Robotic Manipulation Vision-Language-Action Models
Department of Electric Engineering, Korea Advanced Institute of Science and Technology · University Of Science and Technology Beijing · Shanghai Jiao Tong University +1
Trust-Calibrated VLA Policy Post-Training with Evolving Imagination
Oct 12, 2026Wenzhan Li, Yiran Qin, Heng Zhou +3RL Post-Training VLMs for Robotics
Sun Yat-sen University · Shenzhen Loop Area Institute · The Chinese University of Hong Kong, Shenzhen +3
VOTE: Vision-Language-Action Optimization with Trajectory Ensemble Voting
Jul 7, 2025Juyi Lin, Amir Taherin, Arash Akbari +11Robotic Control Efficient VLA Model Inference
Northeastern University, Boston, USA · EmbodyX,San Mateo,USA

Sep 21, 2026J/K move · Enter open · S save

Imagine-RL: Residual-Confidence-Guided Cross-Attention for World-Model-Augmented VLA Reinforcement Learning

Kejia Hu, Wentong Zhai, Bo Zhao, Shuai Liang

Department of Electric Engineering, Korea Advanced Institute of Science and Technology · University Of Science and Technology Beijing · Shanghai Jiao Tong University+1

Robotic Manipulation Vision-Language-Action Models RL for VLMs Model-Based RL

Open paper AskPDF arXiv

Reliable action evaluation in contact-rich manipulation requires looking beyond the current observation to future visual and contact consequences. Existing noise-space reinforcement learning efficiently steers a frozen Vision-Language-Action (VLA) policy, but its critics largely ignore these consequences. We present Imagine-RL, which augments noise-space VLA post-training with action-conditioned visual-torque imagination. For each candidate action chunk, a frozen visual-torque latent world model (VTLWM) autoregressively predicts compact future representations without pixel reconstruction. A current image-state-action query attends to observed histories and predicted futures, while previous-window prediction residuals provide token-wise confidence priors that suppress unreliable future tokens. By combining current evidence with predicted consequences, the action critic better evaluates candidate actions and supervises the actor, while the VLA and VTLWM remain frozen. Across four real-robot tasks with 50 evaluation trials per task, Imagine-RL uses only 100 RL trajectories and improves the average success rate by (23.6%) over DSRL and by (60%) over VLA baselines.

Abstract

VLMs for Robotics

Explore similar work

Imagine-RL: Residual-Confidence-Guided Cross-Attention for World-Model-Augmented VLA Reinforcement Learning

Trust-Calibrated VLA Policy Post-Training with Evolving Imagination

VOTE: Vision-Language-Action Optimization with Trajectory Ensemble Voting