Organizations: Beijing University of Posts and Telecommunications, Beijing, P.R. China · ShiFang Technology Inc., Hangzhou, P.R. China · Institute of Automation, Chinese Academy of Sciences, Beijing, P.R. China · Wuhan Electric Power Technical College, Wuhan, P.R. China
Video world models enable candidate actions to be evaluated through imagined outcomes, but high predicted rewards can favor trajectories with poor execution quality. We introduce Trusting Imagined Futures (TIF), an offline supervision selection method for vision-language-action (VLA) post-training that audits predicted rewards using execution feedback. A multidimensional reward model predicts a utility score for candidate ranking and five dimensions of execution quality. An auditor uses reward predictions and uncertainty to identify overestimation patterns learned from paired imagined videos and real executions. TIF accepts the candidate with the highest predicted utility when its advantage over the existing trajectory exceeds a margin and its audit risk passes a threshold; otherwise, it retains the existing supervision. The retained real trajectories then supervise behavior cloning. Across four LIBERO suites and three training seeds, TIF achieves a mean success rate of 76.25%, improving over training on the existing trajectories by 3.00 percentage points with the same trajectory count and training budget. Adding audit gating yields 1.47 percentage points with the reward model and advantage margin held fixed. The gains persist against advantage ranking and logistic regression risk selection at matched replacement counts. On recorded videos from three real robot tasks, TIF increases clean completion among selected trajectories from 58.33% to 70.00% across 60 candidate pools.
Figures & tables
Figure 1: A candidate may receive a higher predicted utility than Source yet exhibit poorer execution quality. TIF audits evaluation reliability before accepting a replacement.
Figure 2: Overview of TIF. Matched executions supervise reward learning and reliability auditing. During curation, predicted utility and audit risk determine whether to replace Source. The selected real trajectories supervise VLA post-training across all valid steps.
Model
Pairwise ↑
Top-1 ↑
Utility ↑
Scalar
56.88
30.99
.4574
2D
57.45
32.55
.4598
5D
57.79
33.07
.4603
5D w/o pairwise
57.42
30.99
.4598
5D shuffled M/S/E
56.80
29.17
.4550
Table 1: Candidate ranking with different reward supervision. Pairwise and Top-1 accuracies are reported in %. Top-1 measures whether the selected candidate has maximal execution reference utility within its candidate set. Utility is the mean execution reference utility of selected candidates. The final two rows independently ablate 5D.
(a) Default selection rules
Method
Replaced
Better
Worse
Utility ↑
Source
0
0
0
.4444
Reward-only
310
91
219
.4603
Reward+Margin
186
65
121
.4587
TIF
111
52
59
.4602
Table 2: Supervision selection with reliability auditing. Better/Worse count replacements with higher/lower execution reference utility than Source. Utility averages the retained trajectories over all 384 states. (a) Default selection rules. (b) Each rule accepts 111 of the same 310 non-Source reward winners without an advantage margin and retains Source elsewhere. Random reports expected values.
Method
Spatial
Object
Goal
Long
Average
Original OpenVLA
74.80
86.20
77.20
37.80
69.00
Source
80.20±0.92
86.07±1.03
83.73±0.31
43.00±1.06
73.25±0.68
Greedy
78.00±1.11
84.53±0.76
81.67±0.42
41.93±1.33
71.53±0.51
Random
79.07±1.21
85.33±0.42
82.67±0.42
41.93±1.21
72.25±0.48
Reward-only
81.60±0.72
87.40±0.72
82.93±0.64
45.00±1.20
74.23±0.40
Reward+Margin
82.13±1.01
88.27±0.50
83.33±0.50
45.40±1.39
74.78±0.63
Table 3: Policy success on LIBERO (%). Post-training methods share the same policy initialization, trajectory count, and training budget, using all valid time steps. Values are means ± sample standard deviations across three training seeds. Original OpenVLA is evaluated before post-training. Bold indicates the highest mean in each column.
(a) Audit training target
Target
Offline selection
Policy success ↑
Worse ↓
Utility ↑
Signed error regression
56
.4624
75.60±0.46
Below-Source classification
50
.4629
75.85±0.59
High-score mismatch classification
53
.4636
76.25±0.58
Table 4: Audit targets and supervision ablations. Policy success is averaged over four suites (%, mean ± sample SD across three training seeds). (a) Auditors share inputs, training data, and training budget. Offline selection accepts 111 of 310 test proposals without a margin; Worse counts replacements below Source, and Utility averages all 384 test states including Source fallback. Policy training separately uses 230 replacements from training candidates that pass the margin, with counts matched by suite. (b) Each reward model has its own auditor trained under the same protocol. (c) Selected trajectories and training budgets are fixed within each selection method.
Selection rule
Success ↑
Random-K
74.15±0.63
Advantage Top-K
75.55±0.50
Logistic risk-K
75.65±0.52
Low-entropy-K
75.03±0.51
Consistency-K
75.60±0.52
TIF
76.25±0.58
Table 5: Policy success at matched replacement counts. All rules select from the same 348 training candidates that pass the advantage margin, accepting 57/61/54/58 replacements in Spatial/Object/Goal/Long. Each training set contains 230 replacements and 570 Source trajectories. Selections remain unchanged across training seeds. Success is averaged over all four suites (%, mean ± sample SD). TIF uses the result from Tab. 3 .
Figure 3: Real-robot candidate selection. (a) Tasks and setup. (b) Scores and selections, with advantage relative to Source C1: TIF rejects C2 and retains C1. (c,d) Recorded executions: both reach the goal, but only C2 collides with the obstacle.
Task completion
Clean completion
Task
Reward
TIF
Reward
TIF
Push
15/20
16/20
11/20
14/20
One cup
14/20
15/20
11/20
13/20
Relocation
16/20
16/20
13/20
15/20
Total
45/60
47/60
35/60
42/60
Table 6: Selection outcomes on recorded robot videos. Counts refer to selected trajectories, with one selection per candidate pool. Reward denotes Reward-only.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Initial states
Candidates
Pairs
Role
Component
8,000
4
32,000
Reward model and auditor training and evaluation
Natural training
800
5
4,000
Supervision selection for behavior cloning
Natural validation
400
5
2,000
Validation and configuration selection
Natural testing
400
5
2,000
Offline candidate evaluation and selection analysis
Appendix
Table S1 : Dataset sizes and roles. Candidate counts are per initial state. Each pair consists of a recorded execution trajectory and its corresponding imagined video.
Dimension
Evaluation focus
Execution signals
Task progress ( P )
Advancement toward task completion
Changes in success status, environment rewards, approach to the target, and relevant object state changes
Goal correctness ( G )
Agreement with the object and goal specified by the instruction
Target object interaction, approach to the goal location, and satisfaction of task conditions
Manipulation quality ( M )
Effectiveness of grasping, transport, and placement
Gripper engagement, object motion, end-effector motion, and feedback on task-relevant operations
Safety ( S )
Risks associated with abnormal contact and motion
Contact anomalies, excessive actions, action saturation, and abrupt state changes
Execution stability ( E )
Smoothness of actions and state changes
Action smoothness, action reversals, state jitter, and gripper switching
Appendix
Table S2 : Execution quality dimensions and the signals used to construct their labels.
Metric
Definition
Macro-F1
For each quality dimension, the unweighted mean of the F1 scores for the low, medium, and high levels.
Ordinal MAE
Mean absolute error between predicted and reference levels encoded as 0 , 1 , and 2 .
Pairwise accuracy
The fraction of candidate comparisons for which predicted utility agrees with the ordering given by execution reference utility. Comparisons are made within the same initial state.
Top-1 accuracy
The fraction of initial states for which the candidate selected by predicted utility has the highest execution reference utility in its pool.
Selected utility
Mean execution reference utility of the final retained candidates across initial states, including states that retain Source.
Better / Worse
Counts of replacements whose execution reference utility is higher or lower than that of Source, respectively.
Appendix
Table S3 : Evaluation metrics and their aggregation. Execution reference utility is used for candidate evaluation; task success is used for downstream policy evaluation.
Category
Fields
Description
Availability and decoding
feature_available , decode_ok
Feature availability and successful video decoding.
Low motion, very low motion, low contrast, dark or blank images, a blur proxy, and decoding failure.
Appendix
Table S4 : Imagined video diagnostic fields. The shared prefix vq__ is omitted.
Item
Setting
Policy
OpenVLA
Updated parameters
LoRA parameters in attention and MLP modules
Training data
One retained actual execution segment per initial state in the natural training set
Training segments
200 per suite; 800 in total
Supervision steps
All valid execution steps
Sampling
Uniform over segments, then uniform over valid time steps within the selected segment
Appendix
Table S5 : Policy post-training and evaluation configuration.
Metric
P
G
M
S
E
Macro-F1 ↑
.748
.740
.732
.745
.758
Ordinal MAE ↓
.248
.247
.222
.216
.208
Appendix
Table S6 : Prediction of individual quality dimensions. P, G, M, S, and E denote task progress, goal correctness, manipulation quality, safety, and execution stability, respectively.
Reward model input
Pairwise ↑
Top-1 ↑
Utility ↑
Below Source ↓
Initial image + instruction + actions
55.91
29.95
.4527
233
Same inputs + imagined video
57.79
33.07
.4603
219
Same inputs + recorded execution video
64.32
50.52
.4735
165
Appendix
Table S7 : Candidate selection with different reward inputs. Pairwise and Top-1 accuracies are percentages. Utility is the mean execution reference utility of selected candidates across all 384 states. Below Source counts selections with lower execution reference utility than Source from the same initial state.
(a) Event discrimination
Event and population
Positives / Candidates
AUROC ↑
AP ↑
Below Source, all proposals
219/310
.774
.872
High-score mismatch, all proposals
23/310
.77
.30
High-score mismatch, high-score subset
23/142
.73
.38
Appendix
Table S8 : Event discrimination and grouping by audit risk. (a) The high-score subset contains candidates with predicted utility of at least 0.58 . (b) Candidates are ranked by audit risk. Percentages in parentheses are event rates within each group.
Selection rule
Better ↑
Worse ↓
Utility ↑
Random (expected)
32.58
78.42
.4501
Predicted advantage
47
64
.4589
Simple score risk
46
65
.4602
Logistic regression risk
52
59
.4621
Auditor w/o utility
51
60
.4621
Full auditor
58
53
.4636
Appendix
Table S9 : Selection with 111 replacements. Better and Worse count replacements with higher and lower execution reference utility than Source, respectively. Utility is averaged over all 384 test states.
Training target
Mismatch AUROC ↑
Mismatch AP ↑
Below Source AUROC ↑
Signed-error regression
.73
.25
.75
Below-Source classification
.64
.16
.79
High-score mismatch classification
.77
.30
.77
Appendix
Table S10 : Event discrimination with different audit training targets.
Configuration
Seed 7
Seed 11
Seed 23
Mean success (%)
Reference configurations and reward supervision
Source
1464
1479
1452
73.25±0.68
2D complete pipeline
1507
1519
1500
75.43±0.48
TIF (5D)
1521
1538
1516
76.25±0.58
Selection rules at matched replacement counts
Random-K
1484
1495
1470
74.15±0.63
Appendix
Table S11 : Policy success counts for individual training seeds. Each seed is evaluated over 2,000 episodes. Mean success rates are reported in percent with sample standard deviations. The TIF row also supplies the results for matched replacement comparisons and the high-score mismatch target.
Supervision
Long
Average
Scalar
43.33±0.92
74.57±0.33
2D
46.13±1.30
75.43±0.48
5D
48.73±0.83
76.25±0.58
Appendix
Table S12 : Policy success on Long and averaged across all four suites with different reward supervision. Values are percentages, reported as mean ± sample standard deviation across three training seeds.
Task
Goal and completion criteria
Execution quality
Push across line
Push the green block completely beyond the blue target line and leave it in a stable state.
Whether the block fully crosses the line and whether execution collides with or displaces the obstacle.
Exactly-one-cup placement
Place exactly one cup upright and fully supported on the target platform beyond the blue line.
Compliance with the single-cup constraint, interaction with additional cups, and placement stability.
Specified-cup relocation
Move the specified cup from the green platform to the yellow platform and place it upright.
Selection of the specified cup, attempts to manipulate the wrong cup, and final placement stability.
Appendix
Table S13 : Real-robot tasks, completion criteria, and aspects of execution quality.
Candidate
Pred. utility
Audit risk
Advantage
C1: Source
.62
.14
0.00
C2
.83
.72
+0.21
C3
.72
.23
+0.10
C4
.56
.18
−0.06
C5
.66
.31
+0.04
Appendix
Table S14 : Scores for the five candidates in the push example shown in Fig. 3 . Predicted advantage is relative to Source C1.
Reliable action evaluation in contact-rich manipulation requires looking beyond the current observation to future visual and contact consequences. Existing noise-space reinforcement learning efficiently steers a frozen Vision-Language-Action (VLA) policy, but its critics largely ignore these consequences. We present Imagine-RL, which augments noise-space VLA post-training with action-conditioned visual-torque imagination. For each candidate action chunk, a frozen visual-torque latent world model (VTLWM) autoregressively predicts compact future representations without pixel reconstruction. A current image-state-action query attends to observed histories and predicted futures, while previous-window prediction residuals provide token-wise confidence priors that suppress unreliable future tokens. By combining current evidence with predicted consequences, the action critic better evaluates candidate actions and supervises the actor, while the VLA and VTLWM remain frozen. Across four real-robot tasks with 50 evaluation trials per task, Imagine-RL uses only 100 RL trajectories and improves the average success rate by (23.6%) over DSRL and by (60%) over VLA baselines.
Kejia Hu, Wentong Zhai, Bo Zhao +1
Department of Electric Engineering, Korea Advanced Institute of Science and Technology · University Of Science and Technology Beijing · Shanghai Jiao Tong University +1
Vision-Language-Action (VLA) models are often pretrained via imitation learning, making reinforcement learning (RL) post-training important when deployment moves beyond demonstration coverage. Existing refinement paradigms expose a reliability--efficiency tension: model-free refinement avoids simulator bias but is interaction-intensive, whereas model-based refinement can exploit unreliable imagination even as its simulator adapts. We propose Model-Evolving Reinforcement Learning (MERL), a trust-calibrated framework that couples policy optimization with an online-updated simulator comprising a world model and a reward proxy head. Across stages, simulator reliability controls the imagined-data ratio and rollout horizon; within rollouts, chunk-level trust controls replay selection and optimization weight. For recursive imagination without exact grounded continuations, a residual predictor estimates trust without querying future ground truth. On perturbed LIBERO-PRO tasks, MERL reaches 79.7% average success, improves high-threshold learning speed by 10.9% over the matched model-free control, and outperforms same-protocol external baselines across three training seeds. Fixed-data adaptation from 50 demonstrations per task achieves 80.0% average success on two physical-robot tasks.
Wenzhan Li, Yiran Qin, Heng Zhou +3
Sun Yat-sen University · Shenzhen Loop Area Institute · The Chinese University of Hong Kong, Shenzhen +3
Recent large-scale Vision Language Action (VLA) models have shown superior performance in robotic manipulation tasks guided by natural language. However, current VLA models suffer from two drawbacks: (i) generation of massive tokens leading to high inference latency and increased training cost, and (ii) insufficient utilization of generated actions resulting in potential performance loss. To address these issues, we develop a training framework to finetune VLA models for generating significantly fewer action tokens with high parallelism, effectively reducing inference latency and training cost. Furthermore, we introduce an inference optimization technique with a novel voting-based ensemble strategy to combine current and previous action predictions, improving the utilization of generated actions and overall performance. Our results demonstrate that we achieve superior performance compared with state-of-the-art VLA models, achieving significantly higher success rates and 39× faster inference than OpenVLA with 46 Hz throughput on edge platforms, demonstrating practical deployability. The code is available at https://github.com/LukeLIN-web/VOTE.
Juyi Lin, Amir Taherin, Arash Akbari +11
Northeastern University, Boston, USA · EmbodyX,San Mateo,USA