ConfAL-WM: Confidence-Guided Active Learning for Action-Conditioned World Models

Xiang Liu; Kunwei Wu; Miao Liu; Sen Cui; Changshui Zhang

cs.ROAug 26, 2026

ConfAL-WM: Confidence-Guided Active Learning for Action-Conditioned World Models

Authors: Xiang Liu, Kunwei Wu, Miao Liu, Sen Cui, Changshui Zhang

Organizations: Tsinghua University · National University of Singapore

Abstract

Action-conditioned world models have become an important foundation for embodied prediction, planning, and synthetic data generation, but their errors under new task and scene distributions are often concentrated in localized spatiotemporal regions such as robot arms, manipulated objects, contact areas, and occluded objects. This paper presents ConfAL-WM, a confidence-guided active learning framework for post-training embodied world models. Building upon EnerVerse-AC (EVAC), we attach a lightweight confidence probe to UNet decoder features and predict dense confidence maps in the latent space. These maps are aggregated into task-, frame-, and patch-level scores, enabling data-budget allocation and localized training enhancement. Our pipeline trains the probe and warms up EVAC on a small target-domain subset. EVAC-v1 then supplies task-level acquisition and optional frame/patch weighting signals; all selected-data models are initialized from the original pretrained EVAC checkpoint for retraining. Experiments on RoboTwin2.0 at the default 40% data budget show that confidence-guided selection improves post-training quality, while dense frame and patch weighting offers complementary reconstruction and semantic gains compared with scalar reward, progress, and judge-based scoring baselines. A quick visual overview of this work is available at https://ConfAL-WM.github.io.

Figures & tables

Figure 1: An introduction of our work. (A) Given reference frames and action conditions, we use a UNet-based latent diffusion world model as the backbone and train a confidence probe on decoder features to predict dense future risk maps. (B) Confidence guides active learning in two ways: selecting more informative post-training data and enhancing supervision on unreliable frames or patches. (C) Two representative episodes compare the same video frame from the ground truth, warmup model, frame-weighted retraining, and frame-and-patch-weighted retraining.

Figure 2: Training and inference of the confidence probe in the UNet latent diffusion world model. Video pixels are encoded into latent variables, diffused into $z^{(\tau)}$ , processed by the UNet denoiser, and supervised by the denoising target $v^{(\tau)}$ . The confidence probe is attached to selectable decoder features $h_{\mathrm{dec}}^{(\ell)}$ and predicts dense confidence maps.

Figure 3: Direct comparison for a single frame. All operations are performed in the VAE latent space. The lower panels visualize the channel-averaged $\mathbb{E}_{c}\left[\left|h_{\mathrm{dec}}^{(\ell)}\right|\right]$ .

Figure 4: Qualitative confidence visualization. High- and low-agreement episodes from different manipulation tasks are shown. High-risk regions generally coincide with errors around robot arms, manipulated objects, contacts, and occlusions. More examples: see Appendix B.8 .

Figure 5: Confidence-probe ablations and multi-scale risk validity. (a) Feature and threshold-supervision ablations on EVAC-v1, showing episode-mean Brier (top) and paired task-macro EMA-minus-tuned-fixed Brier (bottom). Tuned fixed uses the calibrated $[Q_{.05},Q_{.95}]$ range; error bars denote 95% task-bootstrap intervals, with positive differences favoring fixed supervision. (b) Mean risk versus latent RMS rollout error at frame (top) and task (bottom) levels. (c) AUROC (top) and AUPRC (bottom) for detecting the top- $Q\%$ highest-error patches in latent and pixel spaces.

Property	Metric	Result
Multi-scale ranking	Patch / Frame / Episode Spearman ( $\uparrow$ )	$0.540/0.590/0.595$
High-error detection	AUROC / AUPRC@top- $5\%$ ( $\uparrow$ )	$0.761/0.146$
Spatial agreement	Top- $5\%$ IoU ( $\uparrow$ )	$0.130$
Temporal stability	Adjacent-frame IoU ( $\uparrow$ ) / Flicker ( $\downarrow$ )	$0.740/0.005$
Temporal alignment	Peak correlation ( $\uparrow$ ) / $\|\mathrm{lag}\|$ ( $\downarrow$ )	$0.602/\approx 0$

Table 1: Summary of latent-space confidence validity across multiple scales. Frame- and task-level statistics use mean aggregation.

Figure 6: Main comparison of active-learning scoring and retraining strategies. Each panel compares EVAC-v1 with selection-only retraining and additional frame weighting. Confidence enables frame+patch weighting. Learned-method bars report means over three seeds, and arrows report relative changes from EVAC-v1. Error bars indicate paired-bootstrap 95% confidence intervals.

Scoring	Weighting	Reconstruction ( $\uparrow$ )		Scene ( $\uparrow$ )	Semantics ( $\uparrow$ )			Motion ( $\uparrow$ )
Scoring	Weighting	PSNR ( $\uparrow$ )	SSIM ( $\uparrow$ )	Scene Cons. ( $\uparrow$ )	Logics ( $\uparrow$ )	Sem.-CLIP ( $\uparrow$ )	Sem.-BLEU ( $\uparrow$ )	Traj-HSD ( $\uparrow$ )	Traj-Dyn ( $\uparrow$ )	Traj-nDTW ( $\uparrow$ )
Base EVAC		0.5532	0.5778	0.8757	0.4298	0.8523	0.1799	0.0007	0.0001	0.0006
Base EVAC (Warmup v1)		0.6446	0.7309	0.9047	0.5537	0.8824	0.2708	0.1045	0.0721	0.1439
Selection-only retraining: no additional weighting after selection
Random	None	0.5968	0.6849	0.8524	0.3554	0.8689	0.2372	0.1067	0.0725	0.1518
Oracle error	None	0.6458	0.7542	0.8988	0.5826	0.8875	0.2914	0.0985	0.0587	0.1396

Table 2: Detailed normalized results under the same active-learning setting. Learned-method results are averaged over three seeds (42, 3407, and 123), and oracle uses optimizer seed 42. Methods are separated into selection-only retraining and selection with additional weighting. Green, yellow, and red indicate the best, second-best, and third-best results within each block, respectively.

Variant	Reconstruction ( $\uparrow$ )		Scene ( $\uparrow$ )	Semantics ( $\uparrow$ )			Motion ( $\uparrow$ )
Variant	PSNR ( $\uparrow$ )	SSIM ( $\uparrow$ )	Scene Cons. ( $\uparrow$ )	Logics ( $\uparrow$ )	Sem.-CLIP ( $\uparrow$ )	Sem.-BLEU ( $\uparrow$ )	Traj-HSD ( $\uparrow$ )	Traj-Dyn ( $\uparrow$ )	Traj-nDTW ( $\uparrow$ )
Random	0.5968	0.6849	0.8524	0.3554	0.8689	0.2372	0.1067	0.0725	0.1518
Mean Risk	0.6838	0.7797	0.9322	0.6198	0.8894	0.2754	0.1313	0.0921	0.1780
Tail Risk	0.6317	0.7197	0.9218	0.5579	0.8890	0.3047	0.1207	0.0711	0.1630
Persistent Risk	0.6529	0.7470	0.9230	0.5496	0.8928	0.2858	0.0926	0.0608	0.1342

Table 3: Ablation of selection criteria. All experiments use the default seed 42 and no additional weighting. Green, yellow, and red indicate the best, second-best, and third-best results.

Appendix figures & tables39 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Training and inference of the confidence probe in the UNet latent diffusion world model. The figure follows the same training and inference pipeline as Figure 2 .

Figure 8: Patch-level latent error construction. Channel-wise latent prediction errors are first averaged into a spatial error map and then pooled over the region aligned with each confidence-probe patch.

Panel of Figure 9 — Figure 9: Dynamics of the EMA-based adaptive threshold band. The probe uses a stochastic threshold $\theta\sim\mathcal{U}(l^{(s)},h^{(s)})$ , where $l^{(s)}$ and $h^{(s)}$ are the EMA-tracked lower and upper bounds at training step $s$ , and the black curve shows their midpoint $(l^{(s)}+h^{(s)})/2$ . (a) During the first 100 steps, both bounds decrease rapidly and quickly approach a stable range, reflecting fast adaptation to the current error scale. (b) Over a longer set of sampled training steps, the threshold band shows a clear transition from a fast-changing warmup regime to a slower and more stable regime.

Figure 10: Task-level risk aggregation in latent and pixel spaces. Mean, tail, and persistent risk scores are compared with the mean oracle-error statistics over the 50 prescreened tasks. Dashed lines show linear fits, shaded regions denote confidence intervals, and each panel reports the Spearman correlation. Mean aggregation gives the strongest latent-space correlation, whereas tail and persistent aggregation emphasize different failure patterns.

Panel of Figure 11 — Figure 11: Spatial agreement between high-risk and high-error patches. Top- $k$ IoU and overlap@ $k$ are reported for the top- $1\%$ , $5\%$ , $10\%$ , and $20\%$ patches in (a) latent and (b) pixel spaces. The increasing scores at larger ratios reflect coarser spatial coverage and should not be interpreted as improved localization resolution.

Panel of Figure 12 — Figure 12: ECE and Brier scores under different operational thresholds. Calibration is evaluated by sweeping the correctness threshold $\theta_{\mathrm{eval}}$ in (a) latent space and (b) pixel space. The confidence maps are fixed throughout; only the binary correctness definition changes.

Panel of Figure 13 — Figure 13: Reliability diagrams across operational thresholds. Each subfigure contains reliability diagrams under different choices of $\theta_{\mathrm{eval}}$ : (a) latent space and (b) pixel space. As the threshold becomes looser, the observed correctness generally moves closer to the predicted confidence, although the latent-space probe remains more overconfident near the main operating point.

Panel of Figure 14 — Figure 14: Sensitivity to confidence aggregation and probe conditioning. (a–b) Frame- and task-level Spearman correlations under mean, maximum, percentile, and top- $k$ aggregation in latent and pixel spaces. (c) High-error detection and spatial localization under different probe conditioning thresholds $\theta_{\mathrm{cond}}$ . (d) ECE, Brier score, and mean confidence under the same thresholds. Unlike the operational threshold $\theta_{\mathrm{eval}}$ , changing $\theta_{\mathrm{cond}}$ regenerates the confidence maps.

Scoring	Weighting	Reconstruction ( $\uparrow$ )	Scene ( $\uparrow$ )	Motion ( $\uparrow$ )	Semantics ( $\uparrow$ )
Base EVAC		0.5655 (-17.8%)	0.8757 (-3.2%)	0.0014 (-99.6%)	0.4873 (-14.3%)
Base EVAC (Warmup v1)		0.6878 (0.0%)	0.9047 (0.0%)	0.3205 (0.0%)	0.5690 (0.0%)
Selection-only retraining: no additional weighting after selection
Random	None	0.6409 (-6.8%)	0.8524 (-5.8%)	0.3310 (+3.3%)	0.4872 (-14.4%)
RoboReward 1	None	0.6712 (-2.4%)	0.8470 (-6.4%)	0.3360 (+4.8%)	0.6022 (+5.8%)
GVL 2	None	0.7088 (+3.1%)	0.9057 (+0.1%)	0.3393 (+5.9%)	0.5913 (+3.9%)

Appendix

Table 4: Aggregated active-learning results. Each entry reports the aggregated mean followed by its relative change from EVAC-v1. Results for learned scoring methods are averaged over three seeds.

Scoring	Weighting	$\Delta$ Reconstruction ( $\uparrow$ )	$\Delta$ Scene ( $\uparrow$ )	$\Delta$ Motion ( $\uparrow$ )	$\Delta$ Semantics ( $\uparrow$ )
Base EVAC		[-0.133, -0.112] (-0.122)	[-0.038, -0.020] (-0.029)	–	[-0.113, -0.049] (-0.081)
Base EVAC (Warmup v1)		–	–	–	–
Selection-only retraining: no additional weighting after selection
Random	None	[-0.055, -0.039] (-0.047)	[-0.060, -0.045] (-0.052)	[-0.035, +0.056] (+0.011)	[-0.112, -0.046] (-0.079)
RoboReward 1	None	[-0.020, -0.013] (-0.017)	[-0.063, -0.052] (-0.058)	[-0.006, +0.038] (+0.015)	[+0.016, +0.053] (+0.035)
GVL 2	None	[+0.018, +0.024] (+0.021)	[-0.001, +0.003] (+0.001)	[-0.008, +0.046] (+0.019)	[+0.003, +0.042] (+0.023)

Appendix

Table 5: Paired-bootstrap improvements over EVAC-v1. For each learned scoring method, episode-level paired differences from seeds 42, 3407, and 123 are pooled before bootstrap resampling. Each entry gives the 95% confidence interval followed by the paired mean in parentheses.

Scoring	Weighting	Reconstruction ( $\uparrow$ )		Scene ( $\uparrow$ )	Semantics ( $\uparrow$ )			Motion ( $\uparrow$ )
Scoring	Weighting	PSNR ( $\uparrow$ )	SSIM ( $\uparrow$ )	Scene Cons. ( $\uparrow$ )	Logics ( $\uparrow$ )	Sem.-CLIP ( $\uparrow$ )	Sem.-BLEU ( $\uparrow$ )	Traj-HSD ( $\uparrow$ )	Traj-Dyn ( $\uparrow$ )	Traj-nDTW ( $\uparrow$ )
Base EVAC		0.5532	0.5778	0.8757	0.4298	0.8523	0.1799	0.0007	0.0001	0.0006
Base EVAC (Warmup v1)		0.6446	0.7309	0.9047	0.5537	0.8824	0.2708	0.1045	0.0721	0.1439
Selection-only retraining: no additional weighting after selection
Random	None	0.5968	0.6849	0.8524	0.3554	0.8689	0.2372	0.1067	0.0725	0.1518
RoboReward 1	None	0.6276	0.7067	0.8712	0.6322	0.8895	0.2832	0.1076	0.0662	0.1501

Appendix

Table 6: Detailed normalized active-learning results for seed 42.

Scoring	Weighting	Reconstruction ( $\uparrow$ )		Scene ( $\uparrow$ )	Semantics ( $\uparrow$ )			Motion ( $\uparrow$ )
Scoring	Weighting	PSNR ( $\uparrow$ )	SSIM ( $\uparrow$ )	Scene Cons. ( $\uparrow$ )	Logics ( $\uparrow$ )	Sem.-CLIP ( $\uparrow$ )	Sem.-BLEU ( $\uparrow$ )	Traj-HSD ( $\uparrow$ )	Traj-Dyn ( $\uparrow$ )	Traj-nDTW ( $\uparrow$ )
Base EVAC		0.5532	0.5778	0.8757	0.4298	0.8523	0.1799	0.0007	0.0001	0.0006
Base EVAC (Warmup v1)		0.6446	0.7309	0.9047	0.5537	0.8824	0.2708	0.1045	0.0721	0.1439
Selection-only retraining: no additional weighting after selection
Random	None	0.5968	0.6849	0.8524	0.3554	0.8689	0.2372	0.1067	0.0725	0.1518
RoboReward 1	None	0.6262	0.7290	0.7985	0.6488	0.8863	0.2775	0.1089	0.0732	0.1554

Appendix

Table 7: Detailed normalized active-learning results for seed 3407.

Scoring	Weighting	Reconstruction ( $\uparrow$ )		Scene ( $\uparrow$ )	Semantics ( $\uparrow$ )			Motion ( $\uparrow$ )
Scoring	Weighting	PSNR ( $\uparrow$ )	SSIM ( $\uparrow$ )	Scene Cons. ( $\uparrow$ )	Logics ( $\uparrow$ )	Sem.-CLIP ( $\uparrow$ )	Sem.-BLEU ( $\uparrow$ )	Traj-HSD ( $\uparrow$ )	Traj-Dyn ( $\uparrow$ )	Traj-nDTW ( $\uparrow$ )
Base EVAC		0.5532	0.5778	0.8757	0.4298	0.8523	0.1799	0.0007	0.0001	0.0006
Base EVAC (Warmup v1)		0.6446	0.7309	0.9047	0.5537	0.8824	0.2708	0.1045	0.0721	0.1439
Selection-only retraining: no additional weighting after selection
Random	None	0.5968	0.6849	0.8524	0.3554	0.8689	0.2372	0.1067	0.0725	0.1518
RoboReward 1	None	0.6260	0.7117	0.8712	0.6364	0.8825	0.2838	0.1174	0.0685	0.1607

Appendix

Table 8: Detailed normalized active-learning results for seed 123.

Feature	Supervision	Brier $\downarrow$	BCE $\downarrow$	ECE $\downarrow$	AUROC $\uparrow$
Bottleneck	Tuned fixed	0.089487	0.288859	0.00567	0.78379
Bottleneck	EMA	0.089551	0.289220	0.00639	0.78487
Decoder	Tuned fixed	0.084181	0.271915	0.00402	0.82237
Decoder	EMA	0.084243	0.272314	0.00643	0.82184

Appendix

Table 9: Controlled feature and supervision comparison on EVAC-v1. All rows use seed 42 and identical probe capacity. Tuned fixed denotes the calibration-selected

[Q_{.05},Q_{.95}]

range. Scores are computed on 250 test episodes with

\tau\in\{150,\ldots,599\}

. Lower Brier, BCE, and ECE and higher AUROC indicate better performance. Boldface marks the best values among the four configurations.

Figure 15: Probe performance at shared thresholds. (a) Brier scores for the four core configurations. (b) EMA-minus-tuned-fixed Brier at each threshold; positive values favor fixed supervision. Fixed and EMA supervision yield closely aligned curves, with a larger separation between decoder and bottleneck features. All configurations use the same binary targets

q_{e,t,(i,j)}(\theta_{k})

, constructed from held-out local latent errors.

Feature	Supervision	Cal. Brier	Test Brier	Test ECE	Test AUROC	Low-noise Brier
Bottleneck	Fixed $[0.20,0.70]$	0.154102	0.153483	0.12854	0.77596	0.090326
Bottleneck	Fixed $[Q_{.10},Q_{.90}]$	0.092173	0.089468	0.00670	0.78338	0.057385
Bottleneck	Fixed $[Q_{.05},Q_{.95}]^{\dagger}$	0.092149	0.089487	0.00567	0.78379	0.057210
Bottleneck	Fixed $[Q_{.20},Q_{.80}]$	0.092776	0.090206	0.01182	0.78185	0.058627
Bottleneck	EMA	0.092239	0.089551	0.00639	0.78487	0.057544
Bottleneck	EMA-Qinit	0.092101	0.089477	0.00694	0.78517	0.057503

Appendix

Table 10: Complete single-seed range ablation. All 12 fits use seed 42. Quantiles

Q_{\rho}

are computed from training errors. The dagger marks the fixed range selected by calibration Brier, both when averaging across feature choices and when selecting separately for each feature. EMA-Qinit starts from

[Q_{.10},Q_{.90}]

. Test columns use

\tau\in\{150,\ldots,599\}

; the final column evaluates the same episodes with

\tau\in\{50,\ldots,199\}

	Training-noise range		Low-noise range
Contrast	Difference	95% CI	Difference	95% CI
Decoder $-$ bottleneck (fixed)	$-5.331$	$[-5.657,-4.987]$	$-2.767$	$[-3.156,-2.243]$
Decoder $-$ bottleneck (EMA)	$-5.294$	$[-5.637,-4.934]$	$-3.141$	$[-3.475,-2.762]$
EMA $-$ fixed (decoder)	$+0.072$	$[+0.011,+0.134]$	$-0.016$	$[-0.151,+0.091]$
EMA $-$ fixed (bottleneck)	$+0.035$	$[-0.071,+0.143]$	$+0.358$	$[+0.270,+0.449]$
Feature $\times$ supervision	$+0.037$	$[-0.090,+0.170]$	$-0.374$	$[-0.562,-0.229]$

Appendix

Table 11: Paired task-macro Brier contrasts. All values, including interval endpoints, are multiplied by

10^{3}

. Intervals are percentile 95% task-bootstrap intervals at seed 42. Negative differences favor the first named configuration. Let

B_{a,b}

denote task-macro Brier for feature choice

a

and supervision rule

b

, with

\mathrm{dec}

and

\mathrm{mid}

denoting decoder and bottleneck features. The interaction is

(B_{\mathrm{dec,EMA}}-B_{\mathrm{mid,EMA}})-(B_{\mathrm{dec,fixed}}-B_{\mathrm{mid,fixed}})

Figure 16: Feature and supervision effects across tasks. (a) Decoder-minus-bottleneck Brier contrasts under fixed and EMA supervision. (b) EMA-minus-tuned-fixed contrasts for each feature choice. The panels use different horizontal scales to display the respective effect sizes. Error bars indicate percentile 95% task-bootstrap intervals at seed 42.

Figure 17: Noise-range transfer on held-out episodes. Brier (left) and ECE (right) are evaluated on the same episodes with fixed probe parameters and threshold ranges. Decoder features achieve lower Brier in both noise ranges. Low-noise evaluation shows lower Brier but higher ECE due to changes in local-error distributions and target balance.

Figure 18: Alignment of confidence-derived signals with prediction error. (a) Mean within-episode Spearman correlation between normalized patch weights and local latent MAE. (b) Mean within-episode Spearman correlation between spatially averaged frame risk and frame MAE. Risk is averaged over the nine evaluation thresholds before constructing these diagnostics. All variants use the same weighting transformation, and values are computed before EVAC retraining at seed 42.

Panel of Figure 19 — Figure 19: Qualitative comparison of feature choices and threshold supervision. Two held-out episodes from the hammering task. Each row shows the first ground-truth future frame, pooled latent MAE, and probe responses for decoder features with EMA and tuned fixed supervision, followed by bottleneck features with EMA and tuned fixed supervision. Decoder responses contain finer spatial structure, while EMA and tuned fixed supervision yield similar patterns within each feature choice.

Selection	Reconstruction ( $\uparrow$ )		Scene ( $\uparrow$ )	Semantics ( $\uparrow$ )			Motion ( $\uparrow$ )
Selection	PSNR	SSIM	Scene Cons.	Logics	Sem.-CLIP	Sem.-BLEU	Traj-HSD	Traj-Dyn	Traj-nDTW
Oracle error	0.6458	0.7542	0.8988	0.5826	0.8875	0.2914	0.0985	0.0587	0.1396
Confidence	0.6838	0.7797	0.9322	0.6198	0.8894	0.2754	0.1313	0.0921	0.1780

Appendix

Table 12: Direct-error and confidence-based selection. Both models use optimizer seed 42, 7,298 selected episodes, 4,000 updates, and no loss weighting. Confidence (Selection) uses the seed-42 result in Table 6 , while Table 2 reports the average over three confidence runs. Bold values indicate the better result in each column. All metrics are higher-is-better.

Figure 20: Confidence versus direct-error selection. Bars show Confidence minus Oracle for four aggregate metrics (left) and nine component metrics (right), using the unweighted models in Table 12 . Positive values favor Confidence. Error bars denote paired 95% episode-bootstrap intervals. Most comparisons use 242 common episodes; Sem.-CLIP, Sem.-BLEU, and aggregate Semantics use 190 common episodes.

Budget	Selection	Episodes	Reconstruction $\uparrow$	Scene $\uparrow$	Semantics $\uparrow$	Motion $\uparrow$	Train GPU-h
10%	Random	1,824	0.6341	0.8772	0.5007	0.1701	4.47
10%	Confidence	1,824	0.6995	0.9088	0.5900	0.2502	4.45
20%	Random	3,649	0.6679	0.9003	0.5391	0.2360	4.45
20%	Confidence	3,649	0.6642	0.8812	0.6167	0.1970	4.44
40%	Random	7,298	0.6874	0.8786	0.5576	0.2004	4.44
40%	Confidence	7,298	0.6611	0.8981	0.5987	0.2305	4.47

Appendix

Table 13: Budget sensitivity under 750-update post-training. All models use optimizer seed 42 and no loss weighting. Bold values mark the higher score within each confidence/random pair. Aggregates follow the main-text definitions and use each component’s available mean. Reconstruction, Scene, and Motion cover 62 episodes; CLIP/BLEU cover 56 episodes.

Figure 21: Budget–performance curves under fixed optimization. All metrics are higher-is-better. Confidence and random receive the same 750 updates. In each panel, the gray square marks the 100% full-pool result, and the gray horizontal dashed line extends its mean across budgets for reference. Error bars show 95% episode-bootstrap intervals, using 62 common episodes for most metrics and 56 for CLIP/BLEU. Lines between measured budget points serve as visual guides.

Budget	$\Delta$ Reconstruction $\uparrow$	$\Delta$ Scene $\uparrow$	$\Delta$ Semantics $\uparrow$	$\Delta$ Motion $\uparrow$
10%	$0.0655\ [0.0534,0.0772]$	$0.0316\ [0.0229,0.0396]$	$0.0914\ [0.0247,0.1563]$	$0.0801\ [0.0293,0.1391]$
20%	$-0.0037\ [-0.0186,0.0112]$	$-0.0190\ [-0.0296,-0.0089]$	$0.0768\ [0.0119,0.1425]$	$-0.0389\ [-0.1023,0.0185]$
40%	$-0.0263\ [-0.0411,-0.0120]$	$0.0195\ [0.0031,0.0362]$	$0.0440\ [-0.0164,0.1021]$	$0.0301\ [-0.0570,0.1390]$

Appendix

Table 14: Paired confidence-minus-random differences. Each entry gives the paired mean and its 95% interval from 10,000 episode-bootstrap resamples. Larger differences favor confidence. Reconstruction, Scene, and Motion use 62 common episodes; Semantics uses the 56-episode intersection of its three components. This common-episode calculation can differ slightly from subtracting the component-wise aggregates in Table 13 .

Setting	Value
Task representatives / acquisition score	1 per task / $S_{\mathrm{mean}}$
Rank-based allocation multiplier	$0.25$ – $2.0$
Decoder index $\ell$	5 (zero-based)
Frame-and-patch mixing $\alpha$	0.5
Confidence strength $\lambda_{\mathrm{conf}}$	1.0
Weighting warmup $s_{\mathrm{warm}}$	1,000 updates

Appendix

Table 15: Selection and retraining settings. The weighting parameters apply to the weighted experiments in the main text. The new Oracle and budget-sweep models use unweighted losses.

Episode	Variant	Reconstruction ( $\uparrow$ )	Scene ( $\uparrow$ )	Semantics ( $\uparrow$ )	Motion ( $\uparrow$ )
place burger fries (ep130)	Base EVAC	0.4592 (-27.3%)	0.7890 (-10.0%)	0.3591 (-28.9%)	0.0000 (-100.0%)
	EVAC-v1	0.6318 (0.0%)	0.8770 (0.0%)	0.5050 (0.0%)	0.4570 (0.0%)
	EVAC-v2 Frame	0.6618 (+4.7%)	0.9430 (+7.5%)	0.5221 (+3.4%)	1.0830 (+137.0%)
	EVAC-v2 Fr.+Patch	0.7020 (+11.1%)	0.9370 (+6.8%)	0.5486 (+8.6%)	1.3080 (+186.2%)
place empty cup (ep294)	Base EVAC	0.4605 (-38.6%)	0.7120 (-27.0%)	0.5454 (+2.0%)	0.0000 (-100.0%)
	EVAC-v1	0.7503 (0.0%)	0.9750 (0.0%)	0.5349 (0.0%)	0.5600 (0.0%)

Appendix

Table 16: Aggregated evolution on six representative RoboTwin2.0 episodes. Each entry reports the normalized aggregate followed by its relative change from EVAC-v1. Green and red percentages indicate improvement and degradation, respectively. For Motion, relative changes are omitted when the EVAC-v1 score is zero. Green, yellow, and red cells indicate the best, second-best, and third-best values within each episode.

Figure 22: Training evolution on place_burger_fries (ep130). EVAC-v1 develops a strong blue color shift and the tray contents collapse from the middle of the rollout. EVAC-v2 Frame removes most tray corruption, but the left arm hallucinates an object instead of grasping the burger and the object held by the right arm is visibly deformed. EVAC-v2 Fr.+Patch correctly grasps and places the burger while preserving the fries most faithfully.

Figure 23: Training evolution on place_empty_cup (ep294). In both EVAC-v1 and EVAC-v2 Frame, the arm only partially appears near the image boundary and fails to grasp the cup at the beginning; EVAC-v2 Frame additionally deteriorates toward the end of the rollout. EVAC-v2 Fr.+Patch correctly grasps the cup from the boundary, places it on the plate, and maintains a coherent scene throughout the prediction.

Figure 24: Training evolution on put_object_cabinet (ep282). EVAC-v1 shows moderate color bias and eventually produces a floating arm together with a corrupted cabinet. EVAC-v2 Frame does not resolve these failures: the blue box in the upper-left region disappears and the arm is generated at an incorrect location. EVAC-v2 Fr.+Patch preserves the objects, cabinet structure, and robot geometry most consistently.

Figure 25: Training evolution on stack_blocks_two (ep325). EVAC-v1 exhibits strong color flickering, and the lower blue block disappears when the green block is lifted and stacked. EVAC-v2 Frame largely removes the color shift but fails to keep the green block attached to the gripper and consequently misses the stacking interaction. EVAC-v2 Fr.+Patch resolves both the object-persistence and grasping failures.

Figure 26: Training evolution on stack_bowls_three (ep087). EVAC-v1 suffers from pronounced color drift and repeated red flickering, followed by severe object corruption late in the rollout. EVAC-v2 Frame removes the color shift but still produces substantial bowl deformation, disappearance, and reappearance. EVAC-v2 Fr.+Patch better preserves the bowl geometry and maintains object persistence through the later frames.

Figure 27: Training evolution on turn_switch (ep310). EVAC-v1 produces a reddish robot arm, a bluish background, and implausible floating-arm geometry. EVAC-v2 Frame substantially reduces the color bias, but the arm shape remains distorted. EVAC-v2 Fr.+Patch further improves the local arm geometry and produces an embodiment substantially closer to the ground truth.

Episode	Variant	Reconstruction ( $\uparrow$ )		Scene ( $\uparrow$ )	Semantics ( $\uparrow$ )		Motion ( $\uparrow$ )
Episode	Variant	PSNR ( $\uparrow$ )	SSIM ( $\uparrow$ )	Scene Cons. ( $\uparrow$ )	Sem.-CLIP ( $\uparrow$ )	Sem.-BLEU ( $\uparrow$ )	Traj-HSD ( $\uparrow$ )	Traj-Dyn ( $\uparrow$ )	Traj-nDTW ( $\uparrow$ )
place burger fries (ep130)	Base EVAC	0.4413	0.477	0.789	0.7183	0.000	0.000	0.000	0.000
	EVAC-v1	0.5957	0.668	0.877	0.7879	0.222	0.095	0.001	0.361
	EVAC-v2 Frame	0.6777	0.646	0.943	0.8121	0.232	0.545	0.024	0.514
	EVAC-v2 Fr.+Patch	0.6980	0.706	0.937	0.8182	0.279	0.761	0.002	0.545
place empty cup (ep294)	Base EVAC	0.5100	0.411	0.712	0.8448	0.246	0.000	0.000	0.000

Appendix

Table 17: Detailed normalized EWMBench results for the qualitative evolution study. All metrics are higher-is-better. Green, yellow, and red indicate the best, second-best, and third-best distinct values within each episode, respectively.

Figure 28: More results of qualitative confidence visualization. Best-ranked example (top 1) by patch-level Spearman correlation.

Figure 29: More results of qualitative confidence visualization. Second-highest-ranked example by patch-level Spearman correlation.

Figure 30: More results of qualitative confidence visualization. Lowest-ranked example (top 1) by patch-level Spearman correlation.

Figure 31: More results of qualitative confidence visualization. Second-lowest-ranked example by patch-level Spearman correlation.

Action-Conditioned World Models

Action-conditioned world models predict how environments change under actions to support planning, control, and simulation.

See all 346 papers

Jul 10, 2026

Causally Debiased Latent Action Model for Embodied Action Conditioned World Models

Action-conditioned world models (ACWMs) aim to simulate future observations conditioned on embodied actions, offering a promising foundation for robot planning, policy evaluation, and data augmentation. However, learning controllable ACWMs requires large-scale action-labeled data, which remains costly to collect in the real world. Latent action models (LAMs) mitigate this bottleneck by inferring latent actions from unlabeled videos, but existing LAMs are typically trained with reconstruction-only objectives and therefore entangle action-relevant dynamics with action-irrelevant visual factors such as backgrounds and untouched objects. In this work, we identify this action-irrelevant bias as a key obstacle to controllable ACWMs and introduce evaluation metrics to measure latent-action bias, action following, and robustness. We propose CD-LAM, a causally debiased framework for LAM-based ACWMs. CD-LAM introduces three efficient fine-tuning objectives: embodiment-centric reconstruction, action-centric contrastive learning, and latent space calibration, which together encourage embodiment-focused, action-aware, and calibrated non-collapsed latent action representations. Experiments on 2B and 14B ACWM backbones show that CD-LAM substantially improves latent-action controllability, downstream robot-action following, visual fidelity, and adaptation efficiency, requiring only 6k fine-tuning steps and more than 12

\times

fewer robot-action adaptation updates than the baseline.

Yufan Wei, Kun Zhou, Lingjun Mao +9

Aether AI · University of California, San Diego

Latent Action Learning Causal Representation Learning World Models Action-Conditioned World Models Embodied World Models

Ask PDF

Oct 7, 2026

Cross-Embodiment Robot Foundation World Models with Latent Actions

The diversity of robot embodiments and action spaces makes it challenging to build robot world models that generalize across different embodiments. We introduce the Latent Action-Conditioned Robot World Model (LAC-WM), which operates within a learned unified latent action space shared across diverse embodiments. This unified action space improves the world model's performance when adapted to previously unseen robot embodiments. We compare LAC-WM with an Explicit Action-Conditioned World Model (EAC-WM), which conditions on explicit motion labels. Our results show that explicit action conditioning leads to disjoint action representations across embodiments, limiting downstream performance when adapting to new robots. We evaluate both models on dexterous manipulation tasks and a modified LIBERO benchmark. LAC-WM improves downstream performance over EAC-WM by up to 46.7% on dexterous manipulation and 11.7% on LIBERO. Crucially, the unified latent action space allows LAC-WM's downstream performance to scale positively with the number of embodiments used during pretraining. In contrast, the disjoint action space in EAC-WM leads to decreased performance as the number of pretraining embodiments increases. These results highlight the importance of a unified action space for efficient cross-embodiment learning, addressing a key challenge in robotics. Project website: https://lacwm.github.io/

Huang Huang, Sriram Yenamandra, Arjun Majumdar +9

Department of Computer Science, Stanford University, US · Work partially done while at Meta FAIR Robotics.

World Models for Robotics Action-Conditioned World Models Cross-Embodiment Robot Learning Robot Foundation Models Latent Action Learning

Ask PDF

Sep 13, 2026

World-Action Models for Robot Learning and Control: A Survey

Robots operating in open environments act under partial observability, physical constraints, and dynamic task contexts. Beyond mapping observations and language instructions to actions, they must anticipate how candidate actions may affect future states and task-relevant outcomes. Recent advances in world models, video generation, and Vision-Language-Action (VLA) policies have motivated the development of World-Action Models (WAMs), which couple future world prediction with executable action generation. This survey provides a robotics-oriented review of WAMs. We clarify their scope relative to conventional world models, model-based reinforcement learning, action-conditioned video generation, and reactive VLA policies, and organize existing methods through a unified taxonomy covering representations, transition modeling, action interfaces, architectures, training pipelines, data modalities, and scaling strategies. We further review applications of WAMs in manipulation, navigation, and autonomous driving, and we summarize the datasets, benchmarks, metrics, and protocols used to evaluate WAM systems. Finally, we discuss key challenges in action alignment, world-action factorization, spatial and multi-view consistency, long-horizon memory, neural simulation for closed-loop policy learning, and efficient inference. Taken together, this survey aims to provide a concise technical foundation for integrating predictive world modeling with action generation, toward more reliable embodied robot intelligence. Project page: https://rcl-robotics.github.io/Awesome-World-Action-Models.

Zuxing Lu, Hongjia Zhai, Guanzhi Wang +13

MBZUAI · Caltech · Amazon FAR +4

World Action Models Action-Conditioned World Models World Models for Robotics World Model Learning World Model-Based Planning

Ask PDF

Abstract

Action-Conditioned World Models

Explore similar work

Causally Debiased Latent Action Model for Embodied Action Conditioned World Models

Cross-Embodiment Robot Foundation World Models with Latent Actions

World-Action Models for Robot Learning and Control: A Survey