RoboSeed: Multidimensional Evaluation of Vision-Language-Action and World-Action Models: Capability, Resource Affordability, and Research Evidence

Xin Ma; Furong Chen; Xinlong Man; Bo Zhang; Yalong Guo; Haolan Lei; Chaochen Cui; Hongyuan Zhang

cs.ROOct 29, 2026

RoboSeed: Multidimensional Evaluation of Vision-Language-Action and World-Action Models: Capability, Resource Affordability, and Research Evidence

Authors: Xin Ma, Furong Chen, Xinlong Man, Bo Zhang, Yalong Guo, Haolan Lei, Chaochen Cui, Hongyuan Zhang,

Organizations: Northwestern Polytechnical University · Jiangsu Yunmu Intelligent Manufacturing Co., Ltd. · National University of Singapore · Xi’an University of Posts and Telecommunications ·

Abstract

Choosing a robot action model requires evidence about task capability, adaptation cost, deployment requirements, and the reliability of reported conclusions. Success rates alone offer limited guidance to researchers with different computing budgets and goals. We propose RoboSeed, a multidimensional evaluation framework for vision-language-action (VLA) and world-action models (WAM), within a continuing program to train, evaluate, and openly release robot models with verifiable evidence for model selection. The framework defines five dimensions: capability, downstream training affordability, deployment affordability, locally verified research value, and reproduction evidence. It maps empirical measurements and audit criteria to 0-100 profiles using fixed normalization anchors, with balanced, training-constrained, and deployment-constrained preference settings. Interface and protocol checks establish when comparisons are appropriate. Missing measurements remain explicit, incomplete aggregate scores are withheld, and weighted input coverage identifies evidence gaps. We apply the framework to representative VLA and WAM models, drawing on local training, released-weight evaluation, task-level audits, and an evaluation-system study. The results reveal budget-dependent backbone choices, sensitivity to adaptation recipes, and differences between language and visual task interfaces. Reproducible scoring code and machine-readable evidence support extension to additional models. The study provides dimension-level assessments and priorities for further measurement, while incomplete serving traces and heterogeneous protocols preclude overall numerical rankings.

Figures & tables

Dimension	Evaluation basis	Research decision
Capability ( $P$ )	Clean and Randomized closed-loop task outcomes	Does the candidate meet the required task-success level?
Training affordability ( $T$ )	Measured downstream H200 GPU-hours	Can the team afford adaptation and repeated experiments?
Deployment affordability ( $D$ )	Weight size, inference peak memory and p95 latency	Can the policy fit the serving budget and response requirement?
Research value ( $V$ )	Inspected mechanism, comparators, controlled ablations and transfer evidence	Which conclusions can inform further mechanism research?
Reproduction evidence ( $R$ )	Configuration mapping, task records, coverage, training records and seeds	How well can the reported result be checked and reproduced?

Table 1: The proposed evaluation framework connects five assessment dimensions to community research decisions. Chapter 5 defines the scoring rules and reports available scores; deployment measurements remain incomplete in this snapshot.

Figure 1: The RoboSeed evaluation program. (a) Four VLA/WAM families, their native model configurations and benchmarks supply the empirical assessment matrix. (b) Interface alignment, training measurements, closed-loop outcomes and evidence auditing produce the inputs to the proposed framework. (c) Five-dimensional assessment profiles are accompanied by reusable training and evaluation evidence. Chapter 5 defines scoring anchors, preference weights and missing-data rules; Chapters 3 and 4 provide the supporting model studies. The October 28 audit includes completed ABot Qwen3.5-2B and local LiLa evaluations, with the earlier evidence retained where unchanged.

Artifact	Contents	Purpose
Configuration and environment	Dataset versions, backbone sources, code commits, dependencies, seeds, and effective configuration snapshots	Reconstruct execution conditions
Training process	Updates, batch size, learning rates, frozen modules, loss curves, memory, throughput, wall time, and GPU-hours	Assess budgets and convergence
Model weights	Checkpoints, matching configurations, normalization statistics, file sizes, and checksums	Keep weights and execution interfaces consistent
Rollout process	Scenes, instructions, episodes, action chunks, step limits, concurrency, runtime, retries, and exclusion rules	Verify denominators and execution settings
Results and diagnosis	Task-level counts, failure videos, anomalies, diagnostic evidence, and untested hypotheses	Support verification and improvement

Table 2: RoboSeed artifact protocol. Each release should state its evidence completeness.

Figure 2: ABot-M0 backbone substitution and action generation, redrawn from the implementation interfaces. Feature widths are adapted when replacing the VLM; VGGT is frozen during these experiments. Dark blue arrows denote semantic and action flow, light blue arrows denote geometry flow, and the dashed connection denotes observation feedback from execution. State conditioning is used when supplied by the execution interface.

Benchmark	Backbone	Training protocol	Evaluation coverage
RoboTwin	Qwen3.5-0.8B / 2B / 4B / 9B Qwen3-VL-2B / 4B	150k updates Final weights	tasks $\times$ 100 episodes per scene; 2 scenes
LIBERO	Qwen3.5-4B	k checkpoint	suites; 10 tasks per suite; 50 episodes per task
LIBERO-plus	LIBERO-trained Qwen3.5-4B	Zero-shot No extra training	,030 variants, 1 episode each
RoboCasa GR1	Qwen3.5-4B	k checkpoint	tasks $\times$ 50 episodes
CALVIN	Qwen3.5-4B	34k updates Final weights	ABCD $\rightarrow$ D; 1,000 sequences of up to 5 tasks

Table 3: Verified experimental coverage in the current snapshot. Complete scores are derived from original evaluation artifacts.

Backbone	Clean count	Clean	Randomized count	Randomized	Average
Qwen3.5-0.8B	4038/5000	80.76	4025/5000	80.50	80.63
Qwen3-VL-2B	4183/5000	83.66	4195/5000	83.90	83.78
Qwen3-VL-4B	4285/5000	85.70	4267/5000	85.34	85.52
Qwen3.5-2B	4215/5000	84.30	4197/5000	83.94	84.12
Qwen3.5-4B	4294/5000	85.88	4256/5000	85.12	85.50
Qwen3.5-9B	4199/5000	83.98	4184/5000	83.68	83.83

Table 4: ABot-M0 backbone comparison on RoboTwin. Rates are percentages; Average equally weights the two scenes.

Figure 3: Task performance and training cost. Every configuration uses 150k updates and global batch 64. Bars start at zero. Training GPU-hours equal four GPUs multiplied by training wall time; they exclude evaluation and do not indicate equal-FLOPs budgets.

Backbone	Hours	GPU-h	Samples/s	Mean	Recorded max.	Weights (GB)
				Memory / GPU (GiB)
Qwen3.5-0.8B	43.53	174.1	61.3	$\sim$ 52.27	$\sim$ 55.06	5.04
Qwen3-VL-2B	53.51	214.0	49.8	$\sim$ 54.90	$\sim$ 55.30	7.77
Qwen3-VL-4B	72.03	288.1	37.0	$\sim$ 83.26	$\sim$ 83.93	12.58
Qwen3.5-2B	53.23	212.9	50.1	$\sim$ 58.95	$\sim$ 71.01	8.33
Qwen3.5-4B	98.59	394.4	27.0	$\sim$ 142.50	$\sim$ 143.00	13.28

Table 5: RoboTwin training process. Approximate memory readings retain the original records’ GiB labels; uniformly sampled memory traces are unavailable. Throughput is recomputed from updates, batch size, and complete wall time.

Figure 4: Training resources and rollout details. Memory bars use approximate recorded readings. The high Qwen3.5-4B reading is associated with its runtime implementation. Evaluation times retain each run’s actual concurrency settings and are not measurements of single-policy inference latency.

Figure 5: RoboTwin loss comparison and closed-loop outcomes. (a) All six trajectories use ten-observation moving averages on a common logarithmic scale over 0–150k updates. (b) The same smoothed series are enlarged over 100k–150k updates on a shared linear scale. (c) Dots show the mean of 100 raw observations with

140{,}000<\mathrm{step}\leq 150{,}000

; horizontal lines span their empirical 10th–90th percentiles. These ranges describe within-run variation, not confidence intervals or variation across training seeds. (d) Tail means are paired with independently measured, equally weighted Clean/Randomized success rates for all six evaluated backbones. Lower training loss alone does not establish better task performance.

Backbone	Max. workers/GPU	Clean (h)	Randomized (h)	Episodes
Qwen3.5-0.8B	6	4.15	5.52	10,000
Qwen3-VL-2B	6	4.26	4.88	10,000
Qwen3-VL-4B	4	6.38	6.77	10,000
Qwen3.5-2B	6	4.42	9.10	10,000
Qwen3.5-4B	4	8.68	11.31	10,000
Qwen3.5-9B	4	4.51	6.38	10,000

Table 6: Complete RoboTwin rollout details. Every configuration uses four evaluation GPUs. Clean and Randomized times are workflow wall times and may overlap.

Figure 6: Success rates for selected tasks. The columns identify six evaluated configurations, including both 2B and both 4B backbones. Values are percentages. Precise contact, hanging, handover, and stacking tasks exhibit different responses to backbone scale. All 50 task counts appear in Appendix B .

Suite	Success / total	Rate (%)	Success / total	Rate (%)
	LIBERO		LIBERO-plus
Spatial	487/500	97.40	2072/2402	86.26
Object	500/500	100.00	2100/2518	83.40
Goal	488/500	97.60	1953/2591	75.38
Long	460/500	92.00	1902/2519	75.51
Overall micro-average	1935/2000	96.75	8027/10030	80.03

Table 7: ABot-M0 with Qwen3.5-4B on single-arm tasks and perturbations. LIBERO-plus uses the same 40k LIBERO checkpoint without additional training; denominators differ by suite.

Figure 7: Performance by suite on LIBERO and LIBERO-plus, and by perturbation category on LIBERO-plus. LIBERO uses 50 episodes per task; LIBERO-plus uses one per variant. The two benchmarks are not pooled.

Category	Success / total	Rate (%)
Background Textures	1012/1076	94.05
Camera Viewpoints	1050/1599	65.67
Language Instructions	1289/1537	83.86
Light Conditions	1084/1142	94.92
Objects Layout	1206/1525	79.08
Robot Initial States	942/1550	60.77

Table 8: LIBERO-plus perturbation categories, reaggregated from variant-level outcomes and task classifications.

Figure 8: All 24 RoboCasa GR1 Tabletop tasks. The dashed line marks mean success of 47.83%. Equal counts of 50 episodes per task make macro- and micro-averages identical. Figure labels are abbreviated; full task names are retained in the data tables.

Consecutive tasks completed	Sequence count	Rate (%)
$\geq 1$	889/1000	88.9
$\geq 2$	798/1000	79.8
$\geq 3$	721/1000	72.1
$\geq 4$	646/1000	64.6
$\geq 5$	571/1000	57.1
Average successful sequence length (maximum 5)		3.625

Table 9: CALVIN ABCD

\rightarrow

D: successful prefixes for the final 34k Qwen3.5-4B checkpoint. Every row uses the same 1,000 sequences.

Figure 9: CALVIN chain completion and horizontal pushing. The left panel uses all 1,000 sequences as its denominator. The right panel uses actual reached attempts for each push task; the dashed reference aggregates the other 28 task types. Task frequencies and positions differ, so the reference is diagnostic rather than a matched causal comparison.

Task group	Success / attempts	Rate (%)
Horizontal push	196/423	46.34
Push left	133/217	61.29
Push right	63/206	30.58
Other tasks	3429/3631	94.44

Table 10: CALVIN failure groups. Push left and Push right partition Horizontal push; Other tasks form its complement. Rates are conditional on reaching a task.

Figure 10: LingBot-VLA 2.0 reproduction interfaces. Multi-view images and language enter Qwen3-VL-4B. The action stream uses 32 routed experts with top-4 routing and a shared expert. Depth and video teachers supervise predictive queries during training; dashed arrows denote training supervision or environment feedback.

Setting	Observed configuration
Initialization / backbone	Released LingBot foundation / Qwen3-VL-4B plus sparse action experts
Input / native representation	Three cameras; 55 state/action channels
Optimizer / local batch	Muon; micro-batch 32, global batch 96
Adaptation / training GPUs	,000 updates; three H200 GPUs
Warmup / terminal base LR	Ratio 0.0333 / $5\times 10^{-5}$
Action chunk / integration	predicted/executed actions; ten flow steps

Table 11: LingBot-VLA 2.0 reproduction settings. The local run changes the adaptation budget; it does not repeat the foundation pre-train.

Metric	Official weights	Local 50k checkpoint
Clean successes / episodes	/5000	/5000
Clean success rate	.72%	.78%
Randomized success rate	–	–
Completed / skipped tasks	/ 0	/ 0
Local minus official	$-31.94$ percentage points on Clean

Table 12: LingBot-VLA 2.0 closed-loop results on its common local evaluator. Missing Randomized outcomes remain unevaluated.

Metric	Official weights	Local 50k checkpoint
Reference / local sample exposure	.2M (source recipe)	.8M
Measured adaptation wall time	–	:07:18
Measured H200 training GPU-hours	–	.365
Clean evaluation wall time	.683 h	.925 h
Clean evaluation GPU-hours	.733	.700
Primary weight artifact	.504 GB	.504 GB

Table 13: LingBot resource observations. Training time is the final progress interval; evaluation time includes the complete task queue. Artifact GB do not measure resident BF16 VRAM.

Figure 11: LingBot-VLA 2.0 experimental results. (a) Official and local Clean outcomes use 5,000 episodes each. (b) Total and VLA loss come from all 50,000 update records; light traces display every 83rd record and dark traces use a 200-observation moving average. (c) The six lowest local Clean tasks are compared with the official checkpoint under the same within-family evaluator. (d) Full evaluation queue time uses the same four-GPU, 12-slot configuration. Randomized outcomes and serving latency are unavailable.

Task	Official	Local	Difference
place_cans_plasticbox	/100	/100	-87
handover_block	/100	/100	-90
pick_dual_bottles	/100	/100	-79
put_bottles_dustbin	/100	/100	-70
pick_diverse_bottles	/100	/100	-71
lift_pot	/100	/100	-84

Table 14: Ten lowest local LingBot Clean task results. Each count is out of 100; the difference is local minus official in percentage points. All 50 tasks remain in Appendix D .

Figure 12: OpenWAM reproduction interfaces. Frozen Wan VAE and T5 encoding condition a Wan2.2 world stream and ActionDiT action stream coupled by joint self-attention. Training supervises video and action targets; the execution adapter selects 20 physical end-effector channels from the 80-channel representation.

Setting	Observed configuration
World / action modules	Wan2.2-TI2V-5B plus Wan VAE / ActionDiT
Training views / video window	Three views; 33 frames, stride four; $384\times 320$
Native action / unified channels	D end-effector commands / 80D slots
Updates / local global batch	,000 / 64 (four GPUs $\times$ 16)
Optimizer / learning-rate schedule	AdamW; $10^{-4}$ , cosine, warmup ratio 0.05
Training precision / parallelism	BF16 / ZeRO-2 on four H200 NVL GPUs

Table 15: OpenWAM local adaptation and evaluation settings. The released Foundation initialization is inherited.

Metric	Official weights	Local BS64-80K
Clean successes / episodes	/5000	/5000
Clean success rate	.02%	.30%
Randomized successes / episodes	/5000	/5000
Randomized success rate	.66%	.40%
Two-scene mean	.84%	.35%
Local minus official	$-2.49$ percentage points overall

Table 16: OpenWAM closed-loop outcomes. Official and local checkpoints share the adapted OpenWAM evaluator and each cover 10,000 episodes.

Metric	Official weights	Local BS64-80K
Reference / local sample exposure	.376M (source recipe)	.120M
Measured adaptation wall time	–	:31:01
Measured training GPU-hours	–	.068
Clean evaluation wall time	:04:28	:03:02
Randomized evaluation wall time	:05:37	:42:39
Total evaluation GPU-hours	.672	.046

Table 17: OpenWAM resource observations. Downstream training excludes Foundation pre-train; evaluation-device hours assume the four recorded GPUs are allocated throughout each queue.

Figure 13: OpenWAM experimental results. (a) Separate Clean and Randomized rates use 5,000 episodes per scene and checkpoint. (b) Native total, video and action loss use 80,000 selected console observations, every 133rd raw value, and a 200-observation moving average; console losses are rounded and progress counters have a possible one-update offset. (c) Six weakest local Clean tasks retain the same within-family checkpoint comparison. (d) Evaluation wall time covers both scenes on four GPUs and 16 slots; it is not policy inference latency.

Task	Official C/R	Local C/R	Difference
hanging_mug	/54	/34	-13.5
turn_switch	/65	/57	-17.5
open_microwave	/73	/71	-12.5
blocks_ranking_size	/77	/83	+2.0
press_stapler	/94	/91	-7.0
stack_bowls_three	/80	/82	-1.5

Table 18: Ten lowest local OpenWAM Clean tasks. Each C/R entry is a pair of success counts out of 100 per scene; differences are local minus official two-scene means in percentage points.

Figure 14: LiLa-WAM reproduction interfaces. A frozen DINOv3 encoder provides head-camera features. A task name selects precomputed visual transition tokens; the inspected policy does not use the instruction argument. Compact future-feature prediction and action generation share the task condition. The controller executes 16 of 32 predicted actions with spline smoothing.

Setting	Observed configuration
Visual encoder / policy scale	Frozen DINOv3 ViT-L/16 / approximately 0.5B (source)
Task specification	Task-name-selected precomputed visual transition vector
Input view / size	Head camera / $320\times 240$
Action / internal state channels	/ 16
Prediction / execution horizon	/ 16 actions
Integration / smoothing	Ten flow steps / B-spline

Table 19: LiLa-WAM settings shared by the official-weight and local stage-2 evaluations. Its visual task interface is explicit.

Metric	Official weights	Local stage-2 epoch 4
Clean successes / episodes	/250	/250
Clean success rate	.60%	.80%
Working Wilson 95% interval	.20%–92.80%	.29%–92.14%
Randomized success rate	–	–
Tasks with five successes	/50	/50
Tasks below four successes	/50	/50

Table 20: LiLa-WAM official/local Clean results. Each column represents a complete 250-episode evaluation, with task-name-selected visual conditions.

Training prefix	Saved epochs	Epoch timers (h)	Last epoch total loss
lila_stage1		.35	.0689
lila_stage2		.59	.0641

Table 21: LiLa local training prefixes. Summed epoch timers exclude startup, scans, checkpoint export and partial epochs. Local rollout is now verified, but the epoch timers lack an independent device-allocation trace and are not inserted into the H200 cost score.

Figure 15: LiLa-WAM experimental results. (a) Official/local task-outcome distributions over 50 tasks with five episodes each; compact text reports Clean success, 89.60%/88.80%. (b,c) Local action-MSE and future-feature curves cover saved stage-1 epochs 1–12 and stage-2 epochs 1–4, with separate update axes. Light curves subsample every 39th/13th observation; dark curves use 20-observation moving averages. The local rollout uses the stage-2 epoch-4 checkpoint reached by these training prefixes.

Task	Official / 5	Local / 5	Gap (pp)
place_a2b_left	/5	/5	-40
place_dual_shoes	/5	/5	-60
move_stapler_pad	/5	/5	-20
open_microwave	/5	/5	-40
place_mouse_pad	/5	/5	-20
click_bell	/5	/5	-20

Table 22: Ten lowest local LiLa-WAM Clean task results, retaining the official-weight comparator. A single extra success changes a task’s displayed rate by 20 points; these rows should not be treated as precise task rankings.

Family	Impl.	Comp.	Isolate	Xfer.	Evidence and $V$
ABot-M0					Backbone budget comparison; same LIBERO checkpoint on LIBERO-plus. $V=75$ .
LingBot 2.0					Official/local recipe gap, common local evaluator; no isolated MoE/teacher effect. $V=50$ .
OpenWAM					Official/local adaptation gap on the same execution path; randomized data used in training. $V=50$ .
LiLa-WAM					VTT inspected; official/local stage-2 comparison on a documented common evaluator. $V=50$ .

Table 23: Research-value decisions and their local evidence. A one denotes a satisfied criterion. No local mechanism-isolation experiment meets the stated rule.

Checkpoint	$P$	$T$	$D$	$V$	$R$	$S_{\mathrm{bal}}$	$q$
ABot Q3.5-0.8B	.6	.9	–	.0	.0	–	%
ABot Q3-VL-2B	.8	.5	–	.0	.0	–	%
ABot Q3-VL-4B	.5	.7	–	.0	.0	–	%
ABot Q3.5-2B	.1	.6	–	.0	.0	–	%
ABot Q3.5-4B	.5	.4	–	.0	.0	–	%
ABot Q3.5-9B	.8	.2	–	.0	.0	–	%

Table 24: Complete assessment of the 12 checkpoint units. Q3-VL and Q3.5 abbreviate the backbones.

P

is two-scene capability,

T

training affordability,

D

deployment affordability,

V

research value, and

R

reproduction evidence. A dash denotes unavailable measurements or an incomplete composite.

q

reports available input weight. Measured scores are rounded only for display.

Figure 16: Cost–capability observations and measurement coverage. (a) Only local checkpoints with measured downstream H200 intervals and observed Clean outcomes are plotted. Different initialization, interfaces and protocols prevent a causal scaling interpretation. (b) Bars show available input weight under the balanced profile for all 12 units. Coverage is not an aggregate model score; incomplete scores are shown as dashes in the tables.

Profile	Weights	Use case
Balanced	$(.30,.20,.20,.15,.15)$	General reproduction decision
Training constrained	$(.30,.35,.10,.10,.15)$	Repeated adaptation with limited compute
Deployment constrained	$(.30,.10,.35,.10,.15)$	Serving memory and latency dominate

Table 25: Preference sensitivity. Weights follow the order

(P,T,D,V,R)

and sum to one. These are explicit decision preferences rather than empirically learned constants.

Checkpoint	Balanced	Training constrained	Deployment constrained
ABot Q3.5-0.8B	– / 84%	– / 92%	– / 72%
ABot Q3-VL-2B	– / 84%	– / 92%	– / 72%
ABot Q3-VL-4B	– / 84%	– / 92%	– / 72%
ABot Q3.5-2B	– / 84%	– / 92%	– / 72%
ABot Q3.5-4B	– / 84%	– / 92%	– / 72%
ABot Q3.5-9B	– / 84%	– / 92%	– / 72%

Table 26: Aggregate availability and weighted measurement coverage under each preference profile. Each cell is aggregate score /

q

. The aggregate remains a dash until all required inputs are available.

Setting	Observed configuration
Policy / accelerator	ABot/PX-M0 Qwen3-VL-2B / H200 NVL
Inputs / action interface	Three cameras / 14 absolute joint channels
Chunk / numerical path	actions / BF16, eager attention, ten DDIM steps
Single-GPU timed workload	Randomized tasks $\times$ ten episodes
Original / shared model instances	One per task process / one per GPU
Primary concurrency / batch cap	Six- and eight-worker comparisons / one

Table 27: PX harness systems-study settings. These experiments benchmark an existing VLA policy rather than training another action model.

Entry	Successes	Executed steps	End-to-end seconds
Original PX	/120	,692	.949
Harness W6/B1	/120	,352	.580
Harness W8/B1	/120	,175	.330

Table 28: Single-GPU systems comparison on 12 Randomized tasks, ten episodes each. End-to-end time includes model startup. Historical PX time is reconstructed from task-directory and log timestamps. Seed acceptance and instruction sampling differ across frameworks, so this is an approximate workload comparison.

Deployment	Wall seconds	GPU-hours	Episodes/min
Co-located W8/B1	.641	.279	.18
Co-located W8/B1 per GPU	.499	.344	.62
Split model/simulation W16/B1	.538	.453	.84

Table 29: Harness GPU-allocation experiments on the separate 120-episode subset. Evaluation-only times exclude startup and must not be mixed with Table 28 ’s end-to-end times.

Entry	Successes	Steps	End-to-end (s)	GPU-hours	Episodes/min
Original PX / W6	/500	,145	.99	.326	.17
Harness / W6	/500	,487	.52	.235	.50
Harness / W16	/500	,542	.54	.008	.53

Table 30: Updated two-GPU systems comparison over all 50 Randomized tasks. W denotes workers per GPU; harness batch cap is one. Both GPUs are allocated throughout each end-to-end interval. Seed acceptance and instruction sampling differ across execution paths.

Deployment	Successes	Steps	Rollout (s)	End-to-end (s)
Centralized (1+3 GPUs)	/120	,918	.435	.22
Centralized (1+3 GPUs)	/120	,856	.496	.27
Co-located (4 GPUs)	/120	,997	.647	.01

Table 31: Four-GPU topology and batch-cap observations on the matched 120-episode input manifest. B is the configured maximum model batch, not a measured mean batch size. Rollout-only and startup-inclusive times are separated.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	RoboTwin	LIBERO / LIBERO-plus	RoboCasa GR1
Training updates	k	k / no extra training	k
Global batch
State / action dimensions	/ 14	/ 14 in config; state input to be verified	/ 29
Training action horizon
Integration steps
Training seed

Appendix

Table 32: Key interfaces and evaluation settings. LIBERO-plus entries describe this script snapshot rather than a universal protocol for other implementations.

Evidence	Current status	Follow-up
RoboTwin success	Original results checked against final log counters; 50 tasks per scene	Multiple training seeds and external protocol checks
Configurations and final weights	Six evaluated RoboTwin configurations and dataset-specific final weights; file sizes measured	Versioned release configurations and weight checksums
Loss convergence	Observations extracted from console logs	Standard exports of raw metrics
Training memory	Approximate recorded mean and maximum readings	Raw traces, units, and capacity checks
Rollout time and concurrency	Workflow times and concurrency limits recorded	Policy latency, task time, and worker activity
LIBERO / RoboCasa counts	Reconstructed video verdicts and task denominators	Independent episode tables and exception logs

Appendix

Table 33: ABot evidence completeness after the October 28 refresh. Additional method results are audited separately in Appendix D .

	Q3.5-0.8B		Q3-2B		Q3-4B		Q3.5-2B		Q3.5-4B		Q3.5-9B
Task	C	R	C	R	C	R	C	R	C	R	C	R
adjust_bottle	100	100	100	100	100	100	100	99	100	99	100	100
beat_block_hammer	83	82	94	88	94	92	86	83	89	83	96	87
blocks_ranking_rgb	91	93	93	93	92	94	90	84	98	91	94	90
blocks_ranking_size	57	59	67	66	66	65	54	74	74	69	72	66
click_alarmclock	2	5	5	6	10	29	6	15	22	13	3	9

Appendix

Table 34: Success counts for all 50 RoboTwin tasks.

Setting	Recorded value
Training / evaluation environments	ABCD / D
Demonstrations / frames	,870 / 1,071,743; converted LeRobot v2.1
Updates / batch / seed	,000 / 64 / 42
Frozen encoder	VGGT spatial module
Action expert / integration steps	DiT-B, 16 layers / configured 4
Training / executed action horizon	/ 8 in the inspected shared client

Appendix

Table 35: CALVIN reproduction details from the saved configuration and inspected evaluation interfaces.

Figure 17: Action loss from the final CALVIN training run. All 340 observations at 100-update intervals are retained; the moving average uses ten observations. Nine earlier observations from preliminary runs in the same console file are excluded from this curve and accounted for in the audit. Final action loss is 0.02949.

Task	Success / attempts	Rate (%)
close_drawer	182/182	100.00
lift_blue_block_drawer	19/19	100.00
lift_blue_block_slider	100/123	81.30
lift_blue_block_table	143/143	100.00
lift_pink_block_drawer	11/11	100.00
lift_pink_block_slider	109/128	85.16

Appendix

Table 36: Complete CALVIN conditional task counts for the selected final evaluation.

Task	L-O	L-L	W-O (C/R)	W-L (C/R)	Li-O	Li-L
adjust_bottle	100	99	100/99	96/99	5/5	5/5
beat_block_hammer	91	45	98/99	100/97	4/5	5/5
blocks_ranking_rgb	92	58	100/96	98/100	5/5	5/5
blocks_ranking_size	76	36	78/77	76/83	4/5	5/5
click_alarmclock	64	62	97/100	96/99	5/5	5/5
click_bell	58	18	100/100	100/100	5/5	4/5

Appendix

Table 37: Complete task counts for the new model reproductions.

Robot Foundation Models

Robot Foundation Models are pretrained systems that combine perception, reasoning, and action to support diverse tasks and robot embodiments.

See all 138 papers

Do World Action Models Generalize Better than VLAs? A Robustness Study
Mar 23, 2026Zhanguang Zhang, Zhiyuan Li, Behnam Rahmati +11World Action Models Robustness of VLA Models
Huawei Technologies · University of Toronto
StarVLA- $α$ : Reducing Complexity in Vision-Language-Action Systems
Apr 13, 2026Jinhui Ye, Ning Gao, Senqiao Yang +7Robot Foundation Models Robotic Manipulation
HKUST · XJTU · CUHK +3
IndustrialVLA-Bench: A Traceable Multi-Axis Evaluation of Open Robot Policy Models
Sep 22, 2026Yiqi Wang, Zhifeng Rao, Jiaqi Zhang +7Robotic Policy Evaluation Vision-Language-Action Models
Griffith University, Queensland, Australia · Southern University of Science and Technology, Shenzhen, China · Jiangsu University, Zhenjiang, China +4

Mar 23, 2026J/K move · Enter open · S save

Do World Action Models Generalize Better than VLAs? A Robustness Study

Zhanguang Zhang, Zhiyuan Li, Behnam Rahmati, Rui Heng Yang+10

Huawei Technologies · University of Toronto

World Action Models Robustness of VLA Models World Model Evaluation

Open paper AskPDF arXiv

Robot action planning in the real world is challenging as it requires not only understanding the current state of the environment but also predicting how it will evolve in response to actions. Vision-language-action (VLA), which repurpose large-scale vision-language models for robot action generation using action experts, have achieved notable success across a variety of robotic tasks. Nevertheless, their performance remains constrained by the scope of their training data, exhibiting limited generalization to unseen scenarios and vulnerability to diverse contextual perturbations. More recently, world models have been revisited as an alternative to VLAs. These models, referred to as world action models (WAMs), are built upon world models that are trained on large corpora of video data to predict future states. With minor adaptations, their latent representation can be decoded into robot actions. It has been suggested that their explicit dynamic prediction capacity, combined with spatiotemporal priors acquired from web-scale video pretraining, enables WAMs to generalize more effectively than VLAs. In this paper, we conduct a comparative study of prominent state-of-the-art VLA policies and recently released WAMs. We evaluate their performance on the LIBERO-Plus and RoboTwin 2.0-Plus benchmarks under various visual and language perturbations. Our results show that WAMs achieve strong robustness, with LingBot-VA reaching 74.2% success rate on RoboTwin 2.0-Plus and Cosmos-Policy achieving 82.2% on LIBERO-Plus. While VLAs such as

π_{0.5}

can achieve comparable robustness on certain tasks, they typically require extensive training with diverse robotic datasets and varied learning objectives. The evaluation code for the RoboTwin2.0-Plus benchmark is available at: https://robot-robustness.github.io/RoboTwin2.0-Plus/.

Abstract

Robot Foundation Models

Explore similar work

Do World Action Models Generalize Better than VLAs? A Robustness Study

StarVLA-ααα: Reducing Complexity in Vision-Language-Action Systems

IndustrialVLA-Bench: A Traceable Multi-Axis Evaluation of Open Robot Policy Models

StarVLA- $α$ : Reducing Complexity in Vision-Language-Action Systems