ORDER: A Fictitious-World Benchmark for Domain-Adaptive Embodied AI

Sai Krishna Reddy Sathi; Anuj Tiwari

cs.ROSep 13, 2026

ORDER: A Fictitious-World Benchmark for Domain-Adaptive Embodied AI

Authors: Sai Krishna Reddy Sathi, Anuj Tiwari

Organizations: Indian Institute of Technology Madras, India

Abstract

Adapting language models to new domains via continual pre-training raises a basic evaluation problem: if the training corpus overlaps with what the model already knows, performance gains cannot be cleanly attributed to new learning rather than pre-existing knowledge. This matters most for knowledge-intensive, task-light (KHTL) robot deployments - pharmaceutical dispensing, hazardous-material handling, facility-specific protocols, where the physical task is simple but the governing rules are proprietary and safety-critical, and where extensive live testing is costly or unsafe. We introduce ORDER (Ontology-driven Decision-making for Embodied Reasoning), a benchmark built on a fictitious world: a 342,069-token synthetic corpus defining a self-consistent physics that cannot appear in any model's pre-training data. ORDER pairs a 500-question knowledge test (ORDER-BENCH) with a harder compositional task, ORDER-SPATIAL: ordering objects for safe manipulation across both familiar and entirely novel scenes. GPT-4.1 without adaptation scores below chance on ORDER-SPATIAL (Kendall's tau = 0.441), showing its priors actively conflict with the invented physics. After continual pre-training, small models improve substantially on both familiar and novel scenes alike evidence of genuine world-model induction rather than memorization. We then carry this through to a robot pipeline: models that answer the knowledge test well often cannot produce valid, executable plans without a further skill-adaptation stage, after which small, fully offline models outperform GPT-4.1 even when GPT-4.1 is given retrieval access to the same rules (Kendall's tau = 0.848 vs. 0.606), on a full perception-to-execution loop demonstrated on a simulated iiwa7 arm with human-in-the-loop correction. Throughout, ORDER-SPATIAL performance, not knowledge-test accuracy is what predicts real plan quality.

Figures & tables

Panel of Figure 1 — Fig. 2 : The Order framework. Left: the Order-World knowledge pyramid; all upstream context is injected into downstream generation calls. Centre: the two-stage adaptation pipeline, instantiated across seven K-LoRA variants for three SLM families and then extended with S-LoRA skill fine-tuning to produce Domain-BT-LMs (Sections VII – VIII ). Right: Order-Bench , comprising the MCQ component and the Order-Spatial seen/unseen ordering task, with the metric suite of Section IV-C .

Section	Tokens	Role
Master Ontology (0)	67,389	Physics foundation
Axioms (1)	11,886	Global constraints
Counterfactuals (2)	16,336	Constraint stress-tests
QnA (3)	47,836	Format diversification
Spatial Configs (4)	100,214	Scenes & priority orders
Paraphrases (5)	98,408	Format diversification

TABLE I : Order-World token distribution (

\mathcal{D}=342{,}069

tokens).

Category	$N$	$w_{c}$	$w_{c}N$
Intrinsic Semantics	10	1.0	10.0
QnA	30	1.0	30.0
Axioms	60	1.5	90.0
Spatial (MCQ)	72	2.0	144.0
Pairwise	150	2.5	375.0
Counterfactuals	111	2.5	277.5

TABLE II : Order-Bench MCQ composition and Bloom weights. Total weight mass: 1127.5; random baseline: 25%.

Variant	Description
Normal	Packed causal LM over all $\mathcal{D}$ sections; all-token CLM loss.
Overfit	Normal extended to near-zero training loss ( $\mathcal{L}\!\approx\!0$ ).
DKL *	Base $\,{\leftarrow}\,$ instruct embedding replacement before CPT [ 43 ] .
RAFT	Retrieval-augmented fine-tuning with hard-negative distractors [ 44 ] .
SSR-CLM	Packed CLM on non-spatial entries; loss-masked SSR (Structured Scene Reasoning) on spatial entries (loss on assistant tokens only, preventing format overfitting).
Unified	SSR-CLM $+$ RAFT $+$ DPO pairs (chosen = oracle-faithful, rejected = parametric-bias) in one training script.

TABLE III : K-LoRA knowledge-ingestion variants.

Model	PPL $\downarrow$	$S_{\mathrm{agg}}$	Sp.MCQ
GPT-4.1	NA	70.4	70.8
M-7B	13.72	48.9	65.3
L-8B	21.79	34.7	44.4
Q-4B	24.22	57.0	69.4

TABLE IV : Closed-book zero-shot performance (PPL on a held-out split of

\mathcal{D}

). Sp.MCQ

\,{\gg}\,

S_{\mathrm{agg}}

for all SLMs reflects common-sense ordering priors, not domain knowledge; the gap closes only after CPT (Appendix G ).

Category	GPT-4.1	M-7B	L-8B	Q-4B
Axioms	100.0	96.67	98.33	98.33
Intrinsic	100.0	100.0	100.0	100.0
Pairwise	100.0	98.67	96.67	98.67
Higher-Order	100.0	100.0	97.01	100.0
Counterfact.	98.2	93.69	90.09	96.4
QnA	100.0	100.0	100.0	100.0

TABLE V : Order-Bench MCQ with retrieval (global hybrid BM25

+

dense pipeline, Appendix A-A ). Non-spatial categories saturate (

\geq\!90\%

), validating corpus coherence. Spatial remains harder for smaller models.

	Mistral-7B (M-7B)				LLaMA-3.1-8B (L-8B)				Qwen3-4B (Q-4B)
	Seen (70)		Unseen (183)		Seen (70)		Unseen (183)		Seen (70)		Unseen (183)
Variant	$\tau_{\mathrm{order}}$	par	$\tau_{\mathrm{order}}$	par	$\tau_{\mathrm{order}}$	par	$\tau_{\mathrm{order}}$	par	$\tau_{\mathrm{order}}$	par	$\tau_{\mathrm{order}}$	par
GPT-4.1 (reference)	family-independent: $\tau^{s}_{\mathrm{order}}=0.441$ , $\textsc{par}^{s}=0.301$ ; $\tau^{u}_{\mathrm{order}}=0.415$ , $\textsc{par}^{u}=0.248$
Instruct	0.528	0.368	0.509	0.347	0.513	0.366	0.554	0.345	0.582	0.394	0.597	0.412
Normal	0.49	0.38	0.55	0.36	0.51	0.36	0.54	0.35	0.64	0.44	0.65	0.44
Overfit	0.43	0.36	0.55	0.35	0.54	0.37	0.54	0.35	0.63	0.45	0.64	0.45

TABLE VI : Order-Spatial :

\tau_{\mathrm{order}}

and par for all K-LoRA variants across the three SLM families, on the 70 seen and 183 unseen scenes. Best per family in bold . The random baseline is

\tau_{\mathrm{order}}=0.500

on both splits (Appendix I ); par floors are

0.333

seen and

0.309

unseen. Off-the-shelf Qwen3 already exceeds GPT-4.1; Mistral-RAFT and LLaMA-Unified show inverted seen/unseen gaps.

Fam.	KI	Instruct		Post-KI		Gain
		$\tau^{s}_{\mathrm{order}}$	$\tau^{u}_{\mathrm{order}}$	$\tau^{s}_{\mathrm{order}}$	$\tau^{u}_{\mathrm{order}}$	$\Delta^{s}$	$\Delta^{u}$
M-7B	RAFT	0.528	0.509	0.71	0.73	$+$ 0.18	$+$ 0.21
L-8B	Unified	0.513	0.554	0.77	0.81	$+$ 0.26	$+$ 0.26
Q-4B	Combined	0.582	0.597	0.79	0.74	$+$ 0.21	$+$ 0.14

TABLE VII : Absolute

\tau_{\mathrm{order}}

improvement to the best K-LoRA variant per family. Both seen and unseen improve substantially, providing evidence against memorisation (

H_{\mathrm{A}}

Family	Variant	svr	$\tau_{\mathrm{order}}$	sdhr
L-8B	Instruct	12.2	0.057	53.4
	Normal	70.7	0.524	72.4
	RAFT	31.7	0.216	87.9
	Unified	4.9	0.037	53.4
M-7B	Instruct	2.4	0.019	25.9
	Normal	4.9	0.035	44.8

TABLE VIII : Zero-shot BT generation of KI-models (82 novel samples, no fine-tuning). Near-zero svr for Mistral and LLaMA variants confirms that domain knowledge and BT-generation skill are separable capabilities. GPT-4.1 vanilla is included as a frontier reference.

Fam.	KI Var.	svr	$\tau_{\mathrm{order}}$	par	P-PAR	sdhr
L-8B	Inst. $+$ S-LoRA	100.0	0.830	0.629	36	100.0
	Normal	100.0	0.799	0.675	43	100.0
	Overfit	100.0	0.813	0.702	46	100.0
	DKL	100.0	0.815	0.718	46	100.0
	RAFT	100.0	0.833	0.743	50	100.0
	Unified	98.8	0.848	0.724	49	98.3

TABLE IX : S-LoRA results - Instruct mounting (82 novel samples). Best per family in bold . P-PAR:

n

/82 exactly correct sequences.

Fam.	KI Var.	svr	$\tau_{\mathrm{order}}$	par	P-PAR	sdhr
L-8B	Normal	100.0	0.781	0.649	40	100.0
	Overfit	100.0	0.848	0.736	46	100.0
	DKL	100.0	0.831	0.697	44	96.6
	RAFT	100.0	0.767	0.574	32	98.3
	SSR-CLM	97.6	0.844	0.705	45	91.4
	Unified	75.6	0.601	0.497	31	74.1

TABLE X : S-LoRA results - Matched mounting (82 novel samples). Matching the S-LoRA initialisation to the KI-variant benefits LLaMA (Overfit/SSR-CLM) and uniformly stabilises Qwen; most Mistral KI-variants collapse.

Setup	svr	$\tau_{\mathrm{order}}$
Closed-book best (RAFT, Instruct)	100.0%	0.833
SLM $+$ full assembled context (unadapted)	0.0%	0.000
SLM $+$ two-stage-cs (adapted)	100.0%	0.798

TABLE XI : Context-hijacking failure and recovery (LLaMA-3.1-8B). Injecting the full assembled ontology context directly into the BT-generation call collapses svr from 100% to 0% in the fine-tuned SLM; the two-stage pipeline with a condensed (

\leq

800-token) priority-reasoning call restores functional output, though still below the closed-book peak.

Panel of Figure 13 — Fig. 3 : The two structured retrieval pipelines. Both delegate Stage 1 priority reasoning to a condensed SLM call ( $\leq$ 800 tokens) rather than injecting the full assembled context into the generation call; the CS variant additionally forces an XML prefix to prevent prose-opening failures. Both share the same weighted retrieval index and no-spatial few-shot conditioning.

LLaMA-3.1-8B
Variant / Mount	Pipeline	svr	$\tau_{\mathrm{order}}$	par	CB- $\tau_{\mathrm{order}}$
Base-Inst / Inst.	two-stage	100%	0.819	0.686	0.830
Base-Inst / Inst.	two-stage-cs	100%	0.798	0.663	0.830
Overfit / Match.	two-stage	100%	0.811	0.713	0.848
Overfit / Match.	two-stage-cs	99%	0.801	0.714	0.848
12-Comb. / Match.	two-stage	100%	0.824	0.719	0.806

TABLE XII : Structured retrieval results (82 novel samples). CB-

\tau_{\mathrm{order}}

is the closed-book reference for the same (variant, mounting) pair. Retrieval revives variants that collapsed closed-book, but does not reach the closed-book peak of

0.848

Fig. 4 : End-to-end pipeline snapshots on the iiwa7 in Gazebo. Left: the direct LLM-generated BT XML and execution summary for Configuration #88, given the VLM $\{O,C,P,R\}$ JSON and the NL user input; the tree commits two errors, placing the grey cylinder at priority 1 (correct: priority 3) and emitting the blue disc as a standalone top-level sequence instead of inside the required Fallback accessibility guard. Centre: the corrected BT XML after a single HITL operator re-prompt, in which EnsureAccessible_brown_cube becomes the first child of MainTask and the blue disc appears only as the conditional blocker-removal step inside it. Top right: the initial Gazebo scene for Configuration #88. Bottom right: the iiwa7 arm grasping the first-priority object (the blue disc, the stacking blocker) and placing it in the blue colour bin.

Panel of Figure 16 — Fig. 5 : Worked HITL correction example (Configuration #88). Top: the scene, with the blue disc stacked on the brown cube (blocking access), and the Order priority order. Left: the LLM-generated tree commits two errors: (i) the grey cylinder is placed at P1 instead of P3, and (ii) the blue disc appears as a standalone top-level Sequence rather than inside the required Fallback guard, missing the CheckStackSafety structure entirely. Right: the gold tree places EnsureAccessible_brown_cube as the first Fallback child of MainTask ; the blue disc appears only inside it as the conditional blocker-removal step, and the remaining three children follow the correct Order -derived priority order. Bottom: the single operator correction prompt that drives the Domain-BT-LM to regenerate the gold tree in one pass. The operator either approves the BT with ‘y’ or starts with ‘n’ followed by the correction prompt.

Fam.	KI / S-LoRA	$\tau^{u}_{\mathrm{order}}$	svr	sdhr	$\tau_{\mathrm{order}}$	par	P-PAR
GPT-4.1 Vanilla		0.44	96.3	100	0.480	0.420	22
GPT-4.1 $+$ two-stage-cs		-	-	-	0.606	-	10
LLaMA-3.1-8B - Instruct S-LoRA (closed-book)
L-8B	Unified / Inst.	0.81	98.8	98.3	0.848	0.724	49
L-8B	RAFT / Inst.	0.60	100	100	0.833	0.743	50
L-8B	DKL / Inst.	0.58	100	96.6	0.815	0.718	46

TABLE XIII : Domain-BT-LMs vs. GPT-4.1 (82 novel samples). All Domain-BT-LM results are closed-book (no retrieval at inference); the GPT-4.1 reference uses its best structured pipeline (two-stage-cs).

\tau^{u}_{\mathrm{order}}

is the Order-Spatial unseen concordance from Section VI-E (random baseline

=0.500

), included to trace the knowledge-to-action chain. † SSR-CLM collapses under Instruct mounting (

\textsc{svr}=0

); the Matched result is shown. ‡ Mistral-RAFT/Inst. is the best Mistral result overall.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Knowledge Type	Weight	Order
Counterfactual constraints	1.00	1st
Pairwise inter-object (contact)	0.90	2nd
Dominance axioms	0.85	3rd
Higher-order emergent regimes	0.80	4th
Pairwise intra-object	0.70	5th
Intrinsic primitive semantics	0.55	6th

Appendix

TABLE XIV : Knowledge-type weighting used for context assembly.

	M-7B		L-8B		Q-4B
Variant	PPL	Loss	PPL	Loss	PPL	Loss
Normal	3.82	0.230	7.22	0.820	8.89	1.110
Overfit	3.10	0.021	6.04	0.016	6.79	0.059
DKL	2.07	0.027	4.72	0.015	6.06	0.085
RAFT	3.44	0.823	5.27	1.660	6.53	2.650
SSR-CLM	1.52	0.003	2.52	0.002	4.62	0.001

Appendix

TABLE XV : PPL and training loss for all K-LoRA variants. SSR-CLM achieves the lowest PPL but dissociates from spatial performance. Mistral-Unified collapses silently.

	Seen ( $n=70$ )		Unseen ( $n=183$ )
Metric	Mean	Std	Mean	Std	$t$
Structural
Num. objects	3.29	0.97	3.53	0.97	$-1.76$
Unique primitives	9.53	2.06	9.63	2.47	$-0.34$
Spatial relations	2.29	0.97	2.45	0.98	$-1.19$
Physics (activation counts)

Appendix

TABLE XVI : Complexity distribution across Order-Spatial splits. Seen and unseen are statistically comparable across all axes (Welch

t

-tests: all

|t|<2

p>0.05

). The mild four-object skew in unseen (

30.6\%

vs.

18.6\%

) slightly disfavours unseen performance.

Family	Variant	$\hat{\tau}_{s}$	$z_{s}$	$\hat{\tau}_{u}$	$z_{u}$
M-7B	RAFT	0.710	$+3.51^{***}$	0.730	$+6.20^{***}$
L-8B	Unified	0.770	$+4.52^{***}$	0.810	$+8.39^{***}$
Q-4B	Combined	0.790	$+4.85^{***}$	0.740	$+6.49^{***}$
GPT-4.1	Vanilla	0.441	$-0.99$	0.415	$-2.30^{*}$

Appendix

TABLE XVII : One-sided

z

-test results.

{}^{*}p<0.05

{}^{**}p<0.01

{}^{***}p<0.001

Variant	Intr.	QnA	Axioms	Pair	CFact	HO	Spat.	$S_{\mathrm{agg}}$
Normal	80.0	60.0	58.3	62.0	64.0	53.7	69.4	61.8
Overfit	70.0	70.0	45.0	68.0	64.9	50.8	63.9	61.9
DKL	80.0	66.7	58.3	70.0	65.8	56.7	52.8	63.5
RAFT	70.0	36.7	25.0	54.0	36.9	41.8	44.4	43.8
SSR-CLM		66.7	61.7	69.3	72.1	77.6	43.1	67.7
Unified	10.0	0.0	1.7	3.3	1.8	0.0	0.0	1.8

Appendix

TABLE XVIII : Mistral-7B-Instruct-v0.3 - section-wise MCQ (%).

Variant	Intr.	QnA	Axioms	Pair	CFact	HO	Spat.	$S_{\mathrm{agg}}$
Normal	60.0	30.0	48.3	43.3	48.7	49.3	36.1	45.0
Overfit	80.0	66.7	63.3	59.3	61.3	52.2	45.8	57.5
DKL	90.0	83.3	58.3	78.0	58.6	59.7	61.1	66.5
RAFT	80.0	33.3	40.0	42.7	52.3	46.3	55.6	47.2
SSR-CLM		90.0	88.3	84.0	79.3	70.1	65.3	78.6
Unified	70.0	83.3	60.0	79.3	70.3	71.6	68.1	72.8

Appendix

TABLE XIX : LLaMA-3.1-8B-Instruct - section-wise MCQ (%).

Variant	Intr.	QnA	Axioms	Pair	CFact	HO	Spat.	$S_{\mathrm{agg}}$
Normal	80.0	73.3	83.3	74.0	52.3	55.2	73.6	66.0
Overfit	90.0	83.3	86.7	81.3	56.8	73.1	69.4	72.9
DKL	90.0	83.3	98.3	94.0	75.7	98.5	62.5	86.3
RAFT	90.0	63.3	71.7	63.3	47.7	56.7	59.7	58.8
SSR-CLM		83.3		94.0	78.4	95.5	70.8	87.5
Unified		93.3	90.0	89.3	70.3	91.0	80.6	84.1

Appendix

TABLE XX : Qwen3-4B-Instruct-2507 (thinking mode) - section-wise MCQ (%).

Variant	MMLU-Pro	GSM8K	BBH	IFEval
Instruct	37.5	84.4	51.0	75.0
Normal	33.8	74.5	50.6	56.8
Overfit	34.3	76.3	50.7	57.9
DKL	36.6	82.6	50.4	73.2
RAFT	36.2	83.4	50.3	74.5
SSR-CLM	33.1	76.2	50.6	62.3

Appendix

TABLE XXI : General benchmark retention: LLaMA-3.1-8B-Instruct.

Variant	MMLU-Pro	GSM8K	BBH	IFEval
Instruct	30.6	51.9	44.9	47.9
Normal	26.9	52.5	43.4	44.9
Overfit	27.2	48.9	42.6	46.8
DKL	27.6	48.3	43.1	43.8
RAFT	28.3	50.7	44.9	47.1
SSR-CLM	25.7	33.0	41.2	38.3

Appendix

TABLE XXII : General benchmark retention: Mistral-7B-Instruct-v0.3.

Variant	MMLU-Pro	GSM8K	BBH	IFEval
Instruct	43.9	77.6	54.6	82.4
Normal	43.4	86.4	55.9	81.3
Overfit	43.8	82.4	54.9	79.3
DKL	38.9	74.0	50.8	79.8
RAFT	38.6	73.6	51.4	77.2
SSR-CLM	39.2	76.5	50.8	79.6

Appendix

TABLE XXIII : General benchmark retention: Qwen3-4B-Instruct-2507.

Family	Variant	$\Delta S_{\mathrm{agg}}$	$\Delta\tau^{u}_{\mathrm{order}}$	$\Delta\overline{\mathrm{Gen}}$ $\downarrow$	Profile
L-8B	Unified ⋆	$+38.1$	$\mathbf{+0.26}$	$2.2$	best spatial
L-8B	SSR-CLM	$+43.9$	$+0.19$	$6.4$	best MCQ
L-8B	Combined	$+43.4$	$+0.15$	$8.4$	high risk
M-7B	RAFT ⋆	$-5.1$	$+0.21$	$1.1$	spatial $+$ min. forg.
M-7B	DKL ⋆	$+14.6$	$+0.18$	$4.1$	factual $+$ spatial
M-7B	SSR-CLM	$+18.8$	$+0.00$	$9.3$	MCQ only

Appendix

TABLE XXIV : Retention–domain trade-off summary per family.

\Delta S_{\mathrm{agg}}

and

\Delta\tau^{u}_{\mathrm{order}}

are improvements over the instruct baseline;

\Delta\overline{\mathrm{Gen}}

is the mean absolute drop across the four general benchmarks (0 if improved; lower is better). ⋆ Pareto-recommended: strong domain gains with low forgetting cost.

Panel of Figure 29 — Fig. 6 : The perception layer is a fixed, domain-agnostic prompt over the visual grammar of Eq. 1 . All interaction semantics live in Order-World , not in the VLM, which is why the same perception module serves any KHTL domain expressed over these primitives.

Embodied AI

Embodied AI studies agents that perceive, reason, and act through bodies in physical or simulated environments.

See all 255 papers

Jun 23, 2026

Qwen-AgentWorld: Language World Models for General Agents

A world model predicts environment dynamics based on current observations and actions, serving as a core cognitive mechanism for reasoning and planning. In this work, we investigate how world modeling based on language models can further push the boundaries of general agents. (i) We first focus on building foundation models for agentic environment simulation. We introduce Qwen-AgentWorld-35B-A3B and Qwen-AgentWorld-397B-A17B, the first language world models capable of simulating agentic environments covering 7 domains via long chain-of-thought reasoning. Leveraging more than 10M environment interaction trajectories of 7 domains in real-world environments, we develop Qwen-AgentWorld through a three-stage training pipeline: CPT injects general-purpose world modeling capabilities from the state transition dynamics and augmented professional corpora, SFT activates next-state-prediction reasoning, and RL sharpens simulation fidelity through a tailored framework with hybrid rubric-and-rule rewards. To evaluate language world models, we present AgentWorldBench, a comprehensive benchmark constructed from real-world interactions of 5 frontier models on 9 established benchmarks. Empirical results demonstrate that Qwen-AgentWorld significantly outperforms existing frontier models. (ii) Beyond foundation models, we further investigate two complementary paradigms through which world modeling enhances general agents. First, as a decoupled environment simulator, Qwen-AgentWorld supports scalable and controllable simulation of thousands of real-world environments for agentic RL, yielding gains that surpass real-environment training alone. Second, as a unified agent foundation model, world-model training acts as a highly effective warm-up that improves downstream performance across 7 agentic benchmarks. Code: https://github.com/QwenLM/Qwen-AgentWorld

Yuxin Zuo, Zikai Xiao, Li Sheng +30

Qwen Team

LLM World Models Agentic RL World Model Learning World Model Evaluation

Ask PDF

Oct 15, 2026

GlobeBench: Evaluating Text-Based World Model Capabilities for Scalable Environment Simulations

Text-based world models have recently shown potential to simulate complex RL environment state transitions for a diverse set of use cases. However, no standardized benchmark exists to compare, study and improve these models. Through this paper, we attempt to solve this problem by building GlobeBench, a novel benchmark with 1,632 tasks across five separate domains and nineteen simulation skills. The tasks check whether models can predict the environment's response to a specific action given the full agent trajectory by understanding earlier state changes, following environment rules, and predicting the effects of actions on both related and unrelated objects. We compare the predicted environment response with an execution-derived reference, using a hybrid architecture with both deterministic gates and a semantic judge. Among 20 general-purpose LLMs and specialized text world models, GPT-6 Astra achieves the highest task success rate at 49.57%. We find that model accuracy decreases as actions affect a larger portion of the environment. Specifically, when the agent's changes must propagate to dependent objects, task success drops to an average of 5.1% across all 20 models. Although model strengths differ across tasks, 631 tasks remain unsolved by any model, suggesting a significant gap in general LLM capabilities as a world model.

Nicholas Saban, Charvi Bannur, Mariya Vasileva +1

Patronus AI

World Model Evaluation LLM World Models World Models Action-Conditioned World Models

Ask PDF

Jul 2, 2026

GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluation

Evaluating embodied robot foundation models remains a critical bottleneck; unlike large language models efficiently assessed via digital benchmarks, robotic policies require slow, costly real-world rollouts limited by hardware and human supervision, which has driven interest in world models as surrogate policy evaluators, yet the key properties that make a world model reliable for policy assessment remain poorly understood. This work presents a systematic study of world models for robotic policy evaluation and introduces WMBench, a benchmark constructed from real-robot teleoperation data and matched policy rollouts covering diverse manipulation tasks to enable controlled comparisons across model families, action encodings, rollout horizons, and evaluation metrics. Using WMBench, we analyze 7 video world models, 4 action representation schemes, and over 324,000 simulated policy rollouts paired with real robot executions, further enriching our analysis with large-scale community submissions from the CVPR 2026 GigaBrain Challenge, curated synthetic trajectories, and a training videos spanning more than 12,000 hours. Our experiments deliver three core insights: evaluator quality is dominated by long-horizon, action-faithful rollout consistency rather than short-term visual realism; pretraining gains stem not only from data scale but from balancing general world knowledge with robot-specific controllability; and architectural choices including action encoding, memory design, and evaluator-focused post-training strongly determine alignment with real-world robot behavior. Drawing on these results, we derive a practical design roadmap and realize it in GigaWorld-1, a world model specially optimized for policy evaluation, and we fully release our code, models, datasets, and toolkits to advance scalable evaluation research for embodied foundation models.

GigaWorld Team, Angyuan Ma, Boyuan Wang +24

GigaAI · Tsinghua University

Robot Foundation Models Action-Conditioned World Models Robotic Policy Evaluation Robot Manipulation Benchmarks World Model Evaluation Video World Models

Ask PDF

Abstract

Embodied AI

Explore similar work

Qwen-AgentWorld: Language World Models for General Agents

GlobeBench: Evaluating Text-Based World Model Capabilities for Scalable Environment Simulations

GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluation