VibeWorlding: Can Multimodal Agents Construct 3D Open Worlds End-to-End?

Yansong Ning; Jingwen Ye; Zhongkai Wu; Yang Sun; Yiqin Zhu; Xingyi Li; Weidong Zhang; Hao Liu

cs.AIAug 15, 2026

VibeWorlding: Can Multimodal Agents Construct 3D Open Worlds End-to-End?

Authors: Yansong Ning, Jingwen Ye, Zhongkai Wu, Yang Sun, Yiqin Zhu, Xingyi Li, Weidong Zhang, Hao Liu

Organizations: AI Thrust, HKUST(GZ) · TEG AIPD, Tencent

Abstract

Constructing an interactive 3D open world from a user query is important. However, existing methods are primarily evaluated on idealized, simple queries, making it difficult to systematically analyze and compare how multimodal agents understand user intent, use 3D tools, and reason over textual and visual 3D world information. To this end, we propose VibeWorlding, a unified framework for benchmarking and training vibe worlding agents: a multimodal agent that can autonomously infer user intent, plan scene layout, invoke 3D tools, and reflect on the multimodal feedback in a multi-turn agent-environment interaction process. To achieve this, we first build VWE-BENCH, a benchmark of 2,616 high-quality 3D assets, 323 human-annotated seed 3D worlds, and 6,828 reverse-synthesized multimodal user queries, split into verified queries with ground-truth and unverified queries with carefully designed rubrics. Moreover, we develop VibeWorlding-Gym, a joint multimodal RL post-training framework that integrates (1) a sandbox environment unifying asset retrieval, editing, and image rendering as MCP tools, and (2) a rubric-based verifier that combines physical feasibility and intent fulfillment verification, supporting both fair model evaluation and scalable multimodal RL reward service. Our experiments show that current frontier MLLMs are far from solving the vibe worlding agent task, with even GPT-5.5 and Qwen3.8-Max reaching below 60% success rate, and trace the bottleneck to precise 3D world editing. We further find that RL training can ease this weakness and enable open-source MLLMs to even surpass closed-source frontiers: our VibeWorlder-8B is comparable to frontier MLLMs, while our flagship VibeWorlder-30B-A3B attains the best overall Pass@1 among all evaluated models.

Figures & tables

Figure 1: The overview of VWE-Bench and VibeWorlding-Gym.

Concept	Description	Sample Format	Records
3D asset	The atomic block of a 3D world	(id, name, category, face_count, bbox)	2,616
3D world	A set of placed assets	{(m, pos[x,y,z], rot[x,y,z], sca), ...}	323
Multimodal query	①3D world construction	(textual instruction)	6,828
Multimodal query	②3D world refinement	(W_text, W_image, textual instruction)	6,828
3D tools	asset_retrieve : retrieve 3D asset	asset_retrieve(name, top_k)	5
	asset_add : put a retrieved asset	asset_add(id, name, pos[x,y,z], sca)

Table 1: Illustrative example of key concepts in VibeWorlding .

Figure 2: Illustrative VWE-Bench data. (a) Asset examples ordered by physical size (small to large). (b) Seed 3D world examples ordered by complexity (low to high).

Figure 3: VWE-Bench distributions. (a) assets across the 20 semantic categories; (b) assets are annotated physical-size class; (c) the seed world complexity varies.

Query type	Sub-type	Verified	Unverified	Count
3D world construction	Theme only		✓	322
	Theme + elements		✓	620
	Full blueprint		✓	302
	Distractor		✓	120
3D world refinement	Asset-level edit (precise)	✓		1,710
	Asset-level edit (fuzzy)		✓	1,462

Table 2: The query statistics in VWE-Bench .

Query type	SFT cold-start	RL	Test
	Training
3D world construction	1,129	189	46
3D world refinement	4,438	818	208
Total	5,567	1,007	254

Table 3: The training and test statistics of VWE-Bench.

Figure 4: Overall Pass@1 on VWE-Bench across models on Verified and Unverified tasks.

	3D world construction				3D world refinement
Model	Theme	Th+El	Blupr	Distr	Precise	Fuzzy	Crit.	Guid.	Rest.	Cplx	Overall	Human
Gemini 3.1-pro	8.3	18.2	20.0	50.0	44.4	47.5	57.1	48.0	50.0	33.3	42.7	13.3
Gemini 3.5-flash	66.7	31.8	30.0	50.0	52.8	52.5	52.4	48.0	28.6	28.6	47.4	43.3
GPT-5.5	75.0	50.0	70.0	50.0	60.4	55.7	57.1	56.0	50.0	38.1	57.3	33.3
Claude-Opus-4.8	25.0	4.5	20.0	0.0	42.9	23.0	4.8	20.0	14.3	0.0	24.3	13.3
Kimi-K3	41.7	18.2	20.0	100.0	52.8	54.1	52.4	28.0	35.7	28.6	44.5	26.7

Table 4: Pass@1 (%) by query sub-type on the Test split. For 3D world construction queries, the sub-types are Theme-only, Theme+elements (Th+El), Full-blueprint (Blupr), and Distractor (Distr). For 3D world refinement queries, the sub-types include asset edit (Precise/Fuzzy), Scene critique (Crit.), Scene guidance (Guid.), Scene restatement (Rest.), and Complex description (Cplx). The abbreviations in parentheses correspond to the column names in the table. “Overall” denotes the mean reward over all query types. “Human” denotes the holistic pass rate from a blind human study per model (a sample passes only if human annotators judge it physically feasible, intent-fulfilling, ecologically plausible, and free of response hallucination) shown in Appendix A.4 . Agent-scaffold baselines (SceneWeaver, SAGE, SceneAssistant) use GPT-5.5 as the backbone.

Figure 5: Per-dimension pass rate across existing MLLMs, revealing which capabilities bottleneck. Each of the six capabilities (collision-freeness, height plausibility, ecological plausibility, 3D understanding, 3D reasoning, and retrieval plausibility) is quantified by our dual-constraint verifier.

Figure 6: The Metric dynamics of the VibeWorlder-8B in multimodal RL training process. Solid curves are the flagship run with cold-start (i.e., RL from the cold-start policy), and dashed curves are the w/o cold-start ablation (i.e., RL from the base backbone).

Figure 7: Two dominant failure modes of frontier agents on VWE-Bench . (a) Imprecise 3D distance editing: The query requires moving rock 18 “ $7$ m forward.” The agent achieves the correct displacement magnitude but moves the object in the opposite direction, resulting in a $14$ m positional error under a $0.5$ m tolerance. The initial and edited renders (left and right) and coordinate visualization (center) highlight this directional inversion. (b) Overediting: The query requires adding a tree beside the chest. The agent correctly adds the new tree but silently deletes the existing tree, introducing an unrequested modification that violates the editing constraint.

Figure 8: The VibeWorlding CLI prototype during a multi-turn refinement session on an urban scene, with the interactive terminal (left) shown alongside the browser-based GUI viewer (right). (a) First-turn reasoning: after the user asks to “remove the car in the middle of the road and delete the two green buildings,” the agent parses the request and enumerates the current 3D world state (trees, skyscrapers, military buildings, streetlights) from the scene’s 3D map. (b) Tool execution and response: the agent proactively invokes three delete tool calls, re-renders the 3D world from multiple views to verify that only the intended assets were removed, and returns a natural-language summary. After refinement, the 3D asset count decreases from $14$ to $11$ assets accordingly.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Visual feedback across 3D world construction task. Each row is one round of environment feedback: the five fixed camera views the agent receives after its edits are applied. Row 1 (seen when the agent reasons in turn 3): the planned layout is blocked out—central road, shop row, high-rises across the street—but the lanterns are barely visible and the distant pillars read as stone. Row 2 (seen in turn 4): the agent has replaced its own placeholder pillars with billboards and enlarged the lanterns. Row 3 : the final world, $34$ assets over $11$ types. Every panel in a column shares one crop box, so apparent motion is real motion.

Figure 10: Visual feedback across 3D world refinement task. Row 1 : the five views given to the agent in turn 1. The world holds two instances of the same statue asset, $3.08$ m and $3.37$ m from the altar. Row 2 : after the edit. The dashed marker locates the statue the agent deleted; every other asset is untouched. The scattered rocks and haystacks differ slightly between rows because the sandbox re-samples scatter-group instances at each render, not because the agent moved them.

Ground truth	$k{=}1$	$k{=}5$	$k{=}10$	$k{=}20$	$k{=}1$	$k{=}5$	$k{=}10$	$k{=}20$	$\text{nDCG}@10$	$\text{MRR}@10$	$\text{mAP}@20$
	$\text{hit}@k$				$\text{recall\_full}@k$				$\text{nDCG}@10$	$\text{MRR}@10$	$\text{mAP}@20$
Multi-positive (default)	$89.5$	$98.5$	$99.8$	$99.9$	$31.8$	$59.7$	$73.4$	$85.3$	$83.5$	$93.5$	$75.2$
Anchor-only (stricter)	$50.3$	$79.9$	$89.2$	$94.5$	—	—	—	—	$69.3$	$63.0$	$63.4$

Appendix

Table 5: Retrieval quality on the held-out test split (

1{,}311

queries, scored against the full asset library as candidate pool), for the two ground-truth protocols of Section A.2 . Multi-positive counts every

G_{0}

asset as correct; anchor-only counts a single asset, so

\text{recall\_full}@k

coincides with

\text{hit}@k

and is omitted. All values are percentages.

Aspect	Scope	Scale	Decision rule
Physical feasibility (geometric)
Collision	all unverified	$0/1$	Fails if any asset pair interpenetrates by $\geq 10\,\text{m}^{3}$ beyond a $0.5$ m per-axis slack
Height	all unverified	$0/1$	Geometric pre-check confirmed by the judge; fails on any unsupported floating or ground-clipping asset
Intent fulfillment (MLLM judge)
Ecological plausibility	all unverified	$0/1$	Fails on any asset that objectively violates ecological or commonsense placement; fantasy and sci-fi themes are relaxed
Intent fulfillment	all unverified	$0/1$	Conjunction of the three sub-checks below

Appendix

Table 6: How each verifier aspect of Section 4.2 is scored. Collision and height are decided geometrically (Section A.3.2 ); the intent aspects are scored by an MLLM judge. Intent fulfillment is assessed through three sub-checks, indented below it. “Scope” indicates the query types to which the aspect applies.

Model	Intent $\uparrow$	Ecological $\uparrow$	Physical $\uparrow$	Halluc. $\downarrow$	Intel. $\uparrow$	Human Pass@1 $\uparrow$
Gemini 3.1-pro	36.8	78.9	63.3	52.6	3.32	13.3
Gemini 3.5-flash	66.7	85.7	70.0	23.8	3.14	43.3
GPT-5.5	55.6	94.4	60.0	38.9	2.94	33.3
Claude-Opus-4.8	19.2	84.6	86.7	69.2	4.54	13.3
Kimi-K3	38.1	90.5	70.0	23.8	3.90	26.7
Qwen3.8-Max	77.8	100.0	60.0	11.1	4.56	46.7

Appendix

Table 7: Per-model results from the blind human study. Intent, Ecological, and Physical are

0/1

pass rates (%); Halluc. is the response-hallucination rate (%, lower is better); Intel. is the mean

1

–

5

response-intelligence score; Human Pass@1 is the holistic pass rate (physically feasible and intent-fulfilling and ecologically plausible and non-hallucinatory). Ecological/Intent/Halluc./Intel. are measured on the physically-feasible subset per the early-exit protocol; the Physical column mirrors the automatic collision reference shown to annotators and is not an independent check.

Multimodal Agents

Multimodal agents perceive and reason across text, images, audio, and video to use tools, interact with interfaces, and act in the world.

See all 375 papers

Oct 14, 2026

BlenderWorld: Benchmarking Visual Agents for 3D Creation from Real-World References

Recent advances in multimodal large language models (MLLMs) have enabled them to act as autonomous agents, creating and editing 3D models via code or direct software manipulation. In practical workflows, these models must infer object shapes, materials, spatial layouts, and lighting from diverse references while strictly following instructions to reconstruct, modify, or design 3D content. However, existing benchmarks provide limited coverage of such open-ended requests across real-world scenarios. To bridge this gap, we introduce BlenderWorld, the first comprehensive benchmark dedicated to evaluating frontier models on realistic 3D creation tasks. BlenderWorld comprises 223 tasks spanning architecture, products, natural environments, and scientific visualization, driven by text, image, video, and structured specification inputs. To rigorously assess these complex deliverables, we establish a unified evaluation protocol: we render the submitted Blender projects from multiple viewpoints and evaluate them against 2,651 task-specific binary rubrics alongside holistic visual quality scores. Benchmarking thirteen frontier models reveals that GPT-6-Astra achieves the highest overall fused score of 79.9, followed by Opus-5 (73.0) and GPT-5.6-Sol (67.7). Ultimately, our findings reveal a critical bottleneck: while current models can parse broad instructions, they still profoundly struggle with fine-grained 3D spatial coherence and accurate structural reproduction in Blender, leaving substantial headroom for future 3D-native agentics.

Xinyu Fang, Yuchi Wang, Weihao Yu +12

Zhejiang University · Alibaba Token Hub, Alibaba Group · The Chinese University of Hong Kong +2

3D Scene Generation 3D Reconstruction Benchmarks

Ask PDF

Sep 27, 2026

WorldAgent: Verification-Guided Agentic Physical World Construction

Constructing complex physical worlds from language requires coordinating extensive 3D environments, detailed structures and objects at different spatial scales, and interacting physical processes under both stated goals and implicit physical constraints. We present WorldAgent, an agentic framework for verification-guided physical world construction from a single natural-language prompt, without iterative user debugging. A world construction layer expands the prompt into a structured world specification and uses physical knowledge to build scenes and run numerical simulations. After every step, a verification layer inspects scene geometry and simulation states alongside rendered views. Failed checks guide automatic revisions to the specification and re-execution of the affected steps. Accepted worlds pass the required checks and remain editable for further inspection and resimulation. We introduce AgenticSimBench, on which WorldAgent achieves the best scores among the evaluated agent-based methods on five of seven metrics. In a 26-participant user study, it receives the highest mean ratings across all four criteria.

Caoliwen Wang, Mengdi Wang, Yige Chen +13

University of British Columbia · VAST · Zhejiang University +3

3D Scene Generation Functional 3D Scene Generation

Ask PDF

Jun 8, 2026

SpatialWorld: Benchmarking Interactive Spatial Reasoning of Multimodal Agents in Real-World Tasks

Spatial reasoning is a foundational capability for multimodal large language models (MLLMs) to perceive and operate within the physical world. However, existing benchmarks predominantly rely on passive evaluation (e.g., static VQA) or simulator-specific pipelines, failing to assess general interactive spatial understanding. We introduce SpatialWorld, a unified benchmark designed specifically for evaluating the interactive spatial understanding of multimodal agents in complex real-world tasks. Integrating eight heterogeneous simulation backends under a shared, simulator-agnostic protocol, SpatialWorld features 760 human-annotated tasks across diverse domains (e.g., household routines, travel, social collaboration). Agents must solve tasks under vision-only partial observability, actively gathering egocentric visual evidence and expressing decisions via a unified, text-based action interface native to MLLMs. For reliable evaluation, each task includes a human-validated initial state, a reference trajectory, and a terminal-state verifier. Evaluating 15 advanced agents reveals that robust spatial task solving remains challenging: the strongest model, GPT-5, achieves an average task success rate (TSR) of only 17.4%, while the leading open-source model, Qwen-3.5, reaches 14.1%. Further analysis exposes a clear mismatch between task success and execution efficiency, alongside substantial domain-specific performance variations. These bottlenecks in active exploration and long-horizon planning position SpatialWorld as a rigorous testbed for future spatial agents.

Hongcheng Gao, Hailong Qu, Jingyi Tang +18

Tsinghua University · Chongqing University · Peking University +6

Spatial Reasoning Benchmarks Visual Spatial Reasoning Long-Horizon Agent Evaluation AI Agent Benchmarks Multimodal Agents Spatial Reasoning

Ask PDF

Abstract

Multimodal Agents

Explore similar work

BlenderWorld: Benchmarking Visual Agents for 3D Creation from Real-World References

WorldAgent: Verification-Guided Agentic Physical World Construction

SpatialWorld: Benchmarking Interactive Spatial Reasoning of Multimodal Agents in Real-World Tasks