Scalable world modeling requires coordinating scene-wide goals with fine-grained objects and dynamic characters under limited time and context budgets. We introduce GTA (Generative Tower of Agents), which represents every unit at every level of an adaptive scene hierarchy as a holon, both a self-contained construction task and a composable scene unit. Parallel agents recursively plan the hierarchy top-down, generate assets, and assemble and refine them bottom-up. Uniform interfaces carry scene-wide requirements down the hierarchy, guiding local construction so that each holon fits into the larger one. GTA accelerates scalable world modeling through high-throughput parallel execution while preserving well-structured layouts and fine-grained asset details. GTA then populates the assembled environments with dynamic human characters whose appearance and motion are guided by the scene, yielding realized 4D worlds. Project page: https://zt-zeng-tao.github.io/GTA/
Figures & tables
Figure 1: Building a Tower of Babel in the AI era would require collaboration across humanity and a system to coordinate that vast effort efficiently. Likewise, GTA unifies scene representation across scales and types, which enables more agents to work in parallel as the scene scales up, making world-scale modeling feasible.
Figure 2: A film studio backlot with holarchy. GTA recursively decomposes the backlot (a) into holons such as a cafe (b), which contains a counter area (c) and smaller components such as a coffee station (d), cups and saucers (e), and an espresso machine (f). The labels connect each rendered example to its corresponding holon on the right.
Figure 3: Holons and their interfaces. Each holon group consists of a parent holon and its immediate children and serves as the unit on which agents operate. Groups communicate with one another through parent-facing and child-facing interfaces.
Figure 4: Parallel agentic processing of holons. Once a holon’s dependencies are satisfied, it enters the pending queue. The scheduler dispatches queued groups to available agentic slots, allowing independent holons to execute concurrently.
Prompt Compliance
Physical Validity
Visual Quality (Obj. / Scene)
Scene Type
Method
Quantity Error (T / U) ↓ (%)
Specified Objects ↑ (%)
P1↓ (%)
MPD ↓ (mm)
NAV [ 35 ] ↑
Opening Clearance [ 35 ] ↑
Claude Opus 5.5 ↑
GPT-6.1 Sol ↑
Indoor Scenes
SAGE
32.92 / 31.38
64.29–100.00
15.25
8.02
0.785
0.976
2.55 / 2.08
2.83 / 2.38
SceneSmith
27.78 / 17.50
55.03–100.00
13.64
2.97
0.979
0.988
3.65 / 2.50
3.86 / 3.31
GTA (Ours)
11.67 / 15.82
96.30–100.00
6.45
4.53
0.983
0.987
3.74 / 3.17
3.81 / 3.86
General Scenes
Fable 5.2
13.58 / 22.82
89.92–100.00
15.29
4.58
0.994
0.992
2.70 / 2.85
3.02 / 3.45
Astra-O †
7.88 / 27.97
84.06–100.00
24.38
16.70
0.983
0.989
2.37 / 2.94
2.87 / 3.42
Table 1: Quantitative comparison between different methods. T/U denote relative absolute errors in total instance and distinct semantic type counts; P1 and MPD measure penetration frequency and depth, respectively. Two visual judges assess object- and scene-level quality on a 1–5 scale. ↑/↓ indicates higher/lower is better. Within each group, the best and second-best results are highlighted in bold and underlined , respectively. † Astra-O, Astra-S, and GTA use the same GPT-6 Astra backbone with medium reasoning effort.
Figure 5: Faster generation with higher agentic throughput. The left panel shows wall-clock time per object. The right panel shows system-level token throughput.
Figure 6: Qualitative comparison across different scene types and scales. We compare GTA with baselines on two indoor scenes (A–B) and two outdoor scenes (C–D). Full-scene views and object close-ups show how GTA combines coherent layouts with detailed, scene-specific assets, from darkroom equipment to market stalls and garden sculptures.
Figure 7: Roaming the World with GTA. GTA generates scenes across diverse geographic and cultural settings. Each scene consists of over 1,000 objects. GPT-6 Astra fills the gray-marked backgrounds, with some textures coming from external sources [ 29 , 30 , 4 , 21 ] . Please visit the Project Page for more high-resolution results.
Figure 8: Roaming the World with GTA (continued). Additional scenes from the same series. Please visit the Project Page for more high-resolution results.
Figure 9: Scaling up the agentic slot numbers speeds up scene generation. We compare per-stage and end-to-end speedups for 1–32 slots with the ideal linear S(p)=p . The stacked bars show each stage’s share of total runtime.
Figure 10: GTA refines the details of each object in Stage 3.
Figure 11: Task-driven scene generation for robot training.
Moving a scene generator to a new game engine requires more than exporting its final output: planning, construction tools, and stored experience may also depend on the original backend. We present SceneAgent, a multi-agent framework for reusing the construction process and its experience across backends. A canonical SceneContext maintains the evolving world, while backend adapters isolate coordinate conventions, asset bindings, and export. Structured handoffs let fresh Builder sessions query this persistent state without inheriting the full construction transcript. Across generations, human-approved traces are distilled into asset-ID-free Layout skills and semantic Content skills that can be retrieved and recombined. We introduce GameSceneBench, comprising 69 indoor and outdoor prompts. Under a common asset and evaluation protocol, SceneAgent obtains human tie-adjusted preference scores of 78.2-89.4% against five baselines. Ablations show that reusable skills and the multi-agent protocol primarily improve prompt alignment and scene completeness, with additional runtime. Deployment in a second engine preserves the shared construction protocol and skill library while replacing backend integration components. Together, these results support separating persistent scene state, task-local reasoning, and reusable experience for portable scene construction.
Generating large-scale, freely explorable 3D worlds from open-ended text remains challenging because a system must jointly maintain global spatial coherence, rich local content, and explicit assets suitable for downstream editing and reuse. We present WorldClaw, a fully agentic, coarse-to-fine framework for open-world 3D scene generation. Planning agents translate a text prompt into a structured specification of regions, terrain, assets, materials, and spatial relations. WorldClaw then builds a globally coherent terrain foundation from semantic layouts, reusable assets, generative or procedural materials, and a region-aware height field. For detail-demanding regions, it generates terrain-conditioned compositions, reconstructs editable textured meshes, and recovers their placement on the terrain; render-based agents further refine terrain, objects, appearance, and contacts. Across diverse open-world prompts, WorldClaw produces large-scale scenes with coherent spatial organization, visually compelling local content, and editable instance-level assets while preserving a consistent global terrain structure.
World models for interactive video generation have largely focused on single-agent settings, where future observations are generated from a single control signal. However, many generated environments require multi-agent interaction: multiple players, robots, or embodied agents act simultaneously within a shared space. Scaling world models to such settings requires a principled multi-agent design: agents should remain independently controllable, permutation-symmetric, and support efficient inference while maintaining consistency across time and perspectives. In this paper, we present our generative multi-agent world model for interactive simulation. It introduces Simplex Rotary Agent Encoding, a parameter-free extension of 3D RoPE that represents agents as vertices of a regular simplex in rotary angle space. This gives each agent a distinct phase while making all agents permutation-equivalent, enabling scalable agent identity without learned per-slot identities or a fixed agent ordering. To avoid dense all-to-all attention across agents, we further propose Sparse Hub Attention, where learnable hub tokens mediate token interaction across agents, reducing cross-agent attention cost from quadratic to linear in the number of agents. For real-time rollout, we distill a full-context diffusion teacher into a causal student that generates temporal blocks sequentially with KV caching, enabling action-responsive generation at 24 FPS. Experiments in multiplayer virtual environments show that our model improves video fidelity, action controllability, and inter-agent consistency over slot-based and dense-attention baselines, while generalizing from two to four players without additional training.
Fangfu Liu, Kai He, Tianchang Shen +7
NVIDIA · Tsinghua University · University of Toronto +1