AgiWorld: An Agentic Video World Model for Building Persistent Worlds
Organizations: The University of Texas at Austin · Adobe Research
Abstract
Recent interactive video world models generate visually compelling videos from a single image, following user-provided camera motion and text prompts. However, such low-level control demands substantial user effort to realize a high-level intent, and the generated world struggles to remain persistent as the exploration proceeds. In this work, we present AgiWorld, an agentic video world model that explores and builds a persistent world from a single image and a free-form instruction. For high-level planning, our agent AgiWorld-Agent designs the world and the journey through it in a world state and directs the exploration. For low-level realization, our video world model AgiWorld-WM generates the video chunk by chunk and stores it with its geometry in a spatial memory, from which the agent updates its world state. This closed loop keeps the world faithful to the user's intent and keeps the agent's world state consistent with the generated content throughout the exploration. We co-design the two levels around the shared spatial memory and an interface of metric camera trajectories and structured prompts. In AgiWorld-WM, to handle occlusion and accumulated geometric errors in the spatial memory over long rollouts, we propose occlusion-aware history selection, which retrieves past views by surface visibility, and reliability-based memory maintenance, which prefers reliable geometry and earlier observations. AgiWorld-WM achieves state-of-the-art overall performance on world revisiting and exploration benchmarks, and qualitative results and ablations show that AgiWorld builds persistent worlds from free-form instructions over minute-long journeys. Persistence thus arises not from the video model alone but from its closed loop with the agent.
Figures & tables
| Step | Called | Writes |
| Director | ||
| Trajectory | At initialization, and whenever the route has to change (replan) | Waypoints of the next 60 s |
| Fix | After Trajectory, when the solved path fails a check | Corrected waypoints |
| Prompt | After Trajectory and Fix | Start, middle, and end of the prompt of each chunk |
| Review | After every chunk | Verdicts, small layout edits, stage updates, decision for the next chunk |
| State update : reconstruction | ||
| Field | Meaning |
|---|---|
| Layout element | |
| Name | What the element is, with a fixed feature of the place when several are alike, e.g., “offering table on the passage side”. Words that depend on the camera, such as left or ahead , are not allowed. |
| Kind | Surface, terrain, structure, object, opening, or zone. Surfaces and terrain form the ground, and structures and objects are solids. |
| Shape | Polygon, polyline, path, box, circle, or point, with metric coordinates in the plane and an optional elevation per vertex for ramps and stairs. |
| Elevation, height | Height of the base above , and vertical extent. |
| Appearance | One line that describes what the element looks like. |
| Tool output | Content | Read by |
|---|---|---|
| Depth map of | The depth of as a color map with a metric scale. | Initial reconstruction |
| Point cloud and samples of | The leveled point cloud of from above, and with a grid of points labeled with their . | Initial reconstruction |
| Layout check | For each element, whether its position and height agree with the depth and the point cloud of . | Initial reconstruction |
| Layout projection | The layout drawn as wireframes into or into the last frame of a chunk. | Initial reconstruction, Revision, Review |
| Floor plan | The layout over the accumulated point cloud, with the executed path and the planned path. | All steps except Prompt |
| Trajectory check | Collisions, eye-height and support violations, and violations of the motion bounds on the solved path. | Fix |
| Behavior of AgiWorld-WM | Camera and prompt practice |
|---|---|
| It extends what is already in the middle of the frame while the camera moves smoothly and most of what the previous seconds showed stays in view. | The camera turns gently while it walks, so that most of what the last five seconds showed stays in view. It walks in the direction it looks, and walks backward only for a few seconds. |
| A space that it has never shown is hard to generate at the moment the camera enters it. Walking through a doorway or around a corner onto unseen ground can produce a jump. | A new space is kept in the middle of the frame from the first chunk in which it is visible, and the camera walks toward it without stopping. The prompt of each of these chunks states what is there. The chunk that crosses an opening ends well beyond it. |
| Its memory holds only what it has generated. Parts of a room that were never in view have nothing to stay consistent with. | In a room, the first chunk turns the view across the room, about a quarter turn while walking slowly, so that its walls are in the memory before the camera leaves them. To look back, the camera turns around while walking. |
| It keeps a large space most consistent when the space is shown whole from a distance. Narrow views that each cover a small patch of a room give it little to keep consistent. | Large scenes are shown from far. An element that the instruction names is held in the middle of the frame, from far enough back that the space around it is also in view. |
| A wall or a shelf that fills the frame from close range gives it nothing to continue. | The camera looks where the space ahead opens, down a lane or across a room, rather than at the wall it passes. It moves sideways only while the view stays open. |
| It generates what the prompt names where the frame gives it room, and what it generates can differ from the layout. | The prompt names only what the frames of the chunk will certainly show. The next path is designed on the generated geometry. |
| Check | Bound |
|---|---|
| Collision | A sphere of radius 0.25 m around the camera stays clear of the confirmed solids and of the point cloud. |
| Eye height | The eye stays within 0.7 to 1.4 times its starting height above the walked surface of the layout. |
| Support | The camera does not pass over an area that the layout marks as not walkable. |
| Speed | At most 2.5 m/s, with a change of pace of at most 0.6 m/s 2 . |
| Turning | At most 90 ∘ /s, and at most 25 ∘ /s on average over a chunk. |
| Chunk boundary | Between consecutive planned chunks, the velocity changes by at most 1 m/s and the turn rate by at most 45 ∘ /s. |
Video World Models
Video world models generate and predict action-conditioned visual futures to simulate changing scenes for embodied agents.
See all 285 papers