Mallea: Building, Steering, and Memorizing Worlds with Coding Agent

Xuanyi Liu; Ruiqi Li; Siwei Ma

cs.AIOct 15, 2026

Mallea: Building, Steering, and Memorizing Worlds with Coding Agent

Authors: Xuanyi Liu, Ruiqi Li, Siwei Ma

Abstract

Code world models represent world dynamics through explicit state and executable rules. We investigate how to coordinate object representations, behavioral revisions, and visual memory within this formulation. We present Mallea, a system for building, steering, and memorizing worlds from text. A coding agent infers object interaction requirements from the prompt and jointly plans object structure, properties, and rules, favoring compact representations that support those requirements. The world evolves under the planned rules; optional user actions steer it, while text requests can edit properties or extend capabilities with unaffected state preserved. Object texture memory retains reusable appearance only for whitebox-defined objects and adapts it as those objects change. The program advances world state, while a generative renderer combines current structure with remembered appearance. By organizing construction, intervention, and memory around the same objects, Mallea conditions subsequent observations on the latest state and compatible appearance. Project page: https://lxycopper.github.io/Mallea.

Figures & tables

Figure 1: Three world-model designs. Top: video models generate future frames from visual context. Middle: recent code models use explicit state, rules, and visual memory to guide rendering. Bottom: a coding agent in Mallea chooses each object’s representation (Build), edits its properties and rules (Steer), and manages its stored appearance (Memorize).

Figure 2: Overview of the Mallea system. A coding agent jointly plans representations, fields, and rules. Users may supply action controls or interleave prompts to edit or extend the world. Runtime code advances state and retrieves object textures through a visibility window; generated observations update the store for subsequent revisits. The scenes are schematic illustrations.

Figure 3: Interaction-driven construction. The agent selects geometry and fields together, checks required contacts and motions, and refines only missing structure. The illustrated representations can be combined.

Figure 4: Defining and extending behavior. Code icons denote the agent. (a) Planned fields and rules produce different responses to the same strike under the illustrated material settings. Orange bursts mark contact. (b) A request adds magnetic response to existing iron and wood, preserving unrelated state. Green arrows mark extension and attraction. Mechanism illustrations.

Figure 5: Object texture memory across edits and revisits. The first visit registers textures for whitebox-defined objects A and B. While both are out of view, a text edit changes B’s material. On return, the current view and state select compatible appearance: A’s pattern is reused (green), while B’s invalid texture is refreshed (orange). Generation combines current geometry with valid appearance, then registers reliable object regions back into memory. Off-screen objects retain their entries and continue evolving. Scenes are AI-generated mechanism illustrations, not experimental results.

Figure 6: Qualitative results. Generated video frames (top) and corresponding display whitebox frames (bottom) for 12 scenes. Each sequence is sampled at 0%, 25%, 50%, 75%, and 100% of its frame span. Sampling is normalized independently, without event alignment. Whiteboxes in left-column rows 2–4 were adjusted after video generation.

Code World Models

See all 15 papers

Code World Model: Coding Agent as World Brain
Aug 26, 2026Yiwen Chen, Guosheng Lin, Chi ZhangWorld Models Code World Models
AGI Lab, Westlake University · Nanyang Technological University
Code Plans, Diffusion Renders: Open-Ended Generative World Modeling
Sep 22, 2026Zixun Fang, Yawen Shao, Kai Zhu +7World Models Code World Models
USTC · TongYi Lab
Code as Worlds: Agentic Discovery of Executable World Representations for Physical Reasoning
Aug 27, 2026Hanyang Wang, Yimo Cai, Weiliang Chen +14Code World Models World Models

Aug 26, 2026J/K move · Enter open · S save

Code World Model: Coding Agent as World Brain

Yiwen Chen, Guosheng Lin, Chi Zhang

AGI Lab, Westlake University · Nanyang Technological University

World Models Code World Models World Model Learning Video World Models Action-Conditioned World Models

Open paper AskPDF arXiv

World models aim to simulate how complex environments evolve under actions and events, yet existing video-based world models primarily learn dynamics from visual observations, which reveal outcomes rather than the underlying knowledge, rules, and mechanisms governing world evolution. This makes it difficult to maintain persistent consequences and support coherent, open-ended evolution. We introduce Code World Model, a framework that separates world evolution from visual realization by combining the reasoning and coding capabilities of language models with the generative priors of video models. A coding agent serves as the world brain, reasoning about events and their consequences and generating executable code to maintain persistent world state and perform rule-consistent evolution. To connect executable state with visual generation, we introduce a proxy representation that encodes frame-wise spatiotemporal constraints and is compiled into a proxy video, which conditions a video model to render high-fidelity visual observations. We further develop data pipelines for constructing aligned proxy-observation pairs from gameplay and real-world videos. After fine-tuning on paired gameplay data, MiniMax-H3 follows proxy-based spatiotemporal specifications from simple interactive worlds built by the coding agent while preserving rich visual details and dynamics. These results demonstrate the potential of combining code for persistent world evolution with video models for flexible visual realization, providing a new path toward open-ended world models.

Abstract

Code World Models

Explore similar work

Code World Model: Coding Agent as World Brain

Code Plans, Diffusion Renders: Open-Ended Generative World Modeling

Code as Worlds: Agentic Discovery of Executable World Representations for Physical Reasoning