cs.MAOct 28, 2026

Cinematize from Unreal: Agentic Choreography for Controllable World Simulation

Authors: Zifan Song, Caihua Shan, Kaitao Song, Yifan Yang, Dongsheng Li, Cairong Zhao

Organizations: Tongji University · Microsoft Research Asia

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Controllable world simulation requires more than visually plausible frames: scenes must retain persistent state, coordinated actions, and editable temporal structure. Generative world models offer flexible visual synthesis, but their outputs do not always expose the objects and transitions needed for precise revision. Graphics engines provide executable state and direct control, yet translating language into valid operations requires specialized knowledge of assets, geometry, and interdependent APIs. We present CineU (Cinematize from Unreal), an agentic system for language-driven world orchestration and cinematic choreography within Unreal Engine. From a code-based world modeling perspective, an Unreal project serves as a persistent, inspectable world representation, while agent-generated code specifies operations whose effects can be executed and verified. CineU combines hierarchical orchestration, cascaded asset retrieval and geometry-constrained placement, capability-aligned code synthesis, and multi-stage verification with localized repair. Shared scene and timeline context coordinates entities, actions, camera trajectories, and shot boundaries while supporting dependency-aware revisions. Experiments on scene construction, selective editing, multi-shot cinematography, and component ablations show how specialized execution knowledge and engine feedback improve task completion and targeted recovery. CineU illustrates how executable world state can complement learned visual generation to support controllable and revisable simulated worlds.

Figures & tables

Figure 1: Language-directed cinematic worlds. Three briefs combine scene assets with camera and action requirements: a building reveal, a rooftop dolly zoom followed by an orbit, and attention shifting between statues above and below water. Rendered frames visualize the results; engine-native actors and tracks remain available for subsequent editing.
RequirementImplementationEvidence
R1: Ground instructionsDecompose a brief into operations and dependencies; retain engine objects and runtime context.Compound scene and camera tasks (Section 6.2 ).
R2: Preserve valid stateUpdate targeted materials, bindings, or tracks while retaining unrelated operations.Texture, animation, and entity revisions (Section 6.3 ).
R3: Localize failuresCheck code, trace runtime calls, inspect engine state, and repair the affected function.Verification and repair ablations (Table 4 ).
R4: Synchronize controlsShare timeline references and constraints across animation, transform, and camera operations.Multi-shot, dolly-zoom, and locomotion cases (Sections 6.2 and 6.4 ).
Table 1: System requirements and corresponding mechanisms. The requirements concern translating semantic instructions into operations, maintaining existing state, diagnosing failures, and coordinating coupled controls.
Revision targetAddressable resourceRelationship to retainDependent state to reconsider
MaterialMaterial assignment or appearance parameters.The same actor remains in the same scene and temporal sequence.Retain geometry, transform, animation, and camera tracks when only appearance changes.
AnimationAnimation binding and its track section.Pose progression agrees with the actor’s displacement and action interval.Align clip playback with the existing transform track and shared timing.
EntityActor or asset binding and placement transform.The replacement retains the intended support and scene relationship.Revisit bounds, support contact, and compatibility with the assigned animation.
CameraCamera transform and lens channels.In a dolly zoom, subject framing follows the distance–focal-length relation.Update the coupled tracks together, using the subject position and common time reference.
Shot intervalCamera-cut segment and associated interval.Shot ordering remains coherent with the action being shown.Reconcile adjacent cut boundaries with the shared timeline and action timing.
Table 2: Revision operations and dependency scope. The addressed resource identifies where an edit starts; the retained relationship determines which surrounding state must be reconsidered. A single instruction can require changes to multiple dependent resources.
Figure 2: Multi-tier architecture of CineU , comprising specialized agents, system middleware, and simulation backend. The orchestration agent decomposes briefs and manages cascaded asset retrieval. Specialized agents configure environments, import entities, reason about placement, and coordinate Sequencer tracks. Middleware dispatches capability-specific knowledge, checks code, bridges execution to Unreal, and records runtime traces. Engine-state inspection closes the verification and repair loop.
Figure 3: Character resources at different visual-fidelity levels. High-, medium-, and low-fidelity groups show character resources in front and side views and in representative poses. Appearance selection provides the entity resource used by subsequent placement and animation operations.
Figure 4: Instantiated character actions. Neutral mannequins display representative standing, walking, running, and gesturing configurations. Character resources and animation bindings are represented separately, with compatibility checked during integration.
Decision boundaryEvidence examinedQuestion addressedResulting response
Static validityGenerated syntax, argument types, and supported API signatures.Is the proposed operation well formed for the available interface?Revise the failing function before submitting it to the engine.
Runtime completionInstrumented calls, parameters, return values, and exceptions.Did the submitted operation complete, and where did execution fail?Localize the failing call and provide its trace to the responsible agent.
State conformanceRealized actors, asset bindings, transforms, keyframes, and track timing.Does the resulting state satisfy the represented spatial and temporal requirements?Revise the responsible operation using engine-state feedback; retain validated operations.
Creative judgmentThe rendered viewport and inspectable scene and timeline.Does the realized composition serve the author’s intended expression?Retain the result or revise the brief through language or native engine editing.
Table 3: Evidence and verification decisions. Static checks, runtime traces, and world-state inspection support different repair decisions. Visual inspection evaluates composition and may trigger a revised specification; it is distinct from automated verification.
Figure 5: Multi-shot coordination. Three camera shots expand the scene from interior exploration to an exterior panorama. Camera actors and cut-track segments are coordinated through a shared timeline rather than assembled as independently generated clips.
Figure 6: Selective revision of a shared scene. The base sequence is modified along texture, animation, and entity dimensions. Animation changes require synchronization with displacement; entity replacement requires placement checks. The environment and broader camera composition are retained in the displayed examples.
Figure 7: Illustrative camera-control cases. Outputs for a zebra dolly-zoom request and a continuous gallery tour compare generative video with engine-grounded authoring. The selected examples emphasize camera–lens coupling and scene continuity, contrasting generated frames with an explicit engine scene and camera trajectory.
Figure 8: Coordination and character control. (a) Without orchestration (upper row), the wolf translates with a static pose; coordinated tracks (lower row) align animation with movement. (b) Two level-simulation examples demonstrate locomotion and pose transitions. Both panels are qualitative system examples.
ConfigurationSuccess (%)Program (%)Refinement Cycle
Vanilla Agent11.519.4>> 10
w/o Knowledge18.422.68.7
w/ Unified K\mathcal{K}31.238.57.3
w/ Specialized KvA\mathcal{K}^{\mathcal{A}}_{v}47.856.25.6
w/o Verification16.856.29.4
w/ Single-Stage29.556.26.9
Table 4: Component ablations. Success is end-to-end task completion, program correctness concerns the first generation, and refinement cycles describe iterative correction. Configuration-level values are organized into knowledge, verification, and repair comparisons. Bold entries identify the full-system configuration within each group.
Figure 9: Representative failures. Invalid placement, uninformative camera framing, and interference between tracks illustrate different gaps between executable operations and the intended scene. Each panel connects a visible scene or timeline issue to the operation requiring correction.
More in this topic

Functional 3D Scene Generation

Functional 3D scene generation creates editable three-dimensional environments arranged to support specified activities, interactions, and physical constraints.

See all 36 papers

Explore similar work

CardsList
Jul 2, 2026

WorldDirector: Building Controllable World Simulators with Persistent Dynamic Memory

We present WorldDirector, a highly controllable video world model framework designed for persistent dynamic object memory and unrestricted viewpoint exploration. Unlike existing world models that entangle physical dynamics with pixel rendering and rely on continuous visual observation to sustain motion, our framework explicitly decouples semantic motion orchestration from visual generation. By leveraging an LLM to coordinate 3D trajectories with camera movements and subsequently employing these orchestrated trajectories as control signals for video generation, our approach ensures strict physical logic and appearance stability, successfully preserving the exact visual identities of dynamic entities even when they re-enter the scene after prolonged periods out of view. Experimental results demonstrate that our method supports the synthesis of complex and extended events with unprecedented controllability and persistent dynamic object memory. Project Page: https://worlddirector.github.io/

Hanlin Wang, Hao Ouyang, Qiuyu Wang +10

1HKUST · 2Ant Group · 3ZJU +1

Physical Consistency in Video GenerationCamera-Controlled Video GenerationMemory-Augmented Video GenerationControllable Video GenerationVideo World Models
AskPDF
May 14, 2026

Coding Agent Is Good As World Simulator

World models have emerged as a powerful paradigm for building interactive simulation environments, with recent video-based approaches demonstrating impressive progress in generating visually plausible dynamics. However, because these models typically infer dynamics from video and represent them in latent states, they do not explicitly enforce physical constraints. As a result, the generated video rollouts are not physically plausible, exhibiting unstable contacts, distorted shapes, or inconsistent motion. In this paper, we present an agentic framework constructing physics-based world models through executable simulation code. The framework coordinates planning, code generation, visual review, and physics analysis agents. The planning agent converts the natural language prompt into a structured scene plan, the code agent implements it as executable simulation code, and the visual review agent provide visual feedback while the physics analysis agent checks physical consistency. The code is iteratively revised based on the feedback until the simulation matches the prompt reqirements and physical constraints. Experimental results show that our framework outperforms advanced video-based models in physical accuracy, instruction fidelity and visual quality, which could be applied to various scenarios including driving simulation and embodied robot tasks.

Hongyu Wang, Jingquan Wang, Bocheng Zou +2

Department of Mechanical & Aerospace Engineering, University of Wisconsin-Madison · School of Computer, Data, and Information Sciences, University of Wisconsin-Madison

Robotics SimulationWorld ModelsAgentic Code GenerationPhysics-Based SimulationEmbodied World Models
AskPDF
Oct 18, 2026

UniState: Unifying Memory and Intervention for Controllable World Simulation

World models offer increasingly rich environments for exploration, yet intervening in a world while observing the consequences from freely chosen viewpoints remains challenging. We tackle these challenges by introducing UniState, a framework for controllable world simulation built on a unified world-state representation. This representation unifies spatial memory and user interventions. Users specify object motion directly within spatial memory, while independently choosing the camera trajectory from which the resulting world state is observed. This unification makes the relationship between the scene and the requested changes explicit, removing ambiguity between object motion and viewpoint change from the control specification. Crucially, specifying an intervention does not require prescribing every resulting movement: the video model can synthesize additional motion within controlled objects and plausible interactions with their surroundings. We develop training and sampling strategies that help generated videos follow the specified object motion and camera trajectory while preserving this capacity to generate responses beyond the prescribed transformations. Experiments demonstrate more accurate joint object and camera control than the existing methods, alongside qualitative evidence that intervention-induced scene changes persist across rollouts.

Jaeseok Jeong, Kyungmook Choi, Sohyun Chung +2

Department of Artificial Intelligence, Yonsei University · Familiar

Controllable Video GenerationVideo World ModelsCamera-Controlled Video GenerationTrajectory-Guided Video Generation
AskPDF