ROWBench: Do Video Models Render What the Program Specifies?

Zheng-Hui Huang; Guixu Lin; Yu-Ju Tsai; Jian-Kai Zhu; Fengbo Lan; Yu-Lun Liu; Yung-Yu Chuang; Kaipeng Zhang

Figure 1 : Overview of our extensible framework and benchmark. Our extensible framework pairs visual proxies with ground-truth world records, enabling reliable evaluation of how faithfully generated videos realize prescribed states, rules, and interactions. This assessment would otherwise be difficult. Here, we compare the CLIP score with our proposed interaction metric. CLIP assigns a higher score to the clearly inferior case, whereas one of our metric correctly reflects whether the prescribed interaction is faithfully executed.

Figure 2 : Overview of the world generation pipeline. World synthesis generates replayable trajectories, control compilation produces synchronized conditioning inputs across viewpoints and representations, and the final process validates and exports videos with structured annotations.

Method	Input	Entity control		Logic / State				Camera control			Quality		Temporal		Text
Method	Input	IoU $\uparrow$	CErr $\downarrow$	ISR $\uparrow$	LRA $\uparrow$	gISR $\uparrow$	gLRA $\uparrow$	Rot $\downarrow$	Trans $\downarrow$	CamMC $\downarrow$	Img $\uparrow$	Aes $\uparrow$	Warp $\downarrow$	Smooth $\uparrow$	CLIP $\uparrow$
Verified-FF (40 pairs)
EchoWM ( Zhang et al., 2026 )		0.351	0.055	0.506	0.685	0.168	0.241	2.2	0.151	0.166	67.6	0.607	0.619	0.987	25.2
SANA-WM ( Zhu et al., 2026 )		0.309	0.070	0.590	0.569	0.185	0.180	2.7	0.145	0.168	68.6	0.612	0.871	0.985	25.1
LingBot 2.0 ( Robbyant Team et al., 2026 )		0.327	0.063	0.733	0.794	0.241	0.256	2.8	0.100	0.133	69.9	0.615	1.026	0.978	25.4
AlayaWorld v1.1 ( Team et al., 2026 )		0.303	0.073	0.583	0.622	0.174	0.185	9.4	0.229	0.356	66.8	0.592	0.530	0.990	25.0

Table 1 : Main track. Results on the Verified-FF setting (top, 40 five-second scene-camera pairs) and the Unverified-FF setting (bottom, 90 pairs); the two settings use different scenes, so best and second-best values are marked within each setting in bold and underlined . Input: = first frame checked against the recorded state (Verified-FF), = unchecked first frame, i.e., frame 0 of the MiniMax-H3 output for the same scene and camera (Unverified-FF), = camera pose, = depth, = class-level segmentation, = CWM proxy, = coarse 3D proxy, = colored-OBB proxy. All methods also receive the text prompt.

\uparrow

/

\downarrow

: higher/lower is better. CErr: normalized box-center error; Rot/Trans: camera rotation (degrees)/translation error; Img/Aes: imaging/aesthetic quality; Warp: warping error (

\times 10^{-3}

); Smooth: motion smoothness; CLIP: CLIP-Score; ISR: Interaction Success Rate; LRA: Logic-Render Alignment; gISR/gLRA: ISR/LRA multiplied by BBox IoU per sample and then averaged, i.e., events realized by correctly placed entities.

Method	Input	Entity control		Logic / State				Memory		Camera control
		IoU $\uparrow$	CErr $\downarrow$	ISR $\uparrow$	LRA $\uparrow$	gISR $\uparrow$	gLRA $\uparrow$	Re-IoU $\uparrow$	Re-State $\uparrow$	Rot $\downarrow$	Trans $\downarrow$	CamMC $\downarrow$
C2R ( Gomez-Nogales et al., 2026 )		0.212	0.066	0.519	0.718	0.127	0.170	0.111	0.405	41.9	0.571	1.119
CWM ( Chen et al., 2026b )		0.210	0.070	0.504	0.854	0.111	0.182	0.053	0.886	30.2	0.381	0.793
LynnReal-Omni ( Mao et al., 2026b )		0.280	0.054	0.515	0.897	0.149	0.247	0.103	0.528	22.7	0.273	0.610
Seedance 2.5 ( ByteDance Seed Team, 2026 )		0.175	0.077	0.497	0.777	0.087	0.132	0.131	0.650	39.4	0.242	0.819
MiniMax-H3 ( MiniMax-AI, 2026 )		0.206	0.088	0.553	0.844	0.121	0.179	0.059	0.833	31.8	0.351	0.814

Table 2 : Long-horizon evaluation of controllability and memory. Results on ten 30-second recordings, with one camera per recording and one output per method; best and second-best values per column in bold and underlined . Input: = unchecked first frame, i.e., frame 0 of the MiniMax-H3 output for the same scene and camera, = depth, = CWM proxy, = coarse 3D proxy, = colored-OBB.

\uparrow

/

\downarrow

: higher/lower is better; Rot in degrees. Re-IoU: Reappearance IoU; Re-State: State Persistence of returning entities; other metrics as in Table 1 .

Method	Input	Entity control			Logic / State				Appearance			Camera control
		IoU $\uparrow$	CErr $\downarrow$	Miss $\downarrow$	ISR $\uparrow$	LRA $\uparrow$	gISR $\uparrow$	gLRA $\uparrow$	Comp. $\uparrow$	Cons. $\uparrow$	MEt3R $\downarrow$	Rot $\downarrow$	Trans $\downarrow$	CamMC $\downarrow$
C2R ( Gomez-Nogales et al., 2026 )		0.414	0.039	0.192	0.637	0.750	0.271	0.318	32.2	25.5	0.397	2.2	0.155	0.173
CWM ( Chen et al., 2026b )		0.439	0.037	0.064	0.812	0.950	0.345	0.414	92.9	88.1	0.422	2.0	0.047	0.071
LynnReal-Omni ( Mao et al., 2026b )		0.490	0.038	0.047	0.700	0.900	0.347	0.435	87.6	89.7	0.448	0.4	0.023	0.026
Seedance 2.5 ( ByteDance Seed Team, 2026 )		0.341	0.049	0.089	0.812	0.963	0.284	0.331	82.2	81.0	0.499	1.0	0.076	0.083
MiniMax-H3 ( MiniMax-AI, 2026 )		0.524	0.029	0.049	0.825	0.938	0.428	0.489	94.0	88.6	0.400	0.9	0.034	0.043

Table 3 : Multi-view evaluation of controllability and consistency. Five methods are evaluated on 10 scenes with four synchronized views each (40 clips per method). Views are generated independently; best and second-best values per column in bold and underlined . Input: = unchecked first frame, i.e., frame 0 of the MiniMax-H3 output for the same scene and camera, = depth, = CWM proxy, = coarse 3D proxy, = colored-OBB proxy. All methods also receive the text prompt.

\uparrow

/

\downarrow

: higher/lower is better; Rot in degrees. Comp./Cons.: appearance compliance/cross-view consistency (%); Miss: miss rate; other metrics as in Table 1 .

Figure 3 : Method classes at a glance. Main-track results per method class in Verified-FF (light) and Unverified-FF (dark). Bars show class means, dots the individual methods. Camera-conditioned: EchoWM, SANA-WM, LingBot 2.0, AlayaWorld v1.1; task-specific renderers: Cosmos-Transfer2.5, C2R, CWM, LynnReal-Omni; general-purpose video generation: MiniMax-H3, Seedance 2.5.

Figure 7

Method	Input	Entity control		Logic / State				Camera control			Quality		Temporal		Text
Method	Input	IoU $\uparrow$	CErr $\downarrow$	ISR $\uparrow$	LRA $\uparrow$	gISR $\uparrow$	gLRA $\uparrow$	Rot $\downarrow$	Trans $\downarrow$	CamMC $\downarrow$	Img $\uparrow$	Aes $\uparrow$	Warp $\downarrow$	Smooth $\uparrow$	CLIP $\uparrow$
C2R ( Gomez-Nogales et al., 2026 )		0.437	0.041	0.450	0.725	0.191	0.330	2.7	1.008	1.043	69.6	0.544	0.301	0.994	24.3
CWM ( Chen et al., 2026b )		0.413	0.039	0.650	0.900	0.287	0.381	0.7	0.740	0.744	72.6	0.562	0.157	0.992	25.6
PWM Huang et al. (2026a)		0.602	0.021	0.850	0.875	0.523	0.529	1.1	0.348	0.359	71.0	0.579	0.182	0.993	26.0
LynnReal-Omni ( Mao et al., 2026b )		0.552	0.033	0.625	0.975	0.340	0.535	1.3	0.563	0.584	75.0	0.527	0.613	0.988	25.2
MiniMax-H3 ( MiniMax-AI, 2026 )		0.526	0.035	0.600	0.900	0.302	0.475	5.1	0.770	0.830	73.9	0.556	0.107	0.995	25.4

Table 4 : Comparison with a domain-specific model on armed-confrontation scenes. The 20-scene set includes first-frame conditioning and is designed for comparison with PWM, which is trained only on gunfight scenes (third-person; an armed confrontation in which one person is shot and collapses). Metrics use native model outputs; best and second-best values are bold and underlined . Input: = first frame, = depth, = CWM proxy, = coarse 3D proxy, = colored-OBB proxy. All methods also receive the text prompt. CWM = Code World Model. Metrics as in Table 1 .

Figure 6 : Proxy leakage. The top row shows proxy inputs, and the bottom row shows corresponding generated frames. In these representative failure cases, colored bounding boxes or wireframe outlines from the proxies remain visible in the outputs, revealing unintended leakage of control representations into the rendered appearance.

Method	Input	Entity control		Logic / State				Camera control			Quality		Temporal		Text
Method	Input	IoU $\uparrow$	CErr $\downarrow$	ISR $\uparrow$	LRA $\uparrow$	gISR $\uparrow$	gLRA $\uparrow$	Rot $\downarrow$	Trans $\downarrow$	CamMC $\downarrow$	Img $\uparrow$	Aes $\uparrow$	Warp $\downarrow$	Smooth $\uparrow$	CLIP $\uparrow$
MiniMax-H3 ( MiniMax-AI, 2026 )		0.514	0.030	0.625	0.802	0.304	0.391	4.8	0.072	0.156	70.1	0.655	0.570	0.987	25.1
MiniMax-H3 ( MiniMax-AI, 2026 )		0.507	0.031	0.683	0.665	0.338	0.333	4.5	0.079	0.156	70.7	0.650	0.573	0.987	25.1
MiniMax-H3 ( MiniMax-AI, 2026 )		0.503	0.033	0.733	0.690	0.347	0.339	5.0	0.065	0.155	70.7	0.652	0.574	0.987	25.1

Table 5 : MiniMax-H3 under different proxy representations on a Verified-FF subset. Input: = first frame, = CWM proxy, = coarse 3D proxy, = colored-OBB proxy. Metrics as in Table 1 .

Figure 11

Figure 6.1 : Gallery of all 170 PROWBench cases. Each tile shows the MiniMax-H3 output at the middle of the clip (at 15 s for the 30-second long-horizon recordings); the inset shows the colored-OBB proxy input at the same instant. MiniMax-H3 is used only because it has outputs for every collection; its frames are generated results, not ground truth. Cases are grouped by collection: the Verified-FF (40) and Unverified-FF (90) settings of the main track, the long-horizon (10) and multi-view (10, view A shown) settings of the challenge track, and the domain-specific gunfight set (20). FPP marks first-person views.

Figure 6.2 : Diversity of PROWBench. PROWBench provides diverse camera perspectives, interactions, scene domains, and long-horizon videos.

Figure 6.3 : Dataset composition of PROWBench. 170 evaluation cases, 200 selected camera views, and 600 proxy videos. The panels show interaction categories, requested visual styles, scene domains, camera path length, maximum angular deviation from the initial camera orientation, and camera perspectives; each panel states its denominator (170 episodes or 200 selected views). Interaction labels follow a unified task-level annotation policy and may overlap within a case. Camera statistics cover each full source clip, including both five-second and 30- second sequences. Each selected view provides three synchronized proxy representations derived from the same source world-state records.

Benchmark	Behavior specification	State / trajectory reference	Task / event reference	Replay	Views	Proxies
VBench [ 19 ]	Text prompts	–	Prompt semantics	–	–	–
WorldScore [ 11 ]	Text; camera paths	Target camera paths	Text-specified dynamics	–	–	–
WorldModelBench [ 24 ]	Text instructions	–	Instructions; physics criteria	–	–	–
MIND [ 57 ]	Agent navigation; camera rotation	Actor and camera poses	–	–	–	–
WorldArena [ 44 ]	Text; robot actions	Robot states and actions	Task goals; simulator outcomes	Sim.	–	–
Omni-WorldBench [ 55 ]	Text interactions; optional camera paths	Camera targets where provided	Specified object effects and event order	–	–	–

Table 6.1 : Benchmark protocols and their reference information. Behavior specification summarizes motion and interaction conditions; initial images and video context are omitted for compactness. State/trajectory references and task/event references are distinguished from conditions supplied to a model. Replay requires shared geometry and saved dynamic states sufficient to re-render an episode without rerunning its behavior controllers. Views denotes synchronized videos of the same world evolution; paired first frames or sequential perspective switches do not qualify. Proxies denotes alternative visual representations derived from the same states and cameras. ✓: documented support; –: not documented in the compared protocol or not applicable; Sim.: simulator-based evaluation, without confirmed recorded-state re-rendering; Subset: selected episodes. PROWBench combines multi-entity state and event records with replay and paired observations to evaluate visual realization under fixed world evolution.

	Verified-FF	Unverified-FF
References	<Picture 1> (first frame) and <Video 1> (proxy)	<Video 1> only
Task	keyframe completion and reference generation; the generated clip must start exactly from the provided first frame	reference generation
Appearance source	Qwen3.6-27B description of the verified first frame	authored scene description
Style source	authored style line; the first frame also visually exhibits this style	authored style line
Control rule	for overall style and the appearance of subjects visible in the first frame, the first frame takes precedence over text	the appearance of every subject is specified by text
Structural skeleton	identical: subject definitions, proxy bindings, action timeline, and measured facts

Table 6.2 : Base prompt in the two main-track settings.

Methods	Prompt format
MiniMax-H3, LynnReal-Omni	base prompt verbatim, including the control block, tagged references, subject bindings with legend colors, and the complete timestamped timeline
Seedance 2.5	Verified-FF: converted to Seedance’s reference syntax ( @Image1 , @Video1 ); retains the style line, the appearance of every subject, a “who is who” list identifying each box by color name and legend color, proxy semantics, the timestamped timeline, and field-of-view information; removes the control block, subject tags, retention analysis, and soundscape. Unverified-FF: base prompt with the control block, as for MiniMax-H3 (see System prompt )
C2R, Cosmos-Transfer2.5, EchoWM, SANA-WM, LingBot 2.0, AlayaWorld v1.1	converted to the single-paragraph caption style used to train C2R: an opening description of the style, setting, number of people, and appearance of every subject, followed by the action progression in prose without timestamps and a final quality sentence; removes tags, proxy terminology, and camera-specific text, and is shortened to satisfy C2R’s 512-token text limit
CWM	converted to CWM’s caption format, consisting of appearance and scene , a timestamped protagonist action timeline , and other visible actors ; removes subject tags and proxy-specific wording, while CWM adds its own system prompt for the mixed proxy render and RGB anchor frame

Table 6.3 : Prompt format by method.

Method	Conditioning	Supplied spatial input	Output	FPS	Frames
EchoWM [ 60 ]	RGB first frame, camera trajectory	$1280\times 720$ image	$1280\times 704$	24	121
SANA-WM [ 63 ]	RGB first frame, camera trajectory	$1280\times 720$ image	$1280\times 704$	16	80
LingBot-World 2.0 [ 43 ]	RGB first frame, camera trajectory	$1280\times 720$ image	$1280\times 720$	16	81
AlayaWorld v1.1 [ 46 ]	RGB first frame, camera trajectory; chunk-autoregressive generation, 4 rounds, 4 denoising steps per round	$1280\times 720$ image, resized and cropped to $960\times 544$	$960\times 544$	24	120
Cosmos-Transfer2.5 [ 1 ]	Depth, class-level segmentation	$1280\times 720$ , 24 fps, 121 frames	$1280\times 720$	24	121
C2R [ 15 ]	Coarse 3D proxy video	$832\times 480$ , 16 fps, 81 frames	$832\times 480$	16	81

Table 8.1 : Evaluated Model Configurations and Input–Output Specifications. Figure 8.1 shows the full test condition of one Unverified-FF sample, and Figure 8.2 the conditions of the other settings.

Figure 8.1 : Test condition of one sample. Unverified-FF, handoff scene (follow camera): every conditioning input provided for this sample, the methods that receive it (Table 8.1 ), and excerpts of the text prompt. Proxy frames are shown at 0.5 s; the first frame is frame 0 of the MiniMax-H3 output, which is not checked against the recorded state.

Figure 8.2 : Test conditions of the other settings. One example per setting: Verified-FF (main track), long-horizon and multi-view (challenge track), and the gunfight set used for the PWM comparison. Compared with the Unverified-FF example in Figure 8.1 , the settings differ in the first frame, the clip length and generation protocol, and the number of synchronized views. All methods also receive a text prompt generated from the engine records.

Figure 8.3 : Entity-level control. A pedestrian walking down a city street, third-person follow camera, same first frame for all methods. The semi-transparent white face on each proxy box marks the entity’s front. Dashed: engine reference boxes of the oncoming car (cyan) and the car ahead (orange). The proxy-conditioned models keep both cars in their boxes (IoU 0.62–0.86); EchoWM, conditioned on the camera trajectory only, does not (0.27 and 0.29).

Figure 8.4 : Interaction Success Rate (ISR) and Logic-Render Alignment (LRA) on one scene. Unverified-FF, gate scene. Columns: the colored-OBB proxy and six methods; rows: the three recorded events (approach, opening, passed-through), four to five frames each. Badges below each event: whether the judge, which sees frames from the whole event window, finds the event realized, i.e., both its action and its end state are judged present. Dashed white boxes are the engine reference boxes and solid boxes the boxes detected in the generated video (one color per entity); the overlay gives the mean BBox IoU of each frame. Bottom: ISR, the fraction of realized events, LRA, the fraction of timeline segments whose action is depicted, and the BBox IoU of the whole video. The two can differ, e.g., LingBot 2.0 depicts all three actions (LRA

1.00

) but misses the end state of the first event (ISR

0.67

).

Figure 8.5 : All methods on one scene. Unverified-FF, handoff scene: a courier hands a parcel to a receiver between 1.0 and 2.6 s. For each method, the conditioning input (left pair) and the generated output (right pair) at 0.5 and 2.6 s. Camera-conditioned world models receive frame 0 of the MiniMax-H3 output and the camera trajectory but no proxy; Cosmos-Transfer2.5 receives semantic segmentation (shown) and depth from the CWM proxy representation; CWM additionally receives depth and frame 0 of the MiniMax-H3 output.

Figure 8.6 : All methods on one scene: entering a car. Verified-FF, city scene, follow camera: the protagonist walks to a parked car, gets in (1.1–2.2 s) and drives away (from 2.6 s); frames at 1.6 and 4.0 s. Methods that accept a first frame also receive the verified first frame. EchoWM and SANA-WM leave the protagonist outside the car, and LingBot 2.0 opens the door but the car does not leave, whereas the proxy-conditioned methods drive the car away as recorded.

Figure 8.7 : All methods on one scene: flying a helicopter. Verified-FF, first-person view from a helicopter cockpit that ascends (0–2.3 s), flies forward (2.3–4.1 s) and descends over a harbour town, with an aircraft ahead; frames at 1.5 and 4.8 s. Methods that accept a first frame also receive the verified first frame. EchoWM drops the cockpit and shows the aircraft from outside; Cosmos-Transfer2.5 renders a person facing the camera and C2R a pilot seen from behind, so neither keeps the prescribed first-person view.

Figure 8.8 : All methods on one scene: sitting down and standing up. Verified-FF, indoor scene, follow camera: the protagonist sits down on a chair (0.6–1.5 s), stays seated, and stands up again (3.0–3.9 s); frames at 2.2 s (seated) and 4.5 s (standing). Methods that accept a first frame also receive the verified first frame; EchoWM loses the room and renders the protagonist on an empty background.

Figure 8.9 : Style and trajectory without a first frame. Stair-climbing scene, follow camera, sampled at 0.5, 2.5 and 4.9 s; the text prescribes a graphic, non-photorealistic marker-sketch style. LynnReal-Omni, Seedance 2.5 and MiniMax-H3 render this style with varying fidelity, and CWM inherits it from MiniMax-H3’s frame 0; C2R renders the scene photorealistically although its prompt requests the marker-sketch style. All methods follow the proxy as the protagonist climbs toward the upper landing (pair IoU

0.52

–

0.61

).

Figure 8.10 : Appearance and facing errors without a first frame. (a) Supermarket scene at 3.0 s: dashed boxes are the engine reference boxes of a courier, a student and a tourist, each with a described outfit; the zoomed crops show what each method renders inside each box. MiniMax-H3 exchanges the student’s and the tourist’s outfits; Seedance 2.5 puts the courier’s outfit at the student’s position. (b) Horseback scene at 0.5 s: the horse on the right faces away from the camera; Seedance 2.5 renders it facing the camera.

ROWBench: Do Video Models Render What the Program Specifies?

Abstract

Figures & tables

Explore similar work

Programmable World Model

WBench: A Comprehensive Multi-turn Benchmark for Interactive Video World Model Evaluation

PAWBench: How Far Are We from Probabilistically Aligned World Modeling?