Fight World Model: Customizable Combat through Visual Interaction Proxies

Jiahua Li; Zixu John; Tom Zhong; Fuping Wu; Tianhao Xu; Ruicheng Zhang; Jun Zhou; Zunnan Xu

cs.CVOct 29, 2026

Fight World Model: Customizable Combat through Visual Interaction Proxies

Authors: Jiahua Li, Zixu John, Tom Zhong, Fuping Wu, Tianhao Xu, Ruicheng Zhang, Jun Zhou, Zunnan Xu

Organizations: University of Oxford, UK · Independent Researcher · Imperial College London, UK · Tsinghua University, China ·

Abstract

A playable, programmable game world must maintain persistent state, enforce executable rules, and support extensible gameplay mechanics. Existing video world models can generate plausible combat, yet struggle to keep hit reactions and health updates consistent with game rules. We introduce Fight World Model (FWM), which separates rule-governed combat execution from video rendering. An executable engine maintains state and resolves interactions. A deterministic compiler encodes these outcomes and attempted actions in a spatially and temporally aligned interaction proxy. The proxy aligns action and skill windows, brief hit reactions, and persistent health by fighter identity and timing. These cues can be controlled separately to render interactions under modified rules. Identity-bound references preserve character appearance across side switches. We train FWM on FightWorld Corpus, spanning 52 skill effects, 187 character-appearance variants, and 15 arenas. Without retraining the renderer, FWM supports damage scaling, limited-hit immunity, lifesteal, and multi-stage skill mechanisms beyond the native game. It also allows character and arena customization. Experiments demonstrate the effectiveness of FWM, which outperforms existing game world models in action fidelity, hit reactions, health-state consistency, and character appearance preservation. Our project is available at https://fight-world-model.github.io.

Figures & tables

Figure 2: Overview of FWM. The Proxy Engine executes gameplay rules, and the resulting state trajectories, action windows, and events are compiled into a visual interaction proxy. The proxy and appearance references are encoded with a shared VAE and concatenated with video tokens. Matching spatiotemporal rotary positions align the proxy with the video, while shared identity embeddings bind character references to the corresponding regions in the first video latent and proxy sequence. The DiT processes the combined sequence, and only video tokens are decoded to generate the continuation.

Figure 3: FightWorld Corpus. Native gameplay is expanded with skill effects, character-appearance edits, and scene edits, yielding 112,618 training samples.

Figure 4: Scene Editing in FightWorld Corpus. Each scene pairs a clean reference with an edited training frame. References exclude fighters and HUD elements and may differ in background details.

Figure 5: Character and Scene Reference Conditioning. Interaction rollouts across four character appearances (under default and side-switched stances) and two arena references.

Figure 6: Complete-Information Encodings on the Same Interaction. Held-out rollout under matched executed records ( Table 2 ). Green brackets mark HUD state errors: $\downarrow$ ! unexpected health loss, $\downarrow\!\times$ missing health loss, delayed update, $\downarrow\!\neq$ incomplete update, and $\uparrow$ ! unexpected recovery.

Figure 7: Dual-Character Actions and Responses. Top rails show visible action summaries for both fighters. Per-frame badges indicate realized ( ) or unrealized ( ) fighter actions and responses; ? denotes strategy-level NPC text in ReactiveGWM v1; HUD state icons follow Fig. 6 .

Figure 8: Skill Timing and Event Consequences. Comparison on a projectile skill commanded at $f=20$ ; HUD state icons follow Fig. 6 .

Test Subset	Binding	Adherence $\uparrow$	DINO Similarity $\uparrow$
Non-Switch	Off	78.43	0.47
Non-Switch	On	99.16	0.61
Side-Switch	Off	53.68	0.35
Side-Switch	On	98.27	0.60

Table 1: Effect of Identity Binding. Variants share reference inputs and training budgets, differing only in identity binding. Adherence is a percentage. Values are ranked within each test subset.

Conditioning	Action $\uparrow$ Move / Attack / Skill	Reaction $\uparrow$	Health MAE $\downarrow$	$J\uparrow$
Actions Only	70.27 / 56.22 / 58.64	74.43	3.42	34.18
Actions + Rule Text	71.71 / 68.40 / 57.35	80.16	3.07	46.37
Spatial Proxy Only	99.19 / 92.27 / 46.58	78.28	2.31	33.46
Spatial Proxy + State Tokens	94.48 / 91.23 / 82.42	87.17	1.23	63.28
Spatial Proxy + Text	96.72 / 93.63 / 92.43	92.24	2.78	76.35
FWM Interaction Proxy	99.78 / 99.24 / 99.06	99.79	0.15	98.48

Table 2: Interaction Fidelity across Control Representations. We compare variants with different conditioning signals to assess the effects of interaction information and its encoding. Rates are percentages, and Health MAE is in percentage points of maximum health. Across tables, bold and underlined values mark the best and second-best results.

Method	Action $\uparrow$ Move / Attack / Skill	Reaction $\uparrow$	Health MAE $\downarrow$	Adherence $\uparrow$ Character / Scene	Character DINO $\uparrow$	Scene DINO $\uparrow$
Without Reference-Image Conditioning
H3-WORLD	43.85 / 62.61 / 47.58	48.25	8.75	—	—	—
ReactiveGWM v1	70.15 / 78.98 / 48.55	35.56	4.59	—	—	—
ReactiveGWM v2	50.24 / 59.70 / 16.67	49.21	4.65	—	—	—
With Reference-Image Conditioning
H3-WORLD + Ref.	36.91 / 62.70 / 42.03	42.22	7.57	88.10 / 100.00	0.47	0.68

Table 3: Comparison with Adapted Game World Models. Baselines are adapted to FightWorld Corpus. +Ref denotes the addition of our reference-conditioning branch. Rates are percentages, and Health MAE is in percentage points of maximum health. DINO measures cosine similarity. A dash indicates an inapplicable metric.

Variant	Action $\uparrow$ Move / Attack / Skill	Reaction $\uparrow$	Health MAE $\downarrow$	$J$ $\uparrow$
FWM	99.78 / 99.24 / 99.06	99.79	0.15	98.48
w/o Command Marks	99.07 / 86.42 / 87.87	97.93	0.28	85.26
w/o Damage-Event Cues	99.16 / 98.92 / 98.91	80.42	0.27	78.38
w/o Health Bars	99.84 / 99.08 / 99.12	98.31	2.18	57.24
w/o Skill Event Trigger	99.21 / 99.03 / 79.28	98.07	0.26	78.81

Table 4: Contributions of Proxy Cues. Each ablated variant omits one cue family and is retrained under the same training setting. Rates are percentages, and Health MAE is in percentage points of maximum health.

Default Rules
Encoding	Health MAE $\downarrow$	$J_{\mathrm{rule}}\uparrow$
Spatial Proxy + State Tokens	1.87	89.26
Spatial Proxy + Text	2.42	76.38
FWM	0.16	100.00
Damage Magnitude
Spatial Proxy + State Tokens	1.63 / 2.48	74.62 / 65.37

Table 5: Interaction Fidelity under Rule Changes. Within each rule setting, encodings share executed records and are ranked separately. Default rules use original damage without immunity or lifesteal. Paired values denote reduced / amplified damage. Health MAE is in percentage points of maximum health, and

J_{\mathrm{rule}}

is a percentage.

Figure 9: Skill-Level Rule Mechanisms Beyond Native Combat. Five combat mechanisms absent from native Street Fighter II/III , spanning lingering ground damage, per-projectile hits, staggered area hits, action locks, and spatial confinement (boundary locked).

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Interaction-Proxy Vocabulary. Transient cues and persistent state share fighter identities.

Panel of Figure 11 — Figure 11: Reference-Augmented Comparison (Saitama, Player, No Side Switch). Matched frames corresponding to row 1 of Fig. 5 ; HUD state icons follow Fig. 6 .

Panel of Figure 13 — Figure 13: Reference-Augmented Comparison (Lava Humanoid, NPC, No Side Switch). Matched frames corresponding to row 3 of Fig. 5 ; HUD state icons follow Fig. 6 .

Interactive World Models

Interactive world models predict and generate how environments change in response to actions, supporting simulation, planning, and control.

See all 122 papers

Programmable World Model
Sep 9, 2026Zheng-Hui Huang, Guixu Lin, Jiacheng Lin +8World Models Controllable Video Generation
Alaya Lab
ReactiveGWM: Flexible Control and NPC Reactivity in Game World Models
May 14, 2026Zeqing Wang, Danze Chen, Zhaohu Xing +4World Models Multi-Agent Control
Tencent · National University of Singapore · The Hong Kong University of Science and Technology (Guangzhou) +3
StatePlay: State-Aware Game World Models for Mechanics-Consistent Generation
Jul 29, 2026Zijun Lin, Zeqing Wang, Cheston Tan +2World Models Action-Conditioned World Models
Tencent · Nanyang Technological University · Centre for Frontier AI Research, A*STAR +1

Sep 9, 2026J/K move · Enter open · S save

Programmable World Model

Zheng-Hui Huang, Guixu Lin, Jiacheng Lin, Yi-Chuan Huang+7

Alaya Lab

World Models Controllable Video Generation World Model Evaluation Interactive World Models Video World Models

Open paper AskPDF arXiv

Recent video world models generate increasingly realistic and interactive visual experiences, yet lack reliable mechanisms for maintaining persistent world state and enforcing programmable rules over extended interactions. We introduce Programmable World Model, a framework that decouples world-state evolution from visual observation generation. An agent translates natural-language instructions into executable programs that specify entity states and state-transition rules, enabling direct control over individual entities and their interactions. A lightweight engine executes these programs to update and maintain an explicit, persistent global world state, including off-screen entities and non-visual attributes. To connect world state with visual generation, we introduce state-augmented 3D oriented bounding boxes (OBBs) as an intermediate representation. This representation, together with the target camera trajectory, is deterministically compiled into pixel-aligned spatiotemporal conditioning signals for a pretrained video model serving as the generative renderer. This design allows users to create playable games with predefined mechanics, direct control over individual entities, and persistent world state throughout gameplay. We further introduce CombatStateBench, a benchmark for evaluating programmable world models. On CombatStateBench, our method achieves 94% Count Accuracy and 98% State Accuracy, substantially outperforming existing interactive video world models while supporting coherent long-horizon generation. These results demonstrate the effectiveness of separating explicit state evolution from generative rendering for building persistent, programmable worlds.

Abstract

Interactive World Models

Explore similar work

Programmable World Model

ReactiveGWM: Flexible Control and NPC Reactivity in Game World Models

StatePlay: State-Aware Game World Models for Mechanics-Consistent Generation