ROAR: Benchmarking LMM Agents under Real-Time Constraints in Multiplayer Action-Shooter Games

Dawei Wang; Di Zhao; Jianran Yu; Chengming Zhou; Marci Chi Ma; Xiaoyang Liu; Beining Zhang; Shuliang Chu

cs.MAOct 16, 2026

ROAR: Benchmarking LMM Agents under Real-Time Constraints in Multiplayer Action-Shooter Games

Authors: Dawei Wang, Di Zhao, Jianran Yu, Chengming Zhou, Marci Chi Ma, Xiaoyang Liu, Beining Zhang, Shuliang Chu,

Organizations: Newcastle University · University of Auckland · CTW Inc.

Abstract

Large multimodal models (LMMs) are increasingly deployed as agents that perceive, reason, and act in interactive multi-agent environments. Yet most existing benchmarks for LMM-based agents in multi-agent settings remain turn-based: the environment and other agents pause while an agent performs inference. Such protocols obscure a fundamental challenge of real-time constraints, where the world continues to evolve and delayed decisions may become outdated before execution. Motivated by this limitation, we introduce ROAR, a multiplayer action-shooter game environment for evaluating LMM-based agents in multi-agent settings under real-time constraints. ROAR covers cooperative, competitive, mixed cooperative-competitive, and basic single-agent scenarios. We further propose a Multi-Agent Asynchronous Gym interface that decouples environment evolution from agent reasoning by running the environment and each agent in separate threads, enabling the environment to advance at fixed time intervals without waiting for LMM inference and allowing agents to reason independently without blocking one another. ROAR also features convenient deployment, multimodal observations, and customizable levels. ROAR offers a practical testbed for advancing LMM-based agents beyond static, turn-based evaluation toward real-time multi-agent decision-making.

Figures & tables

Figure 1: ROAR is a multiplayer action-shooter game environment. Top: A third-person view of the agents and the environment in ROAR. Bottom row: First-person views of the four agents shown above during gameplay, with each image corresponding to one agent. Labels (a)–(f) denote the agent’s current health, remaining episode time, minimap, number of defeated opponents, crosshair, and current/maximum ammunition, respectively.

Panel of Table 1 — Table 1: Comparison of ROAR with other benchmarks for LMM-based agents in multi-agent settings. Real-Time indicates whether the environment supports real-time constraints. Player Mode denotes the agent setting, where Coop. , Comp. , Mixed , and Single represent the multi-agent cooperative, multi-agent competitive, multi-agent mixed cooperative-competitive, and single-agent settings, respectively. Observation denotes the observation modalities provided by the environment, where State corresponds to structured game state observations. ✓ denotes partial real-time support; MineLand uses non-blocking, time-driven execution with New/Resume gating, making it neither strictly turn-based nor fully real-time.

Figure 2: Illustration of the four levels in ROAR. Agents on the same team share the same 3D model and are marked with same-colour rectangles. From left to right: (1) Cooperative Level (four agents collaborate to complete the task); (2) Competitive Level (four agents compete against each other); (3) Mixed Cooperative-Competitive Level (two teams of two with intra-team cooperation and inter-team competition); and (4) Basic Single-Agent Level (a single agent completes the task).

Figure 3: Comparison between our Multi-Agent Asynchronous Gym interface and traditional interfaces. Top: In Multi-Agent Asynchronous Gym, the environment and each agent run on separate threads. The environment advances at a fixed interval $\Delta t_{\mathrm{fixed}}$ without waiting for agent inference, while each agent independently reads its latest observation from the observation buffer $\mathcal{B}{\mathrm{obs}}^{i}$ and writes its generated action to the action buffer $\mathcal{B}{\mathrm{act}}^{i}$ . Bottom: Traditional interfaces execute all agents and the environment within a single thread, where the environment must wait for all agents to finish inference before proceeding to the next step, resulting in variable step durations $\Delta t_{t}$ .

Figure 4: The ROAR environment provides a multimodal observation space, including pixels, depth, text, game states, and audio. The action space is a seven-dimensional vector representing horizontal (left–right) and vertical (forward–backward) movement, yaw (left–right) and pitch (up–down) camera rotations, as well as firing, jumping, and minimap usage.

Model	Latency (ms)	$\boldsymbol{\Delta t_{\mathrm{fixed}}}$ (ms)	Coop.	Comp.	Mixed	Single	Avg.	Shooting	Incoming
GPT-5.4	P50: 1458.5 P90: 1946.1	Turn-based	0.31	0.54	0.65	0.38	0.47	0.54	0.26
		250	$0.07$	$0.36$	$0.43$	$0.16$	0.26	0.29	0.71
		40	$0.02$	0.33	0.30	$0.12$	0.19	0.07	0.88
Gemini-3.1 Pro	P50: 3235.2 P90: 3995.3	Turn-based	0.78	0.82	0.85	0.93	0.87	0.82	0.31
		250	$0.13$	0.68	0.71	$0.22$	0.44	0.61	0.66
		40	$0.04$	0.54	0.63	$0.16$	0.34	0.23	0.89

Table 2: Performance of different LMMs under the traditional turn-based interface and our Multi-Agent Asynchronous Gym interface with fixed time intervals of 250 ms and 40 ms. Latency denotes the end-to-end model latency, reported using P50 and P90. Coop. , Comp. , Mixed , and Single report the episode scores on the cooperative, competitive, mixed cooperative-competitive, and basic single-agent levels, respectively. Avg. reports the average episode score across the four levels. Shooting and Incoming denote shooting accuracy and incoming hit rate, respectively. Bold numbers indicate the highest value in each column. Tables with standard errors are provided in the Appendix N

Model	Latency (ms)	$\boldsymbol{\Delta t_{\mathrm{fixed}}}$ (ms)	Coop.	Comp.	Mixed	Single	Avg.	Shooting	Incoming
Gemma-4 31B	P50: 300.6 P90: 603.9	Turn-based	0.24	0.43	0.51	0.38	0.39	0.32	0.41
		250	0.19	0.36	0.47	0.31	0.33	0.27	0.47
		40	0.08	0.25	0.28	0.13	0.19	0.11	0.62
Qwen-3.5 27B	P50: 207.6 P90: 478.9	Turn-based	0.21	0.45	0.54	0.36	0.39	0.33	0.45
		250	0.18	0.40	0.52	0.31	0.35	0.29	0.56
		40	0.11	0.25	0.30	0.15	0.20	0.17	0.67

Table 3: Performance of different open-source LMMs and calibration baselines under the traditional turn-based interface and our Multi-Agent Asynchronous Gym interface with fixed time intervals of 250 ms and 40 ms. Latency denotes the end-to-end model latency, reported using P50 and P90. Coop. , Comp. , Mixed , and Single report the episode scores on the cooperative, competitive, mixed cooperative-competitive, and basic single-agent levels, respectively. Avg. reports the average episode score across the four levels. Shooting and Incoming denote shooting accuracy and incoming hit rate, respectively. Bold numbers indicate the highest value in each column. Tables with standard errors are provided in Appendix N .

Analysis	Model	Variant	Latency P50 / P90 (ms)	Coop. Turn-based	Coop. 250 ms	Coop. 40 ms
Control Mode Ablation	Gemini-3.1 Pro	Decentralized	3235.2 / 3995.3	$0.78\pm 0.03$	$0.13\pm 0.02$	$\mathbf{0.04{\pm}0.01}$
	Gemini-3.1 Pro	Centralized	4910.1 / 6567.5	$\mathbf{0.81{\pm}0.04}$	$\mathbf{0.15{\pm}0.02}$	$0.03\pm 0.01$
	Qwen-3.5 9B	Decentralized	96.6 / 226.5	$0.06\pm 0.02$	$0.00\pm 0.00$	$0.00\pm 0.00$
	Qwen-3.5 9B	Centralized	187.4 / 362.2	$\mathbf{0.07{\pm}0.02}$	$\mathbf{0.01{\pm}0.01}$	$0.00\pm 0.00$
Agentic Framework Ablation	Gemini-3.1 Pro	Direct	3235.2 / 3995.3	$0.78\pm 0.04$	$0.13\pm 0.02$	$\mathbf{0.04{\pm}0.01}$
	Gemini-3.1 Pro	Agentic	5880.1 / 7519.6	$\mathbf{0.85{\pm}0.05}$	$\mathbf{0.14{\pm}0.03}$	$0.03\pm 0.01$

Table 4: The upper part reports the results of centralized and decentralized control in the cooperative level, while the lower part reports the results of direct model invocation and agentic frameworks in the cooperative level. The results in this table are reported with standard errors.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Environment	Model	Latency P50/P90 (ms)	TB	$\boldsymbol{250\,\mathrm{ms}}$	$\boldsymbol{40\,\mathrm{ms}}$	Relative Drop
TeamCraft	Gemini-3.1 Pro	2575.7 / 3498.1	0.45	0.45	0.40	0.11
TeamCraft	Gemma-4 31B	252.0 / 578.2	0.00	0.00	0.00	N/A
TDW-Cook	Gemini-3.1 Pro	2821.9 / 3793.5	0.55	0.55	0.55	0.00
TDW-Cook	Gemma-4 31B	238.4 / 533.8	0.05	0.05	0.05	0.00
ROAR (Ours)	Gemini-3.1 Pro	3128.2 / 4005.5	0.70	0.45	0.05	0.93
ROAR (Ours)	Gemma-4 31B	295.2 / 612.1	0.20	0.15	0.00	1.00

Appendix

Table 5: Comparison of model performance across environments under different real-time constraints. TB denotes the turn-based interface.

Figure 5: Aerial view of the cooperative level in ROAR.

Figure 6: The six types of obstacles in ROAR levels.

Figure 7: The toxic zone used in the competitive and mixed levels of ROAR is shown as the purple fog in the figure; players take damage upon contact.

Figure 8: This figure shows the minimap interface after enlargement.

Figure 9: From left to right: coin, ammo box, chicken, and turret.

Figure 11: From left to right: coin, ammo box, chicken, and turret.

Parameter	Value	Parameter	Value
ToxicZoneDamagePerSecond	35	ToxicZoneDamageRandomRange	5
ToxicZoneShrinkDistancePerSecond	0.245	TurretAttackRadius	13
TurretHealthValue	100	TurretAttackValue	40
TurretAttackRandomRange	5	PlayerHealthValue	100
PlayerAttackValue	20	PlayerAttackRandomRange	5
PlayerAmmoCapacity	30	ShootAudibleRange	100

Appendix

Table 6: Configuration parameters used in the ROAR environment.

Figure 12: Illustration of the interaction process between the environment and our implemented basic ReAct-based LMM agentic framework. The agent maintains a memory module that stores historical observations and actions, and periodically summarizes these historical data into checkpoints. During inference, the agent constructs the current context from the system prompt, checkpoint, latest observation, and task instruction. Based on this context, the agent performs tool calling and then generates the action to be executed. Dashed arrows indicate the information flow of observations and actions between the memory and context modules of the framework. The direct LMM invocation mode consists only of the components marked with asterisks in the figure.

Model	$\boldsymbol{\Delta t_{\mathrm{fixed}}}$	Coop.	Comp.	Mixed	Single	Avg.	Shooting	Incoming
GPT-5.4	TB	$\mathbf{0.31{\pm}0.03}$	$\mathbf{0.54{\pm}0.02}$	$\mathbf{0.65{\pm}0.04}$	$\mathbf{0.38{\pm}0.03}$	$\mathbf{0.47{\pm}0.02}$	$\mathbf{0.54{\pm}0.05}$	$0.26\pm 0.02$
	250	$0.07\pm 0.01$	$0.36\pm 0.03$	$0.43\pm 0.04$	$0.16\pm 0.02$	$0.26\pm 0.03$	$0.29\pm 0.05$	$0.71\pm 0.04$
	40	$0.02\pm 0.01$	$0.33\pm 0.04$	$0.30\pm 0.03$	$0.12\pm 0.02$	$0.19\pm 0.02$	$0.07\pm 0.01$	$\mathbf{0.88{\pm}0.03}$
Gemini-3.1 Pro	TB	$\mathbf{0.78{\pm}0.04}$	$\mathbf{0.82{\pm}0.03}$	$\mathbf{0.85{\pm}0.03}$	$\mathbf{0.93{\pm}0.02}$	$\mathbf{0.87{\pm}0.03}$	$\mathbf{0.82{\pm}0.04}$	$0.31\pm 0.02$
	250	$0.13\pm 0.02$	$0.68\pm 0.04$	$0.71\pm 0.03$	$0.22\pm 0.02$	$0.44\pm 0.03$	$0.61\pm 0.05$	$0.66\pm 0.04$
	40	$0.04\pm 0.01$	$0.54\pm 0.03$	$0.63\pm 0.04$	$0.16\pm 0.02$	$0.34\pm 0.03$	$0.23\pm 0.02$	$\mathbf{0.89{\pm}0.03}$

Appendix

Table 7: Performance of different LMMs with standard errors under the traditional turn-based interface and our Multi-Agent Asynchronous Gym interface with fixed time intervals of 250 ms and 40 ms. TB denotes the turn-based interface. Coop. , Comp. , Mixed , and Single report the episode scores on the cooperative, competitive, mixed cooperative-competitive, and basic single-agent levels, respectively. Avg. reports the average episode score across the four levels. Shooting and Incoming denote shooting accuracy and incoming hit rate, respectively. Bold numbers indicate the best value for each model across the three evaluation interfaces.

Model	$\boldsymbol{\Delta t_{\mathrm{fixed}}}$	Coop.	Comp.	Mixed	Single	Avg.	Shooting	Incoming
Gemma-4 31B	TB	$\mathbf{0.24{\pm}0.03}$	$\mathbf{0.43{\pm}0.04}$	$\mathbf{0.51{\pm}0.04}$	$\mathbf{0.38{\pm}0.02}$	$\mathbf{0.39{\pm}0.03}$	$\mathbf{0.32{\pm}0.04}$	$0.41\pm 0.04$
	250	$0.19\pm 0.03$	$0.36\pm 0.03$	$0.47\pm 0.03$	$0.31\pm 0.03$	$0.33\pm 0.03$	$0.27\pm 0.03$	$0.47\pm 0.04$
	40	$0.08\pm 0.01$	$0.25\pm 0.02$	$0.28\pm 0.03$	$0.13\pm 0.01$	$0.19\pm 0.02$	$0.11\pm 0.02$	$\mathbf{0.62{\pm}0.02}$
Qwen-3.5 27B	TB	$\mathbf{0.21{\pm}0.02}$	$\mathbf{0.45{\pm}0.03}$	$\mathbf{0.54{\pm}0.04}$	$\mathbf{0.36{\pm}0.03}$	$\mathbf{0.39{\pm}0.04}$	$\mathbf{0.33{\pm}0.03}$	$0.45\pm 0.03$
	250	$0.18\pm 0.02$	$0.40\pm 0.04$	$0.52\pm 0.04$	$0.31\pm 0.02$	$0.35\pm 0.03$	$0.29\pm 0.04$	$0.56\pm 0.04$
	40	$0.11\pm 0.01$	$0.25\pm 0.03$	$0.30\pm 0.03$	$0.15\pm 0.02$	$0.20\pm 0.02$	$0.17\pm 0.02$	$\mathbf{0.67{\pm}0.05}$

Appendix

Table 8: Performance of different open-source LMMs and calibration baselines with standard errors under the traditional turn-based interface and our Multi-Agent Asynchronous Gym interface with fixed time intervals of 250 ms and 40 ms. TB denotes the turn-based interface. Coop. , Comp. , Mixed , and Single report the episode scores on the cooperative, competitive, mixed cooperative-competitive, and basic single-agent levels, respectively. Avg. reports the average episode score across the four levels. Shooting and Incoming denote shooting accuracy and incoming hit rate, respectively. Bold numbers indicate the best value for each model across the three evaluation interfaces.

Model	$\boldsymbol{\Delta t_{\mathrm{fixed}}}$	Surv. Time (s)	Kills	Coins	Turrets	Deaths	Eff. Act. Freq.
Gemma-4 31B	TB	$\mathbf{240.82{\pm}7.63}$	$\mathbf{0.85{\pm}0.05}$	$\mathbf{2.90{\pm}0.65}$	$\mathbf{1.43{\pm}0.41}$	$\mathbf{1.35{\pm}0.56}$	$\mathbf{1.00{\pm}0.00}$
	250	$236.95\pm 5.13$	$0.66\pm 0.06$	$2.38\pm 0.52$	$1.15\pm 0.51$	$1.45\pm 0.51$	$0.64\pm 0.06$
	40	$229.24\pm 6.85$	$0.12\pm 0.02$	$1.18\pm 0.47$	$0.83\pm 0.37$	$1.60\pm 0.67$	$0.11\pm 0.01$
Qwen-3.5 27B	TB	$\mathbf{252.27{\pm}6.96}$	$\mathbf{0.90{\pm}0.06}$	$\mathbf{2.83{\pm}0.62}$	$\mathbf{1.48{\pm}0.68}$	$\mathbf{1.38{\pm}0.42}$	$\mathbf{1.00{\pm}0.00}$
	250	$249.83\pm 5.83$	$0.76\pm 0.05$	$2.35\pm 0.58$	$1.18\pm 0.41$	$1.53\pm 0.57$	$0.75\pm 0.04$
	40	$243.11\pm 5.78$	$0.16\pm 0.03$	$1.28\pm 0.37$	$0.93\pm 0.44$	$1.68\pm 0.67$	$0.17\pm 0.02$

Appendix

Table 9: Additional in-game statistics of different open-source LMMs and calibration baselines with standard errors under the traditional turn-based interface and our Multi-Agent Asynchronous Gym interface with fixed time intervals of 250 ms and 40 ms. TB denotes the turn-based interface. Surv. Time denotes survival time in seconds. Kills , Coins , Turrets , and Deaths report the corresponding in-game statistics. Eff. Act. Freq. denotes effective action frequency. Bold numbers indicate the best value for each model across the three evaluation interfaces, with lower values being better for Deaths and higher values for all other metrics.

$\boldsymbol{\Delta t_{\mathrm{fixed}}}$ (ms)	Injected Delay (ms)	Total Latency P50/P90 (ms)	Coop.	Comp. vs. Bot	Mixed vs. Bot	Single	Avg.
TB	0	–	$0.21{\pm}0.02$	$0.42{\pm}0.03$	$0.48{\pm}0.04$	$0.36{\pm}0.03$	$0.37{\pm}0.03$
250	0	$210.4/471.3$	$0.18{\pm}0.02$	$0.38{\pm}0.04$	$0.44{\pm}0.03$	$0.31{\pm}0.02$	$0.33{\pm}0.03$
250	250	$460.4/721.3$	$0.15{\pm}0.02$	$0.34{\pm}0.03$	$0.39{\pm}0.02$	$0.28{\pm}0.03$	$0.29{\pm}0.04$
250	1000	$1210.4/1471.3$	$0.09{\pm}0.02$	$0.21{\pm}0.03$	$0.22{\pm}0.02$	$0.21{\pm}0.02$	$0.18{\pm}0.02$
250	3000	$3210.4/3471.3$	$0.02{\pm}0.01$	$0.16{\pm}0.02$	$0.16{\pm}0.02$	$0.14{\pm}0.02$	$0.12{\pm}0.02$

Appendix

Table 10: Performance under different injected delays and environment stepping settings.

$\boldsymbol{\Delta t_{\mathrm{fixed}}}$ (ms)	Injected Delay (ms)	Shooting	Incoming	Surv. Time (s)	Kills	Coins	Turrets	Deaths	Eff. Act. Freq.
TB	0	$0.32{\pm}0.02$	$0.47{\pm}0.04$	$245.31{\pm}6.46$	$0.70{\pm}0.03$	$2.83{\pm}0.62$	$1.48{\pm}0.68$	$1.38{\pm}0.42$	$1.00{\pm}0.00$
250	0	$0.27{\pm}0.03$	$0.57{\pm}0.03$	$242.42{\pm}5.98$	$0.63{\pm}0.02$	$2.35{\pm}0.58$	$1.18{\pm}0.41$	$1.53{\pm}0.57$	$0.74{\pm}0.04$
250	250	$0.25{\pm}0.02$	$0.60{\pm}0.03$	$226.28{\pm}7.02$	$0.48{\pm}0.03$	$2.03{\pm}0.75$	$1.13{\pm}0.44$	$1.65{\pm}0.43$	$0.49{\pm}0.03$
250	1000	$0.14{\pm}0.02$	$0.72{\pm}0.02$	$195.22{\pm}5.74$	$0.08{\pm}0.02$	$1.28{\pm}0.43$	$0.80{\pm}0.24$	$1.75{\pm}0.61$	$0.20{\pm}0.03$
250	3000	$0.07{\pm}0.01$	$0.86{\pm}0.05$	$135.72{\pm}6.26$	$0.00{\pm}0.00$	$0.03{\pm}0.02$	$0.33{\pm}0.11$	$1.78{\pm}0.59$	$0.08{\pm}0.02$

Appendix

Table 11: Additional in-game statistics under different injected delays and environment stepping settings.

Observation Modalities	Latency P50/P90 (ms)	Coop.	Single	Avg.	Shooting	Incoming
Text only (Turn-based)	1532.2 / 2341.5	0.23 $\pm$ 0.03	0.31 $\pm$ 0.02	0.27 $\pm$ 0.02	0.11 $\pm$ 0.01	0.56 $\pm$ 0.03
Pixel only (Turn-based)	2861.7 / 3601.4	0.66 $\pm$ 0.01	0.72 $\pm$ 0.04	0.69 $\pm$ 0.03	0.79 $\pm$ 0.05	0.39 $\pm$ 0.04
Pixel + Text (Turn-based)	3046.7 / 4008.9	0.78 $\pm$ 0.04	0.93 $\pm$ 0.05	0.86 $\pm$ 0.03	0.82 $\pm$ 0.04	0.31 $\pm$ 0.02
Pixel + Text + Depth (Turn-based)	3663.5 / 4785.6	0.81 $\pm$ 0.03	0.95 $\pm$ 0.04	0.88 $\pm$ 0.03	0.83 $\pm$ 0.03	0.29 $\pm$ 0.02
Pixel + Text + Depth + Audio (Turn-based)	3847.6 / 4965.8	0.85 $\pm$ 0.05	0.95 $\pm$ 0.03	0.90 $\pm$ 0.04	0.85 $\pm$ 0.06	0.28 $\pm$ 0.02
Text only (250 ms)	1532.2 / 2341.5	0.08 $\pm$ 0.01	0.15 $\pm$ 0.01	0.12 $\pm$ 0.02	0.05 $\pm$ 0.04	0.78 $\pm$ 0.07

Appendix

Table 12: Performance under different observation modalities. All results are reported as mean

\pm

standard error. Latency is reported as P50/P90 in milliseconds.

Model	$\boldsymbol{\Delta t_{\mathrm{fixed}}}$	Single	Shooting	Incoming	Latency P50/P90 (ms)
Gemma E4B (Base)	Turn-based	0.06 $\pm$ 0.01	0.00 $\pm$ 0.00	0.67 $\pm$ 0.04	62.1 / 129.8
	250 ms	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	0.69 $\pm$ 0.04	62.1 / 129.8
	40 ms	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	0.75 $\pm$ 0.03	62.1 / 129.8
Gemma E4B (RL fine-tuning)	Turn-based	0.17 $\pm$ 0.02	0.11 $\pm$ 0.02	0.59 $\pm$ 0.03	68.3 / 126.8
	250 ms	0.14 $\pm$ 0.03	0.10 $\pm$ 0.02	0.63 $\pm$ 0.04	68.3 / 126.8
	40 ms	0.08 $\pm$ 0.02	0.07 $\pm$ 0.01	0.70 $\pm$ 0.05	68.3 / 126.8

Appendix

Table 13: Performance of the base and RL fine-tuned Gemma E4B under different real-time settings. All results are reported as mean

\pm

standard error.

N agents	Game State	Pixel	Pixel + Depth	Memory (GB)
	P50 / P90 (ms)	P50 / P90 (ms)	P50 / P90 (ms)
4	2.3 / 3.2	5.8 / 6.1	12.6 / 15.3	0.20
8	6.2 / 9.1	18.7 / 23.4	35.3 / 38.4	0.21
12	15.2 / 17.6	35.9 / 38.5	43.5 / 49.1	0.21
16	22.6 / 27.8	56.5 / 60.9	76.3 / 82.0	0.22

Appendix

Table 14: Environment scalability with increasing agent count. Per-step update times are reported as P50/P90 in milliseconds. Memory is peak environment RAM.

AI Agent Benchmarks

AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.

See all 1,503 papers

Oct 14, 2026

The World Won't Wait: Evaluating and Teaching Agents Under a Live Clock

Large language model (LLM) agents can solve complex interactive tasks, yet reasoning and tool calls take time. In continuously changing environments, an otherwise correct action may arrive too late. We introduce RealtimeBench, with 180 tasks across six game types, to evaluate agents while games continue running during inference. Paired evaluations show that GPT-6.1-sol outperforms the faster Jev when games are paused, but falls behind when they keep running. We propose Two-Timescale Offline Recursive Self-Improvement (TORSI): a slow teacher studies failures and tests corrections in paused games between attempts, then writes reusable instructions for a fast student. The student acts without teacher calls during gameplay, and validation scores determine which updates are adopted. Two rounds raise Jev's average test score from 13.51 to 24.19 on a 0--100 progress scale, improving all six game types. The instructions also improve average scores for three other models without retuning. These findings point toward real-time intelligence: enabling powerful LLM agents to remain effective in a continuously changing world by organizing deliberation, learning, and action across timescales.

Ningxin Pan, Hanyu Li, Yeqing Shen +3

Peking University · Samsung Research, Beijing, China

LLM Agent Self-Improvement LLM Decision-Making LLM Agent Skill Learning Agent Benchmarks Game-Playing Agents

Ask PDF

Oct 10, 2025

LM Fight Arena: Benchmarking Large Multimodal Models via Game Competition

Existing benchmarks for large multimodal models (LMMs) often fail to capture their performance in real-time, adversarial environments. We introduce LM Fight Arena (Large Model Fight Arena), a novel framework that evaluates LMMs by pitting them against each other in the classic fighting game Mortal Kombat II, a task requiring rapid visual understanding and tactical, sequential decision-making. In a controlled tournament, we test six leading open- and closed-source models, where each agent operates controlling the same character to ensure a fair comparison. The models are prompted to interpret game frames and state data to select their next actions. Unlike static evaluations, LM Fight Arena provides a fully automated, reproducible, and objective assessment of an LMM's strategic reasoning capabilities in a dynamic setting. This work introduces a challenging and engaging benchmark that bridges the gap between AI evaluation and interactive entertainment.

Yushuo Zheng, Tongrui Ye, Zicheng Zhang +3

Shanghai Jiao Tong University, Shanghai, China · Shanghai AI Lab, Shanghai, China · Tongji University, Shanghai, China

Multimodal Large Language Models AI Agent Benchmarks Multimodal Model Evaluation

Ask PDF

Apr 8, 2026

GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents

Towards an embodied generalist for real-world interaction, Multimodal Large Language Model (MLLM) agents still suffer from challenging latency, sparse feedback, and irreversible mistakes. Video games offer an ideal testbed with rich visual observations and closed-loop interaction, demanding fine-grained perception, long-horizon planning, and precise control. However, systematically evaluating these capabilities is currently hindered by heterogeneous action interfaces and heuristic verification. To this end, we introduce GameWorld, a benchmark designed for standardized and verifiable evaluation of MLLMs as generalist game agents in browser environments. Two game agent interfaces are studied: (i) computer-use agents that directly emit keyboard and mouse controls, and (ii) generalist multimodal agents that act in a semantic action space via deterministic Semantic Action Parsing. GameWorld contains 34 diverse games and 170 tasks, each paired with state-verifiable metrics for outcome-based evaluation. The results across 18 model-interface pairs suggest that even the best performing agent is far from achieving human capabilities on video games. Extensive experiments of repeated full-benchmark reruns demonstrate the robustness of the benchmark, while further studies on real-time interaction, context-memory sensitivity, and action validity expose more challenges ahead for game agents. Together, by offering a standardized, verifiable, and reproducible evaluation framework, GameWorld lays a robust foundation for advancing research on multimodal game agents and beyond. The project page is at https://gameworld-bench.github.io.

Mingyu Ouyang, Siyuan Hu, Kevin Qinghong Lin +2

National University of Singapore · University of Oxford

AI Agent Evaluation AI Agent Benchmarks Multimodal Agents Game-Playing Agents

Ask PDF

Abstract

AI Agent Benchmarks

Explore similar work

The World Won't Wait: Evaluating and Teaching Agents Under a Live Clock

LM Fight Arena: Benchmarking Large Multimodal Models via Game Competition

GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents