Gym-Anything-RT: Scaling Real-Time Environments for Vision-Language Agents

Pranjal Aggarwal; Sean Welleck

cs.LGOct 27, 2026

Gym-Anything-RT: Scaling Real-Time Environments for Vision-Language Agents

Authors: Pranjal Aggarwal, Sean Welleck

Organizations: Carnegie Mellon University

Abstract

AI agents have made substantial progress in domains where they have a non-trivial amount of time to deliberate between actions, such as mathematics, software engineering, and performing tasks on desktop applications. It is less clear how such models fare in settings that demand real-time interaction, such as playing a video game or operating a physical device. In these settings, the environment state may evolve as the agent is reasoning, or even as the underlying model is performing a forward pass. To gain insight into the real-time interaction capabilities of vision-language agents, we introduce Gym-Anything-RT. Our key idea is to construct visual minigames that demand a portfolio of capabilities required for interactive real-time settings. Gym-Anything-RT uses an agentic pipeline to automatically generate such environments based on a taxonomy of six capabilities required for real-time interaction, including the ability to perceive the environment and act under an evolving environment state. A key feature of Gym-Anything-RT is that it allows for controlled comparison of real-time interaction capabilities: we can run the same minigame in real time, or pause or slow down the environment to relax the requirement for real-time actions. We use Gym-Anything-RT to construct 100 real-time minigames and evaluate AI agents on them. We find large capability gaps in real-time interaction. For example, GPT-6 Astra (xHigh) achieves a pass rate of only 9.0% on the real-time minigames, which rises to 75.0% when the environment is paused. We also identify intriguing behaviors by the most powerful frontier agents: for instance, when given the ability to write code, these agents write code that bypasses the need to perceive and act in real time. We publicly release Gym-Anything-RT and the generated minigames to spur further research on AI agents in real-time interactive settings.

Figures & tables

Figure 1: Examples of real-time minigames generated with Gym-Anything-RT . The 100 minigames in our benchmark cover different combinations of capabilities, with controlled variations in difficulty and interaction.

Panel of Figure 2 — Figure 2: Overview of our construction method. A creator agent implements selected minigames with controllable difficulty and interaction, constructs oracle solutions, and records evidence of their behavior. An independent auditor checks the implementation and evidence, returning issues for revision until the minigame passes the audit. Review agents calibrate difficulty and annotate capability requirements for controlled model evaluation.

Panel of Figure 3 — Figure 3: Rotating On-Screen Keyboard. The model must enter a code by clicking keys that move between observation and action.

Statistic	Count
Real-time minigames	100
Non-real-time counterparts	100
Difficulty levels	5
Interaction modes	2
Configurations	2,000
Real-time	1,000

Table 1: Benchmark statistics.

Panel of Figure 4 — Figure 4: Pass rates (%) on Gym-Anything-RT at difficulty 5 with full interaction in the live setting. Each model is evaluated on the 100 real-time tasks and their 100 non-real-time counterparts. Even the strongest model performs poorly on tasks requiring real-time interaction.

Panel of Figure 5 — Figure 5: Effect of environment speed on Astra. Pass rates increase as the environment slows, with xHigh remaining below its paused performance at $64\times$ slowdown.

Table 7

Panel of Figure 6 — Figure 6: Effect of pausing on Luna. Pausing improves performance when real-time interaction or temporal reasoning is required. $p$ -values are Holm-adjusted.

	Difficulty
	1	2	3	4	5	All
Real-time	15.0	6.0	4.0	4.0	1.0	6.0
Non-real-time	35.0	33.0	22.0	18.0	17.0	25.0
Gap	20.0	27.0	18.0	14.0	16.0	19.0

Table 4: Pass rates (%) of GPT-6 Luna (xHigh) across difficulty levels, with full interaction in the live setting. Gap is the pass rate on the non-real-time counterparts minus the pass rate on the real-time tasks.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Level	Code length	Keys	Keyboard motion
1	3	12 large keys	Gentle, begins after the second character
2	4	20 keys	Begins after the first character
3	5	26 keys	Faster, begins after the first character
4	5	32 keys	Faster, changes direction more often
5	8	32 smaller keys	Fast, sharply irregular, starts immediately

Appendix

Table 5: Difficulty levels of Rotating On-Screen Keyboard. The keyboard rotates in three dimensions with irregularly timed motion that never repeats within an attempt.

Minigame	Simplified	Full
Rotating On-Screen Keyboard	Type the code on the physical keyboard.	Click each on-screen key while it moves.
Clockwork Clutch Safe	Click a labeled release or engage button beneath each seal.	Drag each seal’s clutch lever between its coupled and free positions.
Parallel Grillmaster	Select a food, then click buttons to start cooking and move it to the tray.	Drag each food from the order to the grill and then to the serving tray.
LIDAR Blacksite	Hold labeled movement and turning buttons, and use buttons to scan, pick up, and verify.	Hold movement keys, drag the view to turn, click the scene to scan or pick up the beacon, and walk through the exit gate.

Appendix

Table 6: Examples of simplified and full interaction.

Minigame	T	R	E	Minigame	3D	T	R	E
After Hours at the Reliquary	$\circ$	$\bullet$	$\bullet$	Impossible Ecology		$\bullet$	$\bullet$	$\bullet$
Anthill Front	$\bullet$	$\bullet$	$\bullet$	Impossible Panorama	$\bullet$	$\bullet$	$\bullet$	$\bullet$
Apothecary Dead Reckoning	$\bullet$	$\bullet$	$\bullet$	Isometric Voxel Extraction Mine	$\bullet$	$\circ$	$\bullet$	$\bullet$
Ballast Lantern	$\bullet$	$\bullet$		Lampwrights Program	$\bullet$	$\circ$	$\bullet$
Bandaged Rose Window	$\bullet$	$\bullet$	$\bullet$	Lantern Loft	$\bullet$	$\bullet$	$\bullet$
Blind Corridor Oscilloscope	$\bullet$	$\bullet$	$\bullet$	Lanternfin Dive	$\bullet$	$\bullet$	$\bullet$

Appendix

Table 7: Minigames in Gym-Anything-RT . Capability labels: 3D visual understanding (3D), temporal reasoning (T), reasoning and planning (R), and exploration and interface understanding (E). A capability is marked when it is required at some difficulty level.

\bullet

: required by the real-time minigame and its non-real-time counterpart.

\circ

: required only by the real-time minigame.

\diamond

: required only by the counterpart.

Model	Input	Cached input	Output
Qwen3.5-9B	0.17	–	0.25
Qwen3.8-27B	0.20	0.09	2.50
GPT-6 Luna	0.10	0.01	0.50
GPT-6 Astra	10.00	1.00	50.00

Appendix

Table 8: Prices used to compute cost , in USD per million tokens.

AI Agent Benchmarks

AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.

See all 1,497 papers

Realtime-Bench: Can Frontier Models Solve Real-time Interactive Problems?
Oct 15, 2026Jingyu Tang, Xu Yang, Zhehao Zhang +7Agent Evaluation AI Agent Benchmarks
University of Texas at Austin · The Ohio State University · University of Illinois at Urbana-Champaign +3
The World Won't Wait: Evaluating and Teaching Agents Under a Live Clock
Oct 14, 2026Ningxin Pan, Hanyu Li, Yeqing Shen +3LLM Agent Self-Improvement LLM Decision-Making
Peking University · Samsung Research, Beijing, China
Proact-VL: A Proactive VideoLLM for Real-Time AI Companions
Mar 3, 2026Weicai Yan, Yuhong Dai, Qi Ran +6AI Companions Vision-Language Agents
Zhejiang University · Shenzhen University · South China University of Technology +2

Oct 15, 2026J/K move · Enter open · S save

Realtime-Bench: Can Frontier Models Solve Real-time Interactive Problems?

Jingyu Tang, Xu Yang, Zhehao Zhang, Zheng Wang+6

University of Texas at Austin · The Ohio State University · University of Illinois at Urbana-Champaign+3

Agent Evaluation AI Agent Benchmarks Multimodal Agents

Open paper AskPDF arXiv

Real-time interaction requires agents to perceive evolving environments and act under temporal constraints, yet existing evaluations of real-time streaming models focus on conversational quality and streaming perception, leaving effective action under strict temporal constraints untested. We introduce Realtime-Bench, a suite of carefully curated dynamic tasks where delayed actions directly affect success. Under Wall Clock, where environments advance according to actual inference latency, the best success rate reaches only 44% and the best Pass^3 is only 36%, demonstrating that real-time interaction remains challenging. To separate serving latency from models' real-time streaming capability, we compare Wall Clock with Paused Clock, which removes inference delays as an idealized reference, and introduce Token Clock, which advances simulated time according to generated token counts, testing temporal awareness independently of physical serving speed. We find that temporal costs limit success in evolving environments and that higher reasoning effort has mixed effects under time pressure, reflecting a trade-off between decision quality and timely execution. Based on these results, we discuss future directions for improving real-time multimodal models, including adaptive reasoning with temporal awareness and continuous perception and action.

Abstract

AI Agent Benchmarks

Explore similar work

Realtime-Bench: Can Frontier Models Solve Real-time Interactive Problems?

The World Won't Wait: Evaluating and Teaching Agents Under a Live Clock

Proact-VL: A Proactive VideoLLM for Real-Time AI Companions