AI agents have made substantial progress in domains where they have a non-trivial amount of time to deliberate between actions, such as mathematics, software engineering, and performing tasks on desktop applications. It is less clear how such models fare in settings that demand real-time interaction, such as playing a video game or operating a physical device. In these settings, the environment state may evolve as the agent is reasoning, or even as the underlying model is performing a forward pass. To gain insight into the real-time interaction capabilities of vision-language agents, we introduce Gym-Anything-RT. Our key idea is to construct visual minigames that demand a portfolio of capabilities required for interactive real-time settings. Gym-Anything-RT uses an agentic pipeline to automatically generate such environments based on a taxonomy of six capabilities required for real-time interaction, including the ability to perceive the environment and act under an evolving environment state. A key feature of Gym-Anything-RT is that it allows for controlled comparison of real-time interaction capabilities: we can run the same minigame in real time, or pause or slow down the environment to relax the requirement for real-time actions. We use Gym-Anything-RT to construct 100 real-time minigames and evaluate AI agents on them. We find large capability gaps in real-time interaction. For example, GPT-6 Astra (xHigh) achieves a pass rate of only 9.0% on the real-time minigames, which rises to 75.0% when the environment is paused. We also identify intriguing behaviors by the most powerful frontier agents: for instance, when given the ability to write code, these agents write code that bypasses the need to perceive and act in real time. We publicly release Gym-Anything-RT and the generated minigames to spur further research on AI agents in real-time interactive settings.
Figures & tables
Figure 1: Examples of real-time minigames generated with Gym-Anything-RT . The 100 minigames in our benchmark cover different combinations of capabilities, with controlled variations in difficulty and interaction.
Figure 2: Overview of our construction method. A creator agent implements selected minigames with controllable difficulty and interaction, constructs oracle solutions, and records evidence of their behavior. An independent auditor checks the implementation and evidence, returning issues for revision until the minigame passes the audit. Review agents calibrate difficulty and annotate capability requirements for controlled model evaluation.
Figure 3: Rotating On-Screen Keyboard. The model must enter a code by clicking keys that move between observation and action.
Statistic
Count
Real-time minigames
100
Non-real-time counterparts
100
Difficulty levels
5
Interaction modes
2
Configurations
2,000
Real-time
1,000
Table 1: Benchmark statistics.
Figure 4: Pass rates (%) on Gym-Anything-RT at difficulty 5 with full interaction in the live setting. Each model is evaluated on the 100 real-time tasks and their 100 non-real-time counterparts. Even the strongest model performs poorly on tasks requiring real-time interaction.
Figure 5: Effect of environment speed on Astra. Pass rates increase as the environment slows, with xHigh remaining below its paused performance at 64× slowdown.
Table 7
Figure 6: Effect of pausing on Luna. Pausing improves performance when real-time interaction or temporal reasoning is required. p -values are Holm-adjusted.
Figure 9
Difficulty
1
2
3
4
5
All
Real-time
15.0
6.0
4.0
4.0
1.0
6.0
Non-real-time
35.0
33.0
22.0
18.0
17.0
25.0
Gap
20.0
27.0
18.0
14.0
16.0
19.0
Table 4: Pass rates (%) of GPT-6 Luna (xHigh) across difficulty levels, with full interaction in the live setting. Gap is the pass rate on the non-real-time counterparts minus the pass rate on the real-time tasks.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Level
Code length
Keys
Keyboard motion
1
3
12 large keys
Gentle, begins after the second character
2
4
20 keys
Begins after the first character
3
5
26 keys
Faster, begins after the first character
4
5
32 keys
Faster, changes direction more often
5
8
32 smaller keys
Fast, sharply irregular, starts immediately
Appendix
Table 5: Difficulty levels of Rotating On-Screen Keyboard. The keyboard rotates in three dimensions with irregularly timed motion that never repeats within an attempt.
Minigame
Simplified
Full
Rotating On-Screen Keyboard
Type the code on the physical keyboard.
Click each on-screen key while it moves.
Clockwork Clutch Safe
Click a labeled release or engage button beneath each seal.
Drag each seal’s clutch lever between its coupled and free positions.
Parallel Grillmaster
Select a food, then click buttons to start cooking and move it to the tray.
Drag each food from the order to the grill and then to the serving tray.
LIDAR Blacksite
Hold labeled movement and turning buttons, and use buttons to scan, pick up, and verify.
Hold movement keys, drag the view to turn, click the scene to scan or pick up the beacon, and walk through the exit gate.
Appendix
Table 6: Examples of simplified and full interaction.
Minigame
3D
T
R
E
Minigame
3D
T
R
E
After Hours at the Reliquary
∘
∙
∙
Impossible Ecology
∙
∙
∙
Anthill Front
∙
∙
∙
Impossible Panorama
∙
∙
∙
∙
Apothecary Dead Reckoning
∙
∙
∙
Isometric Voxel Extraction Mine
∙
∘
∙
∙
Ballast Lantern
∙
∙
Lampwrights Program
∙
∘
∙
Bandaged Rose Window
∙
∙
∙
Lantern Loft
∙
∙
∙
Blind Corridor Oscilloscope
∙
∙
∙
Lanternfin Dive
∙
∙
∙
Appendix
Table 7: Minigames in Gym-Anything-RT . Capability labels: 3D visual understanding (3D), temporal reasoning (T), reasoning and planning (R), and exploration and interface understanding (E). A capability is marked when it is required at some difficulty level. ∙ : required by the real-time minigame and its non-real-time counterpart. ∘ : required only by the real-time minigame. ⋄ : required only by the counterpart.
Model
Input
Cached input
Output
Qwen3.5-9B
0.17
–
0.25
Qwen3.8-27B
0.20
0.09
2.50
GPT-6 Luna
0.10
0.01
0.50
GPT-6 Astra
10.00
1.00
50.00
Appendix
Table 8: Prices used to compute cost , in USD per million tokens.
Real-time interaction requires agents to perceive evolving environments and act under temporal constraints, yet existing evaluations of real-time streaming models focus on conversational quality and streaming perception, leaving effective action under strict temporal constraints untested. We introduce Realtime-Bench, a suite of carefully curated dynamic tasks where delayed actions directly affect success. Under Wall Clock, where environments advance according to actual inference latency, the best success rate reaches only 44% and the best Pass^3 is only 36%, demonstrating that real-time interaction remains challenging. To separate serving latency from models' real-time streaming capability, we compare Wall Clock with Paused Clock, which removes inference delays as an idealized reference, and introduce Token Clock, which advances simulated time according to generated token counts, testing temporal awareness independently of physical serving speed. We find that temporal costs limit success in evolving environments and that higher reasoning effort has mixed effects under time pressure, reflecting a trade-off between decision quality and timely execution. Based on these results, we discuss future directions for improving real-time multimodal models, including adaptive reasoning with temporal awareness and continuous perception and action.
Jingyu Tang, Xu Yang, Zhehao Zhang +7
University of Texas at Austin · The Ohio State University · University of Illinois at Urbana-Champaign +3
Large language model (LLM) agents can solve complex interactive tasks, yet reasoning and tool calls take time. In continuously changing environments, an otherwise correct action may arrive too late. We introduce RealtimeBench, with 180 tasks across six game types, to evaluate agents while games continue running during inference. Paired evaluations show that GPT-6.1-sol outperforms the faster Jev when games are paused, but falls behind when they keep running. We propose Two-Timescale Offline Recursive Self-Improvement (TORSI): a slow teacher studies failures and tests corrections in paused games between attempts, then writes reusable instructions for a fast student. The student acts without teacher calls during gameplay, and validation scores determine which updates are adopted. Two rounds raise Jev's average test score from 13.51 to 24.19 on a 0--100 progress scale, improving all six game types. The instructions also improve average scores for three other models without retuning. These findings point toward real-time intelligence: enabling powerful LLM agents to remain effective in a continuously changing world by organizing deliberation, learning, and action across timescales.
Ningxin Pan, Hanyu Li, Yeqing Shen +3
Peking University · Samsung Research, Beijing, China
Proactive and real-time interactive experiences are essential for human-like AI companions, yet face three key challenges: (1) achieving low-latency inference under continuous streaming inputs, (2) autonomously deciding when to respond, and (3) controlling both quality and quantity of generated content to meet real-time constraints. In this work, we instantiate AI companions through two gaming scenarios, commentator and guide, selected for their suitability for automatic evaluation. We introduce the Live Gaming Benchmark, a large-scale dataset with three representative scenarios: solo commentary, co-commentary, and user guidance, and present Proact-VL, a general framework that shapes multimodal language models into proactive, real-time interactive agents capable of human-like environment perception and interaction. Extensive experiments show Proact-VL achieves superior response latency and quality while maintaining strong video understanding capabilities, demonstrating its practicality for real-time interactive applications.
Weicai Yan, Yuhong Dai, Qi Ran +6
Zhejiang University · Shenzhen University · South China University of Technology +2