Organizations: University of Texas at Austin · The Ohio State University · University of Illinois at Urbana-Champaign · NVIDIA · Northwestern University · Steadyworks
Real-time interaction requires agents to perceive evolving environments and act under temporal constraints, yet existing evaluations of real-time streaming models focus on conversational quality and streaming perception, leaving effective action under strict temporal constraints untested. We introduce Realtime-Bench, a suite of carefully curated dynamic tasks where delayed actions directly affect success. Under Wall Clock, where environments advance according to actual inference latency, the best success rate reaches only 44% and the best Pass^3 is only 36%, demonstrating that real-time interaction remains challenging. To separate serving latency from models' real-time streaming capability, we compare Wall Clock with Paused Clock, which removes inference delays as an idealized reference, and introduce Token Clock, which advances simulated time according to generated token counts, testing temporal awareness independently of physical serving speed. We find that temporal costs limit success in evolving environments and that higher reasoning effort has mixed effects under time pressure, reflecting a trade-off between decision quality and timely execution. Based on these results, we discuss future directions for improving real-time multimodal models, including adaptive reasoning with temporal awareness and continuous perception and action.
Figures & tables
Figure 1: Performance of Frontier Models on Realtime-Bench. Pass^3 measures the proportion of tasks successfully completed in all three independent trials, capturing the reliability needed in time-sensitive scenarios where agents often have only one chance to succeed.
Figure 2: Realtime-Bench evaluates models on interactive game and web tasks under three clock protocols. Wall Clock advances environments with actual inference latency, while Paused Clock removes inference delays as an idealized reference. Token Clock (ours) advances simulated time according to generation tokens consumed before action commitment, testing temporal awareness of models independently of physical serving speed.
Figure 3: Task distribution in Realtime-Bench.
Figure 4: Representative tasks in Realtime-Bench. Each panel shows the interface at the observation, the properties that make the task time-critical (①–③), and the window to act on the token-budget clock ( κ=Bf=64 tokens per simulated second). A 200-token response lands at 3.1 s: inside the auction’s 5 s final call (a), but 0.46 s past the one-cell turn window in Snake (b).
Figure 5: Task success under three timing protocols. We evaluate two streaming models and four asynchronous VLMs under (a) Wall Clock, which includes measured inference latency; (b) Paused Clock, which removes inference-time cost; and (c) Token Clock, which applies a standardized time cost per generated token.
Figure 6: Effect of temporal pressure on task success.
Figure 7: Distribution of failure modes across models.
Figure 8: Representative examples of the three failure modes. Each row shows one failed episode, from the states in which the model makes decisions to the resulting outcome. Timestamps are shown in the upper-right corner of each panel.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Action
Arguments
Semantics
click
target, button
Click the specified point.
right_click
target
Right-click the specified point.
double_click , triple_click
target
Apply the corresponding click sequence.
mouse_down , mouse_up
target, duration
Press or release the mouse button.
move
target, duration
Move the pointer.
type
text
Enter the supplied text.
Appendix
Table 1: Shared GUI action schema. Arguments include optional fields; action durations lie in [0,10] seconds and key.repeat lies in [1,100] .
Model type
Model
Run 1 (%)
Run 2 (%)
Run 3 (%)
Latency (s)
Standard Asynchronous Models
GPT-6-Astra (high)
42.00
42.00
44.00
3.989
GPT-6-Astra (low)
42.00
40.00
44.00
3.339
GPT-5.6 Sol (high)
8.00
10.00
14.00
5.735
GPT-5.6 Sol (low)
16.00
14.00
24.00
4.490
Claude Fable 5.1 (high)
16.00
18.00
16.00
6.506
Claude Fable 5.1 (low)
18.00
16.00
14.00
5.475
Appendix
Table 2: Per-run success rates and average decision latency.
Model type
Model
Pass^1
Pass^2
Pass^3
Standard Asynchronous Models
GPT-6-Astra (high)
42.67
38.00
36.00
GPT-6-Astra (low)
42.00
34.00
30.00
GPT-5.6 Sol (high)
10.67
5.33
4.00
GPT-5.6 Sol (low)
18.00
10.00
6.00
Claude Fable 5.1 (high)
16.67
12.67
10.00
Claude Fable 5.1 (low)
16.00
12.00
10.00
Appendix
Table 3: Pass^k results (%). Pass^k measures success on all k attempts, averaged over combinations of the three runs.
Model type
Model
Pass@1
Pass@2
Pass@3
Standard Asynchronous Models
GPT-6-Astra (high)
42.67
47.33
50.00
GPT-6-Astra (low)
42.00
50.00
54.00
GPT-5.6 Sol (high)
10.67
16.00
20.00
GPT-5.6 Sol (low)
18.00
26.00
30.00
Claude Fable 5.1 (high)
16.67
20.67
22.00
Claude Fable 5.1 (low)
16.00
20.00
22.00
Appendix
Table 4: Pass@k results (%). Pass@k measures success on at least one of k attempts, averaged over combinations of the three runs.
Model
B=32
B=64
B=128
B=∞
GPT-Realtime-2
8.0
22.0
18.0
16.0
Gemini Live
4.0
2.0
18.0
22.0
GPT-5.6 Sol
26.0
34.0
32.0
42.0
Claude Opus 5
30.0
44.0
56.0
62.0
Gemini 3.8 Flash
56.0
54.0
58.0
60.0
Kimi-K3
22.0
24.0
58.0
60.0
Appendix
Table 5: Token-rate ablation: overall success rates (%) over all 50 tasks under low reasoning effort. The B=∞ condition removes simulated inference delay and uses the paused setting.
Figure 9: Reasoning longer with less time in Live auction (Gemini Live, TH). (a) With 3 s left, the model generates 88 tokens, and its click lands at 44.4 s. (b) With 1 s left, it generates 544 tokens (8.5 s) for the same click, which would land at 53.5 s. (c) The lot closes at 46 s before the click is applied. The bid field already holds amounts concatenated from earlier inputs.
Figure 10: Inertia in Google Snake (Gemini Live, PH). (a) At 7 s the model turns left on the row below the apple. (b) It then waits at every step, even at 14 s, when the head is directly below the apple. (c) At 16 s it has passed the apple and still waits. (d) The snake hits the left wall at 17 s.
Figure 11: Waiting too long in Shared document (Gemini 3.8 Flash, PH). At the start, the model plans to “calculate the earliest legal date for C6 and verify the save within the remaining five seconds.” (a) At 55 s the edits have stopped and 5 s remain, but the model waits 5 s. (b) The deadline passes with C6 unchanged; the correct date is Oct 21, 2026.
Figure 12: Deliberating through a reachability window in Waitlist booking (Kimi K3, WH). (a) At 0 s, Devon Bell, the longest-waiting client whose request fits, is reachable for 85 s. The model notes “85 seconds to message Devon… The clock is ticking in real time” but takes 44.8 s to open the calendar. (b) At 46 s, with 40 s left, it writes “I need to message Devon NOW”, then takes 42.3 s to click back to the waitlist. The click arrives after Devon’s meeting has started. Run 1 under the same setting succeeds.
Figure 13: Overthinking a timed guess in Wordle (GPT-5.6 Sol, WH). Each guess has a 10-s clock that resets after every submitted guess. (a) At 7 s SLATE has been scored. The first three guesses take 4.4, 6.6, and 7.8 s. (b) At 22 s, just after GLIDE is submitted, the model writes “I need to act quickly before the 3-second mark” but spends 17.2 s choosing OLIVE. (c) At 24 s GLIDE has been scored and the clock has reset, while the model is still generating. (d) The board locks at about 32 s, and OLIVE arrives at 39.3 s. Panels (c) and (d) are taken from the episode recording. The same model solves this task in 2 of 3 WL runs.
Figure 14: Deliberating past the cutoff in Flight gate assignment (Gemini 3.8 Flash, WH). (a) At 9 s three flights are on the board, none of them assigned, and the sequencing cutoff is 19 s away. The model’s reasoning notes that “LLM generation time and network latency are approximately 5-7 seconds. If I wait, real time advances,” yet the call takes 16.6 s and returns wait . (b) At 26 s, with 2 s left, all five flights are still unassigned, and the plan is never committed. The same model solves this task in 2 of 3 WL runs.
Figure 15: Letting room holds lapse in Hotel booking (OpenAI Decisions, Wall Clock). (a) At 7 s the qualifying Harbor King rate is held with 0:02 left, its terms have been reviewed, and Book reviewed room is enabled. The model nevertheless answers wait (0.9-s call). (b) At 9 s the hold has lapsed. (c) The model then holds Courtyard King, which also qualifies, while Harbor King has been repriced to a nonrefundable rate. It answers wait at 11 and 13 s while the new hold runs down. (d) At 15 s this hold has also lapsed. Soon afterwards, the qualifying inventory closes, ending the episode. Runs 2 and 3 of the same model book in time.
Figure 16: Buying after the dip has ended in Market dip purchase (Jev 1.13, Wall Clock). BTC may be bought only while its price is below 60,000.(a)At5sBTCisat58,365, and Jev 1.13 starts the order by focusing the amount field and typing 0.02. (b) At 7 s the price has rebounded to 62,065,yetJev1.13clicksMarketBuyBTCaftera0.4−scall.(c)At8sthequoteshowsatriggerpriceof61,315, and Jev 1.13 confirms it. (d) The order fills at $61,345.66, above the limit. Run 1 follows the same click sequence and succeeds.
Large language model (LLM) agents can solve complex interactive tasks, yet reasoning and tool calls take time. In continuously changing environments, an otherwise correct action may arrive too late. We introduce RealtimeBench, with 180 tasks across six game types, to evaluate agents while games continue running during inference. Paired evaluations show that GPT-6.1-sol outperforms the faster Jev when games are paused, but falls behind when they keep running. We propose Two-Timescale Offline Recursive Self-Improvement (TORSI): a slow teacher studies failures and tests corrections in paused games between attempts, then writes reusable instructions for a fast student. The student acts without teacher calls during gameplay, and validation scores determine which updates are adopted. Two rounds raise Jev's average test score from 13.51 to 24.19 on a 0--100 progress scale, improving all six game types. The instructions also improve average scores for three other models without retuning. These findings point toward real-time intelligence: enabling powerful LLM agents to remain effective in a continuously changing world by organizing deliberation, learning, and action across timescales.
Ningxin Pan, Hanyu Li, Yeqing Shen +3
Peking University · Samsung Research, Beijing, China
AI agents have made substantial progress in domains where they have a non-trivial amount of time to deliberate between actions, such as mathematics, software engineering, and performing tasks on desktop applications. It is less clear how such models fare in settings that demand real-time interaction, such as playing a video game or operating a physical device. In these settings, the environment state may evolve as the agent is reasoning, or even as the underlying model is performing a forward pass. To gain insight into the real-time interaction capabilities of vision-language agents, we introduce Gym-Anything-RT. Our key idea is to construct visual minigames that demand a portfolio of capabilities required for interactive real-time settings. Gym-Anything-RT uses an agentic pipeline to automatically generate such environments based on a taxonomy of six capabilities required for real-time interaction, including the ability to perceive the environment and act under an evolving environment state. A key feature of Gym-Anything-RT is that it allows for controlled comparison of real-time interaction capabilities: we can run the same minigame in real time, or pause or slow down the environment to relax the requirement for real-time actions. We use Gym-Anything-RT to construct 100 real-time minigames and evaluate AI agents on them. We find large capability gaps in real-time interaction. For example, GPT-6 Astra (xHigh) achieves a pass rate of only 9.0% on the real-time minigames, which rises to 75.0% when the environment is paused. We also identify intriguing behaviors by the most powerful frontier agents: for instance, when given the ability to write code, these agents write code that bypasses the need to perceive and act in real time. We publicly release Gym-Anything-RT and the generated minigames to spur further research on AI agents in real-time interactive settings.
There is a growing demand for agentic AI technologies for a range of downstream applications like customer service and personal assistants. For applications where the agent needs to interact with a person, real-time low-latency responsiveness is required; for example, with voice-controlled applications, under 1 second of latency is typically required for the interaction to feel seamless. However, if we want the LLM to reason and execute an agentic workflow with tool calling, this can add several seconds or more of latency, which is prohibitive for real-time latency-sensitive applications. In our work, we propose Speculative Interaction Agents to enable real-time interaction even for agents with complex multi-turn tool calling. We propose Asynchronous I/O, which decouples the core agent reason-and-act thread from waiting for additional information from either the user or environment, thereby allowing for overlapping agentic processing while waiting on external delays. We also propose Speculative Tool Calling as a method to manage task execution when the agent is still unsure if it has received the full information or if additional user information may later be provided. For strong cloud models, our method can be applied out-of-the-box to existing real-time cloud APIs, providing 1.3-1.7× speedups with minor accuracy loss. To enable real-time interaction with small edge-scale models, we also present a clock-based training methodology that adapts the model to handle streaming inputs and asynchronous responses, and demonstrate a synthetic data generation strategy for SFT. Altogether, this approach provides 1.6-2.2× speedups with the Qwen2.5-3B-Instruct and Llama-3.2-3B-Instruct models across multiple tool calling benchmarks.