cs.AIOct 15, 2026
SaveRealtime-Bench: Can Frontier Models Solve Real-time Interactive Problems?
Organizations: University of Texas at Austin · The Ohio State University · University of Illinois at Urbana-Champaign · NVIDIA ·
Abstract
Real-time interaction requires agents to perceive evolving environments and act under temporal constraints, yet existing evaluations of real-time streaming models focus on conversational quality and streaming perception, leaving effective action under strict temporal constraints untested. We introduce Realtime-Bench, a suite of carefully curated dynamic tasks where delayed actions directly affect success. Under Wall Clock, where environments advance according to actual inference latency, the best success rate reaches only 44% and the best Pass^3 is only 36%, demonstrating that real-time interaction remains challenging. To separate serving latency from models' real-time streaming capability, we compare Wall Clock with Paused Clock, which removes inference delays as an idealized reference, and introduce Token Clock, which advances simulated time according to generated token counts, testing temporal awareness independently of physical serving speed. We find that temporal costs limit success in evolving environments and that higher reasoning effort has mixed effects under time pressure, reflecting a trade-off between decision quality and timely execution. Based on these results, we discuss future directions for improving real-time multimodal models, including adaptive reasoning with temporal awareness and continuous perception and action.
Figures & tables
Figure 1: Performance of Frontier Models on Realtime-Bench. Pass^3 measures the proportion of tasks successfully completed in all three independent trials, capturing the reliability needed in time-sensitive scenarios where agents often have only one chance to succeed.
Figure 2: Realtime-Bench evaluates models on interactive game and web tasks under three clock protocols. Wall Clock advances environments with actual inference latency, while Paused Clock removes inference delays as an idealized reference. Token Clock (ours) advances simulated time according to generation tokens consumed before action commitment, testing temporal awareness of models independently of physical serving speed.
Figure 3: Task distribution in Realtime-Bench.
Figure 4: Representative tasks in Realtime-Bench. Each panel shows the interface at the observation, the properties that make the task time-critical (①–③), and the window to act on the token-budget clock ( tokens per simulated second). A 200-token response lands at s: inside the auction’s 5 s final call (a), but s past the one-cell turn window in Snake (b).
Figure 5: Task success under three timing protocols. We evaluate two streaming models and four asynchronous VLMs under (a) Wall Clock, which includes measured inference latency; (b) Paused Clock, which removes inference-time cost; and (c) Token Clock, which applies a standardized time cost per generated token.
Figure 6: Effect of temporal pressure on task success.
Figure 7: Distribution of failure modes across models.
Figure 8: Representative examples of the three failure modes. Each row shows one failed episode, from the states in which the model makes decisions to the resulting outcome. Timestamps are shown in the upper-right corner of each panel.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Action | Arguments | Semantics |
|---|---|---|
| click | target, button | Click the specified point. |
| right_click | target | Right-click the specified point. |
| double_click , triple_click | target | Apply the corresponding click sequence. |
| mouse_down , mouse_up | target, duration | Press or release the mouse button. |
| move | target, duration | Move the pointer. |
| type | text | Enter the supplied text. |
Table 1: Shared GUI action schema. Arguments include optional fields; action durations lie in seconds and key.repeat lies in .
| Model type | Model | Run 1 (%) | Run 2 (%) | Run 3 (%) | Latency (s) |
| Standard Asynchronous Models | GPT-6-Astra (high) | 42.00 | 42.00 | 44.00 | 3.989 |
| GPT-6-Astra (low) | 42.00 | 40.00 | 44.00 | 3.339 | |
| GPT-5.6 Sol (high) | 8.00 | 10.00 | 14.00 | 5.735 | |
| GPT-5.6 Sol (low) | 16.00 | 14.00 | 24.00 | 4.490 | |
| Claude Fable 5.1 (high) | 16.00 | 18.00 | 16.00 | 6.506 | |
| Claude Fable 5.1 (low) | 18.00 | 16.00 | 14.00 | 5.475 |
Table 2: Per-run success rates and average decision latency.
| Model type | Model | Pass^1 | Pass^2 | Pass^3 |
| Standard Asynchronous Models | GPT-6-Astra (high) | 42.67 | 38.00 | 36.00 |
| GPT-6-Astra (low) | 42.00 | 34.00 | 30.00 | |
| GPT-5.6 Sol (high) | 10.67 | 5.33 | 4.00 | |
| GPT-5.6 Sol (low) | 18.00 | 10.00 | 6.00 | |
| Claude Fable 5.1 (high) | 16.67 | 12.67 | 10.00 | |
| Claude Fable 5.1 (low) | 16.00 | 12.00 | 10.00 |
Table 3: Pass^k results (%). Pass^k measures success on all k attempts, averaged over combinations of the three runs.
| Model type | Model | Pass@1 | Pass@2 | Pass@3 |
| Standard Asynchronous Models | GPT-6-Astra (high) | 42.67 | 47.33 | 50.00 |
| GPT-6-Astra (low) | 42.00 | 50.00 | 54.00 | |
| GPT-5.6 Sol (high) | 10.67 | 16.00 | 20.00 | |
| GPT-5.6 Sol (low) | 18.00 | 26.00 | 30.00 | |
| Claude Fable 5.1 (high) | 16.67 | 20.67 | 22.00 | |
| Claude Fable 5.1 (low) | 16.00 | 20.00 | 22.00 |
Table 4: Pass@k results (%). Pass@k measures success on at least one of k attempts, averaged over combinations of the three runs.
| Model | ||||
|---|---|---|---|---|
| GPT-Realtime-2 | 8.0 | 22.0 | 18.0 | 16.0 |
| Gemini Live | 4.0 | 2.0 | 18.0 | 22.0 |
| GPT-5.6 Sol | 26.0 | 34.0 | 32.0 | 42.0 |
| Claude Opus 5 | 30.0 | 44.0 | 56.0 | 62.0 |
| Gemini 3.8 Flash | 56.0 | 54.0 | 58.0 | 60.0 |
| Kimi-K3 | 22.0 | 24.0 | 58.0 | 60.0 |
Table 5: Token-rate ablation: overall success rates (%) over all 50 tasks under low reasoning effort. The condition removes simulated inference delay and uses the paused setting.
Figure 9: Reasoning longer with less time in Live auction (Gemini Live, TH). (a) With 3 s left, the model generates 88 tokens, and its click lands at 44.4 s. (b) With 1 s left, it generates 544 tokens (8.5 s) for the same click, which would land at 53.5 s. (c) The lot closes at 46 s before the click is applied. The bid field already holds amounts concatenated from earlier inputs.
Figure 10: Inertia in Google Snake (Gemini Live, PH). (a) At 7 s the model turns left on the row below the apple. (b) It then waits at every step, even at 14 s, when the head is directly below the apple. (c) At 16 s it has passed the apple and still waits. (d) The snake hits the left wall at 17 s.
Figure 11: Waiting too long in Shared document (Gemini 3.8 Flash, PH). At the start, the model plans to “calculate the earliest legal date for C6 and verify the save within the remaining five seconds.” (a) At 55 s the edits have stopped and 5 s remain, but the model waits 5 s. (b) The deadline passes with C6 unchanged; the correct date is Oct 21, 2026.
Figure 12: Deliberating through a reachability window in Waitlist booking (Kimi K3, WH). (a) At 0 s, Devon Bell, the longest-waiting client whose request fits, is reachable for 85 s. The model notes “85 seconds to message Devon… The clock is ticking in real time” but takes 44.8 s to open the calendar. (b) At 46 s, with 40 s left, it writes “I need to message Devon NOW”, then takes 42.3 s to click back to the waitlist. The click arrives after Devon’s meeting has started. Run 1 under the same setting succeeds.
Figure 13: Overthinking a timed guess in Wordle (GPT-5.6 Sol, WH). Each guess has a 10-s clock that resets after every submitted guess. (a) At 7 s SLATE has been scored. The first three guesses take 4.4, 6.6, and 7.8 s. (b) At 22 s, just after GLIDE is submitted, the model writes “I need to act quickly before the 3-second mark” but spends 17.2 s choosing OLIVE. (c) At 24 s GLIDE has been scored and the clock has reset, while the model is still generating. (d) The board locks at about 32 s, and OLIVE arrives at 39.3 s. Panels (c) and (d) are taken from the episode recording. The same model solves this task in 2 of 3 WL runs.
Figure 14: Deliberating past the cutoff in Flight gate assignment (Gemini 3.8 Flash, WH). (a) At 9 s three flights are on the board, none of them assigned, and the sequencing cutoff is 19 s away. The model’s reasoning notes that “LLM generation time and network latency are approximately 5-7 seconds. If I wait, real time advances,” yet the call takes 16.6 s and returns wait . (b) At 26 s, with 2 s left, all five flights are still unassigned, and the plan is never committed. The same model solves this task in 2 of 3 WL runs.
Figure 15: Letting room holds lapse in Hotel booking (OpenAI Decisions, Wall Clock). (a) At 7 s the qualifying Harbor King rate is held with 0:02 left, its terms have been reviewed, and Book reviewed room is enabled. The model nevertheless answers wait (0.9-s call). (b) At 9 s the hold has lapsed. (c) The model then holds Courtyard King, which also qualifies, while Harbor King has been repriced to a nonrefundable rate. It answers wait at 11 and 13 s while the new hold runs down. (d) At 15 s this hold has also lapsed. Soon afterwards, the qualifying inventory closes, ending the episode. Runs 2 and 3 of the same model book in time.
Figure 16: Buying after the dip has ended in Market dip purchase (Jev 1.13, Wall Clock). BTC may be bought only while its price is below 58,365, and Jev 1.13 starts the order by focusing the amount field and typing 0.02. (b) At 7 s the price has rebounded to 61,315, and Jev 1.13 confirms it. (d) The order fills at $61,345.66, above the limit. Run 1 follows the same click sequence and succeeds.
More in this topic
Agent Evaluation
Agent evaluation measures how reliably AI systems act, use tools, and complete tasks across interactive environments.
See all 144 papers