Retrieving real-time information is a fundamental capability for search-integrated agents in real-world applications. However, existing benchmarks are predominantly static and therefore fail to capture the temporal dynamics of information and the continuously evolving nature of real-world knowledge. To address this limitation, we propose RT-QA, a dynamic evaluation framework that leverages executable code workflows to retrieve up-to-date answers at evaluation time. Specifically, we construct an agent-driven pipeline that autonomously generates code for web crawling and DOM-based answer extraction to produce real-time ground truth. To ensure robust evaluation over time, the pipeline further incorporates a self-repair mechanism to adapt to changes in web page structures. RT-QA spans 12 domains (e.g., Finance, Sports) with 320 Chinese questions categorized into three difficulty levels. Extensive evaluations of state-of-the-art models (e.g., GPT-5.2, GLM-4.7) reveal significant limitations in real-time adaptability: even the best models achieve only 46% accuracy. Our analysis highlights two primary failure modes: (1) Lazy Retrieval, where agents rely on search snippets instead of deeply scanning specific websites for information (20% of failures); and (2) Temporal Confusion, a cognitive error where agents retrieve a historical date (e.g., an event in 2024) and fail to re-anchor to the current time (2026) for subsequent reasoning. These findings suggest that future agents require not just better retrieval strategies, but robust temporal state management.
Figures & tables
Figure 1: Comparison between Static QA and Real-time QA. Unlike static questions (top) where answers are fixed or can be pre-stored, real-time questions (bottom) typically involve relative temporal constraints (e.g., “yesterday” , “past 7 days” ).
Figure 2: An Example of an RT-QA Executed Workflow. The figure demonstrates how a single Python workflow handles dynamic time anchoring. The code (left) explicitly calculates the target date (tomorrow) based on the execution time. Consequently, executing the same workflow on different dates (Oct 31 vs. Nov 1) yields correct, context-aware answers (bottom right), functioning as a dynamic pseudo-API.
Figure 3: Overview of the RT-QA Framework. (a) The Construction Pipeline consists of three steps: preparation and tool definition, Agent-in-the-Loop workflow generation, and final dataset construction. (b) The Evaluation Pipeline, where models and workflows are executed simultaneously to assess real-time accuracy.
Panel A: Domain Distribution (12 Categories)
Domain
#
Domain
#
Finance & Economy
42
Public Safety
34
Weather & Env.
40
News & Media
33
Culture Activities
35
Space & Astronomy
28
Sports Events
28
Energy & Industry
20
Consumer Products
20
Transportation
18
Table 1: RT-QA Statistics. Top: Domain distribution. Bottom: Difficulty definitions based on reasoning hops. Note: Example questions are translated from Chinese to demonstrate the reasoning complexity.
Model
Daily Overall Trend (Timeline)
Level Performance
Total Avg
Jan 10
Jan 11
Jan 12
Jan 13
Jan 14
Jan 15
L1
L2
L3
Open-Weights Models
GLM-4.7
43.6
46.0
46.1
48.4
43.8
43.1
46.9
41.9
46.9
45.2
DeepSeek-V3.2
30.1
41.6
33.9
34.4
34.2
36.3
34.2
34.0
39.3
35.1
Qwen3-235B
24.5
29.9
34.2
25.6
34.5
35.6
33.0
26.9
31.9
30.7
Kimi-K2
37.6
41.5
40.4
40.6
41.0
41.6
43.0
37.4
39.6
40.4
Table 2: Main Results on RT-QA (Jan 10–15, 2026). The table reports the Accuracy (%) evaluated by GPT-5.2 Judge. The left section shows the daily trend. The right section breaks down average performance by different level. Total Avg represents the aggregated accuracy across the entire 6-day period for all questions.
Figure 4: Case Study: Temporal Confusion (L2) vs. Rigorous Planning (L3). Comparison of agent trajectories on the Shenzhen Museum question.
Level
Jan 10
Jan 15
Jan 20 (w/ Time Anchor)
L2
33.0
40.2
40.4
L3
39.0
44.1
37.3
Table 3: Performance of L2 and L3 tasks, with a Time Anchor added to the prompt for the Jan 20 setting.
Figure 5: Fine-grained Distribution of Failure Modes. Failures are decomposed into 7 specific behavioral flaws.
Real-time interaction requires agents to perceive evolving environments and act under temporal constraints, yet existing evaluations of real-time streaming models focus on conversational quality and streaming perception, leaving effective action under strict temporal constraints untested. We introduce Realtime-Bench, a suite of carefully curated dynamic tasks where delayed actions directly affect success. Under Wall Clock, where environments advance according to actual inference latency, the best success rate reaches only 44% and the best Pass^3 is only 36%, demonstrating that real-time interaction remains challenging. To separate serving latency from models' real-time streaming capability, we compare Wall Clock with Paused Clock, which removes inference delays as an idealized reference, and introduce Token Clock, which advances simulated time according to generated token counts, testing temporal awareness independently of physical serving speed. We find that temporal costs limit success in evolving environments and that higher reasoning effort has mixed effects under time pressure, reflecting a trade-off between decision quality and timely execution. Based on these results, we discuss future directions for improving real-time multimodal models, including adaptive reasoning with temporal awareness and continuous perception and action.
Jingyu Tang, Xu Yang, Zhehao Zhang +7
University of Texas at Austin · The Ohio State University · University of Illinois at Urbana-Champaign +3
We present DynaRAG, a retrieval-augmented generation (RAG) framework designed to handle both static and time-sensitive information needs through dynamic knowledge integration. Unlike traditional RAG pipelines that rely solely on static corpora, DynaRAG selectively invokes external APIs when retrieved documents are insufficient for answering a query. The system employs an LLM-based reranker to assess document relevance, a sufficiency classifier to determine when fallback is necessary, and Gorilla v2 -- a state-of-the-art API calling model -- for accurate tool invocation. We further enhance robustness by incorporating schema filtering via FAISS to guide API selection. Evaluations on the CRAG benchmark demonstrate that DynaRAG significantly improves accuracy on dynamic questions, while also reducing hallucinations. Our results highlight the importance of dynamic-aware routing and selective tool use in building reliable, real-world question-answering systems.
Penghao Liang, Mengwei Yuan, Jianan Liu +4
Northeastern University Boston, USA, 02115 · Independent Researcher Milpitas, USA, 95035 · Independent Researcher Austin, USA, 78613 +3
Temporal Knowledge Graph Question Answering (TKGQA) is challenging because it requires multi-hop reasoning under complex temporal constraints. Recent LLM-based approaches have improved semantic modeling for this task, but many still rely on fixed reasoning workflows or costly post-training, which can limit adaptability and make error recovery difficult. We show that enabling an off-the-shelf Large Language Model (LLM) to determine its next action is already effective in a zero-shot setting. Based on this insight, we propose AT2QA, an Autonomous and Training-free Agent for TKG Question Answering. AT2QA empowers the LLM to iteratively interact with the TKG via a generic search tool, inherently enabling autonomous exploration and dynamic self-correction during reasoning. To further elicit the LLM's potential for complex temporal reasoning, we introduce a training-free experience mining mechanism that distills a compact few-shot demonstration library from successful self-generated trajectories. AT2QA also yields a transparent audit trail for every prediction. Experiments on three challenging benchmarks -- MultiTQ, Timeline-CronQuestion, and Timeline-ICEWS-Actor -- show that AT2QA achieves new state-of-the-art performance, surpassing the strongest baselines by 10.7, 4.9, and 11.2 absolute points, respectively. Our code is available at https://github.com/AT2QA-Official-Code/AT2QA-Official-Code
Xufei Lv, Jiahui Yang, Haoyuan Sun +5
Tsinghua University · National University of Defense Technology