Given that AI agents can search the web broadly and deeply in real time, it is tempting to conclude that their outputs account for recent world events. We show that this conclusion does not hold in practice. While agents can search and easily find recent changes, the correctness of their responses depends on whether they search and what they search for. Popular agentic search benchmarks like BrowseComp cannot expose failures wherein agents do not recognize their own assumptions that should be verified, since these benchmarks explicitly state what needs to be found. In contrast, many user requests state only the final information need (e.g., an itinerary for a trip), leaving the agent to discover relevant changes (e.g., the temporary closure of a museum) and account for their consequences. To highlight such gaps, we introduce BlindToChange, a benchmark that measures whether agents proactively discover recent events that affect their responses. Our benchmark comprises requests for recommendations where an obvious candidate is no longer applicable due to a recent real-world event. These events are drawn automatically from Wikidata, allowing BlindToChange to be periodically refreshed. Strikingly, we find a big gap -- as high as 27.5 points, even at high reasoning effort -- in systems' performance, when the recent event is not explicitly mentioned compared to when we ask about it directly. Analyzing search trajectories, we learn that more than half of the failures can be attributed to the fact that agents do not directly search to verify whether a relevant fact has changed. Our benchmark highlights, and tracks, a new axis of progress, wherein systems account for recent information that is not explicitly mentioned in the user request.
Figures & tables
Figure 1: Two probes for the same underlying update that the Bayeux Tapestry Museum is closed until October 2027, posed to gemini-3.1-pro with search enabled. In (a) the request names the museum and asks about its status. The model finds out that the museum is closed and adjusts its recommendation. In (b) the request asks for an itinerary and never mentions the museum. The model recommends it as an afternoon stop and does not check whether it is open. The direct probe shows that the model can find the fact. The indirect probe evaluates whether it discovers the change on its own. The plot on the right shows the gap between direct and indirect probe accuracy.
Demises
Update
Tony Hoare passed away on 5 March 2026.
Direct
Is Tony Hoare, the British computer scientist, alive?
Indirect
I’m putting together an anthology on the history of programming language design, focusing on type safety and legacy design errors. Who are a few veteran British computer scientists who have been outspoken about early language design flaws that we could invite to write a new foreword for the book?
Update
Nathalie Baye passed away on 17 April 2026.
Direct
Is Nathalie Baye, the French actress, alive?
Indirect
We are assembling the jury for a new national grant supporting emerging French filmmakers. Who are some highly decorated veteran French actresses with multiple César wins who would have the gravitas to sit on the panel and review this year’s final project submissions?
Table 1: Examples from BlindToChange for its two types of updates, demises and ends of official tenures. The direct probe asks whether the person is still alive or still holds the position. The indirect probe never mentions the person, yet they would have been an obvious recommendation for the request before the update.
Table 2: Direct (Dir.) and indirect (Ind.) probing accuracy (%) for each model, harness and reasoning effort. The ▼ columns show the drop from direct to indirect accuracy. – marks MiMo-V2.5-Pro (shared, low), which kept calling tools without giving a final answer.
Table 3: Indirect probing accuracy (%) on a common set of questions shared across all models, run using the shared harness with both low and high reasoning effort.
Figure 2: Across models, we observe comparable performance on indirect probes for demises using the shared harness, when grouped by the number of search queries issued. Top, accuracy per group (with at least 9 probes) and its overall accuracy (dashed). Bottom, the share of probes in each group.
Figure 3: Composition of indirect-probe failure modes under the shared harness for (a) demises and (b) tenures. Numbers above bars give the total number of failures.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Stage
Model
Reasoning effort
Indirect probe generation
Gemini-3.1-Pro
High
Direct probe generation
Gemini-3.7-Flash
Low
Response judging
Gemini-3.5-Flash-Lite
Low
Appendix
Table 4: Models and reasoning efforts used at each stage of the BlindToChange construction pipeline.
Demises
Tenures
Human label
Human label
Probe
Judge label
Correct
Incorrect
Correct
Incorrect
Indirect
Correct
99
1
99
1
Incorrect
0
100
2
98
Direct
Correct
100
0
100
0
Incorrect
2
98
0
100
Appendix
Table 5: Confusion matrices comparing the judge’s labels with human labels. For each probe type and category we sampled 100 responses the judge labeled correct and 100 it labeled incorrect.
Model
Reported knowledge cutoff
Source
Claude-Opus-4.7
January 2026
( Anthropic, 2026b )
GPT-5.4
August 31, 2025
( OpenAI, 2026b )
Gemini-3.1-Pro
January 2025
( Google, 2026 )
Qwen3.7-Max
Not reported
–
DeepSeek-V4-Pro
Not reported
–
Kimi-K2.6
Not reported
–
Appendix
Table 6: Reported knowledge cutoff of each evaluated model, as stated by its provider. We resolve each cutoff to the last day of the reported month and substitute it for {knowledge_cutoff} in the shared harness system prompt (Figure 4 ) in YYYY-MM-DD form. For models whose provider reports no cutoff we substitute January 2026, the latest reported cutoff among the models we evaluate. Every event in BlindToChange postdates all of these dates, so the instruction to search for information after the cutoff applies to every probe.
Demises
Tenures
Events
2,078
751
Generated probes
10,390
3,755
Retained probes after filtering
Claude-Opus-4.7
935
299
GPT-5.4
1,304
766
Gemini-3.1-Pro
3,092
1,712
Appendix
Table 7: Number of indirect probes per category before and after the pre-event filter. Generated counts are shared across models. Retained counts are the probes for which all three no-search responses from that model include the affected person.
Model
Low
High
Claude-Opus-4.7
Disabled
High
GPT-5.4
None
High
Gemini-3.1-Pro
Low
High
Qwen3.7-Max
Disabled
Enabled
DeepSeek-V4-Pro
Disabled
High
Kimi-K2.6
Disabled
Enabled
Appendix
Table 8: Mapping from the low and high reasoning settings used throughout the paper to the provider-specific effort value passed to each model.
Figure 5: Indirect probe accuracy on demises under the shared harness for the models not shown in Figure 2 , grouped by the number of search queries issued. Top, accuracy per group (with at least 9 probes) and its overall accuracy (dashed). Bottom, the share of probes in each group.
Verified failure mode
Category
Assigned failure mode
Reasoning
Explicit
Indirect
No/unrelated
Demises
Reasoning error
33
0
1
1
Ineffective but explicit search
1
12
0
0
Indirect search
0
0
42
0
No/unrelated search
0
0
0
70
Tenures
Reasoning error
43
1
1
0
Appendix
Table 9: Confusion matrices comparing the failure mode assigned by the classifier with the failure mode verified by an author. No/unrelated search rows include only failures that issued at least one search query.
Are LLM-based search agents genuinely searching, or using the web to verify what they already know? We study this question on BrowseComp with three diagnostics. Our analysis reveals Intrinsic Knowledge Dependence (IKD): even with tool access, agents often rely on intrinsic knowledge -- information encoded in the model before retrieval -- rather than on external evidence. Agents answer up to 44.5% of BrowseComp questions without tools, generate more than half of their search queries from internally produced hypotheses rather than retrieved leads, and perform worse than closed-book baselines when answer-supporting evidence is removed. These results suggest that static search benchmarks can reward memory-backed verification rather than evidence-driven discovery, conflating what agents already know with what they can find. We then introduce LiveBrowseComp, a deep-search benchmark designed to evaluate agents beyond intrinsic coverage. It contains 335 human-authored questions whose answers depend on facts published within the 90 days preceding benchmark construction, drawn from six updated sources and filtered to exclude globally salient events. On LiveBrowseComp, all evaluated agents fall below 2% closed-book accuracy, search-augmented scores drop by 25-40 points relative to BrowseComp, and prior model rankings no longer reliably predict performance. LiveBrowseComp is available at https://huggingface.co/datasets/Forival/LiveBrowseComp.
Despite their substantial successes, AI agents continue to face fundamental challenges in terms of trustworthiness. Consider deep research agents, tasked with searching for information relevant to a given topic-while AI agents can perform effective information retrieval, there is little guarantee regarding the completeness of this information. Gaps in retrieved information can leave biases that mislead users even if the information they are given is correct and relevant. We introduce SeekerGym, a benchmark designed to evaluate the completeness of information retrieved by AI agents. In addition, SeekerGym also measures how well agents quantify their uncertainty in the completeness of their information; if an agent fails to retrieve all relevant information, it is useful for it to at least quantify how much might be missing. At a high level, each task in SeekerGym is a document (e.g., a Wikipedia article), and the AI agent must issue queries to retrieve passages from that document. Intuitively, the document comprehensively covers a topic, so the ability to retrieve its sections directly measures completeness of information retrieval. In addition to Wikipedia, we also consider machine learning survey papers, where the goal is to retrieve relevant sections of a survey paper. We benchmark several models and algorithms; the best approaches retrieve 42.5% of passages on Wikipedia and 29.2% on ML Surveys, leaving substantial room for improvement.
Remy Kim, Minseung Lee, Shuo Li +1
University of Pennsylvania, Curation Labs · Curation Labs · Google DeepMind +1
The rapid growth of AI agent ecosystems is transforming how complex tasks are delegated and executed, creating a new challenge of identifying suitable agents for a given task. Unlike traditional tools, agent capabilities are often compositional and execution-dependent, making them difficult to assess from textual descriptions alone. However, existing research and benchmarks typically assume well-specified functionalities, controlled candidate pools, or only executable task queries, leaving realistic agent search scenarios insufficiently studied. We introduce AgentSearchBench, a large-scale benchmark for agent search in the wild, built from nearly 10,000 real-world agents across multiple providers. The benchmark formalizes agent search as retrieval and reranking problems under both executable task queries and high-level task descriptions, and evaluates relevance using execution-grounded performance signals. Experiments reveal a consistent gap between semantic similarity and actual agent performance, exposing the limitations of description-based retrieval and reranking methods. We further show that lightweight behavioral signals, including execution-aware probing, can substantially improve ranking quality, highlighting the importance of incorporating execution signals into agent discovery. Our code is available at https://github.com/Bingo-W/AgentSearchBench.
Bin Wu, Arastun Mammadli, Xiaoyu Zhang +1
Centre for Artificial Intelligence, University College London