With the emergence of search-enabled generative QA systems, users are increasingly turning to tools that browse, aggregate, and reconcile evidence across multiple sources on their behalf. Yet many widely used QA benchmarks remain answerable by retrieving a single relevant passage, making them poorly suited for measuring cross-source sensemaking, such as integrating evidence, tracking causal links, and resolving dependencies across facets of a topic. We present iAgentBench, a dynamic ODQA benchmark that targets these higher-level information needs while keeping questions natural and grounded in realistic information-seeking behavior. iAgentBench draws seed topics from real-world attention signals and uses common user intent patterns to construct user-like questions whose answers require combining evidence from multiple sources, not just extracting a single snippet. Each instance is released with traceable evidence and auditable intermediate artifacts that support contamination checks and enable fine-grained diagnosis of failures in retrieval versus synthesis. Experiments across multiple LLMs show that retrieval improves accuracy, but retrieval alone does not reliably resolve these questions, underscoring the need to evaluate evidence use, not just evidence access.
Figures & tables
Figure 1. Overview of the i AgentBench construction pipeline. We (1) sample time-indexed, high-traffic seed queries from public attention signals (GDELT), (2) retrieve a query-conditioned web corpus and extract a claim-like story graph with thematic communities, (3) assign community roles ( Core / Bridge / Satellite ) and build compact artifacts (community cards, connectors, packets) that preserve cross-theme links, and (4) generate and filter standalone ODQA pairs using a panel of LLM judges.
Field
Description
id
Unique identifier for the instance.
query
Seed query (interest-driven topic string).
time_window
Time interval used for topic sourcing and retrieval.
question
Final one-sentence question.
answer
Short objective answer (entity, date, or short phrase).
intent_pattern
Label for the information-seeking intent: explainer , connection , trigger , consequence , stake .
Table 1. Core fields released per iAgentBench instance.
Model
SimpleQA
HotpotQA
iAgentBench
Base
RAG
Refl.
ΔRAG
ΔRefl
Base
RAG
Refl.
ΔRAG
ΔRefl
Base
RAG
Refl.
ΔRAG
ΔRefl
Claude Sonnet 4.5
0.240
0.740
0.716
0.500
− 0.024
0.574
0.700
0.790
0.126
0.090
0.584
0.648
0.682
0.064
0.034
LLaMA 4 Maverick 17B
0.174
0.657
0.826
0.483
0.169
0.456
0.537
0.758
0.081
0.221
0.356
0.532
0.628
0.176
0.096
Mistral Large 3 (675B)
0.196
0.730
0.820
0.534
0.090
0.500
0.654
0.720
0.154
0.066
0.486
0.638
0.564
0.152
− 0.074
Gemma 3 27B
0.092
0.700
0.764
0.608
0.064
0.420
0.548
0.694
0.128
0.146
0.432
0.592
0.570
0.160
− 0.022
Table 2. Accuracy of Base, RAG, & Reflexion across SimpleQA, HotpotQA, and iAgentBench ; ΔRAG=RAG−Base , ΔRefl=Refl−RAG .
Figure 2. Base vs. RAG accuracy across datasets and models. Points above the diagonal indicate gains from evidence access.
Figure 3. Retrieval gains vs. agentic gains, decomposing improvements into ΔRAG=Acc(RAG)−Acc(Base) and ΔRefl=Acc(Refl)−Acc(RAG) . Positive ΔRefl means iteration helps beyond RAG, while negative values indicate regressions.
Despite their substantial successes, AI agents continue to face fundamental challenges in terms of trustworthiness. Consider deep research agents, tasked with searching for information relevant to a given topic-while AI agents can perform effective information retrieval, there is little guarantee regarding the completeness of this information. Gaps in retrieved information can leave biases that mislead users even if the information they are given is correct and relevant. We introduce SeekerGym, a benchmark designed to evaluate the completeness of information retrieved by AI agents. In addition, SeekerGym also measures how well agents quantify their uncertainty in the completeness of their information; if an agent fails to retrieve all relevant information, it is useful for it to at least quantify how much might be missing. At a high level, each task in SeekerGym is a document (e.g., a Wikipedia article), and the AI agent must issue queries to retrieve passages from that document. Intuitively, the document comprehensively covers a topic, so the ability to retrieve its sections directly measures completeness of information retrieval. In addition to Wikipedia, we also consider machine learning survey papers, where the goal is to retrieve relevant sections of a survey paper. We benchmark several models and algorithms; the best approaches retrieve 42.5% of passages on Wikipedia and 29.2% on ML Surveys, leaving substantial room for improvement.
Remy Kim, Minseung Lee, Shuo Li +1
University of Pennsylvania, Curation Labs · Curation Labs · Google DeepMind +1
Recent large language models (LLMs) have shown rapid progress in reading-based question answering (QA), where evidence is explicitly provided or can be trivially retrieved. In contrast, real-world questions are often not paired with accurate evidence documents. The useful evidence resides in massive data lakes, making search a prerequisite for answering. However, there is a lack of comprehensive benchmarks that require both searching and reasoning over large data lakes. To this end, we introduce LakeQA, a comprehensive benchmark for search-centric question answering over data lakes that jointly emphasizes searching and reasoning capabilities. LakeQA is built on a heterogeneous collection of approximately 9.5 TB of text resources from Wikipedia and open-source government data, spanning structured and unstructured data. To ensure task quality, each sample is annotated by at least one Ph.D.-level expert. Each task requires long-horizon multi-hop reasoning with implicit intermediate steps: agents need to discover the correct documents and then compose evidence across sources to produce the answer. Experimental results on seven frontier LLMs demonstrate that LakeQA is challenging. For instance, GPT-5.2 achieves only an exact-match score of 18.37% on LakeQA. Overall, LakeQA provides a realistic testbed for developing LLM agents that can both find and analyze data in modern data lakes.
Haonan Wang, Jiaxiang Liu, Yurong Liu +11
Columbia University · New York University · Barnard College
Deep research requires reasoning over web evidence to answer open-ended questions, and it is a core capability for AI agents. Yet many deep research agents still rely on implicit, unstructured search behavior that causes redundant exploration and brittle evidence aggregation. Motivated by Anthropic's "think" tool paradigm and insights from the information-retrieval literature, we introduce Q+, a set of query and evidence processing tools that make web search more deliberate by guiding query planning, monitoring search progress, and extracting evidence from long web snapshots. We integrate Q+ into the browser sub-agent of Eigent, an open-source, production-ready multi-agent workforce for computer use, yielding EigentSearch-Q+. Across four benchmarks (SimpleQA-Verified, FRAMES, WebWalkerQA, and XBench DeepSearch), Q+ improves Eigent's browser agent benchmark-size-weighted average accuracy by 3.0, 3.8, and 0.6 percentage points (pp) for GPT-4.1, GPT-5.1, and Minimax M2.5 model backends, respectively. Case studies further suggest that EigentSearch-Q+ produces more coherent tool-calling trajectories by making search progress and evidence handling explicit.
Boer Zhang, Mingyan Wu, Dongzhuoran Zhou +6
Meta · Northeastern University, China · University of Oslo +5