iAgentBench: Benchmarking Sensemaking Capabilities of Information-Seeking Agents on High-Traffic Topics

Preetam Prabhu Srikar Dammu; Arnav Palkhiwala; Tanya Roosta; Chirag Shah

cs.CLMar 4, 2026

iAgentBench: Benchmarking Sensemaking Capabilities of Information-Seeking Agents on High-Traffic Topics

Authors: Preetam Prabhu Srikar Dammu, Arnav Palkhiwala, Tanya Roosta, Chirag Shah

Organizations: University of Washington Seattle, Washington, USA · UC Berkeley Berkeley, California, USA

Abstract

With the emergence of search-enabled generative QA systems, users are increasingly turning to tools that browse, aggregate, and reconcile evidence across multiple sources on their behalf. Yet many widely used QA benchmarks remain answerable by retrieving a single relevant passage, making them poorly suited for measuring cross-source sensemaking, such as integrating evidence, tracking causal links, and resolving dependencies across facets of a topic. We present iAgentBench, a dynamic ODQA benchmark that targets these higher-level information needs while keeping questions natural and grounded in realistic information-seeking behavior. iAgentBench draws seed topics from real-world attention signals and uses common user intent patterns to construct user-like questions whose answers require combining evidence from multiple sources, not just extracting a single snippet. Each instance is released with traceable evidence and auditable intermediate artifacts that support contamination checks and enable fine-grained diagnosis of failures in retrieval versus synthesis. Experiments across multiple LLMs show that retrieval improves accuracy, but retrieval alone does not reliably resolve these questions, underscoring the need to evaluate evidence use, not just evidence access.

Figures & tables

Figure 1. Overview of the i AgentBench construction pipeline. We (1) sample time-indexed, high-traffic seed queries from public attention signals (GDELT), (2) retrieve a query-conditioned web corpus and extract a claim-like story graph with thematic communities, (3) assign community roles ( Core / Bridge / Satellite ) and build compact artifacts (community cards, connectors, packets) that preserve cross-theme links, and (4) generate and filter standalone ODQA pairs using a panel of LLM judges.

Field	Description
id	Unique identifier for the instance.
query	Seed query (interest-driven topic string).
time_window	Time interval used for topic sourcing and retrieval.
question	Final one-sentence question.
answer	Short objective answer (entity, date, or short phrase).
intent_pattern	Label for the information-seeking intent: explainer , connection , trigger , consequence , stake .

Table 1. Core fields released per iAgentBench instance.

Model	SimpleQA					HotpotQA					iAgentBench
Model	Base	RAG	Refl.	$\Delta_{\mathrm{RAG}}$	$\Delta_{\mathrm{Refl}}$	Base	RAG	Refl.	$\Delta_{\mathrm{RAG}}$	$\Delta_{\mathrm{Refl}}$	Base	RAG	Refl.	$\Delta_{\mathrm{RAG}}$	$\Delta_{\mathrm{Refl}}$
Claude Sonnet 4.5	0.240	0.740	0.716	0.500	$-$ 0.024	0.574	0.700	0.790	0.126	0.090	0.584	0.648	0.682	0.064	0.034
LLaMA 4 Maverick 17B	0.174	0.657	0.826	0.483	0.169	0.456	0.537	0.758	0.081	0.221	0.356	0.532	0.628	0.176	0.096
Mistral Large 3 (675B)	0.196	0.730	0.820	0.534	0.090	0.500	0.654	0.720	0.154	0.066	0.486	0.638	0.564	0.152	$-$ 0.074
Gemma 3 27B	0.092	0.700	0.764	0.608	0.064	0.420	0.548	0.694	0.128	0.146	0.432	0.592	0.570	0.160	$-$ 0.022

Table 2. Accuracy of Base, RAG, & Reflexion across SimpleQA, HotpotQA, and iAgentBench ;

\Delta_{\mathrm{RAG}}=\mathrm{RAG}-\mathrm{Base}

\Delta_{\mathrm{Refl}}=\mathrm{Refl}-\mathrm{RAG}

Figure 2. Base vs. RAG accuracy across datasets and models. Points above the diagonal indicate gains from evidence access.

Figure 3. Retrieval gains vs. agentic gains, decomposing improvements into

\Delta_{\mathrm{RAG}}=\mathrm{Acc}(\mathrm{RAG})-\mathrm{Acc}(\mathrm{Base})

and

\Delta_{\mathrm{Refl}}=\mathrm{Acc}(\mathrm{Refl})-\mathrm{Acc}(\mathrm{RAG})

. Positive

\Delta_{\mathrm{Refl}}

means iteration helps beyond RAG, while negative values indicate regressions.

Explore similar work

SeekerGym: A Benchmark for Reliable Information Seeking
Apr 18, 2026Remy Kim, Minseung Lee, Shuo Li +1AI Agent Reliability Agentic Search
University of Pennsylvania, Curation Labs · Curation Labs · Google DeepMind +1
LakeQA: An Exploratory QA Benchmark over a Million-Scale Data Lake
Jun 9, 2026Haonan Wang, Jiaxiang Liu, Yurong Liu +11Multi-Hop QA AI Agent Benchmarks
Columbia University · New York University · Barnard College
EigentSearch-Q+: Enhancing Deep Research Agents with Structured Reasoning Tools
Apr 9, 2026Boer Zhang, Mingyan Wu, Dongzhuoran Zhou +6Deep Research Agents Agentic Search
Meta · Northeastern University, China · University of Oslo +5

Apr 18, 2026J/K move · Enter open · S save

SeekerGym: A Benchmark for Reliable Information Seeking

Remy Kim, Minseung Lee, Shuo Li, Osbert Bastani

University of Pennsylvania, Curation Labs · Curation Labs · Google DeepMind+1

AI Agent Reliability Agentic Search AI Agent Benchmarks LLM Uncertainty Estimation IR Evaluation

Open paper AskPDF arXiv

Despite their substantial successes, AI agents continue to face fundamental challenges in terms of trustworthiness. Consider deep research agents, tasked with searching for information relevant to a given topic-while AI agents can perform effective information retrieval, there is little guarantee regarding the completeness of this information. Gaps in retrieved information can leave biases that mislead users even if the information they are given is correct and relevant. We introduce SeekerGym, a benchmark designed to evaluate the completeness of information retrieved by AI agents. In addition, SeekerGym also measures how well agents quantify their uncertainty in the completeness of their information; if an agent fails to retrieve all relevant information, it is useful for it to at least quantify how much might be missing. At a high level, each task in SeekerGym is a document (e.g., a Wikipedia article), and the AI agent must issue queries to retrieve passages from that document. Intuitively, the document comprehensively covers a topic, so the ability to retrieve its sections directly measures completeness of information retrieval. In addition to Wikipedia, we also consider machine learning survey papers, where the goal is to retrieve relevant sections of a survey paper. We benchmark several models and algorithms; the best approaches retrieve 42.5% of passages on Wikipedia and 29.2% on ML Surveys, leaving substantial room for improvement.

Abstract

Explore similar work

SeekerGym: A Benchmark for Reliable Information Seeking

LakeQA: An Exploratory QA Benchmark over a Million-Scale Data Lake

EigentSearch-Q+: Enhancing Deep Research Agents with Structured Reasoning Tools