A Benchmark for Deep Information Synthesis

Debjit Paul; Daniel Murphy; Milan Gritta; Ronald Cardenas; Victor Prokhorov; Lena Sophia Bolliger; Aysim Toker; Roy Miles

cs.AIFeb 24, 2026

A Benchmark for Deep Information Synthesis

Authors: Debjit Paul, Daniel Murphy, Milan Gritta, Ronald Cardenas, Victor Prokhorov, Lena Sophia Bolliger, Aysim Toker, Roy Miles,

Organizations: Huawei Noah’s Ark Lab, UK · Imperial College London · University of Zurich · University of Sheffield ·

Abstract

Large language model (LLM)-based agents are increasingly used to solve complex tasks involving tool use, such as web browsing, code execution, and data analysis. However, current evaluation benchmarks do not adequately assess their ability to solve real-world tasks that require synthesizing information from multiple sources and inferring insights beyond simple fact retrieval. To address this, we introduce DEEPSYNTH, a novel benchmark designed to evaluate agents on realistic, time-consuming problems that combine information gathering, synthesis, and structured reasoning to produce insights. DEEPSYNTH contains 120 tasks collected across 7 domains and data sources covering 67 countries. DEEPSYNTH is constructed using a multi-stage data collection pipeline that requires annotators to collect official data sources, create hypotheses, perform manual analysis, and design tasks with verifiable answers. When evaluated on DEEPSYNTH, 11 state-of-the-art LLMs and deep research agents achieve a maximum F1 score of 8.97 and 17.5 on the LLM-judge metric, underscoring the difficulty of the benchmark. Our analysis reveals that current agents struggle with hallucinations and reasoning over large information spaces, highlighting DEEPSYNTH as a crucial benchmark for guiding future research.

Figures & tables

Figure 1: A sample task from DeepSynth , illustrating that synthesizing knowledge requires agents to perform multiple steps, including web browsing, gathering information from multiple sources, reasoning over them, and generating the final answer.

Figure 2: An overview of our data collection process for building the DeepSynth benchmark

Model	F1 Score	Precision	Recall	Exact Match	LLM Judge Score
LLM Baselines
o4-mini (2025-08)	3.05	2.33	4.39	0.0	0.0
GPT-4.1 (2025-08)	3.46	2.86	4.39	0.0	0.0
o3 (2025-08)	3.29	2.85	3.90	0.0	0.0
GPT-5.1 (2025-08)	3.83	2.98	5.37	0.0	0.0
Gemini-Pro-2.5 (2025-08)	6.25	4.71	9.27	0.0	5.0

Table 2: Performance comparison on the DeepSynth benchmark (Pass@1). F1, Precision, Recall and Exact Match measure the quality of model predictions. LLM Judge (%) reports the average precision. Models with are models or framework which are closed, while are open-source.

Panel of Figure 4 — Figure 4: Performance comparison on the DeepSynth -Dev benchmark. (a) F1 Score measures the quality of model predictions. LLM-Judge (GPT-4.1) reports the average precision as judged by GPT-4.1. Light bars denote LLM baselines; dark bars denote framework-based agents. (b) Best-of-N LLM Judge Scores comparing GPT-4.1 standalone vs. Smolagents (GPT-4.1) across $N\in\{1,3,5\}$ and and Self-Consistency@5 (majority voting).

Model	Performance Metrics
Model	F1	Precision	Recall	EM
OWL ( Full )	5.41	4.62	6.52	1.67
Tool Ablation
$-$ Web Browsing Toolkit	4.80	4.20	5.60	1.67
$-$ Search Toolkit	3.60	2.96	4.61	0.0
$-$ Document Processing Toolkit	4.90	4.50	5.4	1.67

Table 3: Ablation Study. Tool Ablation: Comparing the benefits of using different tools on DeepSynth . Reasoning Chain Ablation: Studying the role of planning given the intermediate steps.

Error Cause	No. of instances
Navigation Error	15
No answer	4
Technical Issue	4
Synthesis Error	16

Table 4: Error analysis for OWL (GPT-

4.1

). Navigation and synthesis errors are the most prominent.

Region	% of Tasks	GPT-4.1	o3-deep-research	Gemini-2.5-Pro	smolagents
Africa	8.3%	0.0	0.0	0.0	0.0
North America	11.7%	4.65	8.00	12.00	8.33
South America	5%	0.0	25.00	0.0	0.00
Asia	29.2%	3.36	12.70	6.50	11.88
Europe	38.3%	3.45	10.83	4.91	5.28
Oceanic	10.8%	8.96	14.43	6.67	24.00

Table 5: Multi-Regional Analysis : Agent performance across region-specific tasks ( F1 score ). NOTE: A question may span multiple regions. “Others” contains tasks without regional association.

Dataset	Real World	Multi Regional	Information Synthesis	Multi-Part Answers
GAIA ( Mialon et al., 2023 )	partial	×	partial	×
AssistantBench ( Yoran et al., 2024 )	✓	×	×	partial
BrowseComp ( Wei et al., 2025 )	×	×	×	×
HLE ( Phan et al., 2025 )	partial	partial	partial	×
DeepSynth	✓	✓	✓	✓

Table 6: Comparison of datasets on various reasoning and retrieval capabilities.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Avg. Time (sec.)
o4-mini (2025-08)	20.8
GPT-4.1 (2025-08)	7.52
GPT-5 (2025-08)	83.41
Gemini-Pro-2.5 (2025-08)	34.10
DeepSeek-R1 (2025-08)	5.2
o3-deep-research (2025-08)	645.39

Appendix

Table 7: Average Time per instance to run our benchmark

	DeepSeek-R1	GPT-4.1	GPT-5.2	Avg. Prop. (%)
Step 1	11.2	10.0	4.1	—
Step 2	12.4	9.8	2.6	97.0
Step 3	3.9	3.3	0.5	100.0
Step 4	1.4	2.4	0.0	100.0
Step 5	0.2	0.0	0.0	100.0
Step 6	0.0	0.0	0.0	100.0

Appendix

Table 8: Intermediate step accuracy and error propagation. Per-step F1 (%) on 40 DeepSynth tasks. Prop. denotes error propagation rate: the percentage of cases where failure at step

k

(F1

<50

) also results in failure at step

k{+}1

. Steps with

n<3

are omitted. GPT-4.1 and DeepSeek-R1 operate without web access; GPT-5.2 operates with web search enabled. All models show steep accuracy decay and near-total error propagation.

Model	Cost (USD)	Output Tokens (Min / Max)
deepseek-R1	$0.041	687.4 (avg)
GPT-4	$0.432	188 (min) / 383 (max)
GPT-5	$5.52	1600–15872 (reasoning), 201–425 (completion)
o4-mini	$1.39	448–5760 (reasoning), 40–392 (completion)
o3 deep-research	$184.61	448–5760 (reasoning), 40–392 (completion)
Gemini-Pro-2.5	$11.78	12995 (avg)

Appendix

Table 9: Model cost and output token ranges.

Framework	Code Interpreter	Web Search	API Calls	Document Processing	Browser Simulation
o3-deep-research (2025-08)	✓	✓	×	×	×
Smolagent (GPT-5)	✓	✓	×	×	×
OWL	×	✓	✓	✓	✓

Appendix

Table 10: Agentic Framework Tool Capabilities Comparison. Note: ✓ indicates capability present; × indicates capability not available. Web Browser functionality is included within Web Search capabilities for applicable frameworks.

Operation	Description	Example
Trend Detection	Identifying patterns, directions, or changes in data over time or across contexts.	Climate events over the last decade to detect a trend of increasing wildfires.
Average	Determining a representative value that summarises numerical data collected from multiple sources.	Synthesising daily temperatures from multiple weather stations to report the average regional temperature.
Correlation	Measuring relationships or associations between two or more variables.	Synthesising data from health studies to find correlations between exercise frequency and cholesterol levels.
Ranking	Ordering items or facts based on specific criteria or importance.	Compiling product reviews from different websites and ranking products by overall rating.
Anomaly Detection	Identifying data points or patterns that significantly deviate from the norm.	Synthesising sensor data from multiple factories to detect unusual machine behaviour.
Counting and Comparing	Quantifying occurrences and comparing values across sources or categories.	Counting positive vs negative mentions of a policy in news articles and comparing the proportions.

Appendix

Table 11: Key operations in information synthesis, their definitions, and examples of application.

Model	Performance Metrics
Model	F1	Precision	Recall	EM
o3	3.29	2.85	3.90	0.0
+ Intermediate Step	12.87	11.38	14.81	7.50
Gemini-Pro-2.5	6.25	4.71	9.27	0.0
+ Intermediate Step	10.40	8.36	13.76	7.50

Appendix

Table 12: Analysis. Studying the role of planning/intermediate steps.

Model	F1 Score	Precision	Recall	Exact Match	LLM Judge Score (GPT-4.1)
LLM Baselines
o4-mini (2025-12)	3.26	2.63	4.29	0.0	2.5
GPT-4.1 (2025-12)	1.81	1.57	2.14	0.0	7.5
o3 (2025-12)	6.33	5.68	7.14	1.25	10.0
GPT-5.1 (2025-12)	6.18	6.72	5.71	1.25	12.5
Gemini-Pro-2.5 (2025-12)	5.87	4.98	7.14	0.0	5.0

Appendix

Table 13: Performance comparison on the DeepSynth -Dev benchmark (Pass@1). F1, Precision, Recall and Exact Match measure the quality of model predictions. LLM Judge (%) reports the average precision. Models with are models or framework which are closed, while are open-source.

Figure 9: Example run using OWL, illustrating errors when trying to collect and reason about data. The agent finds the right URL but fails to query the right data from the website’s database interface. A second attempt tries to download a data file from an incorrect URL, resulting in a not found error .

Questions	Metadata
In the 2021 Canadian federal elections, compare the last opinion polling results by Abacus Data, EKOS Research, Mainstreet Research, Forum Research, Research Co. and Nanos Research before the elections. How accurate is each of these polls as a predictor for the actual number of votes for each party (rounded to two decimals), you should include the 6 largest parties, and ’others’. Rank each polling company by mean-squared error. Your answer should be a JSON object with the keys being the polling company names. The values should be the MSE (rounded to one decimal). The companies should be sorted in increasing value of MSE.	Type: Compare, Rank Region: North America Domain: Sociopolitics
As a Swiss Federal Statistics Officer, I am studying goods transport performance by railway. From 2008 to 2024, what is the average annual change rate for domestic transport, import, export, and transit? Provide the answer as a JSON object where each key is the transport type and each value is the percentage change (rounded to one decimal).	Type: Change Region: Europe Domain: Transportation
According to ASEAN stats, between 2016-2023, which ASEAN countries’ exports in telecommunication, computer and information services had a negative correlation with the total nominal GDP of all ASEAN countries combined? The final answer should be presented as a JSON: keys should be these ASEAN countries, and values should be the pearson correlation value(s) rounded to two decimals.	Type: Correlation Region: Southeast Asian Domain: Economics
I am analysing the relative importance of airports for Qatar Airways. Using crowd-sourced data from the OpenSky Network for December 2020, can you identify which are the five most important airports for Qatar Airways flights - as measured by their pagerank, using the default damping factor suggested in the paper introducing the google search engine. Your output should be a JSON object with the keys being the IATA airport codes, and the values being the Pagerank value rounded to three decimals.	Type: Rank Region: Middle East , Domain: Transportation
I am analyzing New Zealand migration data from 2001 to 2020 to identify anomalies in migration patterns. For each year, count the number of months in which the net migration was negative ( $<$ 0). Please return the results in JSON format, where each key is a year and the value is the number of months with negative net migration.	Type: Counting, Compare Region: New Zealand Domain: Socioeconomics
What are the tokenizer-level compression ratios (measured as bytes per token) for the following sentence: ” Deep Insight Benchmark is an open-source benchmark that evaluates agents’ ability to solve tasks requiring analysis of multi-regional and real-world data. ”, when tokenized using the tokenizers of Llama (meta-llama/Llama-2-7b-hf), Qwen (Qwen/Qwen3-4B-Base) and Apple (apple/FastVLM-1.5B) models? Return the results as a JSON object, where each key is the model name and the value is the compression ratio (rounded to two decimal places).{”Llama”:float,”Qwen”:float,”Apple”:float}	Type: Task Specific Quantity Region: None Domain: Computer Science

Appendix

Table 14: Examples of questions.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,203 papers

A Matter of TASTE: Improving Coverage and Difficulty of Agent Benchmarks
May 27, 2026Tomer Keren, Nitay Calderon, Asaf Yehudai +3Computer-Use Agent Benchmarks Tool-Use Evaluation
Faculty of Data and Decision Science Technion · IBM Research
QUEST: Training Frontier Deep Research Agents with Fully Synthetic Tasks
May 22, 2026Jian Xie, Tianhe Lin, Zilu Wang +16Deep Research Agents LLM Agent Training
The Ohio State University · Amazon AGI SF Lab
SciExplore: Evaluating Autonomous Agents from Scientific Navigation to Information Integration
Jul 23, 2026Yinhao Tang, Youqing Fang, Yanan Sun +6AI Agent Benchmarks Information Retrieval
University of Science and Technology of China · Shanghai AI Laboratory

May 27, 2026J/K move · Enter open · S save

A Matter of TASTE: Improving Coverage and Difficulty of Agent Benchmarks

Tomer Keren, Nitay Calderon, Asaf Yehudai, Yotam Perlitz+2

Faculty of Data and Decision Science Technion · IBM Research

Computer-Use Agent Benchmarks Tool-Use Evaluation Tool-Augmented Language Model Agents Benchmark Construction LLM Agent Evaluation Synthetic Benchmark Generation+1

Open paper AskPDF arXiv

As agent capabilities advance, existing benchmarks, such as

τ^2

-Bench, are becoming increasingly saturated. Yet constructing new benchmark tasks remains complex, costly, and labor-intensive. Moreover, the standard approach, in which scenarios are first written in natural language and then mapped to tool sequences, captures only a narrow subset of the tool-use patterns agents exercise. In this paper, we address these problems by reversing the task construction process. We propose TASTE: Task Synthesis from Tool Sequence Evolution, an automatic method that generates challenging tasks with broader tool-use coverage. TASTE utilizes an Adaptive Contrastive

n

-gram model trained on LLM-judged validity signals. This enables sampling valid tool sequences that cover a vast range of tool combinations. TASTE then selects representative sequences from the pool via clustering, instantiates them into complete benchmark tasks, and refines them through iterative difficulty evolution. Using TASTE, we construct

τ^c

-Bench, a challenging extension of the three domains of

τ^2

-Bench. We evaluate

11

agent/user LLM pairs and find that models nearly saturating

τ^2

-Bench suffer severe performance drops on our tasks (e.g., Gemini-3-Flash falls from

0.82\!-\!0.94

0.28\!-\!0.61

). Beyond increasing difficulty, our generated tasks more than double the number of unique tool combinations agents must execute. Our results suggest high scores on existing benchmarks often reflect saturation rather than robust task-solving ability. By automating the generation of difficult, high-coverage benchmarks, TASTE enables continuous, scalable evaluation of future agents.

Abstract

LLM Agent Evaluation

Explore similar work

A Matter of TASTE: Improving Coverage and Difficulty of Agent Benchmarks

QUEST: Training Frontier Deep Research Agents with Fully Synthetic Tasks

SciExplore: Evaluating Autonomous Agents from Scientific Navigation to Information Integration