DRBENCHER: Can Your Agent Identify the Entity, Retrieve Its Properties and Do the Math?

Young-Suk Lee; Ramon Fernandez Astudillo; Radu Florian

cs.AIApr 10, 2026

DRBENCHER: Can Your Agent Identify the Entity, Retrieve Its Properties and Do the Math?

Authors: Young-Suk Lee, Ramon Fernandez Astudillo, Radu Florian

Organizations: IBM Research

Abstract

Deep research agents increasingly interleave web browsing with multi-step computation, yet existing benchmarks evaluate these capabilities in isolation, creating a blind spot in assessing real-world performance. We introduce DRBENCHER, a synthetic benchmark generator for questions that require both browsing and computation, synthesized answer-first from knowledge-graph chains with no seed passages or provenance text. It enforces four criteria: verifiability (gold answers are computed by executing parameterized code over knowledge-graph values), complexity (multi-hop entity identification, property retrieval, and domain-specific computation), difficulty (a two-stage verification cascade filters out questions solvable by the generating model), and diversity (a greedy max-min embedding filter maximizes coverage). These criteria are realized via a unified answer-first pipeline spanning five domains: biochemistry, financial, geophysical, security, and history. Human evaluation shows 76% validity, with 35% of errors due to outdated knowledge-graph entries, highlighting an inherent limitation of systems that reason over evolving data. Automatic evaluation shows that the strongest frontier model achieves only 20% answer accuracy. Compared to manually constructed benchmarks (BrowseComp+, MATH-500, GPQA), DRBENCHER achieves the highest semantic diversity. We release DrBencher pipeline and the human-verified benchmark data at https://github.com/IBM/DrBencher.

Figures & tables

Panel of Figure 1 — Figure 1: The unified DrBencher pipeline, shared across all five domains ( biochemistry , financial , geophysical , history , security ). A running example using Mount Fuji (atmospheric pressure template) illustrates each stage.

Benchmark	Multihop	HybridQ	Prog. gold	Syn	KG	Comp. calib.
HotpotQA ( Yang et al., 2018 )	✓	✗	✗	✗	✗	✗
MATH ( Hendrycks et al., 2021b )	✗	✗	✓	✗	✗	✗
MuSiQue ( Trivedi et al., 2022 )	✓	✗	✗	✗	✗	✗
LC-QuAD ( Trivedi et al., 2017 )	✓	✗	✗	✗	✓	✗
GAIA ( Mialon et al., 2024 )	✓	✗	✗	✗	✗	✗
AutoBencher ( Li et al., 2025 )	✗	✗	✗	✓	✗	✗

Table 1: Comparison of DrBencher with related benchmarks. HybridQ = each question jointly requires retrieval and computation, Prog. gold = programmatic gold answer, Syn = questions are synthetically generated, KG = clue facts verified against knowledge graph and Wikipedia, Comp. calib. = complexity calibration. ✓ = supported, ✗ = not supported.

Panel of Figure 2 — Figure 2: CCI illustrated for two questions. (a) $E{=}1,P{=}1$ (CCI $=$ 2): the model identifies one entity (Zugspitze) and retrieves one property (elevation). (b) $E{=}1,P{=}2$ (CCI $=$ 3): the model identifies one entity (Los Ángeles, Chile) but must retrieve two properties (population and area) to apply the population density template.

Domain	QAs	Tmpl.	Top.	Art.
Biochem	53	21	20	491
Financial	104	35	47	558
Geophysical	76	33	26	405
History	73	26	19	388
Security	48	14	23	140
Total	354	129	135	1982

Table 2: Benchmark summary. (a) Tmpl.: unique templates; Top.: entity topics; Art.: grounding articles. (b)

E

: entities;

\bar{P}

: mean property lookups; CCI

=

E{+}P

; C/Q: chains/question; H/C: hops/chain; H/Q: hops/question.

Domain	$n$	Err	DB	LLM	MH	Amb	Ent	Acc	Acc*
Geophysical	76	11	1	7	2	1	0	85.5	86.8
History	73	11	4	2	1	3	1	84.9	90.4
Biochemistry	53	16	2	7	1	3	3	69.8	73.6
Security	48	13	1	3	0	7	2	72.9	75.0
Financial	104	35	22	0	0	10	3	66.3	87.5
Total	354	86	30	19	4	24	9	75.7	84.2

Table 3: Human verification accuracy and error breakdown.

n

: questions after diversity filtering; Err: total errors; Acc: accuracy (%); Acc*: accuracy excluding DB errors (%). Error categories—DB: incorrect/stale KG data; LLM: LLM hallucination in fact extraction or question composition; MH: multi-hop property flattening; Amb: ambiguous or non-unique entity description; Ent: entity misidentification.

	Biochem		Financial		Geophysical		History		Security		All
$n$	37		69		65		62		35		268
Model	Ent	Ans	Ent	Ans	Ent	Ans	Ent	Ans	Ent	Ans	Ent	Ans
Claude Opus 4.6	80.2	35.1	94.2	10.6	77.4	20.0	82.8	30.6	98.1	4.8	86.1	20.1
Gemini 2.5 Flash	57.7	28.8	88.9	7.7	72.3	26.7	79.6	24.7	81.0	1.9	77.4	18.4
GPT-5.2	67.6	14.4	93.7	3.9	79.5	10.3	77.4	22.0	87.6	0.0	82.1	10.6
Llama 4 Maverick	64.0	27.9	82.1	6.3	69.7	21.5	81.7	30.6	74.3	1.0	75.5	17.9

Table 4: Entity identification (Ent) and answer accuracy (Ans) (%) across five DrBencher domains. Top: proprietary models; bottom: open-weight. Each model accuracy is the average of 3 runs. McNemar’s test (All,

N{=}268

, majority vote over 3 runs): Opus, Gemini, and Llama form a statistical tie (pairwise p

>

0.05); Opus significantly outperforms GPT5.2, Qwen, and Mistral at p

<

0.001.

Table 7

Domain	GPT	Llama	Shared	%GPT $\subseteq$ Llama	Gold
Biochem	21	41	16	76.2	16/16
Financial	18	37	13	72.2	13/13
Geophysical	41	61	31	75.6	31/31
History	21	34	18	85.7	18/18
Security	12	20	11	91.7	11/11
Total	113	193	89	78.8	89/89

Table 7: Difficulty-filter ablation: gpt-oss-120b vs. Llama 4 Maverick as the V1/V2 verifier, with question generation held fixed. GPT/Llama: questions surviving each filter; Shared: retained by both; %GPT

\subseteq

Llama: fraction of gpt-oss-120b ’s set also in Llama’s; Gold: matching gold answers among shared questions. Gold answers match exactly (89/89), confirming answer-level model independence.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Architecture	MoE, 120B parameters
Quantization	MXFP4
Precision	bfloat16
Tensor parallel size	8
GPU memory utilization	0.9
Max sequence length	131,072 tokens

Appendix

Table 8: Model serving configuration.

Parameter	V1	V2
Samples per question ( $n$ )	10	10
Sampling temperature	0.7	1.0
Accuracy threshold ( $\tau$ )	0.5	0.5
Max agentic iterations ( $K$ )	—	200
Tool timeout	—	60 s
Answer tolerance ( $\epsilon$ )	5%

Appendix

Table 9: Verification hyperparameters. V1: closed-book; V2: agentic with tool calls.

\epsilon{=}5\%

relative tolerance is used during generation-time difficulty filtering; a stricter 2% tolerance is applied during evaluation scoring (Section 5.2 ).

Family	Template	Computation	Properties	Unit
Quantitative modeling	Exponential growth	$\lfloor P(1{+}r)^{t}\rceil$	P1082	people
Quantitative modeling	GDP per capita	$G/P$	P2131, P1082	USD
Scientific inference	Surface gravity	$GM/r^{2}$	P2067, P2120	m/s 2
Scientific inference	Atmospheric pressure	$P_{0}e^{-Mgh/RT}/1000$	P2044	kPa

Appendix

Table 10: Representative reasoning templates (4 of 37)

Panel of Figure 3 — Figure 3: Representative page from the human annotation interface. The tool displays the question, gold answer, gold entity chain, computation code, and grounding evidence with semantic highlighting ( blue : question-relevant entities; green : answer-relevant values). Annotators assign a binary verdict and optional comment.

Domain	$n$	Agr.	$\alpha$
Geophysical	47	78.7	0.25
History	30	86.7	0.43
Biochemistry	32	68.8	0.10
Security	11	81.8	0.56
Financial	39	69.2	0.36
Overall	159	76.1	0.30

Appendix

Table 11: Inter-annotator agreement across all five domains.

n

: questions with overlapping annotations from two annotators; Agr.: raw agreement (%);

\alpha

: Krippendorff’s alpha. Overall statistics are weighted by domain size.

Disagreement source	Count	%
LLM hallucination / factual error (only)	18	47.4
Question ambiguity / non-uniqueness (only)	8	21.1
Both ambiguity + hallucination	7	18.4
Stale KG data	4	10.5
Unclassified	1	2.6
Total	38	100.0

Appendix

Table 12: Breakdown of all 38 inter-annotator disagreements (of 159 matched pairs). Disagreements concentrate in subjective categories (hallucination, ambiguity); objectively verifiable stale data accounts for only 10.5%.

Parameter	Value
Samples per question ( $n$ )	3
Temperature	model default (omitted)
Max response tokens	8,192
Answer tolerance	2% relative (exact for history)
Retry on failure	5 attempts, exponential backoff

Appendix

Table 13: Evaluation inference configuration.

Failure breakdown (% of each model’s failures):
	Opus	Gem.	GPT	Llm.	Qwn.	Mst.
	4.6	2.5	5.2	4 Mav	3	Sm.
Answer acc. (%)	20	17	10	16	8	6
Failures ( $n$ )	214	222	242	224	247	251
Property retrieval	64.0	45.5	54.5	51.8	38.5	51.8
True calculation	15.0	0.5	0.0	19.6	2.4	0.0

Appendix

Table 14: Two-stage failure analysis over the 268 questions (majority vote, 3 runs). Rows below the rule are percentages of each model’s failures. Property/formula (indet.): entity correctly identified and answer wrong, but the reasoning trace was insufficient to distinguish a wrong property value from a wrong formula. Models: Claude Opus 4.6, Gemini 2.5 Flash, GPT-5.2, Llama 4 Maverick, Qwen3-30B-A3B, Mistral-Small-3.2-24B.

	Opus	Gemini	GPT5.2	Llama	Qwen
Gemini	0.243
GPT5.2	<.001 ∗∗∗	<.001 ∗∗∗
Llama	0.155	0.868	0.004 ∗∗
Qwen	<.001 ∗∗∗	<.001 ∗∗∗	0.404	<.001 ∗∗∗
Mistral	<.001 ∗∗∗	<.001 ∗∗∗	0.095	<.001 ∗∗∗	0.522

Appendix

Table 15: McNemar’s test p-values for answer accuracy (All

N{=}268

). Per-question correctness is determined by majority vote across 3 runs. ∗∗ p

<

0.01, ∗∗∗ p

<

0.001.

Domain	$n$	Mean	Med.	CCI=2	CCI=3	CCI=4	CCI=6
Geophysical	65	2.35	2	53	3	8	1
Biochemistry	37	2.81	2	20	4	13	0
History	62	2.82	3	24	25	13	0
Financial	69	2.84	2	37	10	20	2
Security	35	3.49	4	3	12	20	0

Appendix

Table 16: CCI distribution by domain. Mean: mean CCI; Med.: median CCI.

CCI	Biochem	Financial	Geophysical	History	Security	All
2	34.7 (20)	6.0 (37)	13.9 (53)	37.5 (24)	5.6 (3)	18.8
3	6.9 (4)	6.1 (10)	31.5 (3)	15.6 (25)	2.3 (12)	11.1
$\geq$ 4	6.8 (13)	4.8 (22)	11.1 (9)	20.5 (13)	0.6 (20)	7.4

Appendix

Table 17: CCI-controlled accuracy (%) by domain (number of questions in parentheses). Each cell is the average across 6 models

\times

3 runs.

Panel of Figure 4 — Figure 4: Representative QA pairs from the Financial (top) and Security (bottom) domains.

AI Agent Benchmarks

AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.

See all 1,494 papers

DI-Bench: Systematically Generating In-Domain Data Intelligence Benchmarks for Enterprise Agents
Sep 4, 2026Jiangyun Zhang, Kristen Surrao, Torpong Nitayanont +10AI Agent Benchmarks Synthetic Benchmark Generation
WANDR: A Benchmark for Wide and Deep Research
Aug 14, 2026Vitaliy Polshkov, Marcin Pitera, Jeremy Yang +7AI Agent Benchmarks Deep Research Agents
Perplexity
DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports
Jan 13, 2026Ruizhe Li, Mingxuan Du, Benfeng Xu +3Deep Research Agents LLM Agent Evaluation
University of Science and Technology of China · Metastone Technology, Beijing, China

Sep 4, 2026J/K move · Enter open · S save

DI-Bench: Systematically Generating In-Domain Data Intelligence Benchmarks for Enterprise Agents

Jiangyun Zhang, Kristen Surrao, Torpong Nitayanont, Yupei Zhang+9

AI Agent Benchmarks Synthetic Benchmark Generation Data Analysis Agents Knowledge-Intensive QA

Open paper AskPDF arXiv

Evaluating enterprise agents on domain-specific benchmarks is critical, yet public benchmarks rarely evaluate whether agents can integrate business knowledge with analytical computation, and constructing such benchmarks manually is costly. We present DI-Bench, a pipeline for generating realistic benchmarks for data intelligence (DI), the practice of extracting insights from large volumes of enterprise data. To emulate realistic DI tasks that require both computation and knowledge retrieval, DI-Bench builds an artifact linkage graph over data tables, dimensions, metrics, and documents to form questions involving structured data and associated knowledge. Ground truth answers are derived via query execution, followed by LLM question generation and validation. Applied to two public datasets, the pipeline produces a 731-task benchmark covering knowledge retrieval, analytical computation, and rule-grounded reasoning. To show the discriminatory capability and difficulty of the benchmark, we evaluate four models, revealing a substantial finding: models achieve only 32% accuracy when doing computational tasks where retrieved business rules modify the computation.

Abstract

AI Agent Benchmarks

Explore similar work

DI-Bench: Systematically Generating In-Domain Data Intelligence Benchmarks for Enterprise Agents

WANDR: A Benchmark for Wide and Deep Research

DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports