Enterprise agents need retrieval evaluation sets that expose domain-specific failures and guide developers towards efficient deployment tradeoffs. Neither expert-human-authored sets nor current synthetic generation methods provide a methodology for achieving the necessary coverage, reliability, efficiency, and discriminability required for such decision-making. Simultaneously, the field does not have an assessment framework for determining when an evaluation set has satisfied these properties. We introduce an evaluation set assessment framework for these criteria and CoverSynth -- a generator specifically designed to satisfy these criteria without incurring the high cost of human expert labeling. Starting from an unannotated corpus, CoverSynth targets coverage across retrieval demands and system capabilities. Deterministic checks locate evidence quotes in their sources, while model-based checks assess semantic validity to guarantee reliable ground truth. Across five public enterprise-style corpora, our method provides informative variation in difficulty, revealing unseen failure modes such as an inability of agents to handle underspecified or ambiguous requests. On the RepLiQA corpus, 91% of human-generated queries can be answered by all members on a panel of 12 retrieval systems, compared to only 12% of CoverSynth-generated queries. Finally, in a repeated TechQA/RepLiQA study, 12 of 12 within-harness model contrasts have positive nominal 95% intervals on generated sets, compared with 4 of 12 on human sets. CoverSynth makes coverage and evidence quality explicit construction objectives, producing corpus-grounded evaluations that reveal retrieval headroom and differences between agent configurations.
Figures & tables
Figure 1: Overview of CoverSynth. An inferred lens guides chunk annotation and concept-graph construction. Graph communities, personas, and coverage controls guide positive and negative query generation. Model checks assess semantic validity; the export gate enforces exact span localization and record contracts.
Corpus
Source
Queries
AD ↑
Persona Hnorm↑
ZQC ↓
SCC ↑
Difficulty skew ⊙
ERAG-Bench
CoverSynth
150
0.88 ∗
0.98 ∗
0.00
0.96
−0.69
Reference
53
0.70
0.79
0.54
0.86
−0.90
Harvey FK
CoverSynth
150
0.89 ∗
0.99 ∗
0.00
0.76
−0.67∗
Reference
150
0.59
0.87
0.59
0.99
−1.60
TechQA
CoverSynth
128
0.91 ∗
0.99 ∗
0.18
0.87
+0.27
Reference
374
0.41
0.54
0.31
0.54
−0.09
Table 1: Benchmark assessment metrics as defined in Section 4 on the five corpora using both CoverSynth as well as reference benchmarks. Note that persona entropy values are calculated using CoverSynth’s persona set, excluding questions without a matching persona. Computation of reference datasets’ SCC and ZQC are detailed in Section A.10 . Arrows indicate the preferred direction for the first four assessment metrics. For difficulty skew, values closer to zero are preferred (with negative values representing a skew towards hard and positive values representing a skew towards easy). Statistically significant differences for AD, persona coverage, and difficulty skew are marked with ∗ .
ERAG-Bench
Harvey FK
TechQA
eCFR
RepLiQA
Rate
CoverSynth
Human
CoverSynth
MiRAGE
CoverSynth
Human
CoverSynth
MiRAGE
CoverSynth
Human
Sufficiency
0.70
0.64
0.80
0.95
0.72
0.61
0.97
0.95
0.91
0.90
Uniqueness
0.99
1.00
0.68
0.65
0.96
0.96
0.99
0.94
0.90
0.97
Necessity (leave-one-out)
0.58
0.64
0.61
0.24
0.67
— †
0.49
0.41
0.49
— †
Table 2: Gold-span validity, as defined in Section 4 , from oracle-reader probes. The oracle reader (Claude-Opus-5) is a different model from the generator (GPT-5.6 Sol). Uniqueness contrasts each gold span against the three highest lexical-overlap non-gold passages. † The human TechQA and RepLiQA sets carry a single gold span per item, so leave-one-out necessity is trivially 1.00 and not comparable to the multi-span CoverSynth rate.
Figure 2: Single-pass answerability-verdict accuracy by corpus and model under the Claude Code harness, for all five CoverSynth outputs (the same runs as Table 9 ). The first four columns are the negative families; the final column is answerable-item verdict accuracy. Failed runs are excluded.
Figure 3: Retrieval performance on CoverSynth and human evaluation sets. Points show mean span F2 on answerable questions; whiskers show 95% bootstrap intervals. Protocol details are in Section A.14 .
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Method
SME-free
Grounding
Multi-hop
Negatives
Coverage
InPars
✓
–
–
–
–
Promptagator
∼
–
–
–
–
DataGen
✓
∼
–
–
∼
RARE
✓
∼
✓
–
–
CCQGen
✓
–
–
–
✓
OBLIQ-Bench
∼
∼
–
–
✓
Appendix
Table 3: Positioning of CoverSynth against representative synthetic generation methods (references in Section 2 ) on five dimensions: reliance on human annotation or guidance (SME-free), validation of byte-exact text spans (grounding), inclusion of multi-hop and negative questions (multi-hop and negative, respectively), and coverage-aware generation (coverage). ✓: supported; ∼ : partial or heuristic; –: not addressed. Each prior method covers a subset of these properties; CoverSynth is distinguished by combining byte-exact grounding with a formal structural approach to generating multi-hop positives and hard negatives under enforced coverage.
Question intent
What the answer requires
Factual lookup
Retrieve a stated fact, definition, value, threshold, name, or reference precisely, as written (the default when no other intent fits).
Compliance
Whether a given situation satisfies a rule, or what is required to satisfy it: eligibility, permissibility, exemption, or which requirement governs the case.
Procedural
How something is done or constructed; the answer is a method, a sequence of steps, or a process.
Diagnostic
Explains an existing state: the reason or mechanism behind a rule or behavior, or where a defect or failure lies.
Impact
What follows from a state or change: its effect, consequence, penalty, or knock-on obligation.
Temporal
Turns on time or order: what must happen before or after what, a stage, duration, waiting period, frequency, or deadline; or how guidance on a topic changed across dated sources.
Appendix
Table 4: The eight question intents ( scenario_type ) a positive item can realize, each naming the analytical work the answer requires. The generator steers a candidate toward one intent and an adversarial gate confirms the drafted question matches it. Factual lookup is the default, used when no other intent fits.
Failure family
Generation control(s)
Answerability
Desired agent behavior
Corpus-unsupported
Entity, property, condition, time, scope
Unsupported
Abstain with rationale
False premise
Contradicted assertion
Unsupported
Correct the premise
Underspecified
Remove disambiguator
Ambiguous
Request clarification
Ambiguity
Ambiguous reference
Ambiguous
Request clarification
Appendix
Table 5: The negative-question taxonomy: four failure families realized by eight generation controls. Answerability and expected behavior are determined by the family. Only the corpus-unsupported family ships an empty gold set; the others cite the passage that makes the item fail.
Corpus
Documents
Generation Communities
ERAG-Bench
95
95
Harvey FK
137
515
TechQA
500
176
eCFR
120
398
RepLiQA
275
65
Appendix
Table 6: Reported Leiden community counts (communities with at least two concept labels) on the corpus subsets used for the construction profiles.
Property
TechQA
RepLiQA
All items
128
147
Answerable items
100
100
Multi-document answerable items
50
72
Labeled multi-hop answerable items
34
11
Occupied capability cells
28/32
19/32
AD
0.906
0.768
Appendix
Table 7: CoverSynth-generated set composition. Structural counts and AD use the 100 answerable items per corpus; intent presence uses all records. AD is normalized over the declared 32-cell grid. Negative-family counts refer to distinct items, not repeated runs.
Figure 4: Difficulty distribution of answerable questions for CoverSynth and each reference set. Difficulty is IRT item difficulty β from a joint fit per corpus over six systems (three indexed retrievers and three Claude Code agents). Cut-offs for each corpus’s difficulty category were determined using Jenks natural breaks optimization. Hatched segments represent the proportion of hard questions that no system was able to solve.
Corpus
Set
Skew
Δ skew [95% CI]
ERAG-Bench
CoverSynth
−0.69
+0.21[−0.54,+0.92]
Reference
−0.90
Harvey FK
CoverSynth
−0.67
+0.93[+0.37,+1.53]
Reference
−1.60
TechQA
CoverSynth
+0.27
+0.35[−0.21,+0.90]
Reference
−0.09
Appendix
Table 8: Shape of the item difficulty distribution on answerable questions. An item’s failure rate is the share of the six panel systems whose binarized span- F2 falls below the corpus’s pooled mean-matching threshold. Skew is the adjusted Fisher–Pearson sample skewness of item failure rates; positive values indicate mass at the easy end with a tail of harder items. Δ skew gives CoverSynth minus reference with a 95% bootstrap interval (2,000 resamples, drawing items independently within each set).
File
Span
Resource use
Corpus
Model
P
R
F1
P
R
F1
Tool Calls
Tokens (k)
ERAG-Bench
Opus-5
0.52
0.62
0.53
0.17
0.47
0.23
13.7
11.0
Sonnet-5
0.59
0.55
0.54
0.26
0.41
0.29
18.4
13.9
Haiku-4.5
0.43
0.40
0.39
0.20
0.29
0.22
30.0
30.1
Harvey FK
Opus-5
0.46
0.66
0.51
0.19
0.38
0.23
7.6
7.9
Sonnet-5
0.50
0.54
0.49
0.20
0.29
0.22
6.7
8.7
Appendix
Table 9: Retrieval performance on the answerable questions from the five-corpus CoverSynth outputs against a Claude code harness for three differently sized Claude models. File and span retrieval metrics include average precision (P), recall (R), and F1. Resource use is the mean number of tool calls and of evidence-context tokens per question; the turn budget is 50. Failed runs are excluded (Appendix A.13 ).
Figure 5: Five-corpus system comparison: (a) mean span F2 and (b) mean context tokens, on a logarithmic scale. Agent bars summarize the broader ReAct backbone panel, the three-Claude ReAct panel and the three-Claude Claude Code panel; whiskers show the within-group range. The three-Claude agent groups on TechQA and RepLiQA include negative questions, with credit for correct abstention; the tables and repeated study report retrieval scores on answerable questions.
File
Span
Resource use
Corpus
Model
P
R
F1
P
R
F1
Tool Calls
Tokens (k)
ERAG-Bench
Opus-5
0.95
0.88
0.90
—
—
—
10.8
8.8
Sonnet-5
0.91
0.67
0.73
—
—
—
9.9
7.3
Haiku-4.5
0.94
0.56
0.66
—
—
—
10.5
10.9
TechQA
Opus-5
0.83
0.91
0.86
0.23
0.72
0.32
8.1
2.5
Sonnet-5
0.77
0.80
0.78
0.24
0.61
0.31
11.4
4.5
Appendix
Table 10: Retrieval on native ERAG-Bench, TechQA and RepLiQA reference sets using Claude Code. File and span metrics report mean precision (P), recall (R), and F1 . Resource use is the mean number of tool calls and evidence-context tokens per question; the turn budget is 40. Failed runs are excluded ( Section A.13 ). ERAG reference gold is document-level, so span metrics are omitted. MiRAGE chunk metrics appear separately in Table 11 .
File
Chunk
Resource use
Corpus
Model
N
P
R
F1
P
R
F1
Calls
Tokens (k)
Harvey FK
Opus-5
150
0.673
0.616
0.596
0.317
0.491
0.335
7.4
7.740
Sonnet-5
150
0.621
0.485
0.498
0.350
0.365
0.310
6.3
8.489
Haiku-4.5
149
0.636
0.455
0.486
0.411
0.316
0.309
6.4
17.632
eCFR
Opus-5
150
0.997
0.977
0.982
0.625
0.828
0.651
5.4
2.435
Sonnet-5
150
0.907
0.880
0.889
0.712
0.721
0.665
4.9
3.306
Appendix
Table 11: MiRAGE retrieval using native chunk gold on Harvey and eCFR. File and chunk precision (P), recall (R) and F1 use the evidence units named in their column groups. Chunk scores are not directly comparable to character-overlap span scores. Claude Code uses a 40-turn budget. N counts successful answerable runs; Harvey Haiku excludes one failure. Resource columns report mean tool calls and evidence-context tokens.
TechQA
RepLiQA
Metric
CoverSynth
Human
CoverSynth
Human
All nine systems, three repeats
Complete document recall by all systems
23/100
75/150
12/100
118/130
Best-system mean span F2
0.52
0.47
0.47
0.75
Six base agents, three repeats
Span- F2 reliability G(100)
0.94
0.74
0.93
0.86
Appendix
Table 12: Focused comparison on answerable items with span gold. The first block uses all nine systems; complete document recall requires every system to cite all gold documents in every repeat. The second block uses the six base agents. A pair meets the detection criterion when nominal paired tests detect a difference in at least 80% of 400 resamples of 100 items with replacement.
Panel
Corpus
GCS
GHu
ΔG
95% interval
Six agents, three repeats
TechQA
0.942
0.739
+0.203
[+0.050,+0.659]
RepLiQA
0.927
0.857
+0.070
[−0.056,+0.337]
Pooled
—
—
+0.136
[+0.041,+0.414]
Six agents + retrievers
TechQA
0.986
0.988
−0.002
[−0.011,+0.005]
RepLiQA
0.976
0.997
−0.022
[−0.039,−0.016]
Pooled
—
—
−0.012
[−0.022,−0.008]
Appendix
Table 13: Span- F2 reliability and sensitivity to panel choice. CS = CoverSynth; Hu = human. All estimates project to 100 items. Intervals are 95% bootstrap intervals for ΔG=GCS−GHu ; pooled rows average the two corpus differences. The last panel adds BM25, dense, and hybrid retrieval to the six base agents (three repeats).
System
Reps
Span F2
Span P
Span R
File R
CS
Hu
CS
Hu
CS
Hu
CS
Hu
BM25
1
0.16
0.13
0.04
0.03
0.72
0.83
0.83
0.95
Dense
1
0.15
0.12
0.04
0.03
0.61
0.77
0.72
0.95
Hybrid
1
0.17
0.12
0.05
0.03
0.75
0.82
0.84
0.97
ReAct-Haiku
3
0.39
0.47
0.39
0.31
0.43
0.64
0.65
0.86
ReAct-Sonnet
3
0.46
0.41
0.36
0.23
0.57
0.63
0.78
0.78
Appendix
Table 14: TechQA: mean scores on answerable items with span gold (CS = CoverSynth, Hu = human; n = 100 CS, 150 human). Reps = independent runs over the full evaluation set (not agent turns); agent scores average their repeats, and failed runs score 0. CC-Opus-5.5 and CC-Max are single-run extensions outside the nine-system panel ( Section A.16 ).
System
Reps
Span F2
Span P
Span R
File R
CS
Hu
CS
Hu
CS
Hu
CS
Hu
BM25
1
0.19
0.15
0.06
0.03
0.56
0.98
0.71
0.99
Dense
1
0.16
0.15
0.05
0.03
0.49
0.91
0.69
0.99
Hybrid
1
0.20
0.16
0.06
0.04
0.58
0.99
0.78
1.00
ReAct-Haiku
3
0.36
0.70
0.35
0.72
0.39
0.75
0.59
0.98
ReAct-Sonnet
3
0.42
0.74
0.40
0.70
0.45
0.82
0.69
0.99
Appendix
Table 15: RepLiQA: mean scores on answerable items with span gold (CS = CoverSynth, Hu = human; n = 100 CS, 130 human). Reps = independent runs over the full evaluation set (not agent turns); agent scores average their repeats, and failed runs score 0. CC-Opus-5.5 and CC-Max are single-run extensions outside the nine-system panel ( Section A.16 ).
TechQA
RepLiQA
System
Unsupp.
False prem.
Underspec.
Ambig.
Hu unsupp.
Unsupp.
False prem.
Underspec.
Ambig.
Hu unsupp.
n items
12
9
5
2
120
13
9
12
13
38
ReAct-Haiku
0.67
1.00
0.00
0.00
0.54
0.79
0.96
0.06
0.08
0.35
ReAct-Sonnet
0.75
0.89
0.00
0.00
0.79
0.95
0.85
0.14
0.21
0.39
ReAct-Opus
0.81
1.00
0.00
0.00
0.71
0.97
1.00
0.25
0.64
0.43
CC-Haiku
0.81
0.85
0.13
0.00
0.66
0.72
0.85
0.06
0.05
0.26
Appendix
Table 16: Answerability-verdict accuracy on negatives, by failure family (CoverSynth) and on the human ’not in corpus’ negatives. Failed runs count as wrong; agents averaged over repeats. CC-Opus-5.5 and CC-Max are single-run extensions ( Section A.16 ).
Retrieval quality is the primary bottleneck for accuracy and robustness in retrieval-augmented generation (RAG). Current evaluation relies on heuristically constructed query sets, which introduce a hidden intrinsic bias. We formalize retrieval evaluation as a statistical estimation problem, showing that metric reliability is fundamentally limited by the evaluation-set construction. We further introduce semantic stratification, which grounds evaluation in corpus structure by organizing documents into an interpretable global space of entity-based clusters and systematically generating queries for missing strata. This yields (1) formal semantic coverage guarantees across retrieval regimes and (2) interpretable visibility into retrieval failure modes. Experiments across multiple benchmarks and retrieval methods validate our framework. The results expose systematic coverage gaps, identify structural signals that explain variance in retrieval performance, and show that stratified evaluation yields more stable and transparent assessments while supporting more trustworthy decision-making than aggregate metrics.
Recent advancements in generative AI facilitate large-scale synthetic data generation for model evaluation. However, without targeted approaches, these datasets often lack the sociotechnical nuance required for sensitive domains. We introduce NodeSyn, a methodology that pairs a fine-tuned taxonomy generator (TaG) with credible published sources and extracted sensitive attributes to generate queries reflecting authentic societal contexts and nuanced harms. Evaluated across ten open-weight and proprietary LLMs in Medical Advice and Self-Harm using human-validated autoraters, NodeSyn ranked first on 8 of 10 models in each domain. NodeSyn datasets induced average failure rates 3.1 times higher than generic synthetic baselines, 2.4 times higher than structured synthetic benchmarks, and 1.6 times higher than authentic human-authored queries. Ablation studies confirm that, controlling for structural confounds, our granular Level 3 taxonomic expansion significantly drives these failure rates over Level 2 across 9 of 10 models (+29.2% relative lift in Medical Advice and +24.0% in Self-Harm). Variance attribution analysis reveals that failures are driven not only by what topics are queried (taxonomic nodes and lexical specificity), but also by their sociotechnical grounding (affected demographics, occupations, geographies, and querying personas). We embed this method into a research prototype, and open-source (https://github.com/google-research/nodesynth) the code and datasets to enable scalable, high-stakes model evaluation.
Qazi Mamunur Rashid, Xuan Yang, Zhengzhe Yang +5
Google Research, USA · Google, USA · Google DeepMind, USA
Today, tool-calling agents are commonly evaluated or tested on static datasets of execution traces, including input commands, agent responses, and associated tool calls. However, internal production datasets are often insufficient or unusable for testing; for example, they may contain sensitive or proprietary data, or they may be too sparse to support comprehensive testing (especially pre-deployment). In these settings, practitioners are increasingly replacing or augmenting real datasets with synthetic ones for evaluation purposes. A key challenge is quantifying the relation between these synthetic datasets and the real data. We introduce SynAE, an evaluation framework for assessing how well synthetic benchmarks for multi-turn, tool-calling agents replicate and augment the characteristics of real data trajectories. SynAE assesses the validity, fidelity, and diversity of synthetic data across four metric categories: (i) task instructions and intermediate responses, (ii) tool calls, (iii) final outputs, and (iv) downstream evaluation. We evaluate SynAE using recent agent benchmarks and test common synthetic data failure modes via realistic and controlled generation schemes. SynAE detects fine-grained variations in data validity, fidelity and diversity, and shows that no single metric is sufficient to fully characterize synthetic data quality, motivating a multi-axis evaluation of synthetic data for agent testing. A demo of SynAE is available at https://synae-2026-synae-demo.static.hf.space/index.html, with code at https://github.com/wsqwsq/SynAE.