CoverSynth: Coverage-Guided Generation of Retrieval Evaluation Sets

Anahita Sharma; Apaar Shanker; Rishav Chakravarti; Sneha Srinivasan; Vijay Kalmath; Dylan Wilbur; Yuan Li; Sara Bolouki

cs.AIOct 11, 2026

CoverSynth: Coverage-Guided Generation of Retrieval Evaluation Sets

Authors: Anahita Sharma, Apaar Shanker, Rishav Chakravarti, Sneha Srinivasan, Vijay Kalmath, Dylan Wilbur, Yuan Li, Sara Bolouki,

Organizations: Scale AI

Abstract

Enterprise agents need retrieval evaluation sets that expose domain-specific failures and guide developers towards efficient deployment tradeoffs. Neither expert-human-authored sets nor current synthetic generation methods provide a methodology for achieving the necessary coverage, reliability, efficiency, and discriminability required for such decision-making. Simultaneously, the field does not have an assessment framework for determining when an evaluation set has satisfied these properties. We introduce an evaluation set assessment framework for these criteria and CoverSynth -- a generator specifically designed to satisfy these criteria without incurring the high cost of human expert labeling. Starting from an unannotated corpus, CoverSynth targets coverage across retrieval demands and system capabilities. Deterministic checks locate evidence quotes in their sources, while model-based checks assess semantic validity to guarantee reliable ground truth. Across five public enterprise-style corpora, our method provides informative variation in difficulty, revealing unseen failure modes such as an inability of agents to handle underspecified or ambiguous requests. On the RepLiQA corpus, 91% of human-generated queries can be answered by all members on a panel of 12 retrieval systems, compared to only 12% of CoverSynth-generated queries. Finally, in a repeated TechQA/RepLiQA study, 12 of 12 within-harness model contrasts have positive nominal 95% intervals on generated sets, compared with 4 of 12 on human sets. CoverSynth makes coverage and evidence quality explicit construction objectives, producing corpus-grounded evaluations that reveal retrieval headroom and differences between agent configurations.

Figures & tables

Figure 1: Overview of CoverSynth. An inferred lens guides chunk annotation and concept-graph construction. Graph communities, personas, and coverage controls guide positive and negative query generation. Model checks assess semantic validity; the export gate enforces exact span localization and record contracts.

Corpus	Source	Queries	AD $\uparrow$	Persona $H_{\mathrm{norm}}$ $\uparrow$	ZQC $\downarrow$	SCC $\uparrow$	Difficulty skew $\odot$
ERAG-Bench	CoverSynth	150	0.88 ∗	0.98 ∗	0.00	0.96	$-0.69$
ERAG-Bench	Reference	53	0.70	0.79	0.54	0.86	$-0.90$
Harvey FK	CoverSynth	150	0.89 ∗	0.99 ∗	0.00	0.76	$-0.67^{*}$
Harvey FK	Reference	150	0.59	0.87	0.59	0.99	$-1.60$
TechQA	CoverSynth	128	0.91 ∗	0.99 ∗	0.18	0.87	$+0.27$
TechQA	Reference	374	0.41	0.54	0.31	0.54	$-0.09$

Table 1: Benchmark assessment metrics as defined in Section 4 on the five corpora using both CoverSynth as well as reference benchmarks. Note that persona entropy values are calculated using CoverSynth’s persona set, excluding questions without a matching persona. Computation of reference datasets’ SCC and ZQC are detailed in Section A.10 . Arrows indicate the preferred direction for the first four assessment metrics. For difficulty skew, values closer to zero are preferred (with negative values representing a skew towards hard and positive values representing a skew towards easy). Statistically significant differences for AD, persona coverage, and difficulty skew are marked with ∗ .

	ERAG-Bench		Harvey FK		TechQA		eCFR		RepLiQA
Rate	CoverSynth	Human	CoverSynth	MiRAGE	CoverSynth	Human	CoverSynth	MiRAGE	CoverSynth	Human
Sufficiency	0.70	0.64	0.80	0.95	0.72	0.61	0.97	0.95	0.91	0.90
Uniqueness	0.99	1.00	0.68	0.65	0.96	0.96	0.99	0.94	0.90	0.97
Necessity (leave-one-out)	0.58	0.64	0.61	0.24	0.67	— †	0.49	0.41	0.49	— †

Table 2: Gold-span validity, as defined in Section 4 , from oracle-reader probes. The oracle reader (Claude-Opus-5) is a different model from the generator (GPT-5.6 Sol). Uniqueness contrasts each gold span against the three highest lexical-overlap non-gold passages. † The human TechQA and RepLiQA sets carry a single gold span per item, so leave-one-out necessity is trivially

1.00

and not comparable to the multi-span CoverSynth rate.

Figure 2: Single-pass answerability-verdict accuracy by corpus and model under the Claude Code harness, for all five CoverSynth outputs (the same runs as Table 9 ). The first four columns are the negative families; the final column is answerable-item verdict accuracy. Failed runs are excluded.

Figure 3: Retrieval performance on CoverSynth and human evaluation sets. Points show mean span

F_{2}

on answerable questions; whiskers show 95% bootstrap intervals. Protocol details are in Section A.14 .

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Method	SME-free	Grounding	Multi-hop	Negatives	Coverage
InPars	✓	–	–	–	–
Promptagator	$\sim$	–	–	–	–
DataGen	✓	$\sim$	–	–	$\sim$
RARE	✓	$\sim$	✓	–	–
CCQGen	✓	–	–	–	✓
OBLIQ-Bench	$\sim$	$\sim$	–	–	✓

Appendix

Table 3: Positioning of CoverSynth against representative synthetic generation methods (references in Section 2 ) on five dimensions: reliance on human annotation or guidance (SME-free), validation of byte-exact text spans (grounding), inclusion of multi-hop and negative questions (multi-hop and negative, respectively), and coverage-aware generation (coverage). ✓: supported;

\sim

: partial or heuristic; –: not addressed. Each prior method covers a subset of these properties; CoverSynth is distinguished by combining byte-exact grounding with a formal structural approach to generating multi-hop positives and hard negatives under enforced coverage.

Question intent	What the answer requires
Factual lookup	Retrieve a stated fact, definition, value, threshold, name, or reference precisely, as written (the default when no other intent fits).
Compliance	Whether a given situation satisfies a rule, or what is required to satisfy it: eligibility, permissibility, exemption, or which requirement governs the case.
Procedural	How something is done or constructed; the answer is a method, a sequence of steps, or a process.
Diagnostic	Explains an existing state: the reason or mechanism behind a rule or behavior, or where a defect or failure lies.
Impact	What follows from a state or change: its effect, consequence, penalty, or knock-on obligation.
Temporal	Turns on time or order: what must happen before or after what, a stage, duration, waiting period, frequency, or deadline; or how guidance on a topic changed across dated sources.

Appendix

Table 4: The eight question intents ( scenario_type ) a positive item can realize, each naming the analytical work the answer requires. The generator steers a candidate toward one intent and an adversarial gate confirms the drafted question matches it. Factual lookup is the default, used when no other intent fits.

Failure family	Generation control(s)	Answerability	Desired agent behavior
Corpus-unsupported	Entity, property, condition, time, scope	Unsupported	Abstain with rationale
False premise	Contradicted assertion	Unsupported	Correct the premise
Underspecified	Remove disambiguator	Ambiguous	Request clarification
Ambiguity	Ambiguous reference	Ambiguous	Request clarification

Appendix

Table 5: The negative-question taxonomy: four failure families realized by eight generation controls. Answerability and expected behavior are determined by the family. Only the corpus-unsupported family ships an empty gold set; the others cite the passage that makes the item fail.

Corpus	Documents	Generation Communities
ERAG-Bench	95	95
Harvey FK	137	515
TechQA	500	176
eCFR	120	398
RepLiQA	275	65

Appendix

Table 6: Reported Leiden community counts (communities with at least two concept labels) on the corpus subsets used for the construction profiles.

Property	TechQA	RepLiQA
All items	128	147
Answerable items	100	100
Multi-document answerable items	50	72
Labeled multi-hop answerable items	34	11
Occupied capability cells	28/32	19/32
AD	0.906	0.768

Appendix

Table 7: CoverSynth-generated set composition. Structural counts and AD use the 100 answerable items per corpus; intent presence uses all records. AD is normalized over the declared 32-cell grid. Negative-family counts refer to distinct items, not repeated runs.

Figure 4: Difficulty distribution of answerable questions for CoverSynth and each reference set. Difficulty is IRT item difficulty $\beta$ from a joint fit per corpus over six systems (three indexed retrievers and three Claude Code agents). Cut-offs for each corpus’s difficulty category were determined using Jenks natural breaks optimization. Hatched segments represent the proportion of hard questions that no system was able to solve.

Corpus	Set	Skew	$\Delta$ skew [95% CI]
ERAG-Bench	CoverSynth	$-0.69$	$+0.21$ $[-0.54,+0.92]$
ERAG-Bench	Reference	$-0.90$	$+0.21$ $[-0.54,+0.92]$
Harvey FK	CoverSynth	$-0.67$	$+0.93$ $[+0.37,+1.53]$
Harvey FK	Reference	$-1.60$	$+0.93$ $[+0.37,+1.53]$
TechQA	CoverSynth	$+0.27$	$+0.35$ $[-0.21,+0.90]$
TechQA	Reference	$-0.09$	$+0.35$ $[-0.21,+0.90]$

Appendix

Table 8: Shape of the item difficulty distribution on answerable questions. An item’s failure rate is the share of the six panel systems whose binarized span-

F_{2}

falls below the corpus’s pooled mean-matching threshold. Skew is the adjusted Fisher–Pearson sample skewness of item failure rates; positive values indicate mass at the easy end with a tail of harder items.

\Delta

skew gives CoverSynth minus reference with a 95% bootstrap interval (2,000 resamples, drawing items independently within each set).

		File			Span			Resource use
Corpus	Model	P	R	F1	P	R	F1	Tool Calls	Tokens (k)
ERAG-Bench	Opus-5	0.52	0.62	0.53	0.17	0.47	0.23	13.7	11.0
	Sonnet-5	0.59	0.55	0.54	0.26	0.41	0.29	18.4	13.9
	Haiku-4.5	0.43	0.40	0.39	0.20	0.29	0.22	30.0	30.1
Harvey FK	Opus-5	0.46	0.66	0.51	0.19	0.38	0.23	7.6	7.9
	Sonnet-5	0.50	0.54	0.49	0.20	0.29	0.22	6.7	8.7

Appendix

Table 9: Retrieval performance on the answerable questions from the five-corpus CoverSynth outputs against a Claude code harness for three differently sized Claude models. File and span retrieval metrics include average precision (P), recall (R), and F1. Resource use is the mean number of tool calls and of evidence-context tokens per question; the turn budget is 50. Failed runs are excluded (Appendix A.13 ).

Figure 5: Five-corpus system comparison: (a) mean span $F_{2}$ and (b) mean context tokens, on a logarithmic scale. Agent bars summarize the broader ReAct backbone panel, the three-Claude ReAct panel and the three-Claude Claude Code panel; whiskers show the within-group range. The three-Claude agent groups on TechQA and RepLiQA include negative questions, with credit for correct abstention; the tables and repeated study report retrieval scores on answerable questions.

		File			Span			Resource use
Corpus	Model	P	R	F1	P	R	F1	Tool Calls	Tokens (k)
ERAG-Bench	Opus-5	0.95	0.88	0.90	—	—	—	10.8	8.8
	Sonnet-5	0.91	0.67	0.73	—	—	—	9.9	7.3
	Haiku-4.5	0.94	0.56	0.66	—	—	—	10.5	10.9
TechQA	Opus-5	0.83	0.91	0.86	0.23	0.72	0.32	8.1	2.5
	Sonnet-5	0.77	0.80	0.78	0.24	0.61	0.31	11.4	4.5

Appendix

Table 10: Retrieval on native ERAG-Bench, TechQA and RepLiQA reference sets using Claude Code. File and span metrics report mean precision (P), recall (R), and

F_{1}

. Resource use is the mean number of tool calls and evidence-context tokens per question; the turn budget is 40. Failed runs are excluded ( Section A.13 ). ERAG reference gold is document-level, so span metrics are omitted. MiRAGE chunk metrics appear separately in Table 11 .

Corpus	Model	$N$	P	R	$F_{1}$	P	R	$F_{1}$	Calls	Tokens (k)
			File			Chunk			Resource use
Harvey FK	Opus-5	150	0.673	0.616	0.596	0.317	0.491	0.335	7.4	7.740
	Sonnet-5	150	0.621	0.485	0.498	0.350	0.365	0.310	6.3	8.489
	Haiku-4.5	149	0.636	0.455	0.486	0.411	0.316	0.309	6.4	17.632
eCFR	Opus-5	150	0.997	0.977	0.982	0.625	0.828	0.651	5.4	2.435
	Sonnet-5	150	0.907	0.880	0.889	0.712	0.721	0.665	4.9	3.306

Appendix

Table 11: MiRAGE retrieval using native chunk gold on Harvey and eCFR. File and chunk precision (P), recall (R) and

F_{1}

use the evidence units named in their column groups. Chunk scores are not directly comparable to character-overlap span scores. Claude Code uses a 40-turn budget.

N

counts successful answerable runs; Harvey Haiku excludes one failure. Resource columns report mean tool calls and evidence-context tokens.

All nine systems, three repeats
	TechQA		RepLiQA
Metric	CoverSynth	Human	CoverSynth	Human
Complete document recall by all systems	23/100	75/150	12/100	118/130
Best-system mean span $F_{2}$	0.52	0.47	0.47	0.75
Six base agents, three repeats
Span- $F_{2}$ reliability $G(100)$	0.94	0.74	0.93	0.86

Appendix

Table 12: Focused comparison on answerable items with span gold. The first block uses all nine systems; complete document recall requires every system to cite all gold documents in every repeat. The second block uses the six base agents. A pair meets the detection criterion when nominal paired tests detect a difference in at least 80% of 400 resamples of 100 items with replacement.

Panel	Corpus	$G_{\mathrm{CS}}$	$G_{\mathrm{Hu}}$	$\Delta G$	95% interval
Six agents, three repeats	TechQA	0.942	0.739	+0.203	$[+0.050,+0.659]$
	RepLiQA	0.927	0.857	+0.070	$[-0.056,+0.337]$
	Pooled	—	—	+0.136	$[+0.041,+0.414]$
Six agents + retrievers	TechQA	0.986	0.988	$-0.002$	$[-0.011,+0.005]$
	RepLiQA	0.976	0.997	$-0.022$	$[-0.039,-0.016]$
	Pooled	—	—	$-0.012$	$[-0.022,-0.008]$

Appendix

Table 13: Span-

F_{2}

reliability and sensitivity to panel choice. CS = CoverSynth; Hu = human. All estimates project to 100 items. Intervals are 95% bootstrap intervals for

\Delta G=G_{\mathrm{CS}}-G_{\mathrm{Hu}}

; pooled rows average the two corpus differences. The last panel adds BM25, dense, and hybrid retrieval to the six base agents (three repeats).

System	Reps	Span F2		Span P		Span R		File R
		CS	Hu	CS	Hu	CS	Hu	CS	Hu
BM25	1	0.16	0.13	0.04	0.03	0.72	0.83	0.83	0.95
Dense	1	0.15	0.12	0.04	0.03	0.61	0.77	0.72	0.95
Hybrid	1	0.17	0.12	0.05	0.03	0.75	0.82	0.84	0.97
ReAct-Haiku	3	0.39	0.47	0.39	0.31	0.43	0.64	0.65	0.86
ReAct-Sonnet	3	0.46	0.41	0.36	0.23	0.57	0.63	0.78	0.78

Appendix

Table 14: TechQA: mean scores on answerable items with span gold (CS = CoverSynth, Hu = human; n = 100 CS, 150 human). Reps = independent runs over the full evaluation set (not agent turns); agent scores average their repeats, and failed runs score 0. CC-Opus-5.5 and CC-Max are single-run extensions outside the nine-system panel ( Section A.16 ).

System	Reps	Span F2		Span P		Span R		File R
		CS	Hu	CS	Hu	CS	Hu	CS	Hu
BM25	1	0.19	0.15	0.06	0.03	0.56	0.98	0.71	0.99
Dense	1	0.16	0.15	0.05	0.03	0.49	0.91	0.69	0.99
Hybrid	1	0.20	0.16	0.06	0.04	0.58	0.99	0.78	1.00
ReAct-Haiku	3	0.36	0.70	0.35	0.72	0.39	0.75	0.59	0.98
ReAct-Sonnet	3	0.42	0.74	0.40	0.70	0.45	0.82	0.69	0.99

Appendix

Table 15: RepLiQA: mean scores on answerable items with span gold (CS = CoverSynth, Hu = human; n = 100 CS, 130 human). Reps = independent runs over the full evaluation set (not agent turns); agent scores average their repeats, and failed runs score 0. CC-Opus-5.5 and CC-Max are single-run extensions outside the nine-system panel ( Section A.16 ).

	TechQA					RepLiQA
System	Unsupp.	False prem.	Underspec.	Ambig.	Hu unsupp.	Unsupp.	False prem.	Underspec.	Ambig.	Hu unsupp.
n items	12	9	5	2	120	13	9	12	13	38
ReAct-Haiku	0.67	1.00	0.00	0.00	0.54	0.79	0.96	0.06	0.08	0.35
ReAct-Sonnet	0.75	0.89	0.00	0.00	0.79	0.95	0.85	0.14	0.21	0.39
ReAct-Opus	0.81	1.00	0.00	0.00	0.71	0.97	1.00	0.25	0.64	0.43
CC-Haiku	0.81	0.85	0.13	0.00	0.66	0.72	0.85	0.06	0.05	0.26

Appendix

Table 16: Answerability-verdict accuracy on negatives, by failure family (CoverSynth) and on the human ’not in corpus’ negatives. Failed runs count as wrong; agents averaged over repeats. CC-Opus-5.5 and CC-Max are single-run extensions ( Section A.16 ).

Synthetic Benchmark Generation

Synthetic benchmark generation creates controlled, verifiable evaluation tasks and datasets to test models across diverse settings.

See all 287 papers

Coverage, Not Averages: Semantic Stratification for Trustworthy Retrieval Evaluation
Apr 22, 2026Andrew Klearman, Radu Revutchi, Rohin Garg +3RAG Evaluation IR Evaluation
Scale AI.
NodeSyn: Socially Aligned Synthetic Data for AI Evaluation
May 14, 2026Qazi Mamunur Rashid, Xuan Yang, Zhengzhe Yang +5Language Model Generation Evaluation Language Model Safety Evaluation
Google Research, USA · Google, USA · Google DeepMind, USA
SynAE: A Framework for Measuring the Quality of Synthetic Data for Tool-Calling Agent Evaluations
May 21, 2026Shuaiqi Wang, Aadyaa Maddi, Zinan Lin +1Tool-Use Evaluation LLM Agent Evaluation
Carnegie Mellon University · Microsoft Research

Apr 22, 2026J/K move · Enter open · S save

Coverage, Not Averages: Semantic Stratification for Trustworthy Retrieval Evaluation

Andrew Klearman, Radu Revutchi, Rohin Garg, Rishav Chakravarti+2

Scale AI.

RAG Evaluation IR Evaluation

Open paper AskPDF arXiv

Retrieval quality is the primary bottleneck for accuracy and robustness in retrieval-augmented generation (RAG). Current evaluation relies on heuristically constructed query sets, which introduce a hidden intrinsic bias. We formalize retrieval evaluation as a statistical estimation problem, showing that metric reliability is fundamentally limited by the evaluation-set construction. We further introduce semantic stratification, which grounds evaluation in corpus structure by organizing documents into an interpretable global space of entity-based clusters and systematically generating queries for missing strata. This yields (1) formal semantic coverage guarantees across retrieval regimes and (2) interpretable visibility into retrieval failure modes. Experiments across multiple benchmarks and retrieval methods validate our framework. The results expose systematic coverage gaps, identify structural signals that explain variance in retrieval performance, and show that stratified evaluation yields more stable and transparent assessments while supporting more trustworthy decision-making than aggregate metrics.

Abstract

Synthetic Benchmark Generation

Explore similar work

Coverage, Not Averages: Semantic Stratification for Trustworthy Retrieval Evaluation

NodeSyn: Socially Aligned Synthetic Data for AI Evaluation

SynAE: A Framework for Measuring the Quality of Synthetic Data for Tool-Calling Agent Evaluations