An agentic retrieval system issues a sequence of search queries and must decide, at each step, whether the evidence collected so far is enough to stop. Delegating that decision to an LLM verifier or a prompt judge makes stopping reliable, but the verifier then reprocesses the growing evidence after every retrieval step, a substantial repeated cost. We show that most of these calls can be skipped without materially changing answer accuracy: a single threshold on a frozen sentence-embedding coverage margin detects the states in which the evidence is still plainly incomplete, and the verifier is called only on the ambiguous remainder, a gate we call CoVeR (Coverage-based Verifier Routing). Across three multi-hop QA benchmarks, with the evaluation protocol fixed before the full-scale run, the CoVeR-gated agent matches the answer accuracy of both the full-budget agent and the always-verify baseline within a fraction of an EM point. It cuts 62-68% of verifier calls, and 93% in a saturated regime. Routers built on evidence counts, lexical overlap, or BM25 relevance, alone or learned in combination, give weaker overall trade-offs, the gate transfers without re-tuning across deciders and agent scales, and its drafter distills into a 921k-parameter head atop the frozen encoder, leaving no LLM in the routing loop. The same signal cannot replace verification: matching a claim is far easier than deciding the claim is supported.
Figures & tables
Figure 1: Routing the verification bill. (a) The verifier reads all evidence at every step. (b) CoVeR pays only for the ambiguous step (outcome preservation: § 5.1 ). Source: results/raw/e2/ .
Figure 2: CoVeR routes the decider (a routed HotpotQA trajectory). Claims and evidence are embedded by a frozen encoder; one threshold on the coverage margin skips the clearly-uncovered steps and pays for the one ambiguous state, where the swappable decider rules. A skip can only defer stopping — every stop is decider-issued. Source: lead-drawn figure ( figures/Figure.svg , cropped).
EM
Δ EM (pp) vs.
Call cut
Loop Δ
Agree.
Dataset
Full-budget
CoVeR
Always-verify
Full-budget
Always-verify
vs. Always
vs. Full
vs. Always
HotpotQA
0.3060
0.3040
0.3050
−0.20
−0.10
62.2%
−11.9%
.910
2Wiki
0.1840
0.1870
0.1850
+0.30
+0.20
67.8%
−9.8%
.886
MuSiQue †
0.0801
0.0812
0.0812
+0.11
+0.00
93.0%
−0.6%
.976
Table 1: Routing at up to n=1000 , LODO thresholds, trained-verifier decider. Δ EM in pp (full 95% CIs in Table 8 ; formal tests in Appendix B ). Agree. = held-out stop-loop agreement ( ≥0.90 applies at LODO selection). † Saturated (§ 5.1 ). Source: results/stat02/ .
Dataset
Oracle AUROC (CI)
Generated-claim AUROC (CI)
HotpotQA
0.892(0.879,0.904)
0.686(0.659,0.713)
2Wiki
0.895(0.884,0.905)
0.680(0.655,0.703)
MuSiQue
0.831(0.812,0.849)
0.760(0.723,0.792)
Table 2: Margin vs. per-hop coverage ( n=1000 ; clustered bootstrap CIs). A high oracle ceiling with a much weaker deployable floor (non-overlapping CIs): the geometry can express coverage, but matching a generated claim string cannot read it. Oracle and generated-claim targets come from different label runs/positive rates (ceiling/floor contrast, not a controlled ablation). Sources: results/gate01,stat01/ .
Dataset
Embedding
Count
Lexical
BM25
Learned
HotpotQA
0.622
0.000
0.512
0.202
0.435
2Wiki
0.623
0.288
0.562
0.191
0.563
MuSiQue
1.000
0.999
0.999
0.339
1.000
Table 3: Cheaper-router control: max call cut at stop-agreement ≥0.90 (best in bold; MuSiQue saturated). BM25: per-question closed-pool, pool-max normalized; Learned: LR on the cheap signals (§ G ). Sources: results/ctl01,def01,ctl05/ .
Figure 3: Router frontier (descriptive). Achievable verifier-call cut vs. stop-agreement with the always-verify decider, per dataset (envelope: max call cut at agreement ≥ each x ). Shaded region: agreement ≥0.90 , the pre-specified safety bar; the star marks the primary one-threshold operating point (call cut 0.622 at agreement 0.910 on HotpotQA; STAT-02). Per-dataset readings are in the text. Source: results/ctl01/ .
Figure 4: Verification compute stack. Verifier claim-verdicts per question at each routing tier: pure verifier ( 9.4/9.2/11.1 ), + tier-1 call routing ( 3.1/3.0/0.8 ), + tier-2 intra-call ordering ( 2.0/1.9/0.4 ). Tier 1 holds batched; tier 2 saves only unbatched. Source: results/eff01/ .
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Step
Follow-up query
Key retrieved evidence (new that step)
mRtD / mCUbtS
m(ℓ)
Gate ( τ=0.16 )
1
director of Remember the Day
“Remember the Day is a 1941 film released by 20th Century Fox, directed by Henry King…” (also retrieves John Gavin’s bio, 1875–1938, without any link to the film)
.174/.178
.178
skip
2
when did Henry King die
“Henry King (January 24, 1886 – June 29, 1982) was an American actor and film director.”
.174/.178
.178
skip
3
director of Cast Up by the Sea
“ Cast Up by the Sea is a 1916 Australian film directed by John Gavin. ”
.134/.131
.134
call → decider fires
Appendix
Table 6: A routed trajectory (2WikiMultihopQA). The Remember the Day side is covered by step 2, but nothing links Cast Up by the Sea to a director, so the state margin (the least-covered claim) holds above the threshold and the gate skips two verifier calls whose verdicts could only have been “keep searching.” When the missing film–director link arrives at step 3, both per-claim margins collapse, the gate pays for one verdict, and the decider fires. Full-budget runs three further steps to the loop-6 cap; full-budget, always-verify, and CoVeR all return the correct answer ( Cast Up by the Sea , EM 1 ). Per-claim margins recomputed from our archived caches; state margins match the cached values exactly.
Δ EM
NI vs. Full ( ε )
TOST
MDE
Dataset
vs. Full
1
1.5
2
vs. alw-verify
(pp)
HotpotQA
−0.20
×
✓
✓
±1,1.5,2
1.42
2Wiki
+0.30
✓
✓
✓
±1,1.5,2
1.23
MuSiQue
+0.11
✓
✓
✓
degen.
0.54
Appendix
Table 7: Confirmatory statistics. NI is one-sided (lower 95% bound >−ε , α=0.025 ); TOST vs. always-verify holds at the listed margins; MuSiQue’s always-verify delta is a zero-variance degenerate cell and is excluded from the equivalence family. All five non-degenerate tests survive Holm (max adj. p=0.0002 ). Source: results/stat02/ .
vs. full-budget
vs. always-verify
Dataset
Δ EM
95% CI
Δ EM
95% CI
HotpotQA
−0.20
[−1.20,+0.80]
−0.10
[−0.60,+0.40]
2Wiki
+0.30
[−0.60,+1.20]
+0.20
[−0.40,+0.80]
MuSiQue
+0.11
[−0.22,+0.55]
+0.00
[0.00,0.00]
Appendix
Table 8: The full intervals behind the ±1.2 pp bound. Paired bootstrap, 10,000 resamples, seed 13 , in percentage points; the widest interval is HotpotQA’s [−1.20,+0.80] against full-budget. MuSiQue’s always-verify delta is degenerate (the verifier fires on 2.6% of loops, so the gated and always-verify runs answer identically). Source: results/stat02/ci95_plain.json .
Dataset
Stage
pairs/q
vs. pure
HotpotQA
pure decider (every loop)
9.43
—
+ tier 1: call routing
3.56
−62.3%
+ tier 2: intra-call order
2.22
−76.4%
2Wiki
pure decider (every loop)
9.22
—
+ tier 1: call routing
2.97
−67.8%
+ tier 2: intra-call order
1.93
−79.1%
Appendix
Table 9: Verification compute stack (EFF-01, primary one-threshold arm). A pair is one claim-verdict; verdict latency is 116 ms batched / 405 ms unbatched (SRC §48). Tier 2 saves only in the unbatched regime. Source: results/eff01/ .
Dataset
System
calls/q
pairs/q
tok/q
ms/q
batch- 1
batched
HotpotQA
always-verify
4.69
10.95
3100
4532
1822
CoVeR
1.77
3.02
861
1250
502
2Wiki
always-verify
4.59
10.22
2801
4223
1643
CoVeR
1.41
2.67
749
1102
429
MuSiQue
always-verify
5.42
16.05
4563
6692
2665
Appendix
Table 10: Measured verifier workload and wall-clock (LAT-01). Per question, GPU-timed verifier forward passes replayed on the eval populations (gold-order last 400 /dataset; MuSiQue unique-qid). A pair is one (claim, sentence) score-pair; tok/q is verifier input tokens; ms/q is verifier wall-clock (forward passes only). Verdict parity ≥0.985 ; per-pair median 412/413/416 ms batch- 1 , 161 – 166 ms batched. Source: results/lat01/ .
Verification share % , always → CoVeR
Dataset
g=0.25 s
g=1 s
g=4 s
HotpotQA
60.8→28.7
28.0→9.2
8.9→2.5
2Wiki
58.9→25.3
26.4→7.8
8.2→2.1
MuSiQue
66.3→5.4
33.0→1.4
11.0→0.4
Appendix
Table 11: Verification share of trajectory latency ( % ), before and after gating. Computed as V/(V+gL′) from the batched verification wall-clock V (Table 10 ) and each system’s own mean executed loops ( Lc includes the added deferral loops; BE-01); g is the (unmeasured) generator per-loop latency. Source: results/lat01,be01/ .
Protocol
Datasets
n
Main routing (LODO, 3B)
hotpot / 2wiki / musique
1000/1000/911
7 B-agent routing
hotpot / 2wiki / musique
959/996/871
Open-corpus (Setting-B, 3B)
hotpot
1000
Pilot / judge / distilled
hotpot / 2wiki / musique
400/400/384
Appendix
Table 12: Evaluation denominators. All EM is Std-Ext (a frozen Qwen2.5-3B extractor at the stop loop). MuSiQue counts are unique qids (its gold dev split contains duplicate ids); 7 B populations cover questions with a usable 7 B label. Sources: results/e2c,rev01,e2b,gen01/ .
Dataset
Router
Thr.
Cut
Agree.
Cov.-safe
HotpotQA
Embedding
0.15
0.759
0.874
0.740
Count
6
0.458
0.822
0.845
Lexical
0.34
0.858
0.828
0.810
BM25
0.00
0.202
0.964
0.764
2Wiki
Embedding
0.145
0.883
0.801
0.790
Count
5
0.288
0.913
0.843
Appendix
Table 13: LODO operating points of the four routers (CTL-01/DEF-01; held-out n=1000/1000/911 ). Each threshold is selected on the other two datasets’ train- 600 — max call cut subject to agreement ≥0.90 and coverage safety ≥ the decider’s own reference — and applied here unchanged. Cut = decider-call reduction; Agree. = stop-loop agreement with the always-verify decider. As in Table 1 the ≥0.90 bar binds at selection , on the training folds; held-out agreement may fall below it, and here it does for several routers. Count on MuSiQue met no threshold satisfying the coverage-safety reference and fell back to the agreement constraint alone, which admits no skips; BM25’s threshold collapses to the grid floor on all three datasets.
Floor AUROC
Frontier Δ pp
Encoder
Hpt.
2Wiki
MuS.
Hpt.
2Wiki
e5-base-v2
.686
.680
.760
—
—
e5-large-v2
.662
.670
.753
−3.1
+1.6
bge-base
.604
.537
.702
−15.0
−24.9
bge-large
.605
.570
.720
−13.8
−12.2
gte-base
.546
.525
.588
−19.1
−25.4
Appendix
Table 14: Encoder choice (ENC-01). Gate floor AUROC (pooled per dataset) and LODO call-cut frontier at stop-agreement ≥0.90 , as points relative to the deployed e5-base-v2 (—, reference). Each encoder uses its own adaptive threshold grid because cosine scales differ across families. MuSiQue’s frontier saturates at 1.0 for every encoder, so it cannot separate them and is omitted from the frontier columns. Best floor per column in bold.
Arm
Router target texts
Claims consumed by decider
Deployable
Main 3B (Table 1 )
Generated claim strings
Generated expected claims a
Yes
Prompt judge (§ 5.4 )
Generated claim strings
None b
Yes
7 B (Table 4 a)
Generated claim strings
Generated expected claims
Yes
Open-corpus (Table 4 c)
Generated claim strings
Gold expected claims
No a′
CoVeR-D (Table 4 b)
Predicted claim embeddings c
Generated expected claims
Yes
CoVeR-D + judge (this appendix)
Predicted claim embeddings
None (claim-free judge)
Yes
Appendix
Table 15: Claim sources by experiment. The router’s coverage margin is computed against generated expected-claim strings in every routing arm (the gold supporting sentence is an oracle diagnostic only; § 3 ), except CoVeR-D, whose distilled sub- 1 M drafter predicts the target embeddings and leaves no LLM in the routing loop. The decider’s claim source varies: the confirmatory 3B and 7 B arms run the verifier with generated expected claims (fully deployable); only the open-corpus arm uses gold expected claims (disclosed). a Deployable at inference; threshold selection consults a gold coverage-safety reference on the training folds (calibration-time gold use, disclosed in § 4 ). a′ Open-corpus: not deployable, the decider consumes gold expected claims (disclosed). b The judge emits a single sufficiency probability ( pyes ) from the question and the collected sub-question/sub-answer pairs; its prompt contains no claim strings, so it is billed per call rather than per claim. c CoVeR-D’s drafter is a sub- 1 M MLP that predicts the claim-embedding targets from the question embedding (no claim-generation pass); its in-band decider is the generated-claims verifier, unchanged. The NLI-decider arm (Table 5 ) is fully public: its decider reads declarative NLI templates over the same generated-claim fields, with no unreleased weights.
Retrieval-augmented search agents answer multi-hop questions by repeatedly issuing search queries and accumulating evidence. This creates a stopping problem: after the necessary evidence has appeared, further retrieval often adds cost, latency, and distracting context rather than useful information. We frame stopping as evidence coverage rather than generator confidence, and introduce HALT, a lightweight verification-aware policy that leaves the search agent unchanged. Given expected hop claims, HALT stops only when cumulative evidence supports each required claim. Across three multi-hop QA benchmarks, HALT reduces redundant search while largely preserving exact match. We separate a deployable setting, where hop claims are generated from the question, from a diagnostic upper bound that uses gold supporting-fact annotations: generated claims give smaller but still exact-match-preserving savings, while gold claims show the larger savings available when hop targets are clean. Baseline comparisons and ablations show that this behavior is driven by claim-evidence alignment rather than generic sufficiency, fixed stop positions, or lexical overlap. Open-corpus pilots further suggest that HALT abstains when coverage cannot be reliably verified. Overall, evidence coverage provides a practical runtime control signal for improving retrieval-augmented agents without retraining or modifying the host agent.
LLM agents in knowledge intensive question answering take retrieval and reasoning actions with incomplete knowledge about whether their current answer is uncertain, unsupported, or already complete. This produces two failure modes: committing to confident but unsupported answers, which hurts accuracy, and over-retrieving when the evidence in hand already suffices, resulting in wasted compute. To give agents a more complete picture of the state space they are operating in, we introduce calibrated verifier telemetry (CalVerT), which augments the agent's state with additional telemetry: a calibrated self-confidence score and a grounding verifier score. We show that CalVerT can improve agents in both training-free and training-based settings. On four QA benchmarks, we find that CalVerT raises F1 by triggering retrieval in cases where agents over-rely on parametric knowledge, while cutting redundant retrieval in cases where agents have sufficient context to answer. We show that CalVerT can augment existing QA frameworks without training. Moreover, CalVerT also improves trained systems: by simply augmenting an agent's state with telemetry, we observe improvements after reinforcement learning, as compared to an agent with identical training but no CalVerT telemetry.
Retrieval-augmented generation improves the factuality of large language models by grounding responses in retrieved evidence, yet existing evaluation frameworks struggle to provide consistent, fine-grained diagnostics across the diverse spectrum of user queries, ranging from close-ended fact-seeking to open-ended explanatory requests. We propose Q-CARE, a query-agnostic and fully reference-free framework that enables fine-grained assessment by decomposing queries into sub-queries and answers into atomic claims. Q-CARE establishes a unified evaluation principle based on query coverage and claim verifiability, yielding coverage-aware retriever metrics (C-Prec@k, C-nDCG@k) and claim-level generator metrics (Completeness, Conciseness, and Verifiableness). On a human-annotated benchmark spanning eight datasets, Q-CARE achieves higher correlation with human judgments than four existing RAG evaluation metrics, including RAGEval and RAGChecker, proving its effectiveness as a reliable, automated evaluation framework. Code and data are publicly available at https://github.com/DISL-Lab/Q-CaRE-COLM-26.
Jeonghwan Choi, Taewon Yun, Minjeong Ban +3
1Korea Advanced Institute of Science and Technology (KAIST) · 2Cluvion