Controlling Authority Retrieval: A Missing Retrieval Objective for Authority-Governed Knowledge

Andre Bacellar

cs.IRApr 15, 2026

Controlling Authority Retrieval: A Missing Retrieval Objective for Authority-Governed Knowledge

Authors: Andre Bacellar

Abstract

In law, regulatory regimes for pharmaceuticals and software security, newer authorities can revoke older established ones even when semantically distant. We call this CAR: retrieving the currently active authority frontier for a semantic anchor q, that is, front(cl(A_k(q))). This differs from finding the most similar document by relevance score: argmax_d s(q, d). Theorem 4 characterizes when a set R truly covers the active authority set for q with TCA(R, q)=1, providing conditions necessary and sufficient for any retrieved set R: frontier inclusion (front(cl(A_k(q))) contained in R) and no-ignored-superseder (no superseding document exists in the corpus outside R). Proposition 2 shows that TCA@k <= phi(q) * R_anchor(q) in the worst case over any scope-indexed algorithm, proved by an adversarial permutation argument. We evaluated on three real-world datasets: security advisories (Dense TCA@5=0.270, two-stage 0.975), SCOTUS overruling pairs (Dense TCA=0.172, two-stage 0.926), and FDA drug records (Dense TCA=0.064, two-stage 0.774). A GPT-4o-mini experiment shows Dense RAG produces explicit "not patched" claims for 39% of queries where a patch exists; two-stage cuts this to 16%. Four benchmark datasets, domain adapters, and a single-command scorer are released at https://github.com/andremir/car-retrieval.

Figures & tables

Result	Role	Claim (independent of pipeline)
Theorem 4	Correctness	N&S conditions on any $\mathcal{R}$ : frontier inclusion $+$ no-ignored-superseder
Proposition 2	Limit	Hard worst-case ceiling $\varphi(q)$ on all scope-indexed algorithms
Theorem 3 $+$ Cor. 3	Construction	Two-stage is sufficient when $\varphi(q)=1$ ; T1, T2 prove each component necessary
Proposition 3	Cost	$\Omega(\|\mathcal{R}_{\sigma}\|)$ oracle lower bound; RSSG costs $O(\|\mathcal{R}_{\sigma}\|^{2}m)$ without oracle
Theorems 1–2, P1, L1	Context / synthesis	Mismatch, closure necessity, factorization, separability

Table 1

	T0	T1	T2	T3
Examples	250	250	250	250
KB events (avg)	1.0	2.0	3.0	3.0
Supersession hops	0	1	2	2
Cross-provenance	No	No	No	Yes
Shared retrieval corpus: 12,250 events total

Table 1: FinSuperQA statistics.

	Overall				TCA by Hop Type
System	R@5	TCA	Acc	ProvRec	T0	T1	T2	T3
BM25	0.607	0.305	0.503	0.607	1.000	0.000	0.216	0.004
TF-IDF (cosine)	0.621	0.254	0.491	0.621	1.000	0.000	0.012	0.004
Dense (MiniLM)	0.450	0.214	0.247	0.450	0.820	0.000	0.036	0.000
Oracle	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000
DenseReRank	0.333	0.499	0.499	0.333	1.000	0.000	0.980	0.016

Table 2: Main results on FinSuperQA (

k=5

, 12,250-event corpus). TCA

=1.000

for TwoStage and TwoStage+RSSG on T0–T3 is a guaranteed consequence of the entity-disjoint corpus design (each (emp, ticker) pair appears in exactly one example, so the entity bucket always contains clean gold events)—not an empirical surprise. The benchmark is designed to isolate the vocabulary-mismatch failure: BM25, TF-IDF, and Dense all achieve TCA

=0.000

on T1 despite Recall@

k>0

, confirming Theorem 1 for both lexical and semantic retrievers. The contamination ablation (Table 3 ) evaluates the harder setting where the entity-disjoint guarantee is relaxed.

System	TCA	T3	$\Delta$ T3
TwoStage (anchor only)	0.911	0.652	—
TwoStage+RSSG (anchor + RSSG)	0.942	0.776	$+12.4$ pp

Table 3: Results on the contaminated corpus (naive distractor sampling with no entity-disjoint guarantee; enforce_entity_disjoint=False ). The RSSG provides

+12.4

pp on T3 by applying domain rules to disambiguate spurious same-entity distractors without oracle access. Full results in data/contaminated_corpus_results.json .

Metric	Query $\leftrightarrow$ Disclosure	Query $\leftrightarrow$ Patch
Jaccard	0.085	0.060
Jaccard (no stopwords)	0.037	0.014
% positive gap (Jaccard)	96.7%	—
BM25 score	9.73	6.45
% positive gap (BM25)	97.5%	—

Table 4: Vocabulary gap between queries and (a) vulnerability disclosures vs. (b) patch release notes on 120 real security advisories (GHSA + NVD). Disclosures are

1.51\times

more BM25-relevant to the query than patch notes; 97.5% of pairs show positive BM25 gap, 96.7% positive Jaccard gap. The vocabulary gap is a property of natural corpora, not an artifact of our synthetic construction.

System	TCA@5	TCA@10	TCA@20	Mean rank	Median rank
BM25	0.641	0.730	0.761	144.4	3
Dense (MiniLM)	0.132	0.157	0.214	1,234	1,847
Dense (E5-large)	0.069	0.120	0.170	1,113	1,246
HyDE (Llama 3.3 70B)	0.151	0.170	0.170	1,283	1,577
Regex-NER Probe	1.000	1.000	1.000	2.0	2

Table 5: End-to-end retrieval on 159 real GHSA advisories (2,318-document corpus). 95% Wilson CIs: Dense (E5-large) TCA@5 = 0.069 [0.038, 0.119]; Dense (MiniLM) TCA@5 = 0.132 [0.087, 0.194]; HyDE TCA@5 = 0.151 [0.103, 0.216]; Regex-NER Probe TCA@5 = 1.000 [0.977, 1.000]. HyDE’s LLM includes the CVE identifier in 100% of generated hypotheticals, making each hypothetical semantically closer to disclosures than to release notes; adding 70B parameters changes nothing structurally. The Regex-NER Probe matches the oracle upper bound (TCA = 1.000): entity-indexed retrieval is both the correct fix and immediately practical. Artifacts: data/ghsa_realworld_results.json ; HyDE: data/ghsa_hyde_results.json .

System	Disc@5	TCA@5	TCA@10	Mean rank
BM25	0.975	0.138	0.214	609
Dense (MiniLM)	0.937	0.270	0.365	764
Dense (E5-large)	0.962	0.252	0.321	799
HyDE (Llama 3.3 70B)	—	0.151	—	—
Two-Stage ( $k_{1}{=}5$ )	0.975	0.975	—	43.2

Table 6: Free-form query results on 159 real GHSA pairs (2,318-document corpus). Disc@5 = fraction of free-form queries for which BM25 retrieves the correct CVE disclosure in top-5. TCA@

k

of the two-stage pipeline equals exactly Stage-1 Disc@

k

, since the

j

-th retrieved disclosure’s patch note is placed at rank

j

in the final ranking. Artifact: data/ghsa_twostage_results.json .

Model	Params	Family	Disc@5	TCA@5
MiniLM-L6-v2	22M	MiniLM	0.962	0.283
E5-small-v2	33M	E5	0.987	0.264
E5-base-v2	109M	E5	0.950	0.239
E5-large-v2	335M	E5	0.956	0.214
BGE-small	33M	BGE	0.981	0.252
BGE-base	109M	BGE	0.975	0.283

Table 7: Scale amplifies error in the E5 family (TCA@5 monotonically decreases with parameters), while BGE saturates at the same ceiling as MiniLM. No dense model exceeds TCA@5 = 0.283 regardless of scale. The two-stage pipeline achieves 0.975 with no trained parameters , a

3.4\times

improvement over any dense baseline. Artifact: data/ghsa_scale_results.json .

Domain	Query style	Stage-1 Rec@5	BM25	Two-Stage
Security (real, $n{=}159$ )	Generic free-form	0.975	0.138	0.975
CVE (synth, $n{=}250$ )	Generic free-form	0.860	0.000	0.860
CVE (synth, $n{=}250$ )	+ product hint	1.000 †	0.000	0.900
Legal (synth, $n{=}250$ )	Generic free-form	0.312	0.000	0.312
Legal (synth, $n{=}250$ )	+ party-name hint	1.000	0.000	1.000

Table 8: Two-stage pipeline across three domain constructions and two query styles. BM25 direct always fails (TCA@5

\approx

0). Two-Stage TCA@5 equals Stage-1 anchor Recall@5 when entity scope keys are unambiguous (Corollary 2 ). † CVE+product-hint exception: product names are shared across CVEs, so product hints cause false anchor retrievals; free-form queries avoid this. Artifacts: data/cve_synthetic_twostage_results.json , data/legal_twostage_results.json .

System	Finance T1	CVE T1	Legal T1
BM25 / TF-IDF	0.000	0.000	0.000
TwoStage	1.000	1.000	1.000
TwoStage+RSSG	1.000	1.000	1.000

Table 9: T1 TCA across three synthetic domains. All relevance-based retrievers fail (TCA

=0.000

); the two-stage architecture solves T1 exactly in all three.

System	Anchor@5	TCA@5	95% CI
BM25 direct	—	0.893	[0.826, 0.936]
Dense (MiniLM) direct	—	0.172	[0.115, 0.249]
Two-Stage (BM25 anchor)	0.836	0.836	[0.760, 0.891]
Two-Stage (Dense anchor)	0.926	0.926	[0.865, 0.961]

Table 10: End-to-end retrieval on 122 public SCOTUS overruling pairs (244-opinion corpus). BM25 direct achieves 0.893 [0.826, 0.936] because overruling opinions explicitly cite the case they overrule; Dense direct fails (0.172 [0.115, 0.249]) because semantic similarity prefers the original opinion. Two-Stage (Dense anchor) achieves 0.926 [0.865, 0.961]—the CIs of BM25 and Two-Stage overlap (n=122 is underpowered to distinguish the +3.3 pp gap). SCOTUS is the easiest domain for BM25: explicit citation vocabulary bridges the lexical gap. The hard structural failure is Dense (0.172), where semantic similarity actively misdirects. Artifacts: data/legal_scotus_expanded_pairs.json , data/legal_scotus_expanded_results.json .

System	Stage-1 @5	TCA@5	95% CI
BM25 direct	—	0.004	[0.001, 0.014]
Dense (MiniLM) direct	—	0.064	[0.046, 0.089]
Two-Stage (BM25 anchor)	0.774	0.774	[0.735, 0.808]

Table 11: End-to-End retrieval on 500 FDA drug recall pairs (3,000-document corpus). BM25 and Dense (MiniLM) collapse (TCA@5

\leq 0.064

, 95% CI upper bound

\leq 0.089

): therapeutic indication vocabulary has near-zero overlap with recall enforcement vocabulary. Two-Stage TCA@5 equals Stage-1 anchor recall@5 exactly (Theorem 3 ); the 95% CI [0.735, 0.808] reflects the indication ambiguity at Stage-1: multiple drugs share therapeutic categories, and Stage-1 errors cannot be remedied by entity-indexed Stage-2. Artifact: data/fda_recall_pairs.json , data/fda_recall_results.json .

Condition	Patch Recall@5	Says Patched	Conf. Wrong	Uncertain
Dense (MiniLM)	28.3%	61.0%	39.0%	0%
Two-Stage (BM25)	97.5%	82.4%	16.4%	1%

Table 12: GPT-4o-mini answer quality on 159 GHSA pairs. All gold answers are patched . “Says Patched” = LLM asserts the vulnerability is fixed; “Conf. Wrong” = LLM explicitly asserts it is not patched. Dense retrieval causes confident wrong assertions for 39% of queries. Two-stage retrieval cuts this to 16.4% (

-22.6

pp). Artifact: data/llm_downstream_results.json .

Domain	n	BM25	Dense	Two-Stage	vs. Dense
Security (GHSA)	159	0.138	0.270	0.975	$+70.5$ pp
Legal (SCOTUS LII)	122	0.893	0.172	0.926	$+75.4$ pp
Medical (FDA Recall)	500	0.004	0.064	0.774	$+71.0$ pp

Table 13: Real-world benchmark summary (TCA@5 with 95% Wilson CIs). “Two-Stage” uses BM25 anchor in security and medical, Dense anchor in legal (best per-domain). Two-Stage 95% CIs: GHSA [0.937, 0.990], SCOTUS [0.865, 0.961], FDA [0.735, 0.808]. Legal BM25 is high (0.893 [0.826, 0.936]); BM25 and Two-Stage CIs overlap at n

=122

(see § 6.9 ). GHSA and FDA provide the cleaner evidence: both BM25 and Dense collapse (

\leq 0.270

), while Two-Stage recovers to

\geq 0.774

with non-overlapping CIs against both baselines. Medical shows the most severe collapse: therapeutic vocabulary shares no terms with recall enforcement vocabulary (BM25

=0.004

[0.001, 0.014]).

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Rule	Trigger	Target	Scope
R1	BlackoutAnnounced	PreClearanceApproved	Same ticker
R2	BlackoutLifted	BlackoutAnnounced	Same ticker
R3	EmergencyException	BlackoutAnnounced	Emp + ticker
R4	ExceptionRevoked	EmergencyException	Emp + ticker
R5	WatchlistAdded	PreClearanceApproved	Same ticker
R6	WatchlistRemoved	WatchlistAdded	Same ticker

Appendix

Table 14: Supersession rules. Trigger is the later event superseding Target.

Legal IR

Legal information retrieval finds and ranks relevant statutes, cases, patents, and other legal materials for queries and evidence-grounded systems.

See all 53 papers

Re:CAP - Auditing Retrieval Coverage in Production RAG Pipelines
Sep 21, 2026Aviral Joshi, Hanoz Bhathena, Max Nelson +1RAG Evaluation IR Evaluation
Machine Learning Center of Excellence, JPMorgan Chase & Co.
DrugClaw and DrugAudit: A Primary-Source-Grounded Agent and Authority-Aware Benchmark for Drug-Information Question Answering
May 31, 2026Qing Wang, Bo Li, Jialu Liang +3Biomedical QA Agentic RAG
Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Florida, USA · PAMI Research Group, Department of Computer and Information Science, Faculty of Science and Technology, University of Macau, Taipa, Macau, China
CARA: Causally Guided Retrieval Agent for Question Answering
Oct 15, 2026Weicheng Liu, Xiangcheng Wu, Depeng Xu +2Agentic Retrieval Graph-Based Retrieval
University of North Carolina at Charlotte

Sep 21, 2026J/K move · Enter open · S save

Re:CAP - Auditing Retrieval Coverage in Production RAG Pipelines

Aviral Joshi, Hanoz Bhathena, Max Nelson, Saket Sharma

Machine Learning Center of Excellence, JPMorgan Chase & Co.

RAG Evaluation IR Evaluation

Open paper AskPDF arXiv

Retrieval-augmented generation (RAG) is hard to monitor in production: exhaustive relevance labels do not exist for non-stationary multi-million-passage corpora that re-index in real time. As a result, retrieval quality is generally understudied and often deprioritised in favour of generation-oriented metrics. In this work, we propose auditing retrieval coverage by probing for evidence of missing documents rather than enumerating every relevant one. Our method Re:CAP (REtrieval Coverage Audit by iterative Probing) is a reference-free audit loop applied to a deployed RAG pipeline's initial answer and retrieved context: it identifies the topics already covered, generates probing questions for plausibly missing topics, retrieves candidate documents, and applies an LLM-as-judge to retain only those that introduce previously-unretrieved information. On four public benchmarks, Re:CAP recovers 9-29% of gold labels that flat BM25 top-500 cannot reach, rising to 48% on TREC-COVID. On MuSiQue Re:CAP beats flat hybrid top-500 by +12.9 pp on recall at less than half the document budget. An ensemble BM25, dense, and hybrid baseline (top-500 each) still leaves out 21.2% of gold docs on TREC-COVID that Re:CAP recovers; human annotators judge that 78.9% of those structurally distinct documents add new information to the baseline answer (Fleiss

κ

= 0.79, n = 123), and 73.9% on live production traffic (n = 180). End-to-end recall is reproducible to within

\pm

1% across three independent runs, making Re:CAP a stable instrument for periodic retrieval audits.

Abstract

Legal IR

Explore similar work

Re:CAP - Auditing Retrieval Coverage in Production RAG Pipelines

DrugClaw and DrugAudit: A Primary-Source-Grounded Agent and Authority-Aware Benchmark for Drug-Information Question Answering

CARA: Causally Guided Retrieval Agent for Question Answering