Many AI systems need to find every relevant items in a large collection and then organize what they found. The standard way to do this is to tag each item with labels from a curated vocabulary and search on those labels. Those labels are useful for organizing results, but searching on them misses items that do not fit a label and returns too much when labels are broad. We compare this approach with an embedding-based approach: find candidates by meaning using text embeddings, then have a language model keep only the relevant ones. In a corpus of 7,354 randomized trials, label search found 45% of the evidence a systematic review would include, while the new approach found 92%, at a cost per query that rises with search depth. These results support a two-stage architecture that combines broad candidate retrieval with query-specific evidence filtering.
Figures & tables
Fig. 1: The ImpactAI evidence search interface.
Data Field
Example
Paper id
7077
Paper Title
The impact of mRDT malaria tests in Uganda
Intervention–Outcome pairs
1. Pair id
76717
1. Intervention Name
mRDT malaria tests
1. Intervention Description
The intervention involves the introduction of malaria rapid diagnostic tests (mRDTs) in public health centers, where healthcare personnel diagnose malaria using mRDTs instead of relying solely on clinical judgment. ACT is only administered when the mRDT result is positive.
TABLE I: An RCT and two intervention–outcome pairs in ImpactAI.
Fig. 2: Taxonomy-based entity linking and semantic retrieval with relevance filtering.
Dataset
N. Queries
Query annotation
N. Relevant (avg)
Candidates
Relevant Annotation
Filtering Agreement
Beta Testers
100
Manual
24.02
gemini-embeddings top 30
Human
Synthetic
100
Synthetic
23.82
gemini-embeddings top 30
Human
Double
10
Manual
—
gemini-embeddings top 30
2 Humans
Retrieval Quality
gemini-embeddings top 1000 +
TABLE II: Datasets used to evaluate retrieval and relevance filtering.
Model
Reference
Cohen’s κ
Pearson
Micro-F1
Neg. F1
Pos. F1
100 Beta Tester queries
Gemini 2.5
Human
0.5026
0.5037
0.7513
0.7428
0.7594
100 Synthetic queries
Gemini 2.5
Human
0.5045
0.5097
0.7510
0.7534
0.7486
10 double-annotated queries: 276 paired labels
Reviewer A
Reviewer B
0.4340
0.4726
0.7101
0.7436
0.6667
TABLE III: Binary relevance agreement with human labels.
@10
@100
@150
@300
@500
@1000
Model
R
P
F1
R
P
F1
R
P
F1
R
P
F1
R
P
F1
R
P
F1
Taxonomy-based Retrieval
.15
.36
.21
.15
.36
.21
.15
.36
.21
.15
.36
.21
.15
.36
.21
.15
.36
.21
Gemini embeddings
.09
.65
.16
.47
.52
.49
.62
.43
.51
.84
.33
.47
.90
.27
.41
.93
.13
.24
Voyage
.05
.36
.09
.27
.26
.27
.33
.23
.27
.46
.17
.25
.55
.13
.21
.67
.09
.15
e5-large-v2
.05
.33
.08
.19
.20
.20
.23
.17
.20
.32
.13
.18
.39
.09
.15
.50
.06
.11
granite
.05
.40
.09
.26
.26
.26
.32
.23
.26
.42
.17
.24
.50
.12
.20
.62
.08
.14
TABLE IV: Retrieval performance on the Beta Tester Queries. Recall (R), precision (P), and F1 at different retrieval depths.
@10
@100
@150
@300
@500
@1000
Model
R
P
F1
R
P
F1
R
P
F1
R
P
F1
R
P
F1
R
P
F1
Taxonomy-Based Retrieval
.24
.61
.35
.24
.61
.35
.24
.61
.35
.24
.61
.35
.24
.61
.35
.24
.61
.35
Gemini embeddings
.14
.81
.24
.53
.54
.53
.65
.48
.55
.84
.35
.49
.92
.24
.38
.96
.13
.23
Voyage
.09
.46
.15
.34
.29
.31
.41
.25
.31
.53
.19
.28
.63
.14
.24
.76
.09
.17
e5-large-v2
.09
.41
.15
.35
.30
.33
.33
.19
.24
.42
.11
.18
.49
.11
.17
.60
.07
.13
granite
.09
.44
.14
.30
.25
.28
.36
.22
.27
.46
.16
.24
.55
.12
.20
.67
.08
.14
TABLE V: Retrieval performance on the Synthetic Queries benchmark. R: recall; P: precision.
System
K=100
500
1,000
2,500
EL (unbudgeted)
45.1
45.1
45.1
45.1
Retrieval before filtering
67.1
87.7
93.1
96.7
Retrieval + Gemini
61.3
78.8
82.7
85.0
EL ∪ retrieval + Gemini
73.8
84.0
86.0
87.9
TABLE VI: Macro paper recall (%) on the review-derived benchmark.