Does Recursive Knowledge Construction Help Agents Verify Consensus Claims in Science?
Authors: Lun Ai, Yu-Heng Wu, Jonathan Schaul, Leonie Küchenhoff, Denes Turei, Francesco Carli, Fabio Petroni, Aurelien Dugourd, Julio Saez-Rodriguez
Organizations: European Bioinformatics Institute (EMBL-EBI), European Molecular Biology Laboratory, Hinxton, UK. · Institute for Computational Biomedicine, Heidelberg University, Faculty of Medicine, and Heidelberg University Hospital, Heidelberg, Germany. · EMBL Rome, European Molecular Biology Laboratory, Monterotondo, Italy.
Recent scientific agents build knowledge recursively by retrieving evidence, constructing an intermediate state, and judging whether that state contains sufficient information. These agents are evaluated end-to-end, usually without ground truth, so the effect of each design choice is unknown. Here, we examined two design axes: (1) what the recursion builds (the state of evidence) and (2) what controls it (the judge that decides when to stop). To measure them, we defined consensus scientific claim verification and measured these effects on this task, where a system must establish a claim's consensus stance from the open literature. We built a new consensus claim dataset, ProClaim-eval, including 718 claims with SUPPORT and REFUTE gold labels from five expert-curated databases spanning molecular biology, nuclear physics, and astronomy. Our instrument, ProClaim, persists an evidence state outside the model's context, so each axis can be studied with the other held fixed. Against verification systems and general-purpose agent harnesses, ProClaim ranks among the top three, with a significant advantage over the harnesses. We found that the evidence state improves performance over in-context accumulation, and it also matters what is stored in it. In addition, a trained judge that reads the evidence state performs better than fixed-round stopping. A judge trained separately and never exposed to ProClaim-eval achieves the highest agreement with the gold labels on three of five subsets, where it matches the off-the-shelf Claude Sonnet4.6 judge and outperforms Claude Haiku4.5. Our results support recursively maintaining a structured evidence state outside the model's context as an effective design for verifying claims in scientific agents.
Figures & tables
Figure 1: Document-guided vs. consensus scientific claim verification.
Figure 2: Recursive knowledge construction for claim verification. We examine two design choices in Section 4 : (1) an evidence state that the next retrieval is conditioned on, (2) a judge that takes the state as input and decides either to continue or terminate the construction, after which the verdict is given from the final state.
Table 1: Claim diversity in the ProClaim-eval benchmark. We show one example claim per subset, with the subset’s claim count and support / refute split. The five subsets instantiate five distinct single-sentence templates over three scientific domains with a total of 718 claims.
Figure 3: Verdict prediction-label agreement (outlined segments) across three verification settings on the SIGNOR and ConnectomeDB subsets of ProClaim-eval (406 claims), using Claude Sonnet 4.6. Both document-guided and naive retrieval verification do not produce consensus reliably and overproduce refute predictions due to insufficient or noisy information, a limitation that ProClaim addresses. This comparison with ProClaim highlights the benefit of recursive knowledge construction, driven by adaptive retrieval and evidence organisation before deciding the final verdict.
SIGNOR
ConnectomeDB
ExTRI
NUBASE
NASA
Method
Agr. ↑
Abst.
Agr. ↑
Abst.
Agr. ↑
Abst.
Agr. ↑
Abst.
Agr. ↑
Abst.
Random
0.32
0.35
0.34
0.34
0.32
0.34
0.34
0.34
0.31
0.35
LLM Only
Claude Sonnet 4.6
0.63 +0.17
0.26
0.53 +0.26
0.25
0.21 +0.51
0.66
0.05 +0.47
0.95
0.07 +0.59
0.89
Gemini 2.5 Flash
0.59 +0.21
0.26
0.57 +0.21
0.21
0.23 +0.49
0.56
0.31 +0.21
0.07
0.46 +0.20
0.03
Static Retrieval, S2 Top 5
Table 2: Agreement (Agr.) and abstention rate (Abst.) per ProClaim-eval subset. The shaded figure beside each agreement is ProClaim’s agreement minus that row’s, on the same subset; green where ProClaim is ahead, red where it is behind. The highest agreement per subset is underlined.
SIGNOR (40)
ConnectomeDB (40)
ExTRI (40)
NUBASE (40)
NASA (40)
Evidence & Methods
Agr. ↑
Abst.
Agr. ↑
Abst.
Agr. ↑
Abst.
Agr. ↑
Abst.
Agr. ↑
Abst.
No retrieval
0.63
0.30
0.70
0.23
0.28
0.65
0.10
0.90
0.08
0.88
Abstracts
0.78
0.13
0.83
0.10
0.75
0.25
0.28
0.45
0.63
0.23
Full text
0.80
0.08
0.83
0.10
0.83
0.10
0.38
0.28
0.65
0.18
Facts
0.80
0.10
0.93
0.08
0.88
0.05
0.50
0.15
0.73
0.08
Facts + abstracts
0.80
0.08
0.93
0.05
0.83
0.13
0.50
0.15
0.75
0.03
Table 3: Effect of the evidence state. The same LLM (Claude Sonnet 4.6) reads the different information retrieved in the evidence state of ProClaim, against no retrieval. Every row is scored on the same randomly sampled 40 claims per subset. We show the agreement (Agr.) and abstention rate (Abst.) per subset, and the highest agreement per subset is underlined.
SIGNOR (40)
ConnectomeDB (40)
ExTRI (40)
NUBASE (40)
NASA (40)
Judge
Agr. ↑
Abst.
Iter.
Agr. ↑
Abst.
Iter.
Agr. ↑
Abst.
Iter.
Agr. ↑
Abst.
Iter.
Agr. ↑
Abst.
Iter.
Stopping rules that ignore the evidence state
Fixed rounds
0.70
0.15
3.45
0.73
0.25
3.25
0.83
0.15
3.20
0.33
0.18
2.53
0.68
0.20
3.20
Always continue
0.75
0.13
2.68
0.75
0.15
3.15
0.88
0.08
3.40
0.55
0.15
2.65
0.50
0.33
2.60
Random score
0.70
0.20
3.13
0.90
0.08
3.33
0.88
0.08
3.15
0.53
0.23
2.10
0.48
0.28
2.73
Off-the-shelf LLM judges
Table 4: Effect of the sufficiency judge with a fixed evidence state design, on the same 40 randomly sampled claims per subset. Agreement (Agr.), abstention rate (Abst.) and mean judge calls at stop (Iter.) for each stopping rule, the highest agreement per subset is underlined.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
SIGNOR
ConnectomeDB
ExTRI
NUBASE
NASA
Method
F1 ↑
BAcc ↑
F1 ↑
BAcc ↑
F1 ↑
BAcc ↑
F1 ↑
BAcc ↑
F1 ↑
BAcc ↑
Random
0.38
0.32
0.40
0.33
0.38
0.32
0.32
0.34
0.37
0.31
LLM Only
Claude Sonnet 4.6
0.68 +0.17
0.59
0.60 +0.19
0.56
0.31 +0.47
0.25
0.05 +0.49
0.03
0.11 +0.59
0.06
Gemini 2.5 Flash
0.66 +0.19
0.57
0.64 +0.16
0.58
0.31 +0.47
0.28
0.28 +0.25
0.39
0.32 +0.38
0.43
Static Retrieval, S2 Top 5
Appendix
Table 5: Macro-F1 (F1) and balanced accuracy (BAcc) per ProClaim-eval subset, for the same run behind each cell of Table 2 . Random is the mean over its 10 draws. The shaded figure beside each macro-F1 is ProClaim’s macro-F1 minus that row’s, on the same subset; green where ProClaim is ahead, red where it is behind. The highest value per column is underlined.
Statement: A vs. B
Subsets
n
b
c
Δ
p
pHolm
(1) Trained judge vs. Haiku 4.5 judge
3 × 40
120
13
2
+0.092
0.0074
0.022
Trained judge vs. Sonnet 4.6 judge
120
12
8
+0.033
0.50 ∗
(2) Trained judge vs. Fixed rounds
5 × 40
200
35
11
+0.120
< 0.001
0.0021
(3) Facts vs. Abstracts
5 × 40
200
30
7
+0.115
< 0.001
0.024
Facts vs. Full text
200
24
10
+0.070
0.024
(4) ProClaim vs. ReAct + subclaims − state
5 × 40
200
47
12
+0.175
< 0.001
0.0012
Appendix
Table 6: Paired exact McNemar tests of the finding statements, pooled over subsets. n : claims; b : claims A gets right and B gets wrong; c : claims B gets right and A gets wrong; Δ : A’s pooled agreement minus B’s; p : exact two-sided McNemar p -value; pHolm : the statement’s p -value (the larger of its tests), Holm-adjusted across the five statements.
SIGNOR
ConnectomeDB
ExTRI
NUBASE
NASA
Method
In
Out
In
Out
In
Out
In
Out
In
Out
LLM Only
Claude Sonnet 4.6
0.2
0.1
0.2
0.1
0.2
0.1
0.2
0.1
0.2
0.1
Gemini 2.5 Flash
0.2
0.1
0.2
0.1
0.2
0.1
0.2
0.1
0.2
0.1
Static Retrieval, S2 Top 5
Claude Sonnet 4.6
2.1
0.1
2.4
0.2
2.3
0.2
1.2
0.1
1.1
0.1
Appendix
Table 7: Mean input (In) and output (Out) tokens per claim, in thousands, for the run behind each cell of Table 2 . Input tokens include cached input. ProClaim counts its main agent only.
support
refute
Agr. ↑
Abst.
Macro F1 ↑
Iter.
P
R
F1
P
R
F1
Trained judge
0.80
0.12
0.85
3.21
0.82
0.84
0.83
0.98
0.78
0.87
Qwen3.5 9B
0.71
0.18
0.78
2.63
0.78
0.82
0.79
0.95
0.65
0.77
Claude Haiku 4.5
0.76
0.11
0.81
2.48
0.74
0.92
0.82
0.98
0.67
0.79
Appendix
Table 8: Judge ablation on SIGNOR (101 claims), with the evidence state, backbone, retrieval tools and stop threshold held fixed. Agreement (Agr., abstention counted as a disagreement), abstention rate (Abst.), macro F1 over the two gold labels, mean judge calls at stop (Iter.), and precision (P), recall (R) and F1 per gold label.
Figure 4: Effect of the judge on sufficiency scoring and ProClaim termination. (a) The trained judge is conservative, assigning low scores for most claims and driving the agent to collect more evidence across multiple rounds. Qwen 3.5 9B exhibits a bimodal distribution, with a dominant mode near 1.0 and a secondary mode near 0.0. Haiku assigns uniformly high scores with almost no mass near zero. (b) The trained judge’s conservative scoring terminates ProClaim at three and four iterations, while Qwen 3.5 9B’s bimodal pattern produces peaks at both one and four iterations, and Haiku distributes more evenly.