What Can a Leaderboard Certify? Compositional Controllability for Fair Evaluation and Training of Biomedical Literature-Review Agents

Zhaowei Han; Xiang Zhang; Lingxiao Guan; Danqi Hu; Kai Liu; Kevin Chang; Jie Liu

cs.LGSep 26, 2026

What Can a Leaderboard Certify? Compositional Controllability for Fair Evaluation and Training of Biomedical Literature-Review Agents

Authors: Zhaowei Han, Xiang Zhang, Lingxiao Guan, Danqi Hu, Kai Liu, Kevin Chang, Jie Liu

Organizations: University of Michigan, Ann Arbor

Abstract

Leaderboards rank long-horizon agents by their final outputs. Yet a higher score alone does not establish whether two systems are comparable or which stage accounts for the difference. Unequal evidence, inputs, or budgets can affect scores, and statistical corrections do not remove this mismatch. We introduce compositional controllability to address these questions. A comparison window covers one stage, several stages, or the whole agent. Our central result bounds the gap between observed and controlled score differences using only nuisance outside the window. This yields an admissibility test applied before scores are inspected. Inadmissible comparisons are refused. For admissible pairs, an ordering is certified only when the score gap exceeds the combined sampling and nuisance radii; otherwise, it remains undecided. These decisions give each system a rank interval. We introduce BioLitBench, a benchmark of 2,042 biomedical articles represented as structured claim graphs. Among seven published pipelines, a conventional statistical analysis declares a winner in 14 of 21 pairwise comparisons. Yet the top-ranked system alone received the target review's bibliography. To isolate pipeline performance, our test requires matched inputs and a fixed backbone model. It refuses 11 of the 21 comparisons, including every comparison involving the top-ranked system. Seven of the 14 conventional conclusions fall within these refused pairs. The same comparison windows support stage-level training. We train SCRIBE on Qwen3.8-27B using rewards measured at each stage's exit. Under matched evidence, SCRIBE achieves a certified rank interval of [1,2], with certified advantages over all evaluated published pipelines and the evaluated Claude and OpenAI agents. Under same pool, SCRIBE matches the strongest published retriever and is certified above three published pipelines.

Figures & tables

Figure 1: Certified rank intervals. Each bar spans all positions a system can occupy in an ordering consistent with the certified pairwise relations; a single marker denotes a pinned position. Panel (a) is fixed input and panel (b) is same pool; positions are ranked within a panel and are not comparable across panels. Systems are sorted by composite score, shown on the right. Most systems admit an interval of possible ranks rather than a unique rank.

				Capability (fraction of human level)
Backbone	System	Composite	Rank	Synthesis	Reasoning	Planning	Writing	Form
Fixed input: every system receives the review’s reference list
Qwen3.8-27B	SCRIBE	0.606	$1$ – $2$	0.689	0.594	0.319	0.613	0.814
	SCRIBE (untrained)	0.583	$1$ – $3$	0.686	0.601	0.276	0.565	0.786
gpt-5.6-luna	SCRIBE-Luna	0.564	$2$ – $6$	0.700	0.611	0.177	0.568	0.767
	SurveyG	0.516	$4$ – $12$	0.568	0.541	0.272	0.509	0.689

Table 1: Capability scores under fixed input at the primary observation point. Rank denotes the certified rank interval shown in Figure 1 . Within each column, the highest value is bold and the second highest is underlined. Our systems are shaded.

Rule	Pairs decided	What it adds
R0 sorted table	$21$ / $21$	any difference counts
R1 paired $t$ -test	$17$ / $21$	sampling uncertainty
R2 cluster-robust $t$	$16$ / $21$	task clustering
R3 $+$ Bonferroni	$\mathbf{14}$ / $21$	multiplicity; a careful conventional analysis
R4 $+$ admissibility test	$7$ / $10$	$11$ pairs refused before any score is read
R5 $+$ nuisance radius	$\mathbf{7}$ / $10$	certified; the radius is zero, entries are shared

Table 2: What each analysis concludes about the

21

pairs among the seven published pipelines at the primary observation point. R4 and R5 are counted over the

10

admissible pairs.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Articles	$2{,}042$	Fields / subfields / topics	$6$ / $48$ / $230$
Claims	$246{,}102$	Median claims per article	$108$
Typed relations	$832{,}487$	Hard negatives	$103{,}242$
Unique cited papers	$224{,}140$	Review publication years	$2016$ – $2025$
Training / held-out articles	$1{,}842$ / $200$	Evaluation tasks	$50$

Appendix

Table 3: BioLitBench at a glance. Statistics for the

2{,}042

usable graphs.

				Capability (fraction of human level)
Backbone	System	Composite	Rank	Retrieval	Synthesis	Reasoning	Planning	Writing	Form
Same pool: every system searches one frozen corpus
Qwen3.8-27B	SCRIBE	0.487	$1$ – $7$	0.335	0.523	0.545	0.250	0.446	0.826
	SCRIBE (untrained)	0.464	$1$ – $9$	0.335	0.512	0.538	0.232	0.404	0.766
gpt-5.6-luna	SCRIBE-Luna	0.479	$1$ – $6$	0.335	0.577	0.550	0.185	0.386	0.842
	SurveyGen-I	0.460	$1$ – $9$	0.336	0.535	0.485	0.181	0.371	0.850

Appendix

Table 4: Capability scores under same pool at the primary observation point. Conventions as in Table 1 .

				Capability (fraction of human level)
Backbone	System	Composite	Rank	Synthesis	Reasoning	Planning	Writing	Form
(a) Fixed input: every system receives the review’s reference list
Qwen3.8-27B	SCRIBE	0.609	$1$ – $4$	0.695	0.596	0.319	0.619	0.816
	SCRIBE (untrained)	0.585	$1$ – $6$	0.690	0.603	0.276	0.569	0.787
gpt-5.6-luna	SCRIBE-Luna	0.573	$2$ – $8$	0.710	0.622	0.177	0.592	0.763
	AutoSurvey	0.567	$1$ – $8$	0.958	0.769	0.135	0.370	0.605

Appendix

Table 5: The leaderboard at full length (secondary, descriptive). Conventions as in Tables 1 and 4 ; the footnotes of Table 1 apply to panel (a).

	same driver			pipeline mean
Axis	no evidence	shuffled	half	no evidence	shuffled
synthesis	$-$ 0.044	$-$ 0.036	$-$ 0.019	$-$ 0.053	$-$ 0.045
	[ $-$ .087, $-$ .002]	[ $-$ .075, $+$ .003]	[ $-$ .056, $+$ .018]	[ $-$ .088, $-$ .018]	[ $-$ .082, $-$ .008]
reasoning	$-$ 0.061	$-$ 0.035	$-$ 0.027	$-$ 0.081	$-$ 0.055
	[ $-$ .110, $-$ .013]	[ $-$ .100, $+$ .029]	[ $-$ .080, $+$ .027]	[ $-$ .119, $-$ .042]	[ $-$ .109, $-$ .000]
writing	$+$ 0.071	$+$ 0.011	$-$ 0.001	$+$ 0.094	$+$ 0.034

Appendix

Table 6: Evidence ablation at the controlled writing window (

20

tasks, matched length, fraction-of-human-level scale). Each entry is the paired difference between the writer given the full evidence and a control; the 95% interval is below it. Same driver : AutoSurvey’s own writer against its controls. Pipeline mean : the six pipelines’ mean against the same controls. A negative value means the control scores higher.

fixed input (reference bundle; no retrieval axis)
System	synthesis	reasoning	planning	writing	form
SCRIBE (untrained)	0.686 [1,3]	0.601 [1,4]	0.276 [1,4]	0.565 [1,4]	0.783 [1,3]
AutoSurvey	0.638 [1,5]	0.533 [1,6]	0.135 [3,6]	0.483 [1,6]	0.687 [2,6]
SurveyGen-I	0.591 [2,5]	0.510 [2,6]	0.182 [3,6]	0.449 [2,6]	0.746 [1,6]
SurveyG	0.568 [2,6]	0.541 [1,6]	0.272 [1,3]	0.509 [1,5]	0.689 [2,6]
LiRA	0.539 [1,6]	0.495 [1,6]	0.254 [1,6]	0.442 [1,6]	0.716 [1,6]

Appendix

Table 7: Capability scores as a fraction of human level under fixed input , where every arm receives the task’s reference bundle, with certified rank intervals in brackets — the positions consistent with every certified relation. LiRA receives the review’s bibliography in its own input format rather than through the shared allowlist, so this campaign does not pair it with the other arms, and its whole row is unpinned. The same-pool scores are in Table 8 . Scores come from this campaign’s own per-axis scorer; SCRIBE (untrained)’s form score therefore reads

0.783

here and

0.786

on the merged board of Table 1 .

System	synthesis	reasoning	planning	retrieval	form
SurveyGen-I	0.535 [1,6]	0.485 [1,7]	0.181 [3,6]	0.336 [1,1]	0.850 [1,4]
AutoSurvey	0.540 [1,6]	0.444 [1,7]	0.117 [6,7]	0.238 [2,3]	0.766 [1,5]
SurveyForge	0.529 [1,6]	0.439 [1,7]	0.255 [1,5]	0.220 [2,4]	0.760 [1,5]
SCRIBE (untrained, earlier harness)	0.496 [1,6]	0.480 [1,7]	0.257 [1,6]	0.156 [3,6]	0.786 [1,6]
SurveyG	0.487 [1,6]	0.486 [1,7]	0.284 [1,4]	0.155 [4,5]	0.686 [4,7]
DR-Tulu	0.369 [6,7]	0.419 [1,7]	0.251 [1,6]	0.084 [5,7]	0.663 [3,7]

Appendix

Table 8: Capability scores under same pool , where every system retrieves for itself from one frozen corpus and the retrieval axis becomes observable. Conventions as in Table 7 . Writing, which certifies no pair, is not shown. The SCRIBE row uses its earlier retrieval harness; the final harness, used in the leaderboard, reaches retrieval

0.335

(Table 4 ).

Tier	System	Composite	matched	full length
			Certified rank
self-retrieving	SurveyGen-I	0.460	$\mathbf{1}$	$1$ – $3$
(certified: 7 / 7 of 10)	SurveyForge	0.431	$2$ – $4$	$1$ – $3$
	SurveyG	0.425	$2$ – $4$	$\mathbf{4}$
	AutoSurvey	0.419	$2$ – $4$	$1$ – $3$
	LLM $\times$ MR	0.379	$\mathbf{5}$	$\mathbf{5}$

Appendix

Table 9: The published pipelines alone (scaffold family,

10

pairs per tier; compare Tables 1 and 4 , whose larger families give wider intervals). Display rank orders point estimates and is not a claim; certified rank is the interval of positions consistent with the certified relations over every linear extension of their transitive closure, so a single value is a position the data pin down. In the reference-fed tier, each pipeline runs its reference-fed entry. Composite is the equal-weight mean over the tier’s capability axes on the fraction-of-human-level scale, at matched length. The last column gives the certified rank at full length.

Component	Tested on	Key result	Verdict
On the published systems
Attribution bound, Thm. 1 ( F.1 )	two entries per pipeline, $118$ runs	$L_{M}$ transfers to held-out pipelines (coverage $0.79$ – $1.00$ )	$\checkmark$ holds
Bound at a stage window ( F.2 )	one sealed writing input, $20$ tasks; fit on $5$ pipelines, check on $15$ pairs of $6$	$420/420$ cells hold at matched length; $44\%$ vacuous	$\checkmark$ holds; informative on writing
Admissibility, Def. 1 ( F.3 )	five LiRA pairs, measured entry mismatch	radius $0.70$ – $0.73$ against tolerance $0.21$ ; refused on $15/15$ readouts	$\checkmark$ refuses
Non-expansive windows, Cor. 1 ( F.1 , F.2 )	both designs above	$0$ runs expand at matched length; $62/100$ at full length	$\checkmark$ at matched length
Certification rule, Eq. 4 ( F.5 )	sign-flip null; leave one subfield out	false certification $0.035$ / $0.062$ ; no flips	$\checkmark$ near nominal

Appendix

Table 10: Verification summary. Each part of the theory, what it is tested on, and the key result. The subsection with the full test is given in parentheses.

$L_{M}$ design	$L_{M}$	$\xi$	note
ratio $q_{95}$	0.890	0.000	selected design
ratio sup	0.967	0.000	supremum form
affine	0.803	0.085	envelope, slope $+$ slack
per system (ratio $q_{95}$ ): AS $0.911$ ( $n{=}19$ ), SGI $0.910$ ( $19$ ), LMR $0.880$ ( $31$ ), SG $0.863$ ( $49$ )

Appendix

Table 11: The module-sensitivity constant

L_{M}

, measured on the panel. Fitted on the paired gap over

118

paired runs from four pipelines,

x\in[0.74,1.00]

(a) leave-one-system-out
held out	$n$ test	$L_{M}$ (train)	$L_{M}$ (own)	held-out coverage
AutoSurvey	19	0.881	0.911	0.789
SurveyGen-I	19	0.882	0.910	0.842
LLM $\times$ MR	31	0.895	0.880	0.968
SurveyG	49	0.908	0.863	1.000
(b) falsifiability locally expansive runs ( $y>x$ ): $0/118$ max ratio $0.967$ instrument writing stage: $1.74$

Appendix

Table 12: Falsification tests for Assumption 1 . (a) A constant fitted on the other three pipelines, evaluated on the held-out one — the transferability the panel-level radius actually assumes. (b) The non-expansive claim is falsifiable and is not falsified; the instrument’s writing stage is a positive control.

	matched length	full length
fitted $L_{M}$ ( $95$ th pct. / sup)	$0.906$ / $0.940$	$1.042$ / $1.188$
runs with $y>x$	$0/100$	$62/100$
leave-one-pipeline-out coverage (mean / min)	$0.93$ / $0.80$	$0.94$ / $0.90$
bound holds (all cells)	$420/420$	$419/420$
vacuous cells ( $\epsilon_{\mathrm{outside}}\geq 1$ )	$43.6\%$	$46.0\%$
median $\epsilon_{\mathrm{outside}}$ / median $\|\Delta^{\mathrm{obs}}-\Delta^{\star}\|$	$0.955$ / $0.060$	$0.978$ / $0.066$

Appendix

Table 13: The attribution bound at a controlled writing window (

5

self-retrieving pipelines

\times

20

tasks for the fit;

420

pair-by-readout cells for the score-level check).

Pair	$\epsilon_{\mathrm{in},P}$	median $\epsilon_{\mathrm{outside}}$	median $\epsilon_{\mathrm{fair}}$	readouts refused
AutoSurvey – LiRA	$0.975$	$0.725$	$0.208$	$15/15$
SurveyForge – LiRA	$0.968$	$0.720$	$0.208$	$15/15$
SurveyG – LiRA	$0.972$	$0.723$	$0.208$	$15/15$
SurveyGen-I – LiRA	$0.942$	$0.700$	$0.208$	$15/15$
LLM $\times$ MR – LiRA	$0.979$	$0.728$	$0.208$	$15/15$

Appendix

Table 14: Refusals computed from Eq. 3 for LiRA against each self-retrieving pipeline.

Stage $m$	observed $\mathsf{W}$	bound (affine)	bound (ratio)	holds	non-vacuous	tightness
1 retrieval	0.605	0.744	0.744	95 %	yes	0.79
2 synthesis	0.347	0.844	0.891	100 %	yes	0.38
3 organization	0.261	1.312	0.981	100 %	yes	0.26
4 writing	0.685	1.465	3.378	100 %	no	0.21

Appendix

Table 15: Theorem 2 on synthetic four-stage chains built from the human graphs. Observed chain discrepancy against the composed bound (affine at

q_{95}

, and ratio form); fraction of chains on which the bound holds; tightness

=

observed/bound.

Table	View	pairs	certified	family-wise	$c^{\star}$	Jaccard	flips
fixed input	matched length	$45$	$21$	$0.062$	$1.027$ (drops $1$ )	$0.952$ / $0.870$	$0$
same pool	matched length	$45$	$20$	$0.035$	$0.966$ (drops $0$ )	$0.918$ / $0.727$	$0$
fixed input	full length	$45$	$20$	$0.051$	$1.003$ (drops $0$ )	$0.965$ / $0.905$	$0$
same pool	full length	$45$	$16$	$0.041$	$0.980$ (drops $0$ )	$0.877$ / $0.750$	$0$

Appendix

Table 16: Calibration and stability of the leaderboard’s decision rule. Family-wise rates are over

2{,}000

sign-flip draws (nominal

0.05

c^{\star}

: radius scale giving the nominal rate. Jaccard: overlap of the certified set when one of the

23

subfields is left out (mean / minimum).

		retrieval		writing
Pair	$n$	$\mathrm{Ctrl}_{M}$	sup	$\mathrm{Ctrl}_{M}$ (lo–hi)	sup
AutoSurvey – LiRA	50	0.070	0.161	0.504–0.506	0.606
LiRA – SurveyG	49	0.092	0.293	0.509–0.511	0.749
AutoSurvey – SurveyG	49	0.127	0.322	0.392–0.505	0.901
AutoSurvey – LLM $\times$ MR	50	0.424	0.472	0.800–0.940	1.116
LLM $\times$ MR – SurveyG	49	0.429	0.485	0.757–0.941	1.232

Appendix

Table 17:

\mathrm{Ctrl}_{M}

for the reference-fed pairs at

\lambda_{\mathrm{in}}=0.5

(Eq. 19 ); mean over the tasks both ran, with the essential-supremum form. Pairs with the reference-fed SurveyGen-I are omitted, because they were not recomputed after its re-run. Entry lo/hi bracket the entry term where only each system’s distance to the reference is stored. At retrieval the entry is the shared task specification, so the entry term is

0

and

\mathrm{Ctrl}_{M}

measures behaviour inside the window alone.

Property	Requirement	Measured
contraction	ratio $\leq\gamma(1-\beta)=0.45$	$0.449$ , $0.425$ , $0.428$ (random inputs); $0.450$ at convergence	✓
existence / uniqueness	same fixed point from $d_{0}\equiv 0$ and $d_{0}\equiv 1$	18 / 18 iterations, sup gap $5.7\times 10^{-7}$	✓
pseudometric	symmetry, triangle inequality	symmetry gap $0$ ; $0$ / $20{,}000$ triangle violations	✓
zero self-distance	$d^{\dagger}(S,S)=0$	$0$ for all 12 systems	✓

Appendix

Table 18: Theorem 4 and its corollaries on the empirical kernels of the twelve logged configurations.

AI Agent Evaluation

AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.

See all 521 papers

What Does an LLM-Agent Leaderboard Rank Actually Compare?
Sep 7, 2026Wei-Jung HuangLLM Agent Evaluation AI Agent Benchmarks
Independent Researcher United States
Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboard
Sep 30, 2026Michael Hardy, Ruhana Azam, Anka Reuel +2AI Agent Evaluation Agent Evaluation
Stanford University · University of Illinois Urbana-Champaign · Senior Author
Rank Confidence Sequences:Anytime-valid Leaderboards
Sep 26, 2026Hamed Khosravi, Xiaoming HuoAnytime-Valid Inference Benchmark Validity
H. Milton Stewart School of Industrial and Systems Engineering Georgia Institute of Technology, Atlanta, GA 30332, USA

Sep 7, 2026J/K move · Enter open · S save

What Does an LLM-Agent Leaderboard Rank Actually Compare?

Wei-Jung Huang

Independent Researcher United States

LLM Agent Evaluation AI Agent Benchmarks Automated Evaluation

Open paper AskPDF arXiv

An LLM-agent leaderboard invites a familiar inference: an agent ranked above another is the better agent. Public evaluation logs may not support that conclusion when systems differ in task mixture, label source, release detail, or cost rule. We study what leaderboard scores estimate and when they justify pairwise superiority conclusions. Our estimand-aware pairwise procedure states the comparison target and measurement source, checks common support, and evaluates the supported difference using a stated uncertainty rule and practical margin. Controlled checks evaluate the decision labels under known finite-sample conditions and show why uncertainty must be included when judging sensitivity to target reweighting. Across SWE-bench, AgentRewardBench, and tau2-bench, close rank differences are often unresolved; proxy labels and utility rules can also change which system is selected. DataAgentBench and Open Agent show what remains estimable from coarser public records. A leaderboard score summarizes a released evaluation, whereas a fine-grained superiority claim additionally depends on the estimand and uncertainty rule used to interpret the difference.

Abstract

AI Agent Evaluation

Explore similar work

What Does an LLM-Agent Leaderboard Rank Actually Compare?

Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboard

Rank Confidence Sequences:Anytime-valid Leaderboards