What Can a Leaderboard Certify? Compositional Controllability for Fair Evaluation and Training of Biomedical Literature-Review Agents

Zhaowei Han; Xiang Zhang; Lingxiao Guan; Danqi Hu; Kai Liu; Kevin Chang; Jie Liu

cs.LGSep 26, 2026

What Can a Leaderboard Certify? Compositional Controllability for Fair Evaluation and Training of Biomedical Literature-Review Agents

Authors: Zhaowei Han, Xiang Zhang, Lingxiao Guan, Danqi Hu, Kai Liu, Kevin Chang, Jie Liu

Organizations: University of Michigan, Ann Arbor

Abstract

Leaderboards rank long-horizon agents by their final outputs. Yet a higher score alone does not establish whether two systems are comparable or which stage accounts for the difference. Unequal evidence, inputs, or budgets can affect scores, and statistical corrections do not remove this mismatch. We introduce compositional controllability to address these questions. A comparison window covers one stage, several stages, or the whole agent. Our central result bounds the gap between observed and controlled score differences using only nuisance outside the window. This yields an admissibility test applied before scores are inspected. Inadmissible comparisons are refused. For admissible pairs, an ordering is certified only when the score gap exceeds the combined sampling and nuisance radii; otherwise, it remains undecided. These decisions give each system a rank interval. We introduce BioLitBench, a benchmark of 2,042 biomedical articles represented as structured claim graphs. Among seven published pipelines, a conventional statistical analysis declares a winner in 14 of 21 pairwise comparisons. Yet the top-ranked system alone received the target review's bibliography. To isolate pipeline performance, our test requires matched inputs and a fixed backbone model. It refuses 11 of the 21 comparisons, including every comparison involving the top-ranked system. Seven of the 14 conventional conclusions fall within these refused pairs. The same comparison windows support stage-level training. We train SCRIBE on Qwen3.8-27B using rewards measured at each stage's exit. Under matched evidence, SCRIBE achieves a certified rank interval of [1,2], with certified advantages over all evaluated published pipelines and the evaluated Claude and OpenAI agents. Under same pool, SCRIBE matches the strongest published retriever and is certified above three published pipelines.

Figures & tables

Figure 1: Certified rank intervals. Each bar spans all positions a system can occupy in an ordering consistent with the certified pairwise relations; a single marker denotes a pinned position. Panel (a) is fixed input and panel (b) is same pool; positions are ranked within a panel and are not comparable across panels. Systems are sorted by composite score, shown on the right. Most systems admit an interval of possible ranks rather than a unique rank.

				Capability (fraction of human level)
Backbone	System	Composite	Rank	Synthesis	Reasoning	Planning	Writing	Form
Fixed input: every system receives the review’s reference list
Qwen3.8-27B	SCRIBE	0.606	$1$ – $2$	0.689	0.594	0.319	0.613	0.814
	SCRIBE (untrained)	0.583	$1$ – $3$	0.686	0.601	0.276	0.565	0.786
gpt-5.6-luna	SCRIBE-Luna	0.564	$2$ – $6$	0.700	0.611	0.177	0.568	0.767
	SurveyG	0.516	$4$ – $12$	0.568	0.541	0.272	0.509	0.689

Table 1: Capability scores under fixed input at the primary observation point. Rank denotes the certified rank interval shown in Figure 1 . Within each column, the highest value is bold and the second highest is underlined. Our systems are shaded.

Rule	Pairs decided	What it adds
R0 sorted table	$21$ / $21$	any difference counts
R1 paired $t$ -test	$17$ / $21$	sampling uncertainty
R2 cluster-robust $t$	$16$ / $21$	task clustering
R3 $+$ Bonferroni	$\mathbf{14}$ / $21$	multiplicity; a careful conventional analysis
R4 $+$ admissibility test	$7$ / $10$	$11$ pairs refused before any score is read
R5 $+$ nuisance radius	$\mathbf{7}$ / $10$	certified; the radius is zero, entries are shared

Table 2: What each analysis concludes about the

21

pairs among the seven published pipelines at the primary observation point. R4 and R5 are counted over the

10

admissible pairs.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Articles	$2{,}042$	Fields / subfields / topics	$6$ / $48$ / $230$
Claims	$246{,}102$	Median claims per article	$108$
Typed relations	$832{,}487$	Hard negatives	$103{,}242$
Unique cited papers	$224{,}140$	Review publication years	$2016$ – $2025$
Training / held-out articles	$1{,}842$ / $200$	Evaluation tasks	$50$

Appendix

Table 3: BioLitBench at a glance. Statistics for the

2{,}042

usable graphs.

				Capability (fraction of human level)
Backbone	System	Composite	Rank	Retrieval	Synthesis	Reasoning	Planning	Writing	Form
Same pool: every system searches one frozen corpus
Qwen3.8-27B	SCRIBE	0.487	$1$ – $7$	0.335	0.523	0.545	0.250	0.446	0.826
	SCRIBE (untrained)	0.464	$1$ – $9$	0.335	0.512	0.538	0.232	0.404	0.766
gpt-5.6-luna	SCRIBE-Luna	0.479	$1$ – $6$	0.335	0.577	0.550	0.185	0.386	0.842
	SurveyGen-I	0.460	$1$ – $9$	0.336	0.535	0.485	0.181	0.371	0.850

Appendix

Table 4: Capability scores under same pool at the primary observation point. Conventions as in Table 1 .

				Capability (fraction of human level)
Backbone	System	Composite	Rank	Synthesis	Reasoning	Planning	Writing	Form
(a) Fixed input: every system receives the review’s reference list
Qwen3.8-27B	SCRIBE	0.609	$1$ – $4$	0.695	0.596	0.319	0.619	0.816
	SCRIBE (untrained)	0.585	$1$ – $6$	0.690	0.603	0.276	0.569	0.787
gpt-5.6-luna	SCRIBE-Luna	0.573	$2$ – $8$	0.710	0.622	0.177	0.592	0.763
	AutoSurvey	0.567	$1$ – $8$	0.958	0.769	0.135	0.370	0.605

Appendix

Table 5: The leaderboard at full length (secondary, descriptive). Conventions as in Tables 1 and 4 ; the footnotes of Table 1 apply to panel (a).

	same driver			pipeline mean
Axis	no evidence	shuffled	half	no evidence	shuffled
synthesis	$-$ 0.044	$-$ 0.036	$-$ 0.019	$-$ 0.053	$-$ 0.045
	[ $-$ .087, $-$ .002]	[ $-$ .075, $+$ .003]	[ $-$ .056, $+$ .018]	[ $-$ .088, $-$ .018]	[ $-$ .082, $-$ .008]
reasoning	$-$ 0.061	$-$ 0.035	$-$ 0.027	$-$ 0.081	$-$ 0.055
	[ $-$ .110, $-$ .013]	[ $-$ .100, $+$ .029]	[ $-$ .080, $+$ .027]	[ $-$ .119, $-$ .042]	[ $-$ .109, $-$ .000]
writing	$+$ 0.071	$+$ 0.011	$-$ 0.001	$+$ 0.094	$+$ 0.034

Appendix

Table 6: Evidence ablation at the controlled writing window (

20

tasks, matched length, fraction-of-human-level scale). Each entry is the paired difference between the writer given the full evidence and a control; the 95% interval is below it. Same driver : AutoSurvey’s own writer against its controls. Pipeline mean : the six pipelines’ mean against the same controls. A negative value means the control scores higher.

fixed input (reference bundle; no retrieval axis)
System	synthesis	reasoning	planning	writing	form
SCRIBE (untrained)	0.686 [1,3]	0.601 [1,4]	0.276 [1,4]	0.565 [1,4]	0.783 [1,3]
AutoSurvey	0.638 [1,5]	0.533 [1,6]	0.135 [3,6]	0.483 [1,6]	0.687 [2,6]
SurveyGen-I	0.591 [2,5]	0.510 [2,6]	0.182 [3,6]	0.449 [2,6]	0.746 [1,6]
SurveyG	0.568 [2,6]	0.541 [1,6]	0.272 [1,3]	0.509 [1,5]	0.689 [2,6]
LiRA	0.539 [1,6]	0.495 [1,6]	0.254 [1,6]	0.442 [1,6]	0.716 [1,6]

Appendix

Table 7: Capability scores as a fraction of human level under fixed input , where every arm receives the task’s reference bundle, with certified rank intervals in brackets — the positions consistent with every certified relation. LiRA receives the review’s bibliography in its own input format rather than through the shared allowlist, so this campaign does not pair it with the other arms, and its whole row is unpinned. The same-pool scores are in Table 8 . Scores come from this campaign’s own per-axis scorer; SCRIBE (untrained)’s form score therefore reads

0.783

here and

0.786

on the merged board of Table 1 .

System	synthesis	reasoning	planning	retrieval	form
SurveyGen-I	0.535 [1,6]	0.485 [1,7]	0.181 [3,6]	0.336 [1,1]	0.850 [1,4]
AutoSurvey	0.540 [1,6]	0.444 [1,7]	0.117 [6,7]	0.238 [2,3]	0.766 [1,5]
SurveyForge	0.529 [1,6]	0.439 [1,7]	0.255 [1,5]	0.220 [2,4]	0.760 [1,5]
SCRIBE (untrained, earlier harness)	0.496 [1,6]	0.480 [1,7]	0.257 [1,6]	0.156 [3,6]	0.786 [1,6]
SurveyG	0.487 [1,6]	0.486 [1,7]	0.284 [1,4]	0.155 [4,5]	0.686 [4,7]
DR-Tulu	0.369 [6,7]	0.419 [1,7]	0.251 [1,6]	0.084 [5,7]	0.663 [3,7]

Appendix

Table 8: Capability scores under same pool , where every system retrieves for itself from one frozen corpus and the retrieval axis becomes observable. Conventions as in Table 7 . Writing, which certifies no pair, is not shown. The SCRIBE row uses its earlier retrieval harness; the final harness, used in the leaderboard, reaches retrieval

0.335

(Table 4 ).

Tier	System	Composite	matched	full length
			Certified rank
self-retrieving	SurveyGen-I	0.460	$\mathbf{1}$	$1$ – $3$
(certified: 7 / 7 of 10)	SurveyForge	0.431	$2$ – $4$	$1$ – $3$
	SurveyG	0.425	$2$ – $4$	$\mathbf{4}$
	AutoSurvey	0.419	$2$ – $4$	$1$ – $3$
	LLM $\times$ MR	0.379	$\mathbf{5}$	$\mathbf{5}$

Appendix

Table 9: The published pipelines alone (scaffold family,

10

pairs per tier; compare Tables 1 and 4 , whose larger families give wider intervals). Display rank orders point estimates and is not a claim; certified rank is the interval of positions consistent with the certified relations over every linear extension of their transitive closure, so a single value is a position the data pin down. In the reference-fed tier, each pipeline runs its reference-fed entry. Composite is the equal-weight mean over the tier’s capability axes on the fraction-of-human-level scale, at matched length. The last column gives the certified rank at full length.

Component	Tested on	Key result	Verdict
On the published systems
Attribution bound, Thm. 1 ( F.1 )	two entries per pipeline, $118$ runs	$L_{M}$ transfers to held-out pipelines (coverage $0.79$ – $1.00$ )	$\checkmark$ holds
Bound at a stage window ( F.2 )	one sealed writing input, $20$ tasks; fit on $5$ pipelines, check on $15$ pairs of $6$	$420/420$ cells hold at matched length; $44\%$ vacuous	$\checkmark$ holds; informative on writing
Admissibility, Def. 1 ( F.3 )	five LiRA pairs, measured entry mismatch	radius $0.70$ – $0.73$ against tolerance $0.21$ ; refused on $15/15$ readouts	$\checkmark$ refuses
Non-expansive windows, Cor. 1 ( F.1 , F.2 )	both designs above	$0$ runs expand at matched length; $62/100$ at full length	$\checkmark$ at matched length
Certification rule, Eq. 4 ( F.5 )	sign-flip null; leave one subfield out	false certification $0.035$ / $0.062$ ; no flips	$\checkmark$ near nominal

Appendix

Table 10: Verification summary. Each part of the theory, what it is tested on, and the key result. The subsection with the full test is given in parentheses.

$L_{M}$ design	$L_{M}$	$\xi$	note
ratio $q_{95}$	0.890	0.000	selected design
ratio sup	0.967	0.000	supremum form
affine	0.803	0.085	envelope, slope $+$ slack
per system (ratio $q_{95}$ ): AS $0.911$ ( $n{=}19$ ), SGI $0.910$ ( $19$ ), LMR $0.880$ ( $31$ ), SG $0.863$ ( $49$ )

Appendix

Table 11: The module-sensitivity constant

L_{M}

, measured on the panel. Fitted on the paired gap over

118

paired runs from four pipelines,

x\in[0.74,1.00]

(a) leave-one-system-out
held out	$n$ test	$L_{M}$ (train)	$L_{M}$ (own)	held-out coverage
AutoSurvey	19	0.881	0.911	0.789
SurveyGen-I	19	0.882	0.910	0.842
LLM $\times$ MR	31	0.895	0.880	0.968
SurveyG	49	0.908	0.863	1.000
(b) falsifiability locally expansive runs ( $y>x$ ): $0/118$ max ratio $0.967$ instrument writing stage: $1.74$

Appendix

Table 12: Falsification tests for Assumption 1 . (a) A constant fitted on the other three pipelines, evaluated on the held-out one — the transferability the panel-level radius actually assumes. (b) The non-expansive claim is falsifiable and is not falsified; the instrument’s writing stage is a positive control.

	matched length	full length
fitted $L_{M}$ ( $95$ th pct. / sup)	$0.906$ / $0.940$	$1.042$ / $1.188$
runs with $y>x$	$0/100$	$62/100$
leave-one-pipeline-out coverage (mean / min)	$0.93$ / $0.80$	$0.94$ / $0.90$
bound holds (all cells)	$420/420$	$419/420$
vacuous cells ( $\epsilon_{\mathrm{outside}}\geq 1$ )	$43.6\%$	$46.0\%$
median $\epsilon_{\mathrm{outside}}$ / median $\|\Delta^{\mathrm{obs}}-\Delta^{\star}\|$	$0.955$ / $0.060$	$0.978$ / $0.066$

Appendix

Table 13: The attribution bound at a controlled writing window (

5

self-retrieving pipelines

\times

20

tasks for the fit;

420

pair-by-readout cells for the score-level check).

Pair	$\epsilon_{\mathrm{in},P}$	median $\epsilon_{\mathrm{outside}}$	median $\epsilon_{\mathrm{fair}}$	readouts refused
AutoSurvey – LiRA	$0.975$	$0.725$	$0.208$	$15/15$
SurveyForge – LiRA	$0.968$	$0.720$	$0.208$	$15/15$
SurveyG – LiRA	$0.972$	$0.723$	$0.208$	$15/15$
SurveyGen-I – LiRA	$0.942$	$0.700$	$0.208$	$15/15$
LLM $\times$ MR – LiRA	$0.979$	$0.728$	$0.208$	$15/15$

Appendix

Table 14: Refusals computed from Eq. 3 for LiRA against each self-retrieving pipeline.

Stage $m$	observed $\mathsf{W}$	bound (affine)	bound (ratio)	holds	non-vacuous	tightness
1 retrieval	0.605	0.744	0.744	95 %	yes	0.79
2 synthesis	0.347	0.844	0.891	100 %	yes	0.38
3 organization	0.261	1.312	0.981	100 %	yes	0.26
4 writing	0.685	1.465	3.378	100 %	no	0.21

Appendix

Table 15: Theorem 2 on synthetic four-stage chains built from the human graphs. Observed chain discrepancy against the composed bound (affine at

q_{95}

, and ratio form); fraction of chains on which the bound holds; tightness

=

observed/bound.

Table	View	pairs	certified	family-wise	$c^{\star}$	Jaccard	flips
fixed input	matched length	$45$	$21$	$0.062$	$1.027$ (drops $1$ )	$0.952$ / $0.870$	$0$
same pool	matched length	$45$	$20$	$0.035$	$0.966$ (drops $0$ )	$0.918$ / $0.727$	$0$
fixed input	full length	$45$	$20$	$0.051$	$1.003$ (drops $0$ )	$0.965$ / $0.905$	$0$
same pool	full length	$45$	$16$	$0.041$	$0.980$ (drops $0$ )	$0.877$ / $0.750$	$0$

Appendix

Table 16: Calibration and stability of the leaderboard’s decision rule. Family-wise rates are over

2{,}000

sign-flip draws (nominal

0.05

c^{\star}

: radius scale giving the nominal rate. Jaccard: overlap of the certified set when one of the

23

subfields is left out (mean / minimum).

		retrieval		writing
Pair	$n$	$\mathrm{Ctrl}_{M}$	sup	$\mathrm{Ctrl}_{M}$ (lo–hi)	sup
AutoSurvey – LiRA	50	0.070	0.161	0.504–0.506	0.606
LiRA – SurveyG	49	0.092	0.293	0.509–0.511	0.749
AutoSurvey – SurveyG	49	0.127	0.322	0.392–0.505	0.901
AutoSurvey – LLM $\times$ MR	50	0.424	0.472	0.800–0.940	1.116
LLM $\times$ MR – SurveyG	49	0.429	0.485	0.757–0.941	1.232

Appendix

Table 17:

\mathrm{Ctrl}_{M}

for the reference-fed pairs at

\lambda_{\mathrm{in}}=0.5

(Eq. 19 ); mean over the tasks both ran, with the essential-supremum form. Pairs with the reference-fed SurveyGen-I are omitted, because they were not recomputed after its re-run. Entry lo/hi bracket the entry term where only each system’s distance to the reference is stored. At retrieval the entry is the shared task specification, so the entry term is

0

and

\mathrm{Ctrl}_{M}

measures behaviour inside the window alone.

Property	Requirement	Measured
contraction	ratio $\leq\gamma(1-\beta)=0.45$	$0.449$ , $0.425$ , $0.428$ (random inputs); $0.450$ at convergence	✓
existence / uniqueness	same fixed point from $d_{0}\equiv 0$ and $d_{0}\equiv 1$	18 / 18 iterations, sup gap $5.7\times 10^{-7}$	✓
pseudometric	symmetry, triangle inequality	symmetry gap $0$ ; $0$ / $20{,}000$ triangle violations	✓
zero self-distance	$d^{\dagger}(S,S)=0$	$0$ for all 12 systems	✓

Appendix

Table 18: Theorem 4 and its corollaries on the empirical kernels of the twelve logged configurations.

AI Agent Evaluation

AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.

See all 521 papers

Sep 7, 2026

What Does an LLM-Agent Leaderboard Rank Actually Compare?

An LLM-agent leaderboard invites a familiar inference: an agent ranked above another is the better agent. Public evaluation logs may not support that conclusion when systems differ in task mixture, label source, release detail, or cost rule. We study what leaderboard scores estimate and when they justify pairwise superiority conclusions. Our estimand-aware pairwise procedure states the comparison target and measurement source, checks common support, and evaluates the supported difference using a stated uncertainty rule and practical margin. Controlled checks evaluate the decision labels under known finite-sample conditions and show why uncertainty must be included when judging sensitivity to target reweighting. Across SWE-bench, AgentRewardBench, and tau2-bench, close rank differences are often unresolved; proxy labels and utility rules can also change which system is selected. DataAgentBench and Open Agent show what remains estimable from coarser public records. A leaderboard score summarizes a released evaluation, whereas a fine-grained superiority claim additionally depends on the estimand and uncertainty rule used to interpret the difference.

Wei-Jung Huang

Independent Researcher United States

LLM Agent Evaluation AI Agent Benchmarks Automated Evaluation

Ask PDF

Sep 30, 2026

Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboard

Agent evaluations are increasingly used to compare LLMs and inform deployment decisions, yet ranks can reflect not only the model but also the effects of the evaluation conditions such as the scaffolds or tasks. This makes reliability claim-dependent: an evaluation that reliably ranks deployed systems may not reliably rank underlying models. We ask which conclusions current agent evaluations reliably support and what additional evaluation would improve them. We develop a Bayesian variance-decomposition framework for sparse, imbalanced agent leaderboards and apply it to 22 benchmarks from the Holistic Agent Leaderboard and Harbor Index. The framework separates signal, performance differences relevant to the intended claim, from noise, irrelevant variation that can still change rankings. We find: (1) Reliability depends on the measurement goal. Fixed model-scaffold systems are ranked reliably (0.935-0.994), while underlying-model reliability is substantially lower (0.148-0.841). (2) Scaffold choice can change conclusions. Inter-scaffold reliability measures whether scaffolds preserve model rankings, showing that scaffold effects vary substantially across evaluations. (3) More tasks cannot resolve all uncertainty. Even infinitely many similarly constructed tasks improve model-ranking reliability of a benchmark by at most 0.097 when uncertainty is dominated by limited scaffold coverage. (4) Pooling diverse benchmarks can improve cross-task rankings at lower cost. For rankings across diverse agentic tasks, pooling benchmarks raises projected reliability from 0.44 to 0.75 at the same task budget and can reduce projected cost by up to 83%. Evaluation design should follow the intended claim: identify what a score or ranking should mean, diagnose what limits its reliability, and spend evaluation budget on the sources of uncertainty that matter.

Michael Hardy, Ruhana Azam, Anka Reuel +2

Stanford University · University of Illinois Urbana-Champaign · Senior Author

AI Agent Evaluation Agent Evaluation AI Agent Benchmarks

Ask PDF

Sep 26, 2026

Rank Confidence Sequences:Anytime-valid Leaderboards

Leaderboards rank models by their average scores on benchmark items, and they are consulted repeatedly while the evaluation is still running. Existing confidence intervals for a model's rank control their error rate only if they are computed once, after a number of items chosen in advance. If they are recomputed as results arrive, and the evaluation stops once they look decisive, their error rate exceeds its nominal level. Anytime-valid methods keep their guarantees at all sample sizes simultaneously and hence under any stopping rule. They exist for the accuracy of one model, for one pair of models and for the set of models that may be best. For pairwise battles they also give ranks. None gives ranks when all models are scored on the same items, which makes their scores dependent. We construct rank confidence sequences: for every model, a set of ranks that contains its true rank, simultaneously for all models and at all times, at a chosen error level

α

, in finite samples. The construction combines betting e-processes, one for each ordered pair of models, with closed testing over the possible orderings of the models. It allows any dependence between the models' scores on an item. The method has two advantages. A leaderboard can be inspected after every item without inflating its error rate. The evaluation of each model can stop as soon as the question asked about it is answered, which saves compute. When results are examined only once, halfway through or later, little power is lost relative to fixed-sample methods. The paper quantifies these advantages in simulations and on public leaderboard data.

Hamed Khosravi, Xiaoming Huo

H. Milton Stewart School of Industrial and Systems Engineering Georgia Institute of Technology, Atlanta, GA 30332, USA

Anytime-Valid Inference Benchmark Validity

Ask PDF

Abstract

AI Agent Evaluation

Explore similar work

What Does an LLM-Agent Leaderboard Rank Actually Compare?

Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboard

Rank Confidence Sequences:Anytime-valid Leaderboards