SciCurateBench: Curating Experimental Data from Heterogeneous Scientific Literature with Verbal-Action Agents

Minuk Ma; Kaiyun Guo; Jiarui Ding

cs.CLOct 9, 2026

SciCurateBench: Curating Experimental Data from Heterogeneous Scientific Literature with Verbal-Action Agents

Authors: Minuk Ma, Kaiyun Guo, Jiarui Ding

Organizations: University of British Columbia

Abstract

Scientific research increasingly depends on curating experimental data from literature. Scientific data curation requires verifying exact experimental conditions (e.g., specific assays, clinical outcomes, and data availability) across long, heterogeneous files, which is a highly time-consuming process. Existing benchmarks evaluate only broad relevance on short queries, failing to capture this complexity. We introduce SciCurateBench, a benchmark of 200 expert-annotated papers from major life-science journals that challenges systems to verify conjunctive biological experimental criteria from raw, unprocessed scientific literature. We show that plain LLMs and popular reasoning, tool-using, and feedback-based agents all struggle with this task. To study what makes the task tractable, we further propose LangPoli, an integrative system in which a policy agent generates natural language action guidelines that actor agents convert into executable tools. LangPoli showed strong performance on both Paper Classification (83.16% F1 vs. 58.22% for its base LLM) and Structured Information Retrieval, and also improves re-ranking on the public benchmarks of LitSearch, BEIR, and TREC. We further explore curriculum transfer, a method that distills generalizable strategies from solved tasks to accelerate the solving of related problems. We hope this work facilitates research in truly practical scientific AI assistance. The anonymized pre-release can be found from https://anonymous.4open. science/r/scicuratebench-D885/.

Figures & tables

Figure 1: SciCurateBench promotes adaptive tool creation for efficiently exploring, analyzing, and summarizing heterogeneous scientific files, including manuscripts, supplementary data, and raw datasets.

Figure 2: Overview of LangPoli on SciCurateBench. LangPoli creates an adaptive tool based on the verbal action to interact with the environment, while ReAct chooses an action from a fixed set.

Group	Method	Acc (%)	Prec (%)	Rec (%)	F1 (%)	C3 Prec (%)	C3 Rec (%)	C3 F1 (%)
Plain LLM	gpt-4o	65.33	61.08	64.99	61.00 (2.46)	67.38	83.33	74.46
Plain LLM	claude-3.5	67.67	63.48	57.76	58.22 (2.23)	91.67	52.63	66.59
Reasoning	CoT	75.00	71.16	73.13	71.96 (1.97)	77.39	78.06	77.71
Reasoning	Self-Con	73.33	68.56	71.38	69.67 (0.61)	76.96	78.94	77.91
Tool	ReAct	79.67	75.68	79.05	76.95 (1.01)	79.16	86.84	82.81
	ChatCoT	76.59	71.26	74.94	72.21 (0.76)	77.08	87.72	82.02

Table 1: Benchmarking performance on Science Paper Classification. Std for F1 score was provided. The metrics for Class 3 were additionally provided. LangPoli (full) achieves the best accuracy, F1, and Class 3 F1.

Method	Full (%)	Longest 25% (%)	$\Delta$
GPT-4o	61.00	59.47	$-1.53$
Claude-3.5	58.22	43.83	$-14.39$
LangPoli (full)	83.16	79.90	$-3.26$

Table 2: Classification F1 (%) on the longest-quartile papers (top 25% by token count,

>

31K tokens) versus the full benchmark. LangPoli degrades far less than Claude-3.5 on long papers.

Method	F1 (%)	Pat. ID (%)	Resp.	Miss.
	(std)		Rate (%)	Pat. ( $\downarrow$ )
ReAct	53.57 (1.19)	59.73	75.63	5.93
ChatCoT	59.13 (4.40)	65.60	80.43	6.57
LATM	11.50 (4.62)	20.43	29.27	12.10
Reflexion	57.33 (8.81)	62.23	74.37	5.75
SelfRefine	48.47 (5.56)	52.47	60.70	8.02

Table 3: Benchmarking performance on Structured Information Retrieval. F1 measures the completeness of the pairs. Pat. ID measures Acc on sample (patient) ID and Resp. Rate measures Acc on the response rate information. LangPoli (full) achieves the best F1 and misses the fewest patients.

		GPT-4o-mini		GPT-4o
Retrieval	Reranking	Au-S	Avg. S	Au-S	Avg. S
		R@5	R@5	R@5	R@5
BM25	None	62.6	50.0	62.6	50.0
BM25	RankGPT	71.6	65.3	76.8	68.0
BM25	LangPoli	76.8	67.5	82.9	73.0
	Improvement	+5.2	+2.2	+6.1	+5.0

Table 4: Performance on LitSearch. Au-S means the queries that are author-curated (as opposed to computer-generated) and specific. Avg. S means the average result on all specific queries. Improvement=LangPoli – RankGPT. LangPoli improves over RankGPT in every setting.

Method	F1 (%)	Pat. ID	Resp.	Exact	Missed
		(%)	Rate (%)	Matches	Pat.
full $-$ MoCA	62.97	74.77	82.57	10.31	3.81
full $-$ init. explore	65.83	74.70	84.73	11.24	2.53
full $-$ late task guide	67.57	84.27	90.40	11.67	1.70
full $-$ detailed policy	66.53	76.77	86.30	10.96	2.86
full $-$ feedback $+$ refl.	71.33	81.77	91.20	11.60	2.18

Table 5: LangPoli variants analysis on Structured Information Retrieval. Removing MoCA causes the largest F1 drop.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: SciCurateBench dataset statistics: (a) journal distribution, (b) publication years, (c) class distributions for both queries

Method	Acc (%)	Prec (%)	Rec (%)	F1 (%)
Plain LLM	71.28	83.17	70.01	73.46 (1.71)
ReAct	82.83	82.47	76.59	78.46 (5.28)
LangPoli Curriculum (5 iter)	87.12	91.16	84.29	86.81 (1.68)
LangPoli Curriculum (7 iter)	85.80	93.21	82.98	86.95 (1.04)
LangPoli Curriculum (10 iter)	87.12	90.78	85.64	87.55 (1.26)

Appendix

Table 6: Performance boosting by curriculum transfer for scRNA-seq + scTCR-seq classification

		GPT-generated		Author-curated
Retrieval	Reranking	ACL	Non-ACL	ACL	ICLR
BM25	None	37.7	31.7	49.3	69.4
BM25	RankGPT	58.2	53.9	58.2	72.2
BM25	LangPoli	55.7	55.0	64.2	75.0
	Improvement	-2.5	+1.1	+6.0	+2.8
GritLM	None	63.1	61.7	77.6	80.6

Appendix

Table 7: Performance breakdown on LitSearch specific, high-quality queries (Specificity=1, Quality=2) by query source and paper venue. GPT-4o-mini was used as the basis LLM.

Dataset	Base LLM	RGP	LP	$\Delta$
Touche	GPT-4o-mini	34.12	37.50	+3.38
	Qwen2.5-Coder	37.88	40.54	+2.66
	Zephyr-7B	40.89	66.11	+25.22
	Hermes-2-Pro	44.30	39.87	-4.43
	OpenChat-3.5	44.06	41.11	-2.95
	DeepSeek-Coder	28.13	33.05	+4.92

Appendix

Table 8: Performance (NDCG@10) on BEIR subsets with long corpus contexts. RGP = RankGPT, LP = LangPoli.

\Delta

denotes LP – RGP. Bold indicates improvement by LangPoli.

Method	DL19	DL20
BM25	50.58	47.96
RankGPT (gpt-3.5-turbo)	65.80	62.91
RankGPT (gpt-4o-mini)	72.61	67.59
RankGPT (gpt-4o)	75.59	70.56
LangPoli (gpt-4o-mini)	75.93	68.79
$\Delta$ (vs. RankGPT gpt-4o-mini)	+3.32	+1.20

Appendix

Table 9: Performance comparison on TREC-DL benchmarks. NDCG@10 is used as the metric.

\Delta

denotes improvement of LangPoli (gpt-4o-mini) over RankGPT (gpt-4o-mini).

Method	Correct	ID-only	Missed	FP
	(%)	(%)	(%) $\downarrow$	(%) $\downarrow$
ReAct	49.9	6.8	43.3	26.1
ChatCoT	50.8	5.0	44.2	32.5
LATM	4.3	1.9	93.8	60.7
Reflexion	57.1	4.1	38.8	19.0
SelfRefine	41.7	4.3	54.0	28.9

Appendix

Table 10: Confusion breakdown on Structured Information Retrieval. Correct, ID-only, and Missed are computed over ground-truth patients and sum to 100%. FP is computed over predicted patients.

Model	Method	Acc (%)	F1 (%)	C3 F1 (%)	Gain (%)
Claude-3	Plain	25.33	24.93 (1.40)	38.24	+10.36
Claude-3	LangPoli	37.67	35.29 (2.32)	45.27	+10.36
Claude-3.5	Plain	67.67	58.22 (2.23)	66.59	+24.94
Claude-3.5	LangPoli	86.33	83.16 (0.94)	88.19	+24.94
Claude-3.7	Plain	80.00	77.17 (2.22)	79.51	+9.21
Claude-3.7	LangPoli	87.29	86.38 (1.00)	87.65	+9.21

Appendix

Table 11: Performance of plain LLM and LangPoli by different base LLM models on the SciCurateBench dataset

Agent	Exec.	Evid.	Cyclo.	Hal.	LoC	MI
	Succ. (%)	Len.	Complex.	Vol.
Actor 1	100.00	45,780	7.36	543.53	46.77	43.71
Actor 2	100.00	78,271	4.22	385.67	38.17	47.38
Actor 3	100.00	35,871	6.72	533.25	49.16	43.46
LangPoli (full)	100.00	53,320	6.1	397.57	44.70	48.28

Appendix

Table 12: Evaluation of tool creation in Structured Information Retrieval. Actor 1, Actor 2, and Actor 3 are Manuscript Explorer, Supplementary Content Reader, and Large File Summarizer.

Method	Avg. Prompt	Avg. Response	Total	F1 (%)
ReAct	24,318	–	24,318	76.95
LangPoli-Basic	14,491	2,162	16,653	79.76
LangPoli (full)	51,145	4,371	55,516	83.16
Actor 1	15,788	640	16,428	–
Actor 2	3,583	1,081	4,664	–
Actor 3	24,608	488	25,096	–

Appendix

Table 13: Token consumption on SciCurateBench Paper Classification. Prompt and Response columns report average tokens per iteration. Total = Prompt + Response.

Figure 4: Tool usage trajectories on a representative benchmark paper.

AI-Assisted Scientific Research

AI-assisted scientific research uses machine learning and AI agents to support discovery, analysis, experimentation, and scientific communication.

See all 251 papers

Self-prompting and cross-model consensus enable reproducible data extraction from scientific literature with large language models
Aug 19, 2026Valentin Romanov, Monique Bax, Steven NiedererLLM Information Extraction LLM Reliability
National Heart & Lung Institute, Faculty of Medicine, Imperial College London, United Kingdom · Cambridge Stem Cell Institute, JeDrey Cheah Biomedical Centre, Cambridge Biomedical Campus, University of Cambridge, Cambridge, United Kingdom · Department of Medicine, University of Cambridge, Cambridge, United Kingdom +1
SciExplore: Evaluating Autonomous Agents from Scientific Navigation to Information Integration
Jul 23, 2026Yinhao Tang, Youqing Fang, Yanan Sun +6AI Agent Benchmarks Information Retrieval
University of Science and Technology of China · Shanghai AI Laboratory
DISCERN: Can AI Agents Work Like Scientists and Guide Discovery?
Sep 27, 2026Nan Huang, Mario Tapia-Pacheco, Kun Zhou +4AI Agent Reliability Scientific Hypothesis Generation
University of California San Diego · Independent Researcher

Aug 19, 2026J/K move · Enter open · S save

Self-prompting and cross-model consensus enable reproducible data extraction from scientific literature with large language models

Valentin Romanov, Monique Bax, Steven Niederer

National Heart & Lung Institute, Faculty of Medicine, Imperial College London, United Kingdom · Cambridge Stem Cell Institute, JeDrey Cheah Biomedical Centre, Cambridge Biomedical Campus, University of Cambridge, Cambridge, United Kingdom · Department of Medicine, University of Cambridge, Cambridge, United Kingdom+1

LLM Information Extraction LLM Reliability LLM Prompting AI Agents for Scientific Discovery

Open paper AskPDF arXiv

Accurately extracting nuanced, contextualized data from research articles is laborious and time intensive. Here, we investigate the performance of frontier, browser-based large language models (LLMs) to extract highly contextualized information. We demonstrate four escalating workflows, 1) given an expert curated prompt and research articles, most frontier LLMs perform well at data extraction, however can struggle with interpreting scientific context and nuance, 2) given simple instructions, LLMs can author their own prompts which were almost as eNective as expert-written prompts, 3) autonomous discovery of research literature was diNicult, agents either missed or hallucinated references, and 4) LLMs can create new datasets from published guidelines that closely match human-expert judges, but still require a human-in-the-loop. Together, these findings define an auditable division of labour in which experts specify the evidence standard, models cross-check repeated extractions and researchers resolve disputed cases, providing a practical route to scaling scientific data curation without relinquishing expert oversight.

Abstract

AI-Assisted Scientific Research

Explore similar work

Self-prompting and cross-model consensus enable reproducible data extraction from scientific literature with large language models

SciExplore: Evaluating Autonomous Agents from Scientific Navigation to Information Integration

DISCERN: Can AI Agents Work Like Scientists and Guide Discovery?