Using a Human-AI Teaming Approach to Create and Curate Scientific Datasets with the SCILIRE System

Necva Bölücü; Jessica Irons; Changhyun Lee; Brian Jin; Maciej Rybinski; Huichen Yang; Andreas Duenser; Stephen Wan

cs.CLMar 13, 2026

Using a Human-AI Teaming Approach to Create and Curate Scientific Datasets with the SCILIRE System

Authors: Necva Bölücü, Jessica Irons, Changhyun Lee, Brian Jin, Maciej Rybinski, Huichen Yang, Andreas Duenser, Stephen Wan

Organizations: CSIRO, Sydney, Australia · ITIS, University of Málaga, Málaga, Spain

Abstract

The rapid growth of scientific literature has made manual extraction of structured knowledge increasingly impractical. To address this challenge, we introduce SCILIRE, a system for creating datasets from scientific literature. SCILIRE has been designed around Human-AI teaming principles centred on workflows for verifying and curating data. It facilitates an iterative workflow in which researchers can review and correct AI outputs. Furthermore, this interaction is used as a feedback signal to improve future LLM-based inference. We evaluate our design using a combination of intrinsic benchmarking outcomes together with real-world case studies across multiple domains. The results demonstrate that SCILIRE improves extraction fidelity and facilitates efficient dataset creation.

Figures & tables

Tool	Dynamic Sampling	Multi-record Support	Provenance Data	Table Export (CSV, JSON, $\cdots$ )
Elicit	✗ $\ddagger$	✗	✓ $\dagger$	✓
SciSpace	✗ $\ddagger$	✗	✓ $\dagger$	✓
NotebookLM	✗	✓	✓ $\dagger$	✗
Claude.ai	✗	✓	✗	✓
SciLire (Ours)	✓	✓	✓ $\dagger$	✓

Table 1: Comparison of data curation tools’ functionality, highlighting differences from SciLire .

\ddagger

Elicit and SciSpace can be made to accept static examples in column definitions.

\dagger

Elicit, SciSpace and NotebookLM provide paragraph- or sentence-level citations. SciLire provides the degree of alignment with the source, along with relevant paragraphs. Multi-record support indicates if a tool is designed to produce multiple extracted records per document.

Figure 1: SciLire components and AI-augmented curation workflow.

Dataset	0-shot	ICL-10	ICL-50	ICL-100	ICL-all
TDMS	10.14	19.01	23.01	24.54	25.02
SciREX	3.66	13.51	15.49	15.71	18.27
MPEA	29.23	32.52	30.39	30.81	30.64
Diffusion	17.52	17.99	17.59	–	17.20
YSHEAY	5.34	7.87	8.30	8.03	7.93
CCRMG	1.82	2.48	–	–	2.89

Table 2: F 1 results across datasets. LLM: GPT-5. F 1 reported with 0–100 scale; best score is boldfaced . For the full table, see Table 10 .

	SciSpace		Elicit		SciLire
Dataset	0-shot	ICL-S	0-shot	ICL-S	0-shot	ICL-D
TDMS	0.0	0.0	0.0	3.13	3.97	11.76
SciREX	0.0	0.0	1.08	6.49	2.98	18.22
MPEA	13.26	13.26	0.0	0.0	40.67	42.27
Diffusion	0.65	0.65	0.06	0.53	6.80	8.71
YSHEAY	0.0	0.0	2.22	13.33	3.29	5.54

Table 3: F 1 results across datasets comparing SciLire with other data generation tools. F 1 reported with 0–100 scale; best score is boldfaced . SciLire results are based on GPT-5. Abbreviations: ICL-S: ICL static, ICL-D: ICL Dynamic (

n

=all). For the full table, see Table 11 . Results are shown for 10 randomly selected PDFs.

Domain	# Docs	# Records	Edits (0-shot)	Edits (ICL)	Time (0-shot)	Time (ICL)
Agri.	42	96	30	20	6	7
Env. St.	20	20	35	3	13	3
Biochem.	18	56	7	5	11	3
Med Man.	15	15	12	25	6	9

Table 4: User validation behaviour across case studies at the initial (zero-shot) and final pilot phases (with ICL). Edits: Percentage of values edited by the user. Time: Average time (minutes) curating/correcting each PDF.

Figure 2: Average time spent validating data from the first 20 papers across the early adopter user cases.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: The Table & Figure Extraction module.

Model	[email protected]	[email protected]	mAP
Fast R-CNN ( Girshick, 2015 )	96.18	94.20	87.36
Mask R-CNN ( He et al., 2017 )	96.21	94.39	87.57
Cascade R-CNN	97.12	95.40	90.55

Appendix

Table 5: Evaluation results comparing Cascade R-CNN with baseline approaches. The best results are boldfaced .

Model	TEDS	[email protected]
UniTable Peng et al. (2024)	95.23	96.23
UniTable (finetuned on SciTSR)	97.98	96.98
Qwen2.5-VL-72B Hui et al. (2025)	82.72	—

Appendix

Table 6: Evaluation results comparing UniTable with and without finetuning. The best results are boldfaced .

Domain	Dataset	# Documents	Avg. Pages	# Records	Records / Document	Schema Size	Reference
Machine Learning	TDMS	332	10.52	904	2.72	4	Hou et al. (2019)
Machine Learning	SciREX	372	11.83	1,897	5.34	5	Jain et al. (2020)
Material Science	MPEA	264	8.70	1,544	5.85	17	Borg et al. (2020)
	Diffusion	93	14.02	3,533	37.99	18	Zhang et al. (2010)
	YSHEAY	219	15.65	837	3.82	3	Polak et al. (2024)
	CCRMG	24	11.96	43	1.79	3	Polak et al. (2024)

Appendix

Table 7: Statistics of datasets across five domains.

Model	# of Par.	Context Length	Open-Source	Family
GPT-OSS:20b	20B	128K	✓	OpenAI
GPT-OSS:120b	120B	128K	✓	OpenAI
Gemma3:1B	1B	32K	✓	Google
Gemma3:4B	4B	128K	✓	Google
Gemma3:12B	12B	128K	✓	Google
Gemma3:27B	27B	128K	✓	Google

Appendix

Table 8: Details of models used in record generation module (Section 4.2 ) of SciLire .

Dataset	Gemma3:1B	Gemma3:4B	Gemma3:12B	Gemma3:27B	Qwen3:0.6B	Qwen3:4B	Qwen3:14B	Qwen3:32B	Phi-4	GPT-OSS:20B	GPT-OSS:120B	Deepseek-R1-Llama:8B	Deepseek-R1-Llama:70B	Deepseek-R1-Qwen3:14B	Deepseek-R1-Qwen3:32B	GPT-5
TDMS	12.20	12.13	18.80	21.90	17.13	25.09	17.98	21.91	16.10	22.10	23.02	21.19	22.53	26.04	23.13	25.02
SciREX	4.10	3.97	14.45	15.08	7.82	15.69	17.06	16.53	10.56	18.12	17.65	6.26	08.45	15.99	14.56	18.27
MPEA	9.29	10.45	13.45	18.36	8.43	15.23	18.43	17.56	13.54	19.45	16.34	7.21	14.56	12.42	13.21	30.64
Diffusion	0.45	4.20	5.45	8.36	0.68	6.34	9.12	12.25	3.12	14.21	16.34	7.10	9.21	12.10	12.46	17.20
YSHEAY	9.23	10.13	12.21	13.10	17.35	14.25	15.99	16.48	11.24	16.23	14.56	12.48	13.21	12.02	10.45	7.93
CCRMG	5.85	5.23	5.71	3.37	4.96	5.01	5.89	9.03	8.91	6.10	7.03	2.24	3.87	5.02	4.65	2.89

Appendix

Table 9: F 1 results across datasets for multiple LLMs and their ability to benefit from ICL Dynamic sampling (

n

=all). F 1 reported with 0–100 scale; best score is boldfaced . For detailed results, see https://github.com/bolucunecva/scilire .

	Zero-shot				ICL -10				ICL -50				ICL -100				ICL -all
Models	P	R	F 1	ChrF	P	R	F 1	ChrF	P	R	F 1	ChrF	P	R	F 1	ChrF	P	R	F 1	ChrF
TDMS	6.95	18.75	10.14	11.94	13.88	30.14	19.01	13.88	16.47	38.2	23.01	14.70	17.68	40.13	24.54	15.11	18.12	40.38	25.02	15.05
SciREX	2.8	5.29	3.66	7.16	10.67	18.39	13.51	10.16	12.61	20.07	15.49	10.78	12.53	21.06	15.71	11.09	14.52	24.61	18.27	11.63
MPEA	31.12	27.56	29.23	0.74	34.59	30.68	32.52	0.91	32.9	28.24	30.39	0.77	32.45	29.32	30.81	0.74	31.92	29.45	30.64	0.74
Diffusion	33.51	11.86	17.52	1.54	32.51	12.43	17.99	1.58	33.13	11.98	17.59	1.6	–	–	–	–	30.35	12.0	17.20	1.52
YSHEAY	2.94	29.51	5.34	12.32	4.45	34.41	7.87	12.96	4.69	35.96	8.30	13.14	4.54	34.85	8.03	13.11	4.47	35.36	7.93	13.28

Appendix

Table 10: Evaluation results across datasets. Cells marked with ‘–’ indicate that the dataset does not have enough train data for evaluation. All scores are reported on a 0–100 scale, with the best F 1 score highlighted in boldfaced . LLM: GPT-5.

	SciSpace								Elicit								SciLire best (ICL large)
	Zero-shot				ICL (Static)				Zero-shot				ICL (Static)				Zero-shot				ICL (Dynamic)
Dataset	P	R	F 1	ChrF	P	R	F 1	ChrF	P	R	F 1	ChrF	P	R	F 1	ChrF	P	R	F 1	ChrF	P	R	F 1	ChrF
TDMS	0.0	0.0	0.0	2.97	0.0	0.0	0.0	4.36	0.0	0.0	0.0	5.43	5.0	2.27	3.13	7.70	2.4	11.36	3.97	9.42	7.94	22.73	11.76	11.08
SciREX	0.0	0.0	0.0	2.97	0.0	0.0	0.0	2.97	2.0	0.74	1.08	3.73	12.0	4.44	6.49	6.80	2.09	5.19	2.98	8.09	12.43	34.07	18.22	14.21
MPEA	47.06	7.71	13.26	0.24	47.06	7.71	13.26	0.24	0.0	0.0	0.0	0.22	0.0	0.0	0.0	0.29	39.13	42.33	40.67	0.73	39.59	45.35	42.27	0.74
Diffusion	20.99	0.33	0.65	0.35	20.99	0.33	0.65	0.35	1.85	0.03	0.06	0.55	17.28	0.27	0.53	0.96	23.29	3.98	6.80	1.41	27.51	5.17	8.71	1.59

Appendix

Table 11: Evaluation results across datasets comparing SciLire with other data generation tools. All scores are reported on a 0–100 scale, with the best F 1 score highlighted in boldfaced . SciLire results are based on GPT-5.

	SciSpace								SciLire best (ICL large)
	Zero-shot				ICL (Static)				Zero-shot				ICL (Dynamic)
Dataset	P	R	F 1	ChrF	P	R	F 1	ChrF	P	R	F 1	ChrF	P	R	F 1	ChrF
TDMS	0.08	0.03	0.04	2.22	0.0	0.0	0.0	3.02	6.95	18.75	10.14	11.94	18.12	40.38	25.02	15.05
SciREX	0.00	0.00	0.0	1.3	0.0	0.0	0.0	1.91	2.8	5.29	3.66	7.16	14.52	24.61	18.27	11.63
MPEA	50.32	8.60	14.69	0.1	47.62	8.14	13.9	0.12	31.12	27.56	29.23	0.74	31.92	29.45	30.64	0.74
Diffusion	18.60	0.49	0.95	0.27	14.67	0.38	0.75	0.3	33.51	11.86	17.52	1.54	30.35	12.0	17.20	1.52

Appendix

Table 12: Evaluation results across the full datasets comparing SciLire and SciSpace. F 1 reported with 0–100 scale; best score is boldfaced . SciLire results are based on GPT-5.

Figure 5: A screenshot of SciLire for project creation.

Figure 6: A listing of a user’s projects.

Figure 7: The “samples” representing the table output from the iterative workflow.

Figure 8: The table in Sample 1, which has been edited and curated by the user.

Figure 9: A screenshot from the pilot phase (Sample 3) of the AI-augmented curation workflow in the demo.

LLM Information Extraction

Large language model information extraction converts unstructured documents into structured entities, relations, events, and records.

See all 238 papers

SciCurateBench: Curating Experimental Data from Heterogeneous Scientific Literature with Verbal-Action Agents
Oct 9, 2026Minuk Ma, Kaiyun Guo, Jiarui DingAI-Assisted Scientific Research AI Agent Benchmarks
University of British Columbia
Self-prompting and cross-model consensus enable reproducible data extraction from scientific literature with large language models
Aug 19, 2026Valentin Romanov, Monique Bax, Steven NiedererLLM Information Extraction LLM Reliability
National Heart & Lung Institute, Faculty of Medicine, Imperial College London, United Kingdom · Cambridge Stem Cell Institute, JeDrey Cheah Biomedical Centre, Cambridge Biomedical Campus, University of Cambridge, Cambridge, United Kingdom · Department of Medicine, University of Cambridge, Cambridge, United Kingdom +1
SciDataSailor: Deep Scientific Data Exploring
Jul 29, 2026Jiyong Rao, Yicheng Qiu, Chi Zhang +2Scientific QA AI Agent Benchmarks
Shanghai Artificial Intelligence Laboratory · Tongji University

Oct 9, 2026J/K move · Enter open · S save

SciCurateBench: Curating Experimental Data from Heterogeneous Scientific Literature with Verbal-Action Agents

Minuk Ma, Kaiyun Guo, Jiarui Ding

University of British Columbia

AI-Assisted Scientific Research AI Agent Benchmarks LLM Information Extraction Multi-Agent LLM Systems

Open paper AskPDF arXiv

Scientific research increasingly depends on curating experimental data from literature. Scientific data curation requires verifying exact experimental conditions (e.g., specific assays, clinical outcomes, and data availability) across long, heterogeneous files, which is a highly time-consuming process. Existing benchmarks evaluate only broad relevance on short queries, failing to capture this complexity. We introduce SciCurateBench, a benchmark of 200 expert-annotated papers from major life-science journals that challenges systems to verify conjunctive biological experimental criteria from raw, unprocessed scientific literature. We show that plain LLMs and popular reasoning, tool-using, and feedback-based agents all struggle with this task. To study what makes the task tractable, we further propose LangPoli, an integrative system in which a policy agent generates natural language action guidelines that actor agents convert into executable tools. LangPoli showed strong performance on both Paper Classification (83.16% F1 vs. 58.22% for its base LLM) and Structured Information Retrieval, and also improves re-ranking on the public benchmarks of LitSearch, BEIR, and TREC. We further explore curriculum transfer, a method that distills generalizable strategies from solved tasks to accelerate the solving of related problems. We hope this work facilitates research in truly practical scientific AI assistance. The anonymized pre-release can be found from https://anonymous.4open. science/r/scicuratebench-D885/.

Abstract

LLM Information Extraction

Explore similar work

SciCurateBench: Curating Experimental Data from Heterogeneous Scientific Literature with Verbal-Action Agents

Self-prompting and cross-model consensus enable reproducible data extraction from scientific literature with large language models

SciDataSailor: Deep Scientific Data Exploring