Do We Need Bigger Models for Science? Task-Aware Retrieval with Small Language Models

Florian Kelber; Matthias Jobst; Yuni Susanti; Michael Färber

cs.IRApr 2, 2026

Do We Need Bigger Models for Science? Task-Aware Retrieval with Small Language Models

Authors: Florian Kelber, Matthias Jobst, Yuni Susanti, Michael Färber

Organizations: TU Dresden, Germany · FIZ Karlsruhe, Germany · ScaDS.AI, TU Dresden, Germany

Abstract

Scientific knowledge discovery increasingly relies on large language models, yet many existing scholarly assistants depend on proprietary systems with tens or hundreds of billions of parameters. Such reliance limits reproducibility and accessibility for the research community. In this work, we ask a simple question: do we need bigger models for scientific applications? Specifically, we investigate to what extent carefully designed retrieval pipelines can compensate for reduced model scale in scientific applications. We design a lightweight retrieval-augmented framework that performs task-aware routing to select specialized retrieval strategies based on the input query. The system further integrates evidence from full-text scientific papers and structured scholarly metadata, and employs compact instruction-tuned language models to generate responses with citations. We evaluate the framework across several scholarly tasks, focusing on scholarly question answering (QA), including single- and multi-document scenarios, as well as biomedical QA under domain shift and scientific text compression. Our findings demonstrate that retrieval and model scale are complementary rather than interchangeable. While retrieval design can partially compensate for smaller models, model capacity remains important for complex reasoning tasks. This work highlights retrieval and task-aware design as key factors for building practical and reproducible scholarly assistants.

Figures & tables

Panel of Figure 1 — Figure 1: Task-aware retrieval pipeline. Task routing determines which processing strategy and data source are used before response generation. The input query $q$ is classified into a task $t$ , used to retrieve relevant context $\mathcal{E}(q)$ from data sources $\mathcal{D}$ and $\mathcal{G}$ , and passed to a lightweight LLM to generate the response $r$ with citations $\mathbf{C}$ .

Task Category	Retrieval Strategy	Output Type
General QA	Passage retrieval from unarXive; metadata enrichment via Semantic Scholar API	Factual answer with inline citations
Simplification, Summarization	NER-based paper title detection; fuzzy match against unarXive full-text corpus	Simplified or summarized text with optional paper grounding
KG-Fact	SPARQL queries over SemOpenAlex; 18 predefined templates (author/work)	Structured metadata (e.g., h-index, DOI, ORCID, affiliations)

Table 1: Overview of task categories, retrieval strategies, and output in the classification module.

Model	Org $\uparrow$	Cov $\uparrow$	Rel $\uparrow$	Mean $\uparrow$	Length	Citation Quality
						R $\uparrow$	P $\uparrow$	F1 $\uparrow$
Llama3-8B $\dagger$ *	–	–	–	3.79	–	–	–	0.0%
OS-8B *	3.92	4.44	4.02	4.12	578.6	–	–	42.8%
Llama 3.2 3B Instruct $\dagger$	3.96	4.06	4.54	4.19	450	0.00%	0.00%	0.00%
Llama 3.1 8B Instruct $\dagger$	4.01	4.19	4.76	4.32	475	0.00%	0.00%	0.00%
Llama 3.2 3B Instruct	3.67	3.51	3.99	3.72	363	16.49%	19.05%	17.68%

Table 2: Multi-Paper QA Evaluation results (ScholarQABench-Multi dataset). Best value per model type is underlined ; overall best values (excluding external baselines) are bolded ; LoRA FT rows denote LoRA fine-tuning applied to Llama 3.1 8B Instruct; * results from Asai et al. (2024) ;

\dagger

without retrieval.

Model	Orig Acc $\uparrow$	Orig F1 $\uparrow$	OS Acc $\uparrow$	OS F1 $\uparrow$	Zero Acc $\uparrow$	Zero F1 $\uparrow$
BioBERT $\dagger$	68.08	52.72	–	–	–	–
OS-8B *	–	–	76.4	–	–	–
Llama 3.2 3B Instruct	62.40	40.93	49.11	49.04	64.41	58.97
Llama 3.1 8B Instruct	75.60	54.06	45.91	43.35	64.77	59.74
LoRA FT 2ep 10k context	68.00	46.69	58.01	55.83	58.13	57.23
Full FT 5ep 16k context	62.80	38.69	58.01	56.21	60.62	57.24

Table 3: Single-paper QA evaluation results (PubMedQA dataset). Best values are bolded .

\dagger

results from Jin et al. (2019) ; * results from Asai et al. (2024) .

Model	R1 F1 $\uparrow$	R2 F1 $\uparrow$	RL F1 $\uparrow$	BERTScore F1 $\uparrow$	SMOG Index $\uparrow$	Compression Ratio $\uparrow$
CATT Cachola et al. (2020)	44.9	22.6	37.3	–	–	47.3
Llama 3.2 3B Instruct	1.21	0.011	1.21	54.55	24.45	20.72
Llama 3.1 8B Instruct	1.21	0.014	1.21	54.57	24.81	20.79
Full FT 5ep 16k context	1.31	0.014	1.30	54.74	23.26	22.43

Table 4: Text compression evaluation results (SciTLDR dataset). Best values are bolded . R1, R2, and RL denote ROUGE-1, ROUGE-2, and ROUGE-L F1 scores. Scores are averages over test samples.

AI-Assisted Scientific Research

AI-assisted scientific research uses machine learning and AI agents to support discovery, analysis, experimentation, and scientific communication.

See all 253 papers

Large Knowledge Model: A Knowledge Foundation for Agentic Science at Scale
Sep 23, 2026Yuan Huang, Sihan Hu, Hongyu Gu +18KG Construction AI Agents for Scientific Discovery
DP Technology, Beijing, China · Institute of Theoretical Physics, Chinese Academy of Sciences, Beijing, China · International Center for Quantum Materials, School of Physics, Peking University, Beijing, China +4
EGT-KG: Evidence-Grounded Typed KG Retrieval for Practical Scientific QA with Small Language Models
Aug 31, 2026Muran Yu, Jiechao Gao, Yuandong Pan +5Knowledge Graphs Small Language Models
Department of Civil and Environmental Engineering, Stanford University · Department of Civil Engineering, University of Calgary
Eigen-Agent: Adaptive Multi-Agent Scientific Reasoning with Monitor-Based RAG
Sep 25, 2025Xiangru Tang, Wanghan Xu, Yujie Wang +13Scientific Reasoning in Language Models Multi-Agent Reasoning
Yale University · Shanghai Jiao Tong University · Fudan University +4

Sep 23, 2026J/K move · Enter open · S save

Large Knowledge Model: A Knowledge Foundation for Agentic Science at Scale

Yuan Huang, Sihan Hu, Hongyu Gu, Chao Ma+17

DP Technology, Beijing, China · Institute of Theoretical Physics, Chinese Academy of Sciences, Beijing, China · International Center for Quantum Materials, School of Physics, Peking University, Beijing, China+4

KG Construction AI Agents for Scientific Discovery Scientific QA Knowledge Representation Evidence Retrieval Scientific Document Retrieval

Open paper AskPDF arXiv

Agentic science envisions many autonomous agents investigating concurrently while building on a shared, evolving body of scientific knowledge. This requires a knowledge foundation that supports high-concurrency access, preserves traceable and reusable reasoning, and grows incrementally. We propose the Large Knowledge Model (LKM), a growing, agent-native knowledge foundation that provides a general representation of scientific knowledge across disciplines. LKM organizes the scientific literature into reasoning graphs, with claims as the core nodes and associated reasoning chains that make explicit how premises and evidence support conclusions. These source-grounded objects are persistent and addressable; cross-paper links organize them into aligned question, workflow, and evidence views. Newly extracted papers extend the foundation incrementally while preserving existing object identities. Building on this foundation, we develop an agent-native, reasoning-aware scientific retrieval system that retrieves claims together with their reasoning chains and sources, enabling agents to inspect and reuse the evidence underlying scientific conclusions. Across benchmarks, agents using LKM retrieve more evidence, cite more faithfully, and answer scientific questions more accurately: LKM nearly doubles the known supporting and contradicting evidence retrieved on SciFact-Open (818 versus 443 claim-paper pairs), reasoning graphs raise citation F1 on ScholarQABench by more than 5 points over the same retrieved papers, and LKM retrieval improves a fixed answering model by 9.3, 4.2, and 14.7 points over no retrieval on ChemBench, PubMedQA, and SciBench. LKM lays the foundation for a scientific ecosystem in which AI scientists not only recall accumulated knowledge but also extend it, returning new questions, workflows, and evidence to a memory that every subsequent investigation can build on.

Abstract

AI-Assisted Scientific Research

Explore similar work

Large Knowledge Model: A Knowledge Foundation for Agentic Science at Scale

EGT-KG: Evidence-Grounded Typed KG Retrieval for Practical Scientific QA with Small Language Models

Eigen-Agent: Adaptive Multi-Agent Scientific Reasoning with Monitor-Based RAG