CARF: Context-Aware Reasoning Supervision for Scientific Evidence Screening

Nishtha Agarwal; Adam Visokay; Astha Wahi; Kwabena Owusu Aninkora; Amy Gu; Tyler McCormick; Li Liu

cs.CLOct 27, 2026

CARF: Context-Aware Reasoning Supervision for Scientific Evidence Screening

Authors: Nishtha Agarwal, Adam Visokay, Astha Wahi, Kwabena Owusu Aninkora, Amy Gu, Tyler McCormick, Li Liu

Organizations: University of Washington · University of California, Berkeley · Johns Hopkins University · SUNY Downstate Health Sciences University ·

Abstract

Systematic reviews synthesize biomedical evidence to inform clinical and public-health decision-making, but require domain experts to manually screen thousands of studies against detailed eligibility criteria. We investigate whether language models can assist with this bottleneck by learning from expert feedback on title and abstract screening decisions and model-generated reasoning. We introduce the Context-Aware Reasoning Framework (CARF), a human-in-the-loop framework that uses expert feedback on model-generated reasoning to adapt language models to context-dependent screening tasks. We evaluate CARF on systematic reviews of verbal autopsy and cause-of-death ascertainment methods used in global public health mortality surveillance. CARF shows competitive screening performance compared to evaluated models while retaining high sensitivity, and is more inclined to retain ambiguous, context-dependent studies for human review. Together, these findings suggest that expert-corrected reasoning is a promising supervision signal for context-dependent scientific screening, while retaining a role for human oversight.

Figures & tables

Figure 1: Overview of the Context-Aware Reasoning Framework (CARF) applied to the title and abstract screening stage of a systematic review, illustrating four stages: blind model screening, expert correction of disagreements, construction of an expert-reasoned dataset, and reasoning-based model adaptation through LoRA fine-tuning.

Method	Rec.	95% CI	Spec.	Prec.	F1	FP
Fine-tuned (ours, 14B)
CARF	0.964	0.923–0.987	0.857	0.573	0.719	119
Reasoning (teacher-distilled)	0.964	0.923–0.987	0.610	0.330	0.492	325
Label-only	0.910	0.855–0.949	0.873	0.588	0.714	106
Zero-shot frontier
DeepSeek-V3.2	0.940	0.892–0.971	0.821	0.511	0.662	149

Table 1: Include-class screening performance on the held-out test set (

n{=}1{,}000

; 166 eligible studies). CARF achieves the most favorable combination of recall (0.964) and F1 score (0.719) among all evaluated methods, outperforming zero-shot frontier models on both metrics. Fine-tuned results are from a single training run; zero-shot results are from a single inference pass. 95% confidence intervals for recall use the Clopper–Pearson method.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Rec.	Spec.	Prec.	F1	FP	Missed
Fine-tuned (ours, 14B)
Label Conditioned Reasoning	0.958	0.787	0.472	0.632	178	7/166
Label-only	0.910	0.873	0.588	0.714	106	15/166
Zero-shot frontier
Claude Sonnet-5	0.922	0.860	0.567	0.702	117	13/166
DeepSeek-V3.2	0.940	0.821	0.511	0.662	149	10/166

Appendix

Table F1: Include-class screening performance of the label-conditioned reasoning variant compared with the label-only fine-tuned model and zero-shot frontier models on the held-out test set.

Model	FP	Ambig.	Clear	% ambig. (95% CI)
PubMedBERT	56	19	37	33.9 (21.8–47.8)
CARF	119	36	83	30.3 (22.2–39.3)
Claude Sonnet-5	117	33	84	28.2 (20.3–37.3)
Grok-4.3	123	33	90	26.8 (19.2–35.6)
DeepSeek-V3.2	149	40	109	26.8 (19.9–34.7)
Label-only	106	26	80	24.5 (16.7–33.8)

Appendix

Table G1: False positives of each model on the held-out test set, split by the blinded expert rating and ordered by the share rated ambiguous. Ambig.: ambiguous; Clear: clearly ineligible.

Figure G1: False positives by model, split into clearly ineligible and ambiguous articles by a blinded expert rating (single rater; 347 pooled articles).

Rating	Teacher-dist. FP	Also in CARF	Removed
Clearly ineligible	255	82 (32%)	173 (68%)
Ambiguous	70	35 (50%)	35 (50%)

Appendix

Table G2: False positives of the teacher-distilled reasoning model and whether CARF also includes them.

Feedback notes	Rec.	95% CI	Spec.	Prec.	F1	FP
100	0.922	0.870–0.958	0.871	0.586	0.717	108
500	0.964	0.923–0.987	0.836	0.539	0.691	137
1,346 (full)	0.964	0.923–0.987	0.857	0.573	0.719	119

Appendix

Table H1: Screening performance of CARF trained with different amounts of expert feedback, on the 1000 test articles (166 true includes). Single training run per configuration. 95% CI: exact Clopper–Pearson interval for recall.

Figure H1: Include recall and specificity for CARF trained with 100, 500, and all 1,346 expert feedback notes. Specificity points are labelled with the corresponding number of false positives (FP). The full model is evaluated on all 1,000 test articles, as in Table 1 ; Single training run per configuration.

Panel of Figure 9 — Figure I1: Screening outcomes on excluded test articles by whether the model self-corrected its reasoning.

Systematic Review Automation

See all 22 papers

Fine-Tuning A Large Language Model for Systematic Review Screening
Mar 25, 2026Kweku Yamoah, Noah Schroeder, Emmanuel Dorley +2Systematic Review Automation LLM Fine-Tuning
University of Florida, Gainesville, FL, 32611
SciLitBench: Benchmark and Design Principles for LLM-Powered Systematic Literature Reviews
Aug 29, 2026Miguel Zabaleta, Baihan LinSystematic Review Automation LLM Evaluation
Department of AI and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, NY, USA
ReviewPilot: A Multi-Agent System for Screening and Data Extraction in Systematic Reviews
Oct 15, 2026Zhaoqian Xue, Shaoze Zhou, Xuexin Li +10Systematic Review Automation AI-Assisted Scientific Research
University of Arizona · Florida International University · Vanderbilt University +7

Mar 25, 2026J/K move · Enter open · S save

Fine-Tuning A Large Language Model for Systematic Review Screening

Kweku Yamoah, Noah Schroeder, Emmanuel Dorley, Neha Rani+1

University of Florida, Gainesville, FL, 32611

Systematic Review Automation LLM Fine-Tuning Evidence Selection LLM Evaluation

Open paper AskPDF arXiv

Systematic reviews traditionally have taken considerable amounts of human time and energy to complete, in part due to the extensive number of titles and abstracts that must be reviewed for potential inclusion. Recently, researchers have begun to explore how to use large language models (LLMs) to make this process more efficient. However, research to date has shown inconsistent results. We posit this is because prompting alone may not provide sufficient context for the model(s) to perform well. In this study, we fine-tune a small 1.2 billion parameter open-weight LLM specifically for study screening in the context of a systematic review in which humans rated more than 8500 titles and abstracts for potential inclusion. Our results showed strong performance improvements from the fine-tuned model, with the weighted F1 score improving 80.79% compared to the base model. When run on the full dataset of 8,277 studies, the fine-tuned model had 86.40% agreement with the human coder, a 91.18% true positive rate, a 86.38% true negative rate, and perfect agreement across multiple inference runs. Taken together, our results show that there is promise for fine-tuning LLMs for title and abstract screening in large-scale systematic reviews.

Abstract

Systematic Review Automation

Explore similar work

Fine-Tuning A Large Language Model for Systematic Review Screening

SciLitBench: Benchmark and Design Principles for LLM-Powered Systematic Literature Reviews

ReviewPilot: A Multi-Agent System for Screening and Data Extraction in Systematic Reviews