PubMed Reasoner: Dynamic Reasoning-based Retrieval for Evidence-Grounded Biomedical Question Answering

Yiqing Zhang; Xiaozhong Liu; Fabricio Murai

cs.CLMar 28, 2026

PubMed Reasoner: Dynamic Reasoning-based Retrieval for Evidence-Grounded Biomedical Question Answering

Authors: Yiqing Zhang, Xiaozhong Liu, Fabricio Murai

Organizations: PayPal · Worcester Polytechnic Institute

Abstract

Trustworthy biomedical question answering (QA) systems must not only provide accurate answers but also justify them with current, verifiable evidence. Retrieval-augmented approaches partially address this gap but lack mechanisms to iteratively refine poor queries, whereas self-reflection methods kick in only after full retrieval is completed. In this context, we introduce PubMed Reasoner, a biomedical QA agent composed of three stages: self-critic query refinement evaluates MeSH terms for coverage, alignment, and redundancy to enhance PubMed queries based on partial (metadata) retrieval; reflective retrieval processes articles in batches until sufficient evidence is gathered; and evidence-grounded response generation produces answers with explicit citations. PubMed Reasoner with a GPT-4o backbone achieves 78.32% accuracy on PubMedQA, slightly surpassing human experts, and showing consistent gains on MMLU Clinical Knowledge. Moreover, LLM-as-judge evaluations prefer our responses across: reasoning soundness, evidence grounding, clinical relevance, and trustworthiness. By orchestrating retrieval-first reasoning over authoritative sources, our approach provides practical assistance to clinicians and biomedical researchers while controlling compute and token costs.

Figures & tables

Figure 1: RAG, Self-reflection vs. PubMed Reasoner. (a) RAG baselines: uses few-shot exemplars or custom databases but lack retrieval feedback. (b) Self-reflection agents: inspired our proposal; generates responses first and only reflects after completion. (c) PubMed Reasoner: a search-first approach that performs self-critic query refinement, reflective article retrieval in batches w/ early stopping once evidence is sufficient, and evidence-grounded response generation with explicit citations.

Figure 2: PubMed Reasoner stages. (1) Search with Self-Critic Query Refinement. From a user question, MeSH terms and a structured query are proposed. Self-critic evaluates each term for coverage , alignment , and redundancy , iteratively refining the query. (2) Reflective Article Retrieval with Early Stopping. PubMed Reasoner queries PubMed, filters results by title/abstract, and extracts supporting evidence in batches, checking whether accumulated evidence sufficiently covers the question; if so, retrieval is terminated early to save tokens and avoid unnecessary processing. (3) Evidence-Grounded Response Generation. Retained evidence is synthesized into final answer with explicit inline citations, ensuring factual grounding and traceability. More case studies are provided in App. G .

Method		PubMedQA	MMLU-CK
Without retrieval	Gemini-2.5 Pro	75.64%	60.52%
	GPT4o (leaderboard)	75.20%	60.64%
	Human performance	78.00%	–
RAG method	with Gemini	72.30%	58.94%
RAG method	with GPT	73.28%	58.26%
Self- reflection	with Gemini	77.08%	60.67%

Table 1: Accuracy on PubMedQA and MMLU-CK test sets for PubMed Reasoner variants, strong LLMs, RAG method, self-reflection agents, and human performance. Best result per column is bold-faced ; second best is underlined . Additional results for PubMed Reasoner with Qwen variants are provided in App. F .

Metric	PubMedQA					MMLU-CK
	Loss / Tie / Win (%)			Avg. Likert (1–5)		Loss / Tie / Win (%)			Avg. Likert (1–5)
	Gemini	Tie	Ours	Gemini	Ours	Gemini	Tie	Ours	Gemini	Ours
Reasoning Soundness	39.7	5.7	54.6	3.416	3.584	44.0	10.8	45.2	3.307	3.699
Evidence Grounding	40.8	3.8	55.4	3.421	3.601	25.2	11.7	64.1	3.209	3.595
Clinical Relevance	39.2	7.4	53.4	3.438	3.584	34.4	20.0	45.6	3.525	3.732
Trustworthiness	38.7	5.4	55.9	3.424	3.587	35.3	15.9	48.8	3.386	3.712

Table 2: Explanation quality evaluation on PubMedQA (left) and MMLU-CK (right) test sets: Gemini w/o retrieval vs. PubMed Reasoner. Win/tie/loss rates from pairwise comparisons judged by GPT-4; average Likert scores (1–5).

	PubMedQA				MMLU-CK
	Input	Output	LLM	Search	Input	Output	LLM	Search
Method	Tokens	Tokens	Calls	Calls	Tokens	Tokens	Calls	Calls
LLM	542.70	144.90	1	0	90.24	98.64	1	0
RAG method	$\times$ 1.64	$\times$ 2.40	2	1	$\times$ 4.00	$\times$ 3.06	2	1
Self-reflection	$\times$ 225.27	$\times$ 13.60	13.08	3.52	$\times$ 1914.53	$\times$ 24.35	14.61	4.77
PubMed Reasoner	$\times$ 97.25	$\times$ 12.67	7.61	2.49	$\times$ 1098.26	$\times$ 24.86	10.72	3.59

Table 3: Cost comparison on PubMedQA and MMLU-CK using a shared GPT-4 backbone for PubMed Reasoner and baselines. For PubMedQA, input and output token counts are reported as ratios relative to the direct LLM baseline. For each metric, the lower value between the self-reflection agent and PubMed Reasoner is bolded .

Method	Precision	Recall
Gemini-2.5 Pro	0.9422	0.6848
+Self-reflection	0.9745	0.7900
+PubMed Reasoner	0.9826	0.8025
GPT-4o	0.9874	0.7052
+Self-reflection	0.9562	0.7928
+PubMed Reasoner	0.9868	0.8532

Table 4: Search quality on PubMedQA. Precision/recall computed by comparing set of predicted MeSH terms from the final query against gold MeSH annotations.

Method / Config.	Accuracy	EGR
PubMed Reasoner	77.26%	82.64%
w/o self-critic	75.60%	64.46%

Table 5: Ablation on PubMedQA with PubMed Reasoner (Gemini): impact of self-critic module. Evidence-Grounded Response Rate (EGR) : fraction of questions whose final response cites

\geq 1

extracted findings that pass coverage and alignment checks in reflective stage.

Metric	w/o RR	Ours
Accuracy (%)	77.24	77.26
Reasoning Soundness	3.422	3.554
Evidence Grounding	3.554	3.427
Clinical Relevance	3.443	3.573
Trustworthiness	3.432	3.432

Table 6: Ablation on PubMedQA with PubMed Reasoner (Gemini): impact of reflective retrieval (RR).

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Model/Method	PubMedQA	MMLU-CK
Qwen2.5-1.5B
Without retrieval	68.08%	55.67%
Self-reflection agent	68.06%	51.44%
PubMed Reasoner	67.24%	51.42%
Qwen2.5-7B
Without retrieval	70.20%	56.64%

Appendix

Table 7: Accuracy on PubMedQA and MMLU-CK test sets for PubMed Reasoner Qwen2.5 variants, strong LLMs, and self-reflection agents. Best result per column is bold-faced .

Figure 3: Effectiveness of the reflective integration stage. Distribution of retrieval depth before early stopping.

Metric	Loss / Tie / Win (%)			Avg. Likert (1–5)
Metric	w/o	Tie	Ours	w/o	Ours
Reasoning Soundness	19.6	4.0	76.4	3.422	3.554
Evidence Grounding	26.2	54.3	19.4	3.554	3.427
Clinical Relevance	17.6	23.8	58.6	3.443	3.573
Trust- worthiness	10.0	75.8	14.2	3.432	3.432

Appendix

Table 8: Explanation quality evaluation on PubMedQA test sets. Left: win/tie/loss rates from pairwise comparisons judged by GPT-4.1. Right: average Likert scores (1–5). We compare PubMed Reasoner with and without the reflective retrieval stage.

Method	Accuracy (%)
Abstract Only	67.20
Paper Summary	64.12
PubMed Reasoner	77.26

Appendix

Table 9: Ablation study comparing reduced-context variants to our selective early-stopping framework on PubMedQA.

Biomedical QA

Biomedical question answering develops systems that answer health and life-science questions using clinical knowledge, research literature, and structured data.

See all 51 papers

BELIEF: Structured Evidence Modeling and Uncertainty-Aware Fusion for Biomedical Question Answering
May 17, 2026Chang Zong, Hao Ning, Siliang Tang +2Healthcare Evidence-Grounded Reasoning
School of Computer Science and Technology, Zhejiang University of Science and Technology, Hangzhou 310023, China · College of Artificial Intelligence, Zhejiang University, Hangzhou 310027, China · Zhejiang Key Laboratory of Biomedical Intelligent Computing Technology, Zhejiang University of Science and Technology, Hangzhou 310023, China
When Retrieval Helps and Distracts: Evaluating Evidence-Generating LLMs for Biomedical Claim Verification
Aug 2, 2026Pritam Deka, Prabhjot SinghExplainable Artificial Intelligence Automated Fact-Checking
Queen’s University Belfast United Kingdom · University of Texas at Austin Austin, Texas, USA
Hybrid-IR: Dual-Path Hybrid Retrieval with Iterative Reasoning for Complex Medical Question Answering
Jun 24, 2026Sheng Wan, Jiahui Zhang, Zicheng Zhao +1Retrieval-Augmented Generation Iterative RAG
College of Artificial Intelligence, Nanjing Agricultural University, Nanjing, China · School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China

May 17, 2026J/K move · Enter open · S save

BELIEF: Structured Evidence Modeling and Uncertainty-Aware Fusion for Biomedical Question Answering

Chang Zong, Hao Ning, Siliang Tang, Jie Huang+1

School of Computer Science and Technology, Zhejiang University of Science and Technology, Hangzhou 310023, China · College of Artificial Intelligence, Zhejiang University, Hangzhou 310027, China · Zhejiang Key Laboratory of Biomedical Intelligent Computing Technology, Zhejiang University of Science and Technology, Hangzhou 310023, China

Healthcare Evidence-Grounded Reasoning Dempster-Shafer Theory Biomedical QA LLM Uncertainty Estimation Medical LLMs+1

Open paper AskPDF arXiv

Biomedical question answering often requires decisions from retrieved literature whose relevance, quality, and support for candidate answers are uneven. Most retrieval-augmented large language model (LLM) methods feed this literature to the model as flat text, leaving evidence reliability and remaining uncertainty largely implicit. We propose BELIEF, a structured evidence modeling and uncertainty-aware fusion framework for closed-set biomedical question answering. Rather than treating retrieved documents as undifferentiated context, BELIEF converts them into evidence objects that record clinical attributes, source quality, question relevance, support strength, and the associated candidate hypothesis. These evidence objects provide a shared basis for two complementary reasoning paths. The symbolic path constructs reliability-weighted basic probability assignments based on Dempster--Shafer (D-S) theory over a finite answer space and performs uncertainty-aware symbolic evidence fusion to estimate belief and residual uncertainty. The neural path uses the same structured evidence for LLM-based semantic inference, while a reliability-aware arbitration module reconciles the symbolic and neural outputs according to belief strength, uncertainty, evidence reliability, and semantic consistency. Experiments on PubMedQA, MedQA, and MedMCQA with five general-purpose LLM backbones show that BELIEF obtains the best result in 25 of 30 backbone--dataset--metric settings. Comparisons with biomedical-domain models indicate that BELIEF is competitive on MedQA and MedMCQA, while specialized biomedical pretraining remains advantageous on PubMedQA. Ablation, complementarity, uncertainty-stratified, and cost analyses further show that BELIEF improves retrieved-evidence utilization by making evidence structure, path disagreement, and decision uncertainty explicit.

Abstract

Biomedical QA

Explore similar work

BELIEF: Structured Evidence Modeling and Uncertainty-Aware Fusion for Biomedical Question Answering

When Retrieval Helps and Distracts: Evaluating Evidence-Generating LLMs for Biomedical Claim Verification

Hybrid-IR: Dual-Path Hybrid Retrieval with Iterative Reasoning for Complex Medical Question Answering