Taking the PULSE of Medical Reasoning: Step-Level Retrieval Timing for Knowledge-Grounded Inference

Danyang Zhao; Sendong Zhao; Yanrui Du; Haochun Wang; Ming Ma; Wenhao Li; Jingbo Wang; Jiatong Liu

cs.CLOct 14, 2026

Taking the PULSE of Medical Reasoning: Step-Level Retrieval Timing for Knowledge-Grounded Inference

Authors: Danyang Zhao, Sendong Zhao, Yanrui Du, Haochun Wang, Ming Ma, Wenhao Li, Jingbo Wang, Jiatong Liu,

Organizations: Harbin Institute of Technology

Abstract

Reliable medical reasoning requires LLMs to dynamically access external biomedical evidence to improve factual trustworthiness and reasoning accuracy. However, indiscriminate retrieval can introduce noise and latency, raising a critical retrieval-timing problem: at each reasoning step, the system must decide whether the model's internal knowledge is reliable or external evidence is needed. We propose PUlSE, a lightweight probing method that uses the model's internal representations as intrinsic signals for step-level retrieval control. Our posterior trajectory analysis reveals that erroneous steps exhibit substantially higher inter-layer volatility than correct ones, suggesting unstable internal knowledge. Based on this finding, PUlSE trains a compact probe on inter-layer geometric features to detect factual errors upon step completion, and uses it as an adaptive trigger for retrieval and step regeneration. Across four medical benchmarks, PUlSE-timed retrieval matches exhaustive per-step retrieval in accuracy while reducing retrieval calls by 72%, and can transfer to an unseen legal domain, indicating potential domain-agnostic generality. Overall, our study provides an initial step toward shifting retrieval-trigger design from surface-level confidence estimates to internal activation dynamics for adaptive access.

Figures & tables

Figure 1: Layer-wise PCA trajectories of hidden states. Bold lines: group means; faint lines: individual samples. (a) Unmatched. (b) Complexity-matched.

Figure 2: Overview of PUlSE . Layer-wise geometric trajectory features of each reasoning step are fed into an MLP probe to detect factual errors. Flagged steps trigger retrieval, whose accuracy gain interventionally validates the detection. Output-level signals (purple) serve as baselines.

Backbone: Qwen2.5-7B-Instruct
	MedMCQA (ID)					MMLU-Pro (Biology) (OOD)
Method	Prec.	Rec.	F1	AUROC	PR AUC	Prec.	Rec.	F1	AUROC	PR AUC
PPL	.4433	.6160	.5155	.6112	.4768	.4643	.0138	.0268	.4776	.2200
MaxProb	.4480	.5436	.4912	.6043	.4727	.5417	.0138	.0269	.4838	.2224
Entropy	.4538	.6289	.5272	.6264	.4932	.2727	.0477	.0812	.4695	.2177
CoE-R	.4419	.5436	.4875	.5843	.4204	.2774	.6140	.3822	.5981	.2808

Table 1: Factual error detection performance on Qwen2.5 and Llama3.1. Threshold-independent metrics (AUROC, PR AUC) are reported as primary metrics. Baseline thresholds are selected via Youden’s J statistic; probe-based methods use a default threshold of 0.5. Best results per dataset are in bold .

Backbone: Qwen2.5-7B-Instruct
	MedMCQA		JAMA		MMLU-Pro (Health)		MMLU-Pro (Biology)
Method	Acc (%)	#Ret $\downarrow$	Acc (%)	#Ret $\downarrow$	Acc (%)	#Ret $\downarrow$	Acc (%)	#Ret $\downarrow$
CoT	57.00	0	50.30	0	57.21	0	72.52	0
IRCoT	85.33	5.06	53.87	4.38	65.28	5.34	72.80	5.51
DRAGIN	58.33	3.52	53.74	5.70	58.19	4.16	73.78	2.76
SeaKR	79.00	2.43	54.91	4.06	58.07	2.83	63.88	2.56

Table 2: Main results on four medical QA benchmarks using BM25 retrieval. Acc: accuracy (%). #Ret: average number of retrieval calls per test instance. Bold : best; underline : second best within each backbone group (excluding CoT for #Ret).

Figure 3: Accuracy–efficiency Pareto front on the four medical QA benchmarks (averaged). Top : Qwen2.5-7B-Instruct; Bottom : Llama3.1-8B-Instruct. The $x$ -axis reports the mean number of retrieval calls per question and the $y$ -axis reports average accuracy.

	Detection		End-to-End
Feature	AUROC	PR AUC	Acc(%)	Avg #Ret
Mag	.6565	.3547	75.31	2.51
Angle	.6866	.3903	75.59	2.94
Mag + Ang	.7273	.4316	76.43	2.62

Table 3: Feature ablation on MMLU-Pro (Biology) with Qwen2.5-7B-Instruct. Probe is trained on MedMCQA and evaluated out-of-distribution on MMLU-Pro (Biology).

Figure 4: Layer-interval AUROC on the MedMCQA training-validation split (Qwen2.5-7B-Instruct).

Method	Acc (%)	Avg #Ret
CoT	28.52	0
IRCoT	39.96	4.48
PUlSE	42.23	2.36

Table 4: Cross-domain transfer to MMLU-Pro (Law) with Qwen2.5-7B-Instruct. The probe is trained solely on medical data and applied without adaptation.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Role	Questions	Avg Steps	Domain
Train/Valid	1173	3.7	Medicine
Test (ID)	300	4.9	Medicine
Test (OOD)	717	5.9	Biology

Appendix

Table 5: Question-level statistics across all datasets.

Split	Total	No-Error	Error	Error Ratio
Train/Valid	4348	3216	1132	26.0%
Test (ID)	1483	944	539	36.4%
Test (OOD)	4257	3314	943	22.1%

Appendix

Table 6: Step-level statistics for probe training and evaluation. Error Ratio denotes the proportion of steps labeled as containing a factual error.

Panel of Figure 5 — Figure 5: A complete annotated example showing the question, segmented reasoning steps, and step-level factual error labels with justifications.

Panel of Figure 6 — Figure 6: Annotation Prompt Template

				PULSE		ReProbe
Dataset	Subset	Steps	Error (%)	AUROC	PR-AUC	AUROC	PR-AUC
MedMCQA	All aligned	1,464	36.3	.746	.583	.728	.588
	Non-hedging	1,365	35.1	.744	.570	.732	.575
MMLU-Pro (Biology)	All aligned	4,097	22.4	.712	.417	.694	.438
	Non-hedging	3,662	21.2	.704	.386	.697	.431
MedBullets †	All aligned	844	35.4	.752	.564	.736	.609

Appendix

Table 7: Detection before and after filtering explicit uncertainty expressions. “All aligned” is the diagnostic pool before filtering; “Non-hedging” excludes lexicon-matched steps. PR-AUC uses trapezoidal integration. Error prevalence is reported because filtering changes class balance.

PULSE variant	Baseline	Difference	95% CI	Wins/losses	Exact $p$	Holm $p$
Step	IRCoT	+4.70	[2.38, 7.15]	209/138	.000163	.000490
Step	ReProbe	+2.38	[0.40, 4.43]	140/104	.024849	.024849
Front	IRCoT	+4.96	[2.65, 7.35]	207/132	.000055	.000219
Front	ReProbe	+2.65	[0.66, 4.63]	139/99	.011315	.022630

Appendix

Table 8: Paired JAMA comparisons (

n=1,511

), retaining the main table’s answer-scoring convention. Differences and confidence intervals are in percentage points. Wins/losses count questions answered correctly only by PULSE or only by the baseline. All four comparisons remain significant after Holm correction within this family.

Method	Acc. (%)	Avg. Ret.
CoT (no retrieval)	51.40	0.00
PUlSE (Ours)	47.60	2.25
IRCoT	45.40	3.85
SeaKR	42.40	3.43

Appendix

Table 9: Accuracy (%) on 500 JAMA questions using the public MedRAG Textbook corpus. All retrieval-augmented methods perform below the no-retrieval CoT baseline in this corpus–benchmark setting.

Method	Acc. (%)	Avg. Ret.
CoT (no retrieval)	61.67	0.00
IRCoT	61.33	5.57
SeaKR	61.33	3.66
PUlSE (Ours)	63.50	0.96

Appendix

Table 10: Results on the reserved MedQA sample using the public StatPearls corpus (

n=600

). Avg. Ret. denotes the average number of retrieval calls per question.

Panel of Figure 7 — Figure 7: Prompt template for corpus construction.

Benchmark	# Statements
JAMA	20,225
MMLUPro_Health	4,522
MMLUPro_Biology	4,262
MedMCQA (test, 300 questions)	1,503

Appendix

Table 11: Corpus statistics (number of unique knowledge statements per benchmark).

Method	Acc. (%) $\uparrow$	Avg. Ret. $\downarrow$
CoT †	50.30	0.00
IRCoT †	53.87	4.38
Random ( $p=0.55$ )	55.32	2.81
PUlSE ${}_{\mathrm{front}}^{\dagger}$	58.83	3.00

Appendix

Table 12: Diagnostic random-trigger control on JAMA with Qwen2.5-7B-Instruct (

n=1{,}511

). Random uses a per-step trigger probability of

0.55

and is averaged over three seeds.

\dagger

denotes results reproduced from Table 2 .

Panel of Figure 8 — Figure 8: Case-study question from JAMA Clinical Challenge.

Panel of Figure 9 — Figure 9: Case study of PUlSE on a JAMA Clinical Challenge question.

Panel of Figure 10 — Figure 10: Case study of ReProbe on the same question.

Adaptive Retrieval

Adaptive retrieval adjusts what to retrieve, which strategy to use, and when to stop based on query needs and available evidence.

See all 37 papers

When to Retrieve During Reasoning: Adaptive Retrieval for Large Reasoning Models
Apr 29, 2026Dongxin Guo, Jikun Wu, Siu Ming YiuRetrieval-Augmented Generation Retrieval-Augmented Reasoning
The University of Hong Kong Hong Kong, China · Brain Investing Limited Hong Kong, China · Stellaris AI Limited Hong Kong, China
Learning When to Trust Medical Knowledge: Regression-Safe Routing for Clinical Reasoning
Oct 14, 2026Hui Zhang, Yujia Zeng, Yuhao XinMedical Reasoning LLM Routing
Beijing Institute of Technology · Shanghai Jiao Tong University · Xi’an Jiaotong University
PubMed Reasoner: Dynamic Reasoning-based Retrieval for Evidence-Grounded Biomedical Question Answering
Mar 28, 2026Yiqing Zhang, Xiaozhong Liu, Fabricio MuraiBiomedical QA Agentic RAG
PayPal · Worcester Polytechnic Institute

Apr 29, 2026J/K move · Enter open · S save

When to Retrieve During Reasoning: Adaptive Retrieval for Large Reasoning Models

Dongxin Guo, Jikun Wu, Siu Ming Yiu

The University of Hong Kong Hong Kong, China · Brain Investing Limited Hong Kong, China · Stellaris AI Limited Hong Kong, China

Retrieval-Augmented Generation Retrieval-Augmented Reasoning Multi-Hop Reasoning Large Reasoning Models

Open paper AskPDF arXiv

Large reasoning models such as DeepSeek-R1 and OpenAI o1 generate extended chains of thought spanning thousands of tokens, yet their integration with retrieval-augmented generation (RAG) remains fundamentally misaligned. Current RAG systems optimize for providing context before reasoning begins, while reasoning models require evidence injection during multi-step inference chains. We introduce ReaLM-Retrieve, a reasoning-aware retrieval framework that addresses this mismatch through three key innovations: (1) a step-level uncertainty detector that identifies knowledge gaps at reasoning-step granularity rather than token or sentence level; (2) a retrieval intervention policy that learns when external evidence maximally benefits ongoing reasoning; and (3) an efficiency-optimized integration mechanism that reduces per-retrieval overhead by 3.2x compared to naive integration. Experiments on MuSiQue, HotpotQA, and 2WikiMultiHopQA demonstrate that ReaLM-Retrieve achieves on average 10.1% absolute improvement in answer F1 over standard RAG (range: 9.0-11.8% across the three benchmarks) while reducing retrieval calls by 47% compared to fixed-interval approaches like IRCoT (all improvements significant at p<0.01, paired bootstrap). On the challenging MuSiQue benchmark requiring 2-4 hop reasoning, our method achieves 71.2% F1 with an average of only 1.8 retrieval calls per question. Analysis shows that ReaLM-Retrieve also improves retrieval quality itself, achieving 81.3% Recall@5 with consistently higher precision and MRR than fixed-interval baselines on supporting evidence, establishing new state-of-the-art efficiency-accuracy trade-offs for reasoning-intensive retrieval tasks.

Abstract

Adaptive Retrieval

Explore similar work

When to Retrieve During Reasoning: Adaptive Retrieval for Large Reasoning Models

Learning When to Trust Medical Knowledge: Regression-Safe Routing for Clinical Reasoning

PubMed Reasoner: Dynamic Reasoning-based Retrieval for Evidence-Grounded Biomedical Question Answering