Counting Clues: A Lightweight Probabilistic Baseline Can Match an LLM

Furong Jia; Yuan Pu; Finn Guo; Monica Agrawal

cs.CLDec 14, 2025

Counting Clues: A Lightweight Probabilistic Baseline Can Match an LLM

Authors: Furong Jia, Yuan Pu, Finn Guo, Monica Agrawal

Organizations: Duke University

Abstract

Large language models (LLMs) excel on multiple-choice clinical diagnosis benchmarks, yet it is unclear how much of this performance reflects underlying probabilistic reasoning. We study this through questions from MedQA that ask for the most likely diagnosis. We introduce Frequency-Based Probabilistic Ranker (FBPR), a lightweight method that scores options with a smoothed Naive Bayes over concept-diagnosis co-occurrence statistics from a large corpus. When co-occurrence statistics are sourced from Dolma, the pretraining corpus for OLMo, and RedPajama, an open reproduction of the LLaMA pretraining-data recipe, FBPR achieves comparable performance to the corresponding LLMs. Furthermore, direct LLM inference and FBPR exhibit complementary strengths, correctly answering largely different subsets of questions with overlap barely above chance. Combining them with a simple hybrid method yields a notable gain over either method alone. These findings show that explicit probabilistic baselines can account for a substantial portion of the benchmark performance on diagnosis questions, while serving as historically grounded reference points and complementary signals for hybridization.

Figures & tables

Figure 1: Frequency-Based Probabilistic Ranker: (1) concept extraction from the question, (2) corpus frequency retrieval, and (3) compute a Naive Bayes score to select the most likely diagnosis.

Method	Accuracy
Random Chance	20%
FBPR on Dolma	46.7%
OLMo Instruct 7B	44.1%
Hybrid (OLMo + Dolma)	47.3%
FBPR on RedPajama	44.5%
LLaMA 65B	47.0%

Table 1: Prediction accuracy on five-option MedQA diagnosis subset (

n

= 719).

Figure 2: Distribution of the rank of the correct diagnosis of each question by Dolma-based FBPR with

k=5

on MedQA diagnosis subset.

Figure 3: A successful example of FBPR on a MedQA diagnosis question. The diagnosis occurrence and (diagnosis, concept) co-occurrence frequencies are retrieved by Infini-gram from v4_rpj_llama_s4 index, i.e., from the RedPajama corpus. By Infini-gram design, some of these frequencies are estimates.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Accuracy
Random chance	25.0%
FBPR on Dolma ( $k=5$ )	52.7%
FBPR on Dolma (unrestricted $k$ )	52.4%
OLMo Instruct 7B	54.9%

Appendix

Table 2: Prediction accuracy on the four-option MedMCQA diagnosis subset (

n=639

). Concepts for FBPR are extracted by Qwen2.5-32B-Instruct (Appendix 3 ), and OLMo Instruct 7B was evaluated with the zero-shot prompt in Appendix D

Figure 4: Distribution of the number of concepts per question extracted by GPT-4o when prompt contains no restriction.

Text summary	GPT-4o phrase-level concepts	Google Healthcare API entities
Patient with double vision, difficulty climbing stairs, and weakness when brushing hair	double vision; difficulty climbing stairs; weakness brushing hair; symptoms worse after exercise; symptoms improve with rest; drooping right upper eyelid	double vision; weakness; hair; symptoms; worse; drooping; eyelid; diminished; motor strength; upper extremities
Patient with Crohn disease on natalizumab with difficulty walking and ataxia	crohn disease; difficulty walking; gait ataxia; limb ataxia; natalizumab	Crohn disease; difficulty walking; Crohn disease; natalizumab; intestinal symptoms; severe; unresponsive; therapies; gait; limb; ataxia

Appendix

Table 3: Examples illustrating why phrase-level extraction is better matched to exact-match co-occurrence retrieval than fragmented clinical entity extraction.

Extractor	Corpus	$k$	all	Step 1	Step 2&3
GPT-4o	Dolma	5	46.7%	51.2%	44.3%
GPT-4o	Dolma	unrestricted	45.1%	47.7%	43.6%
GPT-4o	RedPajama	5	44.5%	46.1%	43.6%
GPT-4o	RedPajama	unrestricted	40.9%	40.2%	41.3%
Qwen2.5-32B-Instruct	Dolma	5	44.0%	46.1%	42.8%
Qwen2.5-32B-Instruct	Dolma	unrestricted	43.4%	47.3%	41.3%

Appendix

Table 4: Accuracy of FBPR on MedQA diagnosis subset (

n

=719), based on occurrence frequency from different corpora and with two settings of

k

, stratified on the medical licensing exam domain. Concepts are extracted by GPT-4o (main setting) or by Qwen2.5-32B-Instruct (Appendix 3 ).

	LLaMA 65B + RedPajama			OLMo Instruct + Dolma
Method	Acc	$\Delta_{\text{LLM}}$	$\Delta_{\text{FBPR}}$	Acc	$\Delta_{\text{LLM}}$	$\Delta_{\text{FBPR}}$
LLM	47.0	—	—	44.1	—	—
FBPR alone	44.5	—	—	46.7	—	—
Rank gate (main)	50.8	+3.7	+6.3	47.3	+3.2	+0.6
Confidence gate, $\tau{=}0.99$	48.4	+1.4	+3.9	49.2	+5.2	+2.5
Confidence gate, $\tau{=}0.95$	47.4	+0.4	+2.9	48.8	+4.7	+2.1

Appendix

Table 5: Accuracy (%) and accuracy change relative to the LLM baseline (

\Delta_{\text{LLM}}

) and to FBPR-alone baseline (

\Delta_{\text{FBPR}}

) for the three hybrid variants on both corpus-model pairs at three confidence thresholds

\tau

. Bold accuracy marks each pair’s strongest configuration.

	LLaMA 65B + RedPajama			OLMo Instruct + Dolma
$m$	Acc	$\Delta_{\text{LLM}}$	$\Delta_{\text{FBPR}}$	Acc	$\Delta_{\text{LLM}}$	$\Delta_{\text{FBPR}}$
1	44.5	$-2.5$	$\phantom{+}0.0$	46.7	$+2.6$	$\phantom{+}0.0$
2	48.3	$+1.3$	$+3.8$	49.4	$+5.3$	$+2.6$
3 (main)	50.8	$+3.7$	$+6.3$	47.3	$+3.2$	$+0.6$
4	50.5	$+3.5$	$+6.0$	46.7	$+2.6$	$\phantom{+}0.0$
5	48.8	$+1.8$	$+4.3$	44.1	$\phantom{+}0.0$	$-2.6$

Appendix

Table 6: Rank-gate accuracy (%) and change relative to the LLM baseline (

\Delta_{\text{LLM}}

) and FBPR-alone baseline (

\Delta_{\text{FBPR}}

) for

m

\in\{1,\ldots,5\}

m

=\lceil|D|/2\rceil=3

(paper main) is the peak on LLaMA + RedPajama;

m

=2

is the peak on OLMo + Dolma. The corpus-dependent best

m

mirrors the direction asymmetry of the confidence threshold

\tau

: when the LLM is the stronger baseline, the larger

m

wins; when FBPR is stronger, the smaller

m

wins. The boundary cases

m

=1

(= FBPR alone,

\Delta_{\text{FBPR}}=0

) and

m

=5

(

\approx

LLM alone,

\Delta_{\text{LLM}}\approx 0

) provide reference.

	LLaMA 65B + RedPajama		OLMo + Dolma
Method	$c>\tau$ ( $n{=}228$ )	$c\leq\tau$ ( $n{=}491$ )	$c>\tau$ ( $n{=}197$ )	$c\leq\tau$ ( $n{=}522$ )
LLM regex	49.6	45.8	40.6	45.4
FBPR alone	50.0	42.0	59.4	42.0
Rank gate	54.0	49.3	49.2	46.6
Confidence gate	50.0	47.7	59.4	45.4
Combined gate	50.0	49.3	59.4	46.6

Appendix

Table 7: Per-bin accuracy (%) at

\tau=0.99

. By construction, the combined gate matches the confidence gate in the

c>\tau

bin and the rank gate in the

c\leq\tau

bin. Bold marks the strongest method per bin/pair.

Figure 5: Confusion matrix between FBPR on Dolma and OLMo Instruct 7B on MedQA diagnosis subset.

Figure 6: Distribution of the rank of the correct diagnosis of each question by RedPajama-based FBPR with

k=5

or unrestricted

k

Figure 7: Stacked histogram of per-question softmaxed top-1 scores calculated by Dolma-based FBPR with

k=5

. A larger softmaxed top-1 score indicates that FBPR’s chosen diagnosis is much more strongly favored by frequency-based Naive Bayes relative to the other candidates.

Figure 8: A success example (left) and a failure example (right) of FBPR on MedQA diagnosis subset. The diagnosis occurrence and (diagnosis, concept) co-occurrence frequencies are retrieved by Infini-gram from v4_rpj_llama_s4 index, i.e., from the RedPajama corpus. By Infini-gram design, some of these frequencies are estimates.

Method	Accuracy	95% CI
FBPR on Dolma	46.7%	[43.1, 50.3]
OLMo Instruct 7B	44.1%	[40.5, 47.7]
Hybrid (OLMo + Dolma)	47.3%	[43.7, 50.9]
FBPR on RedPajama	44.5%	[40.9, 48.1]
LLaMA 65B	47.0%	[43.4, 50.6]
Hybrid (LLaMA 65B + RedPajama)	50.8%	[47.0, 54.4]

Appendix

Table 8: Accuracy on the MedQA diagnosis subset (

n=719

) with 95% bootstrap confidence intervals.

Language Model Reasoning Evaluation

See all 94 papers

RPRO: Ranked Preference Reinforcement Optimization for Enhancing Medical QA and Diagnostic Reasoning
Aug 31, 2025Chia-Hsuan Hsu, Jun-En Ding, Hsin-Ling Hsu +5Medical LLMs Medical QA
Department of Computer Science and Information Engineering, National Taiwan University of Science and Technology, Taipei City, Taiwan · Department of Systems Engineering, Stevens Institute of Technology, Hoboken, NJ, USA · Far Eastern Memorial Hospital, New Taipei City, Taiwan +4
Medical Reasoning with Large Language Models: A Survey and MR-Bench
Mar 17, 2026Xiaohan Ren, Chenxiao Fan, Wenyin Ma +4Medical Reasoning Clinical Reasoning
School of Cyber Science and Technology, University of Science and Technology of China, Anhui, China · School of Artificial Intelligence and Data Science, University of Science and Technology of China, Anhui, China · Department of Infectious Disease, The First Affiliated Hospital of USTC, Anhui, China +2
Fathom-Vaidya: Advancing Medical Reasoning with Rubric-Based Rewards
Sep 21, 2026Kalash Shah, Kunal Singh, Snehan J +1RL for Language Model Reasoning Rubric-Based RL
Fractal AI Research

Aug 31, 2025J/K move · Enter open · S save

RPRO: Ranked Preference Reinforcement Optimization for Enhancing Medical QA and Diagnostic Reasoning

Chia-Hsuan Hsu, Jun-En Ding, Hsin-Ling Hsu, Chih-Ho Hsu+4

Department of Computer Science and Information Engineering, National Taiwan University of Science and Technology, Taipei City, Taiwan · Department of Systems Engineering, Stevens Institute of Technology, Hoboken, NJ, USA · Far Eastern Memorial Hospital, New Taipei City, Taiwan+4

Medical LLMs Medical QA RL for Language Model Reasoning Preference Optimization Clinical Reasoning

Open paper AskPDF arXiv

Medical question answering requires advanced reasoning that integrates domain knowledge with logical inference. However, existing large language models (LLMs) often generate reasoning chains that lack factual accuracy and clinical reliability. We propose Ranked Preference Reinforcement Optimization (RPRO), a novel framework that combines reinforcement learning with preference-driven reasoning refinement to enhance clinical chain-of-thought (CoT) performance. RPRO distinguishes itself from prior approaches by employing task-adaptive reasoning templates and a probabilistic evaluation mechanism that aligns model outputs with established clinical workflows, while automatically identifying and correcting low-quality reasoning chains. Unlike traditional pairwise preference methods, RPRO introduces a groupwise ranking optimization based on the Bradley--Terry model and incorporates KL-divergence regularization for stable training. Experiments on PubMedQA, MedQA-USMLE, and a real-world clinical dataset from Far Eastern Memorial Hospital (FEMH) demonstrate consistent improvements over strong baselines. Remarkably, our 2B-parameter model outperforms much larger 7B--20B models, including medical-specialized variants. These findings demonstrate that combining preference optimization with quality-driven refinement provides a scalable and clinically grounded approach to building more reliable medical LLMs.

Abstract

Language Model Reasoning Evaluation

Explore similar work

RPRO: Ranked Preference Reinforcement Optimization for Enhancing Medical QA and Diagnostic Reasoning

Medical Reasoning with Large Language Models: A Survey and MR-Bench

Fathom-Vaidya: Advancing Medical Reasoning with Rubric-Based Rewards