Benchmarking decision models against generative large language models on medical multiple-choice questions

Oscar Freyer; Daniel Glauert; Moritz Hofmann; Max Ostermann; Stephen Gilbert; Jakob Nikolas Kather

cs.CLOct 28, 2026

Benchmarking decision models against generative large language models on medical multiple-choice questions

Authors: Oscar Freyer, Daniel Glauert, Moritz Hofmann, Max Ostermann, Stephen Gilbert, Jakob Nikolas Kather

Organizations: Else Kröner Fresenius Center for Digital Health, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany · Independent researcher, Berlin, Germany · Denkmal.ai, Berlin, Germany · Faculty of Business and Economics, TUD Dresden University of Technology, Dresden, Germany ·

Abstract

Medical workflows often require selecting among predefined options. We compared 13 decision models with 15 generative large language model configurations, including different reasoning settings, using 1,273 four-option MedQA questions under text-only conditions. Accuracy ranged from 67.71% to 89.08% for decision models and from 78.32% to 96.86% for generative configurations. Among the tested configurations, Mercury Decide had the lowest mean final-attempt cost (USD 0.0069 per 1,000 questions), while Clef Omni had the shortest median request duration (0.291 seconds). Decision-model probabilities distinguished correct from incorrect answers, but error rates among answers assigned probabilities of at least 0.90 ranged from 1.71% to 12.77%. Answer agreement across three runs ranged from 92.30% to 100%, while probability repeatability varied. Decision models offered resource advantages alongside lower peak accuracy; selecting models for medical workflows requires considering probability quality and repeatability together with accuracy and the resources needed to detect and correct errors.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,427 papers

Evaluating Small Open LLMs for Medical Question Answering: A Practical Framework
Apr 12, 2026Avi-ad Avraam BuskilaLLM Evaluation LLM Reliability
Department of Information Science and Applied Artificial Intelligence, Bar-Ilan University, Ramat-Gan, Israel
The Value of System 1 Models in Medicine
Oct 22, 2026Xiufeng Yang, Konstantinos Vrettos, Zeineb Ben Chaaben +17Clinical Language Model Evaluation Multimodal Model Evaluation
Department of Diagnostic and Interventional Radiology, School of Medicine and Health, Technical University of Munich, Munich, Germany · Artificial Intelligence and Translational Imaging Lab, Department of Radiology, School of Medicine, University of Crete, Heraklion, Greece · Department of Medical Imaging, University Hospital of Heraklion, Heraklion, Crete, Greece +6
Domain Fine-Tuning vs. Retrieval-Augmented Generation for Medical Multiple-Choice Question Answering: A Controlled Comparison at the 4B-Parameter Scale
Apr 26, 2026Avi-ad Avraam BuskilaRetrieval-Augmented Generation Healthcare
Department of Information Science and Applied Artificial Intelligence, Bar-Ilan University, Ramat-Gan, Israel

Apr 12, 2026J/K move · Enter open · S save

Evaluating Small Open LLMs for Medical Question Answering: A Practical Framework

Avi-ad Avraam Buskila

Department of Information Science and Applied Artificial Intelligence, Bar-Ilan University, Ramat-Gan, Israel

LLM Evaluation LLM Reliability Clinical Language Model Evaluation Medical LLMs Biomedical QA

Open paper AskPDF arXiv

Incorporating large language models (LLMs) in medical question answering demands more than high average accuracy: a model that returns substantively different answers each time it is queried is not a reliable medical tool. Online health communities such as Reddit have become a primary source of medical information for millions of users, yet they remain highly susceptible to misinformation; deploying LLMs as assistants in these settings amplifies the need for output consistency alongside correctness. We present a practical, open-source evaluation framework for assessing small, locally-deployable open-weight LLMs on medical question answering, treating reproducibility as a first-class metric alongside lexical and semantic accuracy. Our pipeline computes eight quality metrics, including BERTScore, ROUGE-L, and an LLM-as-judge rubric, together with two within-model reproducibility metrics derived from repeated inference (N=10 runs per question). Evaluating three models (Llama 3.1 8B, Gemma 3 12B, MedGemma 1.5 4B) on 50 MedQuAD questions (N=1,500 total responses) reveals that despite low-temperature generation (T=0.2), self-agreement across runs reaches at most 0.20, while 87-97% of all outputs per model are unique -- a safety gap that single-pass benchmarks entirely miss. The clinically fine-tuned MedGemma 1.5 4B underperforms the larger general-purpose models on both quality and reproducibility; however, because MedGemma is also the smallest model, this comparison confounds domain fine-tuning with model scale. We describe the methodology in sufficient detail for practitioners to replicate or extend the evaluation for their own model-selection workflows. All code and data pipelines are available at https://github.com/aviad-buskila/llm_medical_reproducibility.

Abstract

LLM Evaluation

Explore similar work

Evaluating Small Open LLMs for Medical Question Answering: A Practical Framework

The Value of System 1 Models in Medicine

Domain Fine-Tuning vs. Retrieval-Augmented Generation for Medical Multiple-Choice Question Answering: A Controlled Comparison at the 4B-Parameter Scale