MS-Exam-Gen: Source-Grounded Benchmark Construction for Evaluating LLMs on Textual Multiple Sclerosis MRI Knowledge

Abdul Basit; Muhammad Abdullah Hanif; Muhammad Shafique

cs.AIOct 5, 2026

MS-Exam-Gen: Source-Grounded Benchmark Construction for Evaluating LLMs on Textual Multiple Sclerosis MRI Knowledge

Authors: Abdul Basit, Muhammad Abdullah Hanif, Muhammad Shafique

Organizations: eBRAIN Lab, Division of Engineering, New York University Abu Dhabi (NYUAD), Abu Dhabi, UAE

Abstract

Biomedical large language model (LLM) evaluation requires auditable assessment of narrow, evolving, source-grounded subspecialty knowledge. Multiple sclerosis MRI (MS-MRI) provides a high-stakes textual-knowledge test case because correct reasoning requires current diagnostic criteria, standardized acquisition and reporting knowledge, longitudinal monitoring concepts, lesion morphology, and recognition of difficult mimics. We present MS-Exam-Gen, a reproducible framework for constructing and auditing a text-based multiple-choice question (MCQ) benchmark for MS-MRI knowledge; it does not evaluate direct MRI image interpretation. MS-Exam-Gen targets source-grounded criteria, protocols, reporting, and differential diagnosis. The framework combines expert-source indexing, exam-oriented topic induction, evidence-grounded MCQ generation, automated quality audits, a same-family consistency screen, and empirical calibration. From a 66-source corpus indexed into 4,289 retrieval chunks, the pipeline produced a locked 3,058-item candidate benchmark spanning 16 topics and 53 subtopics. Evaluation across 12 primary LLM endpoints yielded 36,696 item-level predictions and separated performance over a 42.8-percentage-point accuracy range (89.7% to 46.9%). Across these endpoints, 25.5% of items were missed by at least four. Post-generation audits showed that refreshed construction reduced measurable answer cues, while option-order testing showed that absolute MCQ scores remain position-sensitive. Generated construction labels remain metadata rather than validated psychometric categories. Because expert adjudication and full option-order counterbalancing remain future work, MS-Exam-Gen is not a clinically certified examination. It should be interpreted as an automatically filtered, source-grounded candidate benchmark and reproducible audit workflow for item-level and topic-specific LLM evaluation.

Figures & tables

Family	Representative examples	Useful for	Limitation for this study	MS-MRI text	MS-MRI evidence	MCQ audit / calib.
Broad medical QA	MedQA, PubMedQA, MMLU, MedMCQA, and MultiMedQA [ 5 , 6 , 7 , 16 , 2 ]	Broad text/exam QA	Not organized around MS-MRI criteria, protocols, reporting, or mimics.	✗	Partial	✗
Clinical/multimodal suites	MultiMedQA-style and broader medical benchmark suites [ 2 , 17 ]	Clinical QA and scenarios	Coverage and provenance vary; usually not a locked MS-MRI source-grounded MCQ bank.	✗	Variable	Variable
Radiology VQA/report	Radiology foundation-model and report-generation benchmarks [ 18 , 19 ]	Image, VQA, reports	Image/report focused; not designed for guideline-source textual MCQ auditing.	✗	Partial	Variable
MS imaging tasks	MSSEG and longitudinal lesion segmentation [ 11 , 12 ]	Lesion segmentation	Domain-specific imaging tasks, but image labels/masks rather than textual source evidence.	✗	Partial	✗
LLM-generated evals	Model-written, MT-Bench, G-Eval, LLM-as-judge methods; MCQ-bias analyses [ 13 , 14 , 15 , 20 , 21 ]	Generation and judging	Scalable, but source grounding and clinical safeguards are variable.	✗	Variable	Variable
MS-Exam-Gen	This work	Text MCQ construction	Candidate benchmark only; expert adjudication and full option-order counterbalancing remain future work.	✓	✓	✓

TABLE I: Positioning of MS-Exam-Gen relative to related benchmark families.

Fig. 1: MS-Exam-Gen construction and audit workflow. Sources are indexed with metadata, sampled through seeded retrieval, organized into an exam-oriented taxonomy, used for source-grounded MCQ generation, and filtered by automated audits and a same-family consistency screen with regeneration of malformed or cueable items. Algorithm 1 additionally details benchmark locking, model evaluation, and empirical calibration. Construction labels are metadata; the locked bank is a candidate benchmark, not a clinical examination.

Stage	Input	Accepted/flagged	Locked
Standard Easy/Medium	650 each	646 each	646 each
Standard Hard	650	643	0
Refreshed Hard/synthesis	900 each	883 each	883 each
Legacy Hard/synthesis pool	1,293	1,217 flagged	0

TABLE II: Construction and audit trail for the locked BHI candidate benchmark.

Component	Value
Corpus/taxonomy	66 sources; 4,289 chunks; 16 topics; 53 subtopics
Locked bank	3,058 MCQs: Easy/Medium 646 each; Hard/synthesis 883 each
Final audit	0 schema errors; 0 exact duplicates; 54 retained near-duplicate transparency flags
Evaluation	12 primary endpoints plus GPT-5 mini sensitivity; 36,696 primary/39,754 total responses; 0 parse failures

TABLE III: Final benchmark and evaluation summary.

Fig. 2: Benchmark auditing and empirical calibration. (A) The selector was fitted to the replaced legacy and locked Hard/synthesis strata. The dashed chance line applies only to the first two accuracy metrics; the third is item prevalence. Lower key predictability supports reduced measurable cueing. (B) Primary empirical difficulty is the number of 12 non-construction-overlap endpoints incorrect per item.

Fig. 3: Topic-level accuracy across 12 primary endpoints. All 16 submitted topic labels and item counts are retained. Blue brackets mark three semantically overlapping label pairs and report their accuracy-independent parent means. Cells report percentages, and the final column reports each original topic’s mean.

Model	Easy	Med.	Hard	Synth.	Avg.	95% CI
Gemini 2.5 Flash	91.0	88.4	88.1	91.3	89.7	86.0–92.9
Llama 4 Maverick	88.4	87.2	87.8	90.4	88.5	85.0–91.7
DeepSeek Chat	87.8	86.8	86.2	87.0	86.9	83.1–90.2
Claude Haiku 4.5	86.7	85.1	81.1	89.5	85.6	81.3–89.4
Gemini 2.5 Flash Lite	85.1	81.9	84.9	85.2	84.4	79.2–88.9
Qwen3 14B	83.3	82.5	83.5	83.8	83.3	78.3–87.8

TABLE IV: Primary construction-label-resolved model-panel results on the locked 3,058-item benchmark.

Explore similar work

May 6, 2026eess.IV

MRI-Eval: A Tiered Benchmark for Evaluating LLM Performance on MRI Physics and GE Scanner Operations Knowledge

Background: Existing MRI LLM benchmarks rely mainly on review-book multiple-choice questions, where top proprietary models already score highly, limiting discrimination. No systematic benchmark has evaluated vendor-specific scanner operational knowledge central to research MRI practice. Purpose: We developed MRI-Eval, a tiered benchmark for relative model comparison on MRI physics and GE scanner operations knowledge using primary multiple-choice questions (MCQ), with stem-only and primed diagnostic conditions as complementary analyses. Methods: MRI-Eval includes 1365 scored items across nine categories and three difficulty tiers from textbooks, GE scanner manuals, programming course materials, and expert-generated questions. Five model families were evaluated (GPT-5.4, Claude Opus 4.6, Claude Sonnet 4.6, Gemini 2.5 Pro, Llama 3.3 70B). MCQ was primary; stem-only removed options and used an independent LLM judge; primed stem-only tested responses to incorrect user claims. Results: Overall MCQ accuracy was 93.2% to 97.1%. GE scanner operations was the lowest category for every model (88.2% to 94.6%). In stem-only, frontier-model accuracy fell to 58.4% to 61.1%, and Llama 3.3 70B fell to 37.1%; GE scanner operations stem-only accuracy was 13.8% to 29.8%. Conclusion: High MCQ performance can mask weak free-text recall, especially for vendor-specific operational knowledge. MRI-Eval is most informative as a relative comparison benchmark rather than an absolute competency measure and supports caution in using raw LLM outputs for GE-specific protocol guidance.

Perry E. Radau

Department of Radiology, University of Calgary · Child and Adolescent Imaging Research (CAIR) Program · Alberta Children’s Hospital Research Institute +1

Diagnostic Benchmark Large Language Model Evaluation Multiple-Choice Questions

Ask PDF

Jun 10, 2026cs.CL

Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance?

Large language models (LLMs) in medicine are mainly evaluated using multiple-choice question answering (MCQA), which can overestimate real clinical ability due to guessing strategies and answer biases. To address these limitations, we introduce an expanded and more challenging benchmark based on Polish medical exams, adding over 15,000 questions, two new domains, and four structural modifications that reduce MCQA-specific artifacts and better test reasoning. We evaluate 21 LLMs and show that evaluation design strongly affects results. Under our harder setup, the best model (Qwen3.5-122B) drops by 28.4 and 31 pp on English and Polish exams, respectively. Despite low evidence of data contamination, standard MCQA scores do not reliably reflect true medical competence. To facilitate further research, we make our benchmark publicly available.

Antoni Lasik, Jakub Pokrywka, Łukasz Grzybowski +7

NASK National Research Institute · Adam Mickiewicz University · ARAAI Poland +4

Diagnostic Benchmark Multiple-Choice Questions Assessment Competence

Ask PDF

May 12, 2026cs.CL

MedHopQA: A Disease-Centered Multi-Hop Reasoning Benchmark and Evaluation Framework for LLM-Based Biomedical Question Answering

Evaluating large language models (LLMs) in the biomedical domain requires benchmarks that can distinguish reasoning from pattern matching and remain discriminative as model capabilities improve. Existing biomedical question answering (QA) benchmarks are limited in this respect. Multiple-choice formats can allow models to succeed through answer elimination rather than inference, while widely circulated exam-style datasets are increasingly vulnerable to performance saturation and training data contamination. Multi-hop reasoning, defined as the ability to integrate information across multiple sources to derive an answer, is central to clinically meaningful tasks such as diagnostic support, literature-based discovery, and hypothesis generation, yet remains underrepresented in current biomedical QA benchmarks. MedHopQA is a disease-centered multi-hop reasoning benchmark consisting of 1,000 expert-curated question-answer pairs introduced as a shared task at BioCreative IX. Each question requires synthesis of information across two distinct Wikipedia articles, and answers are provided in an open-ended free-text format. Gold annotations are augmented with ontology-grounded synonym sets from MONDO, NCBI Gene, and NCBI Taxonomy to support both lexical and concept-level evaluation. MedHopQA was constructed through a structured process combining human annotation, triage, iterative verification, and LLM-as-a-judge validation. To reduce leaderboard gaming and contamination risk, the 1,000 scored questions are embedded within a publicly downloadable set of 10,000 questions, with answers withheld, on a CodaBench leaderboard. MedHopQA provides both a benchmark and a reusable framework for constructing future biomedical QA datasets that prioritize compositional reasoning, saturation resistance, and contamination resistance as core design constraints.

Rezarta Islamaj, Robert Leaman, Joey Chan +13

National Library of Medicine, Division of Intramural Research, Bethesda, MD, US · University of Illinois at Urbana-Champaign, Department of Computer Science, Urbana, IL, US · University of Michigan Medical School, Ann Arbor, Michigan, US

Biomedical Text Clinical Reasoning Training Diagnostic Benchmark Multiple-Choice Questions Reasoning Benchmark

Ask PDF