Evidence-Grounded Subspecialty Reasoning: Evaluating a Curated Clinical Intelligence Layer on the 2025 Endocrinology Board-Style Examination

Amir Hosseinian; MohammadReza Zare Shahneh; Umer Mansoor; Gilbert Szeto; Kirill Karlin; Nima Aghaeepour

cs.AIFeb 17, 2026

Evidence-Grounded Subspecialty Reasoning: Evaluating a Curated Clinical Intelligence Layer on the 2025 Endocrinology Board-Style Examination

Authors: Amir Hosseinian, MohammadReza Zare Shahneh, Umer Mansoor, Gilbert Szeto, Kirill Karlin, Nima Aghaeepour

Organizations: January AI · Stanford University

Abstract

Background: Large language models have demonstrated strong performance on general medical examinations, but subspecialty clinical reasoning remains challenging due to rapidly evolving guidelines and nuanced evidence hierarchies. Methods: We evaluated January Mirror, an evidence-grounded clinical reasoning system, against frontier LLMs (GPT-5, GPT-5.2, Gemini-3-Pro) on a 120-question endocrinology board-style examination. Mirror integrates a curated endocrinology and cardiometabolic evidence corpus with a structured reasoning architecture to generate evidence-linked outputs. Mirror operated under a closed-evidence constraint without external retrieval. Comparator LLMs had real-time web access to guidelines and primary literature. Results: Mirror achieved 87.5% accuracy (105/120; 95% CI: 80.4-92.3%), exceeding a human reference of 62.3% and frontier LLMs including GPT-5.2 (74.6%), GPT-5 (74.0%), and Gemini-3-Pro (69.8%). On the 30 most difficult questions (human accuracy less than 50%), Mirror achieved 76.7% accuracy. Top-2 accuracy was 92.5% for Mirror versus 85.25% for GPT-5.2. Conclusions: Mirror provided evidence traceability: 74.2% of outputs cited at least one guideline-tier source, with 100% citation accuracy on manual verification. Curated evidence with explicit provenance can outperform unconstrained web retrieval for subspecialty clinical reasoning and supports auditability for clinical deployment.

Figures & tables

Figure 1: Distribution of clinical reasoning domains in ESAP 2025. The left panel shows the total number of questions involving each domain. The main panel displays intersection sizes for domain combinations, sorted by frequency. The matrix indicates which domains comprise each intersection. Treatment-related reasoning was most prevalent (n=73), followed by pathophysiology (n=60) and diagnosis (n=57). The most common domain combination was risk/prognosis with treatment (n=16).

System	Web Access	Evidence Base	Architecture
Mirror	No	Curated endocrinology corpus	Evidence-grounded ensemble
GPT-5.2	Yes	Real-time web retrieval	Monolithic LLM
GPT-5	Yes	Real-time web retrieval	Monolithic LLM
Gemini-3-Pro	Yes	Real-time web retrieval	Monolithic LLM

Table 1: Systems evaluated and their capabilities

Figure 2: Comparative performance on ESAP 2025 endocrinology examination. Mirror achieved 87.5% accuracy compared to 74.6% for the best-performing frontier LLM (GPT-5.2) and 62.3% for the human reference (ESAP respondent mean). Optional dashed line indicates an internal upper-bound reference condition, provided for context. Error bars represent 95% confidence intervals. * $p<0.05$ vs. Mirror .

Figure 3: Performance stratified by question difficulty (based on human accuracy). Mirror maintained strong performance across all difficulty tiers, with particular advantage on hard questions (human accuracy $<$ 50%) where it achieved 76.7% compared to 53.3% for GPT-5.2 and 39.5% human mean accuracy.

Figure 4: Paired win/loss analysis comparing Mirror to each baseline. For each comparison, “wins” indicate questions Mirror answered correctly while the baseline erred; “losses” indicate the reverse. Mirror achieved a net positive margin against all baselines, with the largest advantage over Gemini-3-Pro (19 net wins).

Question Type	Mirror	GPT-5.2	GPT-5	Gemini-3-Pro	n †
Diagnosis	91.2	77.6	73.7	71.8	57
Diagnostic Testing	85.1	70.0	68.7	65.5	47
Treatment	86.3	70.8	72.2	64.4	73
Risk/Prognosis	86.8	80.0	81.6	71.1	38
Pathophysiology	86.7	74.7	73.8	76.1	60
Overall	87.5	74.6	74.0	69.8	120

Table 2: Accuracy (%) by question type

Figure 5: Performance by question type across systems. Mirror demonstrated consistent performance across all question categories, with the largest advantage in treatment questions (15.5 percentage point difference vs. GPT-5.2).

Figure 6: Evidence provenance profile of Mirror outputs. The left panel shows the highest evidence tier cited per question: society guidelines (41.7%), peer-reviewed literature (33.8%), pivotal clinical trials (20.9%), and other sources (3.6%). The right panel indicates that 74.2% of answers included at least one guideline-tier citation, demonstrating the system’s reliance on authoritative clinical guidance.

System	Clinical Scenario	Error Category
Mirror	Pituitary macroadenoma with elevated prolactin and GH; selected dopamine agonist instead of confirming acromegaly	Incomplete biochemical evaluation before treatment
Mirror	Complete androgen insensitivity syndrome; prioritized bone density assessment over gonadal imaging	Metabolic workup prioritized over malignancy surveillance
Mirror	Opioid-induced hypogonadism with low gonadotropins; recommended obesity pharmacotherapy instead of pituitary imaging	Treatment prioritized over diagnostic workup for central hypogonadism
Mirror	Lactation-associated osteoporotic fracture with suboptimal calcium intake; recommended cessation of breastfeeding	Overaggressive intervention when conservative management indicated
GPT-5.2	Gestational diabetes with suboptimal control on basal insulin; selected outdated A1c target instead of current pregnancy-specific glucose thresholds	Guideline threshold: parametric knowledge conflicted with updated ADA recommendations
Gemini-3-Pro	Type 2 diabetes with CKD and albuminuria on multiple agents; failed to prioritize SGLT2 inhibitor per KDIGO guidelines	Multi-step integration: did not synthesize cardiorenal benefit hierarchy across comorbidities

Table 3: Representative error cases by system

	Case 1: Gestational Diabetes	Case 2: Acute Adrenal Hemorrhage
Clinical Scenario	Pregnant woman (14 weeks) with history of gestational diabetes and documented needle phobia causing insulin nonadherence in prior pregnancy, resulting in preeclampsia and preterm delivery. Now requires pharmacotherapy.	Previously healthy man presenting with acute abdominal pain, headache, and hypotension days after adenoviral-vector COVID-19 vaccine. CT shows bilateral adrenal hemorrhage. Labs show thrombocytopenia and markedly elevated D-dimer.
Correct Answer	Metformin	IV hydrocortisone
Mirror Answer	Metformin ✓	IV hydrocortisone ✓
GPT-5.2 Answer	Subcutaneous insulin	IV immunoglobulin
Mirror Reasoning	“Treatment must be feasible and patient-specific. This patient has documented history of skipping insulin due to needle phobia, which already resulted in serious complications. Per ADA guidelines, metformin is an acceptable alternative when insulin is not feasible.”	“CT appearance consistent with acute adrenal hemorrhage. Patient is critically unwell with tachycardia and elevated lactate. Immediate priority is treating life-threatening adrenal insufficiency with IV hydrocortisone.”
GPT-5.2 Reasoning	“Insulin is the preferred/standard pharmacotherapy in pregnancy because it does not cross the placenta. Address adherence by teaching partner to administer injections.”	“Presentation consistent with vaccine-induced immune thrombotic thrombocytopenia (VITT). Give IVIG to block Fc receptor-mediated platelet activation.”

Table 4: Side-by-side reasoning comparison: Mirror vs GPT-5.2

Explore similar work

When Cases Get Rare: A Retrieval Benchmark for Off-Guideline Clinical Question Answering
May 20, 2026Doeun Lee, Muge Zhang, Yi Yu +11LLM Grounding Healthcare
The Ohio State University · The Ohio State University Wexner Medical Center · University of Chicago Medical Center
Medical Reasoning with Large Language Models: A Survey and MR-Bench
Mar 17, 2026Xiaohan Ren, Chenxiao Fan, Wenyin Ma +4Medical Reasoning Clinical Reasoning
School of Cyber Science and Technology, University of Science and Technology of China, Anhui, China · School of Artificial Intelligence and Data Science, University of Science and Technology of China, Anhui, China · Department of Infectious Disease, The First Affiliated Hospital of USTC, Anhui, China +2
CliniCARE-Bench: Clinical Calibrated Audit of Medical Reasoning in EHR
Aug 7, 2026Veronica Chatrath, Bryan Zhu, George Pu +16Evidence-Grounded Reasoning Clinical Reasoning
Scale AI · Emory University · University of California, Santa Cruz +4

May 20, 2026J/K move · Enter open · S save

When Cases Get Rare: A Retrieval Benchmark for Off-Guideline Clinical Question Answering

Doeun Lee, Muge Zhang, Yi Yu, Ashish Manne+10

The Ohio State University · The Ohio State University Wexner Medical Center · University of Chicago Medical Center

LLM Grounding Healthcare Clinical QA RAG Evaluation Medical Reasoning

Open paper AskPDF arXiv

Across medical specialties, clinical practice is anchored in evidence-based guidelines that codify best studied diagnostic and treatment pathways. These pathways routinely fall short for the long tail of real-world care not covered by guidelines. Most medical large language models (LLMs), however, are trained to encode common, guideline-focused medical knowledge in their parameters. Current evaluations test models primarily on recalling and reasoning with this memorized content, often in multiple-choice settings. Given the fundamental importance of evidence-based reasoning in medicine, it is neither feasible nor reliable to depend on memorization in practice. To address this gap, we introduce OGCaReBench, a free-form retrieval-focused benchmark aimed at evaluating LLMs at answering clinical questions that require going beyond typical guidelines. Extracted from published medical case reports and validated by medical experts, OGCaReBench contains long-form clinical questions requiring free-text answers, providing a systematic framework for assessing open-ended medical reasoning in rare, case-based scenarios. Our experiments reveal that even the best-performing baseline (GPT-5.2) correctly answers only 56% of our benchmark with specialized models only reaching 42%. Augmenting models with retrieved medical articles improves this performance to up to 82% (using GPT-5.2) highlighting the importance of evidence-grounding for real-world medical reasoning tasks. This work thus establishes a foundation for benchmarking and advancing both general-purpose and medical LLMs to produce reliable answers in challenging clinical contexts.

Abstract

Explore similar work

When Cases Get Rare: A Retrieval Benchmark for Off-Guideline Clinical Question Answering

Medical Reasoning with Large Language Models: A Survey and MR-Bench

CliniCARE-Bench: Clinical Calibrated Audit of Medical Reasoning in EHR