Comparative Analysis of Large Language Models in Healthcare

Subin Santhosh; Farwa Abbas; Hussain Ahmad; Claudia Szabo

cs.CLApr 11, 2026

Comparative Analysis of Large Language Models in Healthcare

Authors: Subin Santhosh, Farwa Abbas, Hussain Ahmad, Claudia Szabo

Organizations: Adelaide University, Australia · South Australian Health and Medical Research Institute, Australia

Abstract

Background: Large Language Models (LLMs) are transforming artificial intelligence applications in healthcare due to their ability to understand, generate, and summarize complex medical text. They offer valuable support to clinicians, researchers, and patients, yet their deployment in high-stakes clinical environments raises critical concerns regarding accuracy, reliability, and patient safety. Despite substantial attention in recent years, standardized benchmarking of LLMs for medical applications has been limited. Objective: This study addresses the need for a standardized comparative evaluation of LLMs in medical settings. Method: We evaluate multiple models, including ChatGPT, LLaMA, Grok, Gemini, and ChatDoctor, on core medical tasks such as patient note summarization and medical question answering, using the open-access datasets, MedMCQA, PubMedQA, and Asclepius, and assess performance through a combination of linguistic and task-specific metrics. Results: The results indicate that domain-specific models, such as ChatDoctor, excel in contextual reliability, producing medically accurate and semantically aligned text, whereas general-purpose models like Grok and LLaMA perform better in structured question-answering tasks, demonstrating higher quantitative accuracy. This highlights the complementary strengths of domain-specific and general-purpose LLMs depending on the medical task. Conclusion: Our findings suggest that LLMs can meaningfully support medical professionals and enhance clinical decision-making; however, their safe and effective deployment requires adherence to ethical standards, contextual accuracy, and human oversight in relevant cases. These results underscore the importance of task-specific evaluation and cautious integration of LLMs into healthcare workflows.

Figures & tables

Fig. 1: Applications of LLMs in Healthcare

Study	Models Evaluated	Task Types	Datasets Used	Evaluation Metrics
Kung et al. (2023) [ 20 ]	GPT-3.5 only	MCQ (USMLE)	USMLE Step 1-3	Pass/Fail, Accuracy
Cascella et al. (2023) [ 36 ]	ChatGPT only	Open-ended QA	Proprietary clinical prompts	Qualitative only
Singhal et al. (2023) [ 37 ]	Med-PaLM 2 vs GPT-4	MCQ, long-form QA	MedQA, MedMCQA, HealthSearchQA	Accuracy, human ratings
Nori et al. (2023) [ 38 ]	GPT-4 only	MCQ	USMLE, MedQA	Accuracy only
Lievin et al. (2024) [ 39 ]	GPT-3/4, PaLM	MCQ reasoning	MedQA, MedMCQA	Accuracy, chain-of-thought
Van Veen et al. (2023) [ 40 ]	6 LLMs	Summarization only	Proprietary clinical notes	ROUGE, BERTScore

TABLE I: Comparative Analysis of Related LLM Healthcare Evaluation Studies

	MedMCQA	PubMedQA	Asclepius
Type	MCQ	Yes/No/Maybe	Clinical text
Task	Question Answers	Question Answers	Summarization
Size	183k questions	1k labeled	2k notes
Primary Metrics	Accuracy, Macro F1	Accuracy, Macro F1	BLEU, ROUGE-L, BERTScore

TABLE II: Datasets and task mapping.

Fig. 11: (Left) Normalized performance heatmap showing scaled scores. (Right) Radar plot displaying multi-dimensional capabilities of LLMs.

Model	MedMCQA (Acc)	MedMCQA (F1)	PubMedQA (Acc)	PubMedQA (F1)	Asclepius (BLEU)	Asclepius (ROUGE-L)	Asclepius (BERT F1)
ChatDoctor	0.814	0.812	0.711	0.367	12.573	0.271	0.222
Grok	0.833	0.832	0.762	0.327	6.886	0.195	0.125
GPT-4o	0.739	0.737	0.592	0.283	7.981	0.260	0.155
Gemini	0.732	0.730	0.592	0.290	10.895	0.239	0.220
LLaMA	0.711	0.711	0.794	0.379	9.767	0.257	0.186

TABLE III: Comprehensive Results of LLM Performance Across All Experimental Scenarios

Explore similar work

Feb 16, 2026

Assessing Large Language Models for Medical QA: Zero-Shot and LLM-as-a-Judge Evaluation

Recently, Large Language Models (LLMs) have gained significant traction in medical domain, especially in developing a QA systems to Medical QA systems for enhancing access to healthcare in low-resourced settings. This paper compares five LLMs deployed between April 2024 and August 2025 for medical QA, using the iCliniq dataset, containing 38,000 medical questions and answers of diverse specialties. Our models include Llama-3-8B-Instruct, Llama 3.2 3B, Llama 3.3 70B Instruct, Llama-4-Maverick-17B-128E-Instruct, and GPT-5-mini. We are using a zero-shot evaluation methodology and using BLEU and ROUGE metrics to evaluate performance without specialized fine-tuning. Our results show that larger models like Llama 3.3 70B Instruct outperform smaller models, consistent with observed scaling benefits in clinical tasks. It is notable that, Llama-4-Maverick-17B exhibited more competitive results, thus highlighting evasion efficiency trade-offs relevant for practical deployment. These findings align with advancements in LLM capabilities toward professional-level medical reasoning and reflect the increasing feasibility of LLM-supported QA systems in the real clinical environments. This benchmark aims to serve as a standardized setting for future study to minimize model size, computational resources and to maximize clinical utility in medical NLP applications.

Shefayat E Shams Adib, Ahmed Alfey Sani, Ekramul Alam Esham +2

Department of Computer Science and Engineering, Islamic University of Technology, Gazipur, Bangladesh

Medical QA Medical LLMs Clinical Language Model Evaluation LLM Evaluation

Ask PDF

Sep 13, 2026

A primer on evaluation methods for large language models in healthcare

Large language models (LLMs) have a growing range of applications in medicine, and their evaluation is critical for ensuring they provide benefit and not harm. This evaluation can be more challenging than traditional machine learning for many reasons, including probabilistic and open-ended outputs, and behavior that shifts with prompt design and accumulated context. This review covers four key areas of LLM evaluation: principles of study design, statistical methods, capability evaluation and clinical context evaluation. Capability evaluation considers different benchmarks, including multiple-choice, agentic and multi-turn benchmarks, alongside operational metrics like token usage. Clinical context evaluation addresses establishing accuracy of free text outputs, such as human review and LLM-as-a-judge, and clinical trial approaches. Across sections, we describe underlying concepts and potential pitfalls, while emphasizing the importance of aligning evaluation methods with the research question. Together, this article aims to provide a pragmatic basis for designing and executing rigorous evaluations of healthcare LLMs.

Suzannah E McKinney, Phuc Vu, Samuel A Justice +5

Mass General Brigham AI, Mass General Brigham, Boston, MA, USA · Generative AI Laboratory, School of Informatics, University of Edinburgh, Edinburgh, UK · University of Edinburgh Centre for Cardiovascular Science, Edinburgh, UK +2

LLM Evaluation Clinical Language Model Evaluation Medical LLMs

Ask PDF

Feb 27, 2026

When Metrics Disagree: Automatic Similarity vs. LLM-as-a-Judge for Clinical Dialogue Evaluation

As Large Language Models (LLMs) are increasingly integrated into healthcare to address complex inquiries, ensuring their reliability remains a critical challenge. Recent studies have highlighted that generic LLMs often struggle in clinical contexts, occasionally producing misleading guidance. To mitigate these risks, this research focuses on the domain-specific adaptation of Llama-2-7B using the Low-Rank Adaptation (LoRA) technique. By injecting trainable low-rank matrices into the Transformer layers, we efficiently adapted the model using authentic patient-physician transcripts while preserving the foundational knowledge of the base model. Our objective was to enhance precision and contextual relevance in responding to medical queries by capturing the specialized nuances of clinical discourse. Due to the resource-intensive nature of large-scale human validation, the model's performance was evaluated through a dual-track framework: Track A utilized traditional lexical similarity metrics (e.g., BLEU, ROUGE), while Track B employed an "LLM-as-a-Judge" paradigm using GPT-4 for semantic assessment. Our results demonstrate that while the LoRA-enhanced model achieved significant improvements across all quantitative lexical dimensions, a profound disagreement surfaced in the GPT-4 evaluation, which marginally favored the baseline model's conversational flow. This metric divergence underscores a pivotal finding: traditional automated scores may not fully reflect clinical utility. Consequently, we propose that while automated metrics and LLM judges serve as valuable developmental proxies, rigorous validation by human medical experts remains an indispensable requirement for the safe deployment of LLMs in healthcare settings.

Bian Sun, Zhenjian Wang, Orvill de la Torre +1

Heinz College, Carnegie Mellon University · School of Computer Science, Carnegie Mellon University

Clinical Language Model Evaluation Generative AI Evaluation Domain Adaptation for LLMs Medical QA LLM-as-a-Judge

Ask PDF