Assessing Large Language Models for Medical QA: Zero-Shot and LLM-as-a-Judge Evaluation

Shefayat E Shams Adib; Ahmed Alfey Sani; Ekramul Alam Esham; Ajwad Abrar; Tareque Mohmud Chowdhury

cs.CLFeb 16, 2026

Assessing Large Language Models for Medical QA: Zero-Shot and LLM-as-a-Judge Evaluation

Authors: Shefayat E Shams Adib, Ahmed Alfey Sani, Ekramul Alam Esham, Ajwad Abrar, Tareque Mohmud Chowdhury

Organizations: Department of Computer Science and Engineering, Islamic University of Technology, Gazipur, Bangladesh

Abstract

Recently, Large Language Models (LLMs) have gained significant traction in medical domain, especially in developing a QA systems to Medical QA systems for enhancing access to healthcare in low-resourced settings. This paper compares five LLMs deployed between April 2024 and August 2025 for medical QA, using the iCliniq dataset, containing 38,000 medical questions and answers of diverse specialties. Our models include Llama-3-8B-Instruct, Llama 3.2 3B, Llama 3.3 70B Instruct, Llama-4-Maverick-17B-128E-Instruct, and GPT-5-mini. We are using a zero-shot evaluation methodology and using BLEU and ROUGE metrics to evaluate performance without specialized fine-tuning. Our results show that larger models like Llama 3.3 70B Instruct outperform smaller models, consistent with observed scaling benefits in clinical tasks. It is notable that, Llama-4-Maverick-17B exhibited more competitive results, thus highlighting evasion efficiency trade-offs relevant for practical deployment. These findings align with advancements in LLM capabilities toward professional-level medical reasoning and reflect the increasing feasibility of LLM-supported QA systems in the real clinical environments. This benchmark aims to serve as a standardized setting for future study to minimize model size, computational resources and to maximize clinical utility in medical NLP applications.

Figures & tables

Fig. 1: Methodology pipeline for evaluating LLM performance on medical question answering tasks.

Question: Hello doctor, I have recently been diagnosed with type 2 diabetes. Could you provide me with a simplified explanation of how this condition affects my body and what steps I can take to keep my blood sugar levels stable throughout the day?

Answer: Hello, Welcome to icliniq.com. I read your query and understand your concern. If you have already been diagnosed with diabetes, it is mandatory to keep it under control at all times with the help of diet. Exercise and maintain an active lifestyle and medications prescribed by the treating doctor. Walking, yoga, and exercise as health permits started gradually will help. Check your blood sugar levels and other parameters like liver function test, kidney function test, and lipid profile regularly. Watch what you eat. Small frequented meals might help. Stay away from sweets, sugar, honey, jaggery, marmalade, coconut water, juices, biscuits, rusk, and junk packaged food. Avoid potatoes, white rice, green peas, beetroot, and carrot. Take the dieticians opinion for a detailed diet chart depending on height weight religion culture body mass index likes and dislikes. Avoid stress and tension. Deep breathing and meditation will help. Go for green leafy vegetables, salad, and fruits take only papaya, guava, and kiwi. Replace white rice with brown rice and bread and flour with multi-grains. Avoid fried oily junk alcohol smoking. Take foods with low glycemic index. I hope I cleared all your doubts. Thank you.

TABLE I: Sample Question-Answer Pair from the iCliniq Dataset

Model	BLEU-1	BLEU-4	ROUGE-1	ROUGE-L
GPT-3.5	0.0243	0.0009	0.0022	0.0019
T5	0.0984	0.0080	0.0017	0.0014
GPT-2	0.0998	0.0085	0.0019	0.0012

TABLE III: Baseline MedLM Model Performance on the iCliniq Dataset (from prior work) [ 23 ]

Model	Overall Score
Llama 3.3 70B Instruct	4.40
Llama-4-Maverick 17B 128E	4.23
Llama-3-8B-Instruct	3.77
Llama 3.2 3B	3.20
GPT-5-mini	3.16

TABLE IV: LLM-as-a-Judge Overall Performance Rankings

Explore similar work

Comparative Analysis of Large Language Models in Healthcare
Apr 11, 2026Subin Santhosh, Farwa Abbas, Hussain Ahmad +1Medical LLMs Clinical Language Model Evaluation
Adelaide University, Australia · South Australian Health and Medical Research Institute, Australia
Evaluating Small Open LLMs for Medical Question Answering: A Practical Framework
Apr 12, 2026Avi-ad Avraam BuskilaLLM Evaluation LLM Reliability
Department of Information Science and Applied Artificial Intelligence, Bar-Ilan University, Ramat-Gan, Israel
Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance?
Jun 10, 2026Antoni Lasik, Jakub Pokrywka, Łukasz Grzybowski +7LLM Evaluation LLM Reliability
NASK National Research Institute · Adam Mickiewicz University · ARAAI Poland +4

Apr 11, 2026J/K move · Enter open · S save

Comparative Analysis of Large Language Models in Healthcare

Subin Santhosh, Farwa Abbas, Hussain Ahmad, Claudia Szabo

Adelaide University, Australia · South Australian Health and Medical Research Institute, Australia

Medical LLMs Clinical Language Model Evaluation Medical QA Text Summarization

Open paper AskPDF arXiv

Background: Large Language Models (LLMs) are transforming artificial intelligence applications in healthcare due to their ability to understand, generate, and summarize complex medical text. They offer valuable support to clinicians, researchers, and patients, yet their deployment in high-stakes clinical environments raises critical concerns regarding accuracy, reliability, and patient safety. Despite substantial attention in recent years, standardized benchmarking of LLMs for medical applications has been limited. Objective: This study addresses the need for a standardized comparative evaluation of LLMs in medical settings. Method: We evaluate multiple models, including ChatGPT, LLaMA, Grok, Gemini, and ChatDoctor, on core medical tasks such as patient note summarization and medical question answering, using the open-access datasets, MedMCQA, PubMedQA, and Asclepius, and assess performance through a combination of linguistic and task-specific metrics. Results: The results indicate that domain-specific models, such as ChatDoctor, excel in contextual reliability, producing medically accurate and semantically aligned text, whereas general-purpose models like Grok and LLaMA perform better in structured question-answering tasks, demonstrating higher quantitative accuracy. This highlights the complementary strengths of domain-specific and general-purpose LLMs depending on the medical task. Conclusion: Our findings suggest that LLMs can meaningfully support medical professionals and enhance clinical decision-making; however, their safe and effective deployment requires adherence to ethical standards, contextual accuracy, and human oversight in relevant cases. These results underscore the importance of task-specific evaluation and cautious integration of LLMs into healthcare workflows.

Abstract

Explore similar work

Comparative Analysis of Large Language Models in Healthcare

Evaluating Small Open LLMs for Medical Question Answering: A Practical Framework

Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance?