cs.CLApr 8, 2026

To Adapt or not to Adapt, Rethinking the Value of Medical Knowledge-Aware Large Language Models

Authors: Ane G. Domingo-Aldama, Iker De La Iglesia, Maitane Urruela, Aitziber Atutxa, Ander Barrena

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

BACKGROUND: Recent studies have shown that domain-adapted large language models (LLMs) do not consistently outperform general-purpose counterparts on standard medical benchmarks, raising questions about the need for specialized clinical adaptation. METHODS: We systematically compare general and clinical LLMs on a diverse set of multiple choice clinical question answering tasks in English and Spanish. We introduce a perturbation based evaluation benchmark that probes model robustness, instruction following, and sensitivity to adversarial variations. Our evaluation includes, one-step and two-step question transformations, multi prompt testing and instruction guided assessment. We analyze a range of state-of-the-art clinical models and their general-purpose counterparts, focusing on Llama 3.1-based models. Additionally, we introduce Marmoka, a family of lightweight 8B-parameter clinical LLMs for English and Spanish, developed via continual domain-adaptive pretraining on medical corpora and instructions. RESULTS: The experiments show that clinical LLMs do not consistently outperform their general purpose counterparts on English clinical tasks, even under the proposed perturbation based benchmark. However, for the Spanish subsets the proposed Marmoka models obtain better results compared to Llama. CONCLUSIONS: Our results show that, under current short-form MCQA benchmarks, clinical LLMs offer only marginal and unstable improvements over general-purpose models in English, suggesting that existing evaluation frameworks may be insufficient to capture genuine medical expertise. We further find that both general and clinical models exhibit substantial limitations in instruction following and strict output formatting. Finally, we demonstrate that robust medical LLMs can be successfully developed for low-resource languages such as Spanish, as evidenced by the Marmoka models.

Figures & tables

Figure 1: Instance of the CareQA-En dataset.
Figure 2: Overview of the Marmoka variants based on Llama 3.1-8B-Instruct. Four final models were developed: two trained on medical corpora (one in Spanish and the other in English), and two obtained by merging these models with an intermediate Marmoka model trained using medical instruction data.
Figure 3: Percentage of answers correctly formatted according to task instructions for each evaluated model. Models shown in orange are excluded from further experiments due to a low rate of instruction adherence.
More in this topic

Medical LLMs

Medical large language models are AI systems adapted to clinical and biomedical data for health reasoning, information extraction, and communication.

See all 396 papers

Explore similar work

CardsList
Jun 17, 2026

Trade-offs in Medical LLM Adaptation: An Empirical Study in French QA

The development of large language models (LLMs) has led to an increased focus on their adaptation to specialized domains and languages, yet the effectiveness of domain adaptation strategies remains unclear. We present a study of medical domain adaptation using French medical question-answering (QA) as a case study. We compare continual pretraining (CPT), supervised fine-tuning (SFT), and their combination across three model families, multiple sizes, and three initialization types, explicitly disentangling adaptation effects from base model choice. We evaluate both multiple-choice (MCQA) and open-ended QA (OEQA) under greedy and constrained decoding using automatic metrics and LLM-as-a-Judge evaluation. For MCQA, CPT+SFT most often achieves the best scores, but gains over SFT are small and frequently not statistically significant, making SFT a strong and cost-effective default. For OEQA, CPT consistently improves overlap-based metrics, while SFT often degrades generation quality; instruction tuning and CPT+SFT are preferred by LLM-based evaluation. Cross-lingual experiments further show effective transfer from French adaptation to English benchmarks. Overall, we provide practical guidelines for selecting adaptation strategies under computational constraints.

Ikram Belmadani, Oumaima El Khettari, Carlos Ramisch +3

Aix-Marseille Univ., CNRS, LIS UMR 7020, 13000 Marseille, France · Nantes Univ., École Centrale Nantes, CNRS, LS2N UMR 6004, 44000 Nantes, France · Grenoble Alpes Univ., CNRS, INRIA, Grenoble INP, LIG UMR 5217, 38000 Grenoble, France

LLM Fine-TuningDomain-Adaptive PretrainingDomain Adaptation for LLMsMedical QAClinical Language Model EvaluationMedical LLMs
AskPDF
Jun 10, 2026

Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance?

Large language models (LLMs) in medicine are mainly evaluated using multiple-choice question answering (MCQA), which can overestimate real clinical ability due to guessing strategies and answer biases. To address these limitations, we introduce an expanded and more challenging benchmark based on Polish medical exams, adding over 15,000 questions, two new domains, and four structural modifications that reduce MCQA-specific artifacts and better test reasoning. We evaluate 21 LLMs and show that evaluation design strongly affects results. Under our harder setup, the best model (Qwen3.5-122B) drops by 28.4 and 31 pp on English and Polish exams, respectively. Despite low evidence of data contamination, standard MCQA scores do not reliably reflect true medical competence. To facilitate further research, we make our benchmark publicly available.

Antoni Lasik, Jakub Pokrywka, Łukasz Grzybowski +7

NASK National Research Institute · Adam Mickiewicz University · ARAAI Poland +4

LLM EvaluationLLM ReliabilityMedical QALanguage Model Bias EvaluationClinical Language Model Evaluation
AskPDF
Jul 31, 2026

Bridging the English-Arabic Medical Knowledge Gap: Targeted Low-Rank Adaptation via Causal Layer Selection

Large Language Models (LLMs) perform strongly in English medical tasks but degrade substantially in Arabic, a gap widely attributed to limited training data. We systematically investigate this assumption via tuned lens probing and causal activation patching, and find that Arabic medical knowledge is present in intermediate model representations but fails to surface at the output. This mechanistic insight motivates a targeted adaptation strategy: rather than fine-tuning the full network, we propose Targeted Low-Rank Adaptation (TLoRA), restricted to the layer window where cross-lingual representations diverge, upstream of the output layers where the failure manifests. We evaluate TLoRA on multiple-choice medical QA, where our approach outperforms full-network LoRA, zero-shot, and few-shot baselines. We further evaluate it on short-answer generation and multi-turn clinical dialogue, where it performs competitively without the need for task-specific finetuning. We additionally introduce AraClinicDialog, a clinician-constructed Arabic medical dialogue benchmark in MSA with validated variants across four Arabic dialects. Together, these contributions demonstrate that mechanistic diagnosis can serve as a practical guide for targeted adaptation in underrepresented-language medical LLMs.

Chaimae Abouzahir, Musa Khan, Hala Ali-Hassan +7

New York University Abu Dhabi · Cleveland Clinic Abu Dhabi

LLM Fine-TuningLow-Rank AdaptationMedical QALow-Resource Language ProcessingCausal Interventions in Language ModelsMedical LLMs
AskPDF