Importance of Prompt Optimisation for Error Detection in Medical Notes Using Language Models

Craig Myles; Patrick Schrempf; David Harris-Birtill

cs.CLFeb 25, 2026

Importance of Prompt Optimisation for Error Detection in Medical Notes Using Language Models

Authors: Craig Myles, Patrick Schrempf, David Harris-Birtill

Organizations: University of St Andrews, St Andrews, United Kingdom · Canon Medical Research Europe Ltd., Edinburgh, United Kingdom

Abstract

Errors in medical text can cause delays or even result in incorrect treatment for patients. Recently, language models have shown promise in their ability to automatically detect errors in medical text, an ability that has the opportunity to significantly benefit healthcare systems. In this paper, we explore the importance of prompt optimisation for small and large language models when applied to the task of error detection. We perform rigorous experiments and analysis across frontier language models and open-source language models. We show that automatic prompt optimisation with Genetic-Pareto (GEPA) improves error detection over the baseline accuracy performance from 0.669 to 0.785 with GPT-5 and 0.578 to 0.690 with Qwen3-32B, approaching the performance of medical doctors and achieving state-of-the-art performance on the MEDEC benchmark dataset. Code available on GitHub: https://github.com/CraigMyles/clinical-note-error-detection

Figures & tables

Figure 1: Diagram highlighting our experiment workflow, showing that we utilise the MS training and validation subsets from the MEDEC dataset ( Ben Abacha et al., 2025 ) for prompt optimisation with GEPA Agrawal et al. (2025) . The data contains medical text (indicated by the Report text) with binary labels indicating whether or not the text contains an error. The MS and UW test sets are used for evaluation only. We evaluate seven different models, including the GPT-5 frontier model ( OpenAI, 2025 ) and various open-source Qwen3 models ( Yang et al., 2025 ) , in 28 different configurations.

Panel of Figure 2 — Figure 2: Illustrative example from the MS-Train dataset, selected from the shortest 1% of erroneous examples in the dataset. The narrative contains one injected diagnostic medical error (sentence 2). We show the corresponding correction for clarity.

Figure 3: Distribution of sample categories across MEDEC dataset splits.

Figure 4: Performance of different inference–reflector pairs using GEPA optimisation on the MEDEC-MS validation set. a) shows the absolute performance. b) shows the difference between the base prompt and optimised prompt performance.

Model	MS-test	UW-test	MS+UW
Benchmark paper doctors
Doctor #1 Ben Abacha et al. (2025)	0.813	0.760	0.796
Doctor #2 Ben Abacha et al. (2025)	0.689	0.772	0.716
Benchmark paper models (P#1 prompt)
Claude 3.5 Sonnet Ben Abacha et al. (2025)	0.675	0.750	0.702
o1-preview Ben Abacha et al. (2025)	0.729	0.576	0.675

Table 1: Detection accuracy on the MEDEC MS-test and UW-test subsets, and a combined MS+UW score. Paper models use the original benchmark results on each subset; our implementations use the P#1 prompt with three random seeds (mean

\pm

standard deviation). The MS+UW column for our results is a weighted mean using

N_{\mathrm{MS}}=597

and

N_{\mathrm{UW}}=328

. Bold items highlight the best metric in each column, double underlined items show the second best metric, underlined items show the third best metric.

Figure 5: Benchmark P1 accuracy on the combined MEDEC test sets, with MS+UW weighted by their respective sample counts; comparing previously benchmarked Ben Abacha et al. (2025) ; Saeed (2024) (orange) vs our P1 benchmarks (blue) as well as GEPA optimised.

Figure 6: GEPA prompt optimisation performance across reflector and inference model pairings on the MEDEC MS test set (top) and the out of distribution MEDEC UW test set (bottom). Left panels report absolute error detection accuracy for the best GEPA optimised prompt found for each pairing. Right panels show the corresponding improvement over the P1 baseline (optimised minus baseline).

Figure 7: Confusion matrices highlighting shift in performance before and after the introduction of GEPA optimisation for both Qwen3-32B and GPT-5. In both the Qwen3-32B and GPT-5 examples, GEPA optimisation was run with GPT-5 as the reflector model.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Validation performance on MEDEC-MS using P1, measured by error detection accuracy. Qwen3 results are mean with standard deviation bars across three random seeds.

Figure 9: Validation performance on MEDEC-UW using P1, measured by error detection accuracy. Qwen3 results are mean with standard deviation bars across three random seeds.

Explore similar work

MedRECT: A Bilingual Medical Reasoning Benchmark for Error Correction in Clinical Texts
Nov 1, 2025Naoto Iwase, Hiroki Okuyama, Junichiro IwasawaMultilingual Language Model Evaluation Healthcare
Preferred Networks, Inc., Tokyo, Japan · School of Medicine, Nagoya University, Nagoya, Japan
When Chain-of-Thought Backfires: Evaluating Prompt Sensitivity in Medical Language Models
Mar 26, 2026Binesh Sadanandan, Vahid BehzadanPrompt Sensitivity LLM Prompting
SAIL Lab, University of New Haven, West Haven, CT 06516, USA
MedGuards: Multi-Agent System for Reliable Medical Error Detection and Correction
Jun 24, 2026Congbo Ma, Hu Wang, Yichun Zhang +1LLM Guardrails LLM Safety
New York University Abu Dhabi · Khalifa University · New York University

Nov 1, 2025J/K move · Enter open · S save

MedRECT: A Bilingual Medical Reasoning Benchmark for Error Correction in Clinical Texts

Naoto Iwase, Hiroki Okuyama, Junichiro Iwasawa

Preferred Networks, Inc., Tokyo, Japan · School of Medicine, Nagoya University, Nagoya, Japan

Multilingual Language Model Evaluation Healthcare Clinical Language Model Evaluation Clinical NLP Medical LLMs

Open paper AskPDF arXiv

Large language models (LLMs) show promise in medical applications, but their ability to detect and correct errors in clinical texts remains under-evaluated, particularly beyond English. We introduce MedRECT, a bilingual benchmark for Japanese and English that formulates medical error handling as three subtasks: error detection, error sentence extraction, and error correction. MedRECT-ja contains 663 samples derived from the Japanese Medical Licensing Examinations, while the separately sourced MedRECT-en contains 458 samples curated from MEDEC. We evaluate 11 LLMs across 17 configurations that cover proprietary and open-weight models, medical-domain specialization, and multiple reasoning settings. Qwen3-32B scores higher in its thinking mode than in its non-thinking mode on error detection F1 and sentence extraction accuracy in both subsets, with sentence extraction accuracy higher by 24.5 percentage points on MedRECT-ja and 10.3 on MedRECT-en. Several leading general-purpose reasoning models outperform all three evaluated medical-domain models on these two subtasks. Most models have lower point estimates on the Japanese subset, although absolute scores are not directly comparable because the subsets differ in source material and error distributions. LoRA fine-tuning yields higher sentence extraction accuracy and higher point estimates on all three reference-based correction similarity metrics in both languages. MedRECT provides an open, reusable evaluation resource for studying medical error correction and reasoning across Japanese and English. Our dataset and code are available at https://github.com/pfnet-research/medrect.

Abstract

Explore similar work

MedRECT: A Bilingual Medical Reasoning Benchmark for Error Correction in Clinical Texts

When Chain-of-Thought Backfires: Evaluating Prompt Sensitivity in Medical Language Models

MedGuards: Multi-Agent System for Reliable Medical Error Detection and Correction