A medical coding language model trained on clinical narratives from a population-wide cohort of 1.8 million patients
Authors: Joakim Edin, Sedrah Butt Balaganeshan, Annike Kjølby Kristensen, Lars Maaløe, Ioannis Louloudis, Søren Brunak
Organizations: Section for Health Data Science and AI, Department of Public Health, University of Copenhagen, Denmark · Corti, Copenhagen, Denmark · Novo Nordisk Foundation Center for Protein Research, University of Copenhagen · Danish Research Institute for Suicide Prevention, Mental Health Centre Copenhagen, Denmark
Medical coding translates clinical documentation into standardized codes for billing, research, and public health, but manual coding is time-consuming and error-prone. Existing automation efforts rely on small datasets that poorly represent real-world patient heterogeneity. We trained a language model on 5.8 million electronic health records from 1.8 million patients across nearly all specialties in Eastern Denmark (2006--2016) to predict ICD-10 codes from clinical notes, medications, and laboratory results. Evaluated on 270,000 held-out patients, the model achieved a micro F1 of 71.8% and a top-10 recall of 95.5%. Performance varied by specialty (F1: 53--91%), with higher scores in specialties with well-defined diagnostic criteria. Codes appearing predominantly as secondary diagnoses had markedly lower F1 scores. For three such codes (suicide-related behaviors, weight disorders, and hypertension), the model identified thousands of uncoded cases, of which 76-86% were confirmed valid upon manual review, suggesting systematic under-coding rather than model error. These findings suggest under-coding of secondary diagnoses in Eastern Denmark during this period, with potential implications for epidemiological research, public health surveillance, and understanding of multimorbidity. Similar time constraints and reimbursement structures in other healthcare systems suggest this may not be isolated to this dataset. The model can automate coding for approximately 50% of cases and provide accurate suggestions for most others, and may offer a practical solution to help capture missed secondary conditions.
Figures & tables
ICD-10 Level 3
Level 5
Training data
50K
100K
200K
500K
1M
1.8M
1M
F1 Micro
59.9%
61.7%
65.8%
69.0%
70.1%
71.8%
54.6%
F1 Macro
25.6%
26.3%
30.5%
41.0%
43.6%
47.3%
10.1%
Exact Match Ratio
37.0%
42.5%
47.3%
51.7%
52.0%
54.6%
33.2%
Recall@5
82.2%
85.3%
87.8%
89.8%
90.5%
91.4%
78.4%
Recall@10
88.7%
90.9%
93.0%
94.4%
94.9%
95.5%
86.0%
Table 1: Model performance when trained on progressively larger data sets. All models are evaluated on the same test set using the human-annotated codes as the ground-truth labels.
Figure 1: The model’s F1 score for each specialty. Each dot is the model’s median F1 score for a specific department. The figure reveals a pattern in which the top specialties often involve planned admissions, while the bottom specialties represent patients with an average of many comorbidities. Departments with more than one hundred examples in the test set are included only.
Figure 2: a) The F1 score (y-axis) versus the number of occurrences in the training data (x-axis) for each ICD-10 code (level-3). b) The frequency of codes for secondary diagnoses in the training data (x-axis) and their F1 score (y-axis).
Figure 3: a) Recall@5 for primary diagnoses and secondary diagnoses, and b) Recall@10. Recall@5 and Recall@10 measure the frequency of human-annotated codes among the model’s top 5 and top 10 predictions.
Figure 4: The data preprocessing pipeline for the study.
Figure 5: The interface we used for our manual explainability analysis. The clinical note is shown to the left, and the model’s predictions are to the right. When hovering the mouse over a code prediction, the AttInGrad explanations are highlighted in the text of the clinical note. By visualizing the factors that caused the model prediction, we could quickly identify the elements that led to the human-model disagreements. The text in this example is AI-generated and does not relate to the data used in the study.
Clinical coding maps clinical documentation to standardized medical codes, an essential yet time-consuming administrative task that could benefit from automation. Current models on ICD coding are typically optimized for codes from a specific ICD version. However, in reality, ICD systems evolve continuously, and different versions are adopted across time periods and regions. Moreover, ICD coding suffers from the long-tail problem, and rare code performance can be a bottleneck for developing implementable models. We examine whether it is viable to train version-independent models by combining data annotated in different ICD versions, which may help address these challenges. We add ICD-9 data to the training of a modified label-wise attention model for ICD-10 prediction, and find that despite the version mismatch, adding ICD-9 yields a 27% increase in micro F1 for 18K rare ICD codes compared to training on ICD-10 alone. On 8K frequent ICD-10 codes, the multi-version training also substantially improves macro metrics, with far fewer model parameters.
Improving the accuracy and reliability of medical coding reduces clinician burnout and supports revenue cycle processes, freeing providers to focus more on patient care. However, automating the assignment of ICD-10-CM and CPT codes from clinical documentation remains a challenge due to heterogeneous records, nuanced coding guidelines, and long-tail distributions. Large language models have been proposed to help or automate specific medical coding tasks. However, foundation models are not explicitly trained for medical coding and zero-shot coding has yielded poor results. We investigate whether a modern open-weight foundation model can be adapted for an expert-level medical coding task using privacy-preserving synthetic training data derived from electronic health records. We fine-tune Llama 3-70B on pairs of clinical notes and gold codes generated from EHR-grounded templates and coding policies, then evaluate exact-code prediction for ICD-10-CM and CPT. A zero-shot baseline with the unadapted model achieved an F1 score of 0.18 for exact code match. After fine-tuning on the synthetic corpus, exact-match F1 exceeded 0.70, representing a large absolute gain across both code systems. Notably, performance remained high on complex categories that often require multi-step clinical reasoning and code composition, including Advanced Illness and Frailty classes, and the model retained its performance on medical comprehension tasks. These results indicate that synthetic, policy-aware data can efficiently teach a general-purpose large language model to support precise medical coding without exposing protected health information. The approach offers a practical path for training coding agents safely and iteratively on specific tasks that represent real-world populations.
Automatic medical coding assigns ICD codes to clinical notes, but it remains challenging due to long documents, imbalanced label distributions, and diverse terms. These challenges are especially severe for rare codes, which have limited training instances and are easily confused with semantically similar labels. We introduce CoLa-ICD, a knowledge-enhanced framework for long-tail prediction. CoLa-ICD enriches ICD labels with external terms, models dependencies among related codes, and learns stronger alignment between label semantics and clinical evidence for long-tail prediction. Experiments show that CoLa-ICD improves long-tail prediction with larger gains in larger and sparser label spaces and achieves state-of-the-art performance in AUC, F1, and P@k. Our code is available at https://github.com/youwillbethebest/Cola-ICD.