cs.CLMar 6, 2026

Training a Large Language Model for Medical Coding Using Privacy-Preserving Synthetic Clinical Data

Authors: John Cook, Michael Wyatt, Peng Wei, Iris Chin, Santosh Gupta, Van Zyl Van Vuuren, Richie Siburian, Amanda Spicer, 

Organizations: Veradigm · Snowflake

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Improving the accuracy and reliability of medical coding reduces clinician burnout and supports revenue cycle processes, freeing providers to focus more on patient care. However, automating the assignment of ICD-10-CM and CPT codes from clinical documentation remains a challenge due to heterogeneous records, nuanced coding guidelines, and long-tail distributions. Large language models have been proposed to help or automate specific medical coding tasks. However, foundation models are not explicitly trained for medical coding and zero-shot coding has yielded poor results. We investigate whether a modern open-weight foundation model can be adapted for an expert-level medical coding task using privacy-preserving synthetic training data derived from electronic health records. We fine-tune Llama 3-70B on pairs of clinical notes and gold codes generated from EHR-grounded templates and coding policies, then evaluate exact-code prediction for ICD-10-CM and CPT. A zero-shot baseline with the unadapted model achieved an F1 score of 0.18 for exact code match. After fine-tuning on the synthetic corpus, exact-match F1 exceeded 0.70, representing a large absolute gain across both code systems. Notably, performance remained high on complex categories that often require multi-step clinical reasoning and code composition, including Advanced Illness and Frailty classes, and the model retained its performance on medical comprehension tasks. These results indicate that synthetic, policy-aware data can efficiently teach a general-purpose large language model to support precise medical coding without exposing protected health information. The approach offers a practical path for training coding agents safely and iteratively on specific tasks that represent real-world populations.

Figures & tables

Figure 1: Figure 1. Exact-match ICD-10-CM and CPT coding. F1 scores for the fine-tuned Llama-3-70B model evaluated on held-out synthetic (A) ICD-10-CM and (B) CPT datasets. Results reflect baseline inference without prompt engineering or semantic retrieval assistance.
Figure 2: Figure 2. ICD-10-CM coding performance across hierarchical levels. ICD-10-CM performance is reported at increasing levels of diagnostic specificity, from coarse category identification to exact ICD-10 code (Level 3) and exact code with supporting evidence attribution (Level 4). Performance is highest at the coarsest level and declines gradually as diagnostic specificity increases. CPT coding performance, evaluated independently using exact set matching, achieves an F1 score of 0.736. Results reflect baseline inference without prompt engineering or semantic retrieval assistance.
Figure 3: Breakdown of ICD-10-CM coding performance across clinical domains and diagnostic groupings. (a) Weighted mean F1 score by clinical domain for Advanced Illness, Frailty, and Social Determinants of Health (SDoH). Error bars represent the weighted standard deviations. (b) Lowest-performing ICD-10-CM category-level codes with at least 10 evaluation cases. (c) Mean F1 score by ICD-10-CM chapter. Results are computed over 761 held-out synthetic clinical charts.
Figure 4: Figure 4. Training workflow including data augmentation and sequence packing. A. Synthetic notes and linked ICD-10-CM and CPT labels are bundled. The training dataset is augmented by concatenating multiple notes to increase contextual difficulty and formatted using structured prompts. This amounts to 30% of the original volume of notes. Data augmentation and packing are applied during training only. B. Samples are prepared in a format including system prompt, instruction, and output format. C. Samples are packed into fixed-length sequences to reduce padding and improve computational efficiency. D. The resulting sequences are used for supervised fine-tuning of the Llama-3-70B model.
Figure 5: Figure 5. Relationship between ICD-10-CM training frequency and evaluation performance. Scatter plot of category-level F1 score in the evaluation set versus the number of occurrences in the training data (categories with ≥\geq 10 evaluation cases). Low-frequency categories exhibit high variance in performance, while higher-frequency categories consistently achieve strong F1 scores, indicating a frequency threshold effect rather than a linear relationship.
Figure 6: Figure 6. Human expert evaluation of ICD-10-CM coding. Expert-validated ICD-10-CM coding accuracy on 100 synthetic charts.
CategoryPrecisionRecallF1
Advanced Illness (n=896)0.9008 (0.1008)0.8380 (0.1184)0.8630 (0.1039)
Frailty (n=2128)0.8727 (0.1086)0.8749 (0.1303)0.8727 (0.1173)
SDoH (n=1176)0.7716 (0.1391)0.7640 (0.1578)0.7668 (0.1491)
Combined (n=4200)0.8504 (0.1268)0.8360 (0.1441)0.8410 (0.1329)
Combined w/o SDoH (n=3024)0.8810 (0.1071)0.8639 (0.1280)0.8698 (0.1136)
Table 1: Table 1. ICD-10-CM coding performance by clinical domain. Weighted mean and standard deviation of precision, recall, and F1 score for Advanced Illness, Frailty, and Social Determinants of Health (SDoH). Metrics were first computed at the ICD-10 code level and then aggregated across codes using case-count–weights. Values reflect performance of the fine-tuned model on held-out synthetic evaluation data.
BenchmarkBaselineFine-tunedDiff
Average.8286.7994-0.0292
MedMCQA (acc).7196.6842-0.0354
MedQA (acc).7879.7133-0.0746
MMLU Anatomy.8370.7852-0.0518
MMLU Clinical Knowledge.8415.8377-0.0038
MMLU College Biology.9167.9097-0.0070
Table 2: Performance on medical comprehension benchmarks before and after fine-tuning. Accuracy on standard medical question-answering and reasoning benchmarks for the baseline and fine-tuned Llama 3-70B models. Results indicate modest changes following fine-tuning, with no evidence of catastrophic degradation in general medical knowledge.
CategoryPrecisionRecallF1
ICD-10 (Level 0)0.3963 (0.0172)0.9301 (0.0160)0.5365 (0.0168)
ICD-10 (Level 3)0.3153 (0.0178)0.8621 (0.0208)0.4416 (0.0186)
ICD-10 (Level 1)0.3309 (0.0178)0.8820 (0.0199)0.4618 (0.0185)
ICD-10 (Level 2)0.3157 (0.0178)0.8621 (0.0208)0.4422 (0.0185)
Table 3: Table 3. : Expert-validated ICD-10-CM coding performance by hierarchy level. Mean and standard error of precision, recall, and F1 score for the fine-tuned model with post-processing, evaluated against expert-accepted labels across 100 synthetic notes spanning Advanced Illness, Frailty, and Social Determinants of Health. Predictions were filtered to retain only domain-relevant codes prior to evaluation. ICD-10 hierarchy level ranges from Level 0 to Level 3. Metrics were aggregated at the chart level.
More in this topic

Automated Medical Coding

Automated medical coding uses clinical records and coding rules to assign standardized diagnosis and procedure codes for billing, care, and research.

See all 23 papers

Explore similar work

CardsList
Jun 11, 2026

Can Post-Training Turn LLMs into Good Medical Coders? An Empirical Study of Generative ICD Coding

Automated International Classification of Diseases (ICD) coding is a core medical-coding task for billing, epidemiology, and clinical decision support. Generative large language models (LLMs) are often reported as weak medical coders, but this finding mainly comes from inference-time settings such as prompting, retrieval, reranking, or tool use, leaving the role of task-specific post-training underexplored. We present a controlled empirical study of post-training for generative ICD coding, comparing discriminative baselines with LLM coders across prompting, supervised fine-tuning, and reinforcement learning under a common protocol and metric set. To our knowledge, this is the first study to evaluate RL-based post-training for generative LLM coders in ICD coding. We further introduce PHI, a diagnostic curriculum that extends GRPO to refine missed-code cases. Our results show that prompting-only evaluation substantially underestimates the potential of LLMs for ICD coding. SFT provides the main capability jump, GRPO further improves code-set prediction beyond SFT, and PHI provides targeted gains on macro-level performance. These findings suggest that the main bottleneck is not the generative formulation alone, but how the model is adapted and optimized for full-taxonomy recall. We release our code, data splits, and checkpoints at https://github.com/AlexandreWANG915/LLM4ICD.

Ziqing Wang, Weihao Li, Shijie Chen +2

Northwestern University

Supervised Fine-TuningAutomated Medical CodingRL for Generative ModelsLLM Post-TrainingMedical LLMs
AskPDF
Apr 9, 2026

Enhancing LLM Medical Coding with Structured External Knowledge

Accurate medical coding requires consulting authoritative resources such as the ICD tabular list and coding guidelines. Existing LLM-based automated methods largely rely on LLMs' internal knowledge, which is prone to hallucination and cannot keep pace with guideline updates. We introduce RAG-Coding, an agentic, training-free method that augments LLMs with structured external knowledge: the tabular list is encoded as a knowledge graph capturing hierarchical and instructional code relationships, and the guidelines are distilled into concise, code-specific summaries rather than retrieved as raw text. To enable our study, we also introduce MDACE-2025, expert re-annotations of the MDACE dataset under the 2025 ICD-10-CM/PCS guidelines, adding code sequencing and justification comments. On MDACE, RAG-Coding outperforms the best LLM-based baseline by 3--13% in micro-F1 across five LLM backbones, and achieves comparable micro- and macro-F1 to the supervised state-of-the-art, with higher recall (++11%) at the cost of precision (−-6%). On MDACE-2025, RAG-Coding outperforms all baselines, demonstrating effective generalisation to updated guidelines. Ablations confirm stepwise gains, highlighting the importance of integrating structured external knowledge for LLM-based medical coding.

Yidong Gan, David D. Nguyen, Yang Lin +4

Oracle Health and AI · The University of Sydney · CSIRO Data61

Automated Medical CodingMedical LLMsKnowledge Augmentation for Language ModelsGraph Retrieval-Augmented Generation
AskPDF
Feb 27, 2026

A medical coding language model trained on clinical narratives from a population-wide cohort of 1.8 million patients

Medical coding translates clinical documentation into standardized codes for billing, research, and public health, but manual coding is time-consuming and error-prone. Existing automation efforts rely on small datasets that poorly represent real-world patient heterogeneity. We trained a language model on 5.8 million electronic health records from 1.8 million patients across nearly all specialties in Eastern Denmark (2006--2016) to predict ICD-10 codes from clinical notes, medications, and laboratory results. Evaluated on 270,000 held-out patients, the model achieved a micro F1 of 71.8% and a top-10 recall of 95.5%. Performance varied by specialty (F1: 53--91%), with higher scores in specialties with well-defined diagnostic criteria. Codes appearing predominantly as secondary diagnoses had markedly lower F1 scores. For three such codes (suicide-related behaviors, weight disorders, and hypertension), the model identified thousands of uncoded cases, of which 76-86% were confirmed valid upon manual review, suggesting systematic under-coding rather than model error. These findings suggest under-coding of secondary diagnoses in Eastern Denmark during this period, with potential implications for epidemiological research, public health surveillance, and understanding of multimorbidity. Similar time constraints and reimbursement structures in other healthcare systems suggest this may not be isolated to this dataset. The model can automate coding for approximately 50% of cases and provide accurate suggestions for most others, and may offer a practical solution to help capture missed secondary conditions.

Joakim Edin, Sedrah Butt Balaganeshan, Annike Kjølby Kristensen +3

Section for Health Data Science and AI, Department of Public Health, University of Copenhagen, Denmark · Corti, Copenhagen, Denmark · Novo Nordisk Foundation Center for Protein Research, University of Copenhagen +1

Automated Medical CodingMedical LLMs
AskPDF