Call-Efficient ICD-10 Code Prediction via Supervised Candidate Generation and Selective LLM Invocation
Authors: Ilia Kopanichuk, Daniil Kartashev, Yuri Bolkonsky, Petr Anokhin, Vladimir Makharev, Iaroslav Bespalov, Dmitry V. Dylov
Organizations: AIRI, Moscow, Russia · Moscow Institute of Physics and Technology, Dolgoprudny, Russia · Lomonosov Moscow State University, Moscow, Russia · Innopolis University, Innopolis, Russia · Skolkovo Institute of Science and Technology, Moscow, Russia
Automatic assignment of ICD-10 codes from short clinical diagnosis text is a practical medical text classification and normalization task. The task is difficult because diagnosis phrases are short and noisy, the ICD-10 label space is hierarchical, and the class distribution is strongly imbalanced. This paper presents a call-efficient ICD-10 code prediction pipeline that combines transformer embeddings, tree-based supervised candidate generation, candidate-level reranking, and a gated LLM assistance. The evaluation uses a leakage-safe validation protocol: model and threshold selection are based on out-of-fold cross-validation predictions, while final reporting is performed on a locked test set with no shared source-file groups. The supervised conditional parent-code candidate generator reaches 74.32% Top-1 accuracy on the locked test set and 93.05% Top-10 candidate coverage. A direct zero-shot GPT-5.6-Sol baseline reaches 81.01% Top-1 accuracy, but requires an invocation of this large model for every diagnosis. The final deployable system, Pre-LLM fusion, reaches 81.24% OOF-CV accuracy and 83.19% locked-test accuracy while calling the LLM for only 31.97% of cases. This improves the locked-test error rate from 25.68% for the supervised parent-code baseline to 16.81%, corresponding to a 34.5% relative error reduction. The results show that the most effective role of the LLM is controlled reranking on uncertain examples rather than unrestricted coding or full-dataset invocation.
Figures & tables
Work or system
Published setting and reported result
Relation to the present study
RuCCoD BERT pipeline ( Nesterov et al. 2025 )
Russian NER plus entity-level ICD-10 assignment; F1 = 0.525 and accuracy = 0.356.
The closest Russian academic benchmark; our work studies direct ICD-10 letter and parent-code prediction from diagnosis text.
Zero-shot retrieval-augmented ICD assignment; F1 = 0.458 and accuracy = 0.297.
Shows that retrieval helps LLM baselines, but supervised candidate generation remains important for Russian ICD linking.
RuBioBERT / RuBioRoBERTa ( Yalunin, Nesterov, and Umerenkov 2022 )
Russian biomedical encoders; RuCCoD reports F1 = 0.77 for diagnosis-entity extraction.
Useful for entity recognition and representation learning; our experiments use transformer embeddings and tree-based classifiers.
Russian ICD-10 chapter classifier and per-letter subgroup model series ( Dmitry43243242 2026a ; Dmitry43243242 2026b )
Public short-text classifiers; the chapter model reports top-1 accuracy = 0.7121 and top-3 accuracy = 0.8724.
These models predict coarse chapters or letter-specific subgroups, while our analysis focuses on parent-code prediction and selective LLM use.
Table 1: Closest Russian-language ICD-10 coding analogues and the position of the present study. The results are not ranked against the present model because the datasets and label spaces differ.
Train
Locked test
Size of sample
8,552
1,511
Number of unique parent codes
637
330
Table 2: Train-test dataset statistics for the final split. Parent codes correspond to the three-character ICD-10 code prefixes.
Figure 1: Training-set distribution by ICD-10 letter. The frequencies are strongly imbalanced, with letters such as J, K, M, and N dominating the dataset.
Figure 2: Final call-efficient pipeline for ICD-10 parent-code prediction. The supervised candidate generator produces a shared top-10 list. A no-call candidate-ranker route and a selectively invoked LLM-reranker route then converge in the Pre-LLM fusion selector, which outputs the final parent code. Green denotes supervised or no-call processing, while blue denotes selectively invoked LLM processing.
Figure 3: Letter-level distribution and per-class quality diagnostics. Frequent letters are overrepresented among predictions, while rare letters are underrepresented.
Figure 4: Normalized confusion matrix for ICD-10 letter prediction. Rows correspond to true letters and columns correspond to predicted letters. Off-diagonal cells reveal systematic error destinations.
Figure 5: Relationship between class support and OvR quality metrics. Recall and F1-score increase strongly with support, while precision is less support-dependent.
Method
Top-1
Top-3
Top-5
Top-10
MRR@5
Supervised parent stage
74.32%
88.15%
91.00%
93.05%
81.42%
Full LLM selection
80.41%
91.46%
92.46%
93.05%
85.85%
Table 3: Parent-code candidate-generation and reranking quality on the locked test set. The full-invocation LLM setting serves as an accuracy reference and requires an LLM call for every case; it is not the final selective system.
Method
OOF-CV
Locked test
LLM calls
Zero-shot GPT-5.6-Sol
—
81.01%
100.00%
Pre-LLM gate
79.51%
82.13%
31.97%
Candidate ranker
77.42%
78.56%
0.00%
Pre-LLM fusion
81.24%
83.19%
31.97%
Table 4: Final parent-code prediction results, including the direct zero-shot large-model reference. OOF-CV is used for model and threshold selection; locked-test accuracy is the final held-out score. LLM calls are reported on the locked test set; an em dash denotes that OOF-CV was not evaluated for the zero-shot reference.
Figure 6: Locked-test Top-1 accuracy across supervised, full-LLM, and deployable selective variants. Pre-LLM fusion provides the strongest deployable result.
Figure 7: Rare-class behavior and dominant error routes. Rare classes are most strongly affected through reduced recall and F1-score, and their errors are often redirected to frequent ICD-10 letters.
Figure 8: Reliability diagram for letter-level predictions. Confidence is informative, but the model is not fully calibrated across all probability bins.
Automated International Classification of Diseases (ICD) coding is a core medical-coding task for billing, epidemiology, and clinical decision support. Generative large language models (LLMs) are often reported as weak medical coders, but this finding mainly comes from inference-time settings such as prompting, retrieval, reranking, or tool use, leaving the role of task-specific post-training underexplored. We present a controlled empirical study of post-training for generative ICD coding, comparing discriminative baselines with LLM coders across prompting, supervised fine-tuning, and reinforcement learning under a common protocol and metric set. To our knowledge, this is the first study to evaluate RL-based post-training for generative LLM coders in ICD coding. We further introduce PHI, a diagnostic curriculum that extends GRPO to refine missed-code cases. Our results show that prompting-only evaluation substantially underestimates the potential of LLMs for ICD coding. SFT provides the main capability jump, GRPO further improves code-set prediction beyond SFT, and PHI provides targeted gains on macro-level performance. These findings suggest that the main bottleneck is not the generative formulation alone, but how the model is adapted and optimized for full-taxonomy recall. We release our code, data splits, and checkpoints at https://github.com/AlexandreWANG915/LLM4ICD.
Accurate coding of tumor diagnoses with ICD-10-GM and ICD-O-3 is essential for structured cancer documentation in Germany. Smaller open-weight LLMs are appealing for privacy-preserving automation but often struggle with coding accuracy in German-language contexts. This study investigates whether instruction-based fine-tuning on public datasets improves the coding accuracy of open-weight LLMs for German tumor diagnosis texts. The evaluation uses coded diagnoses from the local tumor documentation system as test data. In a systematic data quality assessment, the upper limit for ICD-10 coding performance was estimated at 60-79% for exact and 81-94% for partial (three-character codes only) derivation. As training data, over 500,000 question-answer pairs were created based on the ICD-10-GM, ICD-O-3, and OPS catalogues. Eight open-weight models from the Qwen, Llama, and Mistral families (7-70 B parameters) were fine-tuned. ICD-10-GM accuracy rose from 1.4-24% to 41-58%, and partial accuracy from 31-74% to 73-83%. The accuracy of ICD-O-3 topography coding also improved but started and remained considerably lower with an exact accuracy of 22-40% and a partial accuracy of 56-67% after fine-tuning. Malformed code outputs dropped to 0% for all models. Tumor-diagnosis recognition reached 99%. Accuracy correlated positively with model size, but gaps between small and large models narrowed after fine-tuning. The reasoning mode in Qwen3 generally yielded a lower performance than fine-tuning and was over 100 times slower. Our findings highlight the potential of leveraging public catalogues to build instruction datasets that improve LLMs in medical documentation tasks. The complete training dataset and the best-performing checkpoints of the fine-tuned models are available from https://huggingface.co/datasets/stefan-m-lenz/ICDOPS-QA-2024.
Stefan Lenz, Lakisha Ortiz Rosario, Georg Vollmar +4
Institute of Medical Biostatistics, Epidemiology and Informatics, University Medical Center of the Johannes Gutenberg University Mainz · University Cancer Center of the Johannes Gutenberg University Mainz · Third Department of Medicine, University Medical Center of the Johannes Gutenberg University Mainz
Generative approaches to automatic International Classification of Diseases (ICD) coding use large language models (LLMs) to predict ICD codes from clinical documents. However, original ICD code strings are not ideal generation targets because the ICD hierarchy does not always reflect clinical relationships, and LLMs struggle to interpret and generate these codes. We introduce SemICD, a framework that trains LLMs to generate Semantic IDs (SIDs) instead. SemICD constructs SIDs by hierarchically clustering embeddings of textual code profiles, allowing semantically related codes to share prefixes while each complete SID uniquely identifies one ICD code. To align the SID tokens with clinical semantics, we employ two auxiliary tasks for bidirectional generation between textual code profiles and SIDs. Experiments on MIMIC-III and MIMIC-IV with four LLMs show that SemICD outperforms direct code generation, improving Micro-F1 by 1.34-8.00 points. Our code is available at https://github.com/youwillbethebest/SemICD.
Yihang Cheng, Qian Zhang, Yonghui +3
Andie · Massey University Palmerston North, New Zealand · South China Normal University Guangzhou, China +1