Call-Efficient ICD-10 Code Prediction via Supervised Candidate Generation and Selective LLM Invocation
Authors: Ilia Kopanichuk, Daniil Kartashev, Yuri Bolkonsky, Petr Anokhin, Vladimir Makharev, Iaroslav Bespalov, Dmitry V. Dylov
Organizations: AIRI, Moscow, Russia · Moscow Institute of Physics and Technology, Dolgoprudny, Russia · Lomonosov Moscow State University, Moscow, Russia · Innopolis University, Innopolis, Russia · Skolkovo Institute of Science and Technology, Moscow, Russia
Automatic assignment of ICD-10 codes from short clinical diagnosis text is a practical medical text classification and normalization task. The task is difficult because diagnosis phrases are short and noisy, the ICD-10 label space is hierarchical, and the class distribution is strongly imbalanced. This paper presents a call-efficient ICD-10 code prediction pipeline that combines transformer embeddings, tree-based supervised candidate generation, candidate-level reranking, and a gated LLM assistance. The evaluation uses a leakage-safe validation protocol: model and threshold selection are based on out-of-fold cross-validation predictions, while final reporting is performed on a locked test set with no shared source-file groups. The supervised conditional parent-code candidate generator reaches 74.32% Top-1 accuracy on the locked test set and 93.05% Top-10 candidate coverage. A direct zero-shot GPT-5.6-Sol baseline reaches 81.01% Top-1 accuracy, but requires an invocation of this large model for every diagnosis. The final deployable system, Pre-LLM fusion, reaches 81.24% OOF-CV accuracy and 83.19% locked-test accuracy while calling the LLM for only 31.97% of cases. This improves the locked-test error rate from 25.68% for the supervised parent-code baseline to 16.81%, corresponding to a 34.5% relative error reduction. The results show that the most effective role of the LLM is controlled reranking on uncertain examples rather than unrestricted coding or full-dataset invocation.
Figures & tables
Work or system
Published setting and reported result
Relation to the present study
RuCCoD BERT pipeline ( Nesterov et al. 2025 )
Russian NER plus entity-level ICD-10 assignment; F1 = 0.525 and accuracy = 0.356.
The closest Russian academic benchmark; our work studies direct ICD-10 letter and parent-code prediction from diagnosis text.
Zero-shot retrieval-augmented ICD assignment; F1 = 0.458 and accuracy = 0.297.
Shows that retrieval helps LLM baselines, but supervised candidate generation remains important for Russian ICD linking.
RuBioBERT / RuBioRoBERTa ( Yalunin, Nesterov, and Umerenkov 2022 )
Russian biomedical encoders; RuCCoD reports F1 = 0.77 for diagnosis-entity extraction.
Useful for entity recognition and representation learning; our experiments use transformer embeddings and tree-based classifiers.
Russian ICD-10 chapter classifier and per-letter subgroup model series ( Dmitry43243242 2026a ; Dmitry43243242 2026b )
Public short-text classifiers; the chapter model reports top-1 accuracy = 0.7121 and top-3 accuracy = 0.8724.
These models predict coarse chapters or letter-specific subgroups, while our analysis focuses on parent-code prediction and selective LLM use.
Table 1: Closest Russian-language ICD-10 coding analogues and the position of the present study. The results are not ranked against the present model because the datasets and label spaces differ.
Train
Locked test
Size of sample
8,552
1,511
Number of unique parent codes
637
330
Table 2: Train-test dataset statistics for the final split. Parent codes correspond to the three-character ICD-10 code prefixes.
Figure 1: Training-set distribution by ICD-10 letter. The frequencies are strongly imbalanced, with letters such as J, K, M, and N dominating the dataset.
Figure 2: Final call-efficient pipeline for ICD-10 parent-code prediction. The supervised candidate generator produces a shared top-10 list. A no-call candidate-ranker route and a selectively invoked LLM-reranker route then converge in the Pre-LLM fusion selector, which outputs the final parent code. Green denotes supervised or no-call processing, while blue denotes selectively invoked LLM processing.
Figure 3: Letter-level distribution and per-class quality diagnostics. Frequent letters are overrepresented among predictions, while rare letters are underrepresented.
Figure 4: Normalized confusion matrix for ICD-10 letter prediction. Rows correspond to true letters and columns correspond to predicted letters. Off-diagonal cells reveal systematic error destinations.
Figure 5: Relationship between class support and OvR quality metrics. Recall and F1-score increase strongly with support, while precision is less support-dependent.
Method
Top-1
Top-3
Top-5
Top-10
MRR@5
Supervised parent stage
74.32%
88.15%
91.00%
93.05%
81.42%
Full LLM selection
80.41%
91.46%
92.46%
93.05%
85.85%
Table 3: Parent-code candidate-generation and reranking quality on the locked test set. The full-invocation LLM setting serves as an accuracy reference and requires an LLM call for every case; it is not the final selective system.
Method
OOF-CV
Locked test
LLM calls
Zero-shot GPT-5.6-Sol
—
81.01%
100.00%
Pre-LLM gate
79.51%
82.13%
31.97%
Candidate ranker
77.42%
78.56%
0.00%
Pre-LLM fusion
81.24%
83.19%
31.97%
Table 4: Final parent-code prediction results, including the direct zero-shot large-model reference. OOF-CV is used for model and threshold selection; locked-test accuracy is the final held-out score. LLM calls are reported on the locked test set; an em dash denotes that OOF-CV was not evaluated for the zero-shot reference.
Figure 6: Locked-test Top-1 accuracy across supervised, full-LLM, and deployable selective variants. Pre-LLM fusion provides the strongest deployable result.
Figure 7: Rare-class behavior and dominant error routes. Rare classes are most strongly affected through reduced recall and F1-score, and their errors are often redirected to frequent ICD-10 letters.
Figure 8: Reliability diagram for letter-level predictions. Confidence is informative, but the model is not fully calibrated across all probability bins.
Institute of Medical Biostatistics, Epidemiology and Informatics, University Medical Center of the Johannes Gutenberg University Mainz · University Cancer Center of the Johannes Gutenberg University Mainz · Third Department of Medicine, University Medical Center of the Johannes Gutenberg University Mainz