CardioVascBench:Prioritized Diagnosis Generation from Real-World Chinese Cardiovascular Records
Organizations: School of Electronics and Communication Engineering, Sun Yat-sen University, Shenzhen Campus, Shenzhen, China · Department of Network Intelligence, Pengcheng Laboratory, Shenzhen, China
Abstract
Identifying the primary diagnosis alone does not fully characterize a model's diagnosis-generation performance. We introduce CardioVascBench, a retrospective benchmark for prioritized diagnosis generation from real-world Chinese cardiovascular inpatient records. It contains 3,262 de-identified cases, with 2,174 for training and 1,088 for evaluation, linking clinical narratives and textual ECG reports to expert-verified, ordered discharge diagnoses standardized to ICD-10. An English translation of the evaluation set supports aligned cross-lingual comparison. Our evaluation protocol combines terminology normalization and rank-aware scoring to separately assess primary-diagnosis identification, reference-diagnosis coverage, and list precision. Evaluation across 36 systems shows that higher primary-diagnosis accuracy does not necessarily imply higher list precision against discharge references. We also provide CardioKG-Agent, a knowledge-enhanced baseline combining entity normalization, constrained graph retrieval, and guideline evidence. In a staged comparison of three baseline models, supervised fine-tuning plus the agent achieves a 73.5% higher mean composite score than fine-tuning alone, representing a system-level gain. These results measure retrospective agreement with recorded diagnoses; agreement between metric rankings and physician judgments remains to be validated. CardioVascBench supports research on diagnosis prioritization and knowledge-enhanced generation.
Figures & tables
| Model | nDCG @10 | Precision @10 | Recall @10 | Top-1 | Top-3 | Score |
|---|---|---|---|---|---|---|
| Medical-specialized LLMs | ||||||
| BioMistral-7B | 49.2 | 14.2 | 13.5 | 7.6 | 7.6 | 11.4 |
| HuatuoGPT-o1-7B | 59.7 | 28.7 | 37.1 | 24.3 | 27.1 | 25.4 |
| Lingshu-7B | 61.9 | 32.0 | 40.9 | 24.7 | 27.7 | 28.2 |
| II-Medical-8B | 58.9 | 32.1 | 38.7 | 22.3 | 23.9 | 28.0 |
| HuatuoGPT-o1-72B | 62.2 | 18.3 | 45.0 | 36.6 | 41.3 | 19.5 |
| Config. | NDCG@10 | Precision@10 | Recall@10 | Top-1 | Score | Score |
|---|---|---|---|---|---|---|
| Baseline (base LLMs) | 59.70 | 23.30 | 38.17 | 24.53 | 21.73 | – |
| +SFT | 60.13 | 30.50 | 37.60 | 21.20 | 26.40 | +21.5% |
| +SFT + KG_agent | 65.63 | 52.70 | 51.43 | 45.57 | 45.80 | +73.5% |
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Rank | Diagnosis Name | Count | Proportion |
|---|---|---|---|
| 1 | Coronary Heart Disease | 2,054 | 26.4% |
| 2 | Coronary Atherosclerotic Heart Disease | 592 | 7.6% |
| 3 | Chest Pain | 491 | 6.3% |
| 4 | Chest Tightness | 444 | 5.7% |
| 5 | Unstable Angina | 439 | 5.6% |
| 6 | Hypertension Stage 3 | 326 | 4.2% |
| Model | NDCG @10 | Precision @10 | Recall @10 | Top-1 Acc | Top-3 Acc | Final Score |
|---|---|---|---|---|---|---|
| Proprietary Models | ||||||
| GPT-4o | 63.8 | 12.6 | 44.6 | 30.6 | 36.3 | 14.9 |
| GPT-5.2 | 66.1 | 11.9 | 48.8 | 39.0 | 45.8 | 14.8 |
| Grok-4 | 63.6 | 13.9 | 45.5 | 34.9 | 39.2 | 16.1 |
| Claude-sonnet-4.5 | 65.0 | 11.4 | 53.0 | 47.1 | 55.6 | 15.1 |
| Qwen3-max | 62.8 | 14.7 | 45.3 | 32.5 | 39.0 | 16.7 |
Medical Diagnosis
Medical diagnosis uses clinical data, tests, and images to identify diseases, assess risk, and guide care.
See all 271 papers