LingLanMedBench: A Unified Benchmark for Evaluating Large Language Models in Traditional Chinese Medicine

Rui Hua; Yu Wei; Zixin Shu; Kai Chang; Dengying Yan; Jianan Xia; Hui Zhu; Xiaodong Li

cs.AIFeb 2, 2026

LingLanMedBench: A Unified Benchmark for Evaluating Large Language Models in Traditional Chinese Medicine

Authors: Rui Hua, Yu Wei, Zixin Shu, Kai Chang, Dengying Yan, Jianan Xia, Hui Zhu, Xiaodong Li,

Organizations: Department of Computer Science and Technology, Beijing Jiaotong University, Beijing, China · Tianjin Tasly Digital Chinese Medicine Technology Co., Ltd., Tianjin, China · Tasly Biopharmaceuticals Co., Ltd., Tianjin, China · State Key Laboratory of Chinese Medicine Modernization, Tianjin, China · Institute of Liver Diseases, Hubei Key Laboratory of the theory and application research of liver and kidney in traditional Chinese medicine, Hubei Provincial Hospital of Traditional Chinese Medicine, Wuhan, China · Affiliated Hospital of Hubei University of Chinese Medicine, Wuhan, China · Hubei Province Academy of Traditional Chinese Medicine, Wuhan, China · Xiyuan Hospital, China Academy of Chinese Medical Sciences, Beijing, China · Beijing Research Institute of Chinese Medicine, Beijing University of Chinese Medicine, Beijing, China · The First Affiliated Hospital, Henan University of Chinese Medicine, Zhengzhou, China · Institute of Chinese Medicine Epidemic Disease, Beijing University of Chinese Medicine, Beijing, China · China Institute for History of Medicine and Medical Literature, China Academy of Chinese Medical Sciences, Beijing, China · Department of Gastroenterology, Guang’anmen Hospital, China Academy of Chinese Medical Sciences, Beijing, China · China Academy of Chinese Medical Sciences, Beijing, China

Abstract

Reliable evaluation of large language models in traditional Chinese medicine remains limited because existing benchmarks mainly focus on examination-style question answering or isolated generative tasks. We developed LingLanMedBench, a unified benchmark comprising 25,620 expert-verified items across 13 subtasks and five domains, covering knowledge, language understanding, information extraction, and diagnostic-therapeutic decision-making. LingLanMedBench standardizes task formats and evaluation metrics and further incorporates synonym-tolerant clinical-label matching, empirically derived hard subsets, and decision-recognition reformulations of open-ended clinical tasks. Fourteen open-source and proprietary models were evaluated under a standardized zero-shot protocol. Performance varied substantially across domains, model families, task formulations, and difficulty levels. The leading overall score decreased from 51.1 on the full benchmark to 31.9 on the hard subsets. Prescription performance also differed markedly between open-ended generation, with F1 scores of 32.7 and 20.3, and decision recognition, with accuracies of 90.6 and 56.8 on the full and hard subsets, respectively. These findings reveal a substantial gap between knowledge-oriented performance and clinically faithful reasoning. LingLanMedBench provides a standardized and reproducible framework for characterizing current capabilities, identifying systematic limitations, and tracking meaningful progress in large language models for traditional Chinese medicine.

Figures & tables

Figure 1: Overview of the LingLanMedBench construction pipeline. The left panel summarizes data sources; the center panel illustrates data processing and curation; the right panel presents the task taxonomy covered in LingLanMedBench together with the unified evaluation metrics. Abbreviations: HMC (Human-machine collaboration).

Figure 2: Subtask distribution and dataset sizes in LingLanMedBench. Abbreviations: FTK (Fundamental TCM Knowledge), CPMK (Chinese Patent Medicine Knowledge), TLE (TCM Licensing Examination), DR (Decision Recognition), IE (Information Extraction), DTR (Diagnostic–Therapeutic Reasoning), SC (single-choice).

Dom.	Subtask	Metric	DeepSeek-R1	DeepSeek-V3.1	GPT-5	Qwen3-235B	Qwen3-32B	Baichuan-M2	Qwen3-Next	GPT-OSS-120B
TLE	Comprehensive	Accuracy	95.0 / 78.3	95.5 / 81.0	87.6 / 52.3	95.0 / 77.8	91.2 / 62.3	89.8 / 58.3	93.8 / 73.0	58.1 / 14.5
FTK	Single-choice	Accuracy	86.1 / 46.5	86.1 / 45.3	85.4 / 44.5	85.0 / 45.3	86.1 / 44.3	83.4 / 35.8	82.1 / 33.3	70.4 / 28.2
FTK	Multiple-choice	Accuracy	64.8 / 31.5	62.4 / 28.0	53.9 / 19.3	61.3 / 25.0	56.3 / 23.3	48.6 / 14.0	52.6 / 14.8	35.3 / 9.8
FTK	Multiple-choice	Precision	92.5 / 82.4	91.9 / 80.5	90.6 / 75.3	93.0 / 80.9	90.3 / 77.5	90.5 / 75.4	89.4 / 71.6	81.2 / 63.5
FTK	Multiple-choice	Recall	92.1 / 79.3	92.0 / 80.5	87.2 / 69.8	89.1 / 73.7	90.9 / 78.0	84.5 / 66.1	84.7 / 65.2	83.2 / 65.6
FTK	Multiple-choice	F1	91.0 / 77.8	90.6 / 77.5	87.3 / 69.3	89.5 / 73.7	89.1 / 74.8	85.3 / 66.4	84.7 / 64.2	80.0 / 61.3

Table 1: LingLanMedBench results covering all subtasks. Each entry reports full-set / Hard-subset performance (left/right). Bold values denote the highest score for each subtask–metric. Average refers to the mean of all metrics, excluding MAE.

Dom.	Subtask	Metric	Qwen3-32B	Qwen3-30B	Qwen3-14B	Qwen3-8B	Qwen3-4B	GPT-5-Mini	GPT-OSS-20B
TLE	Comprehensive	Accuracy	91.2 / 62.3	88.8 / 55.3	87.6 / 50.0	84.6 / 40.5	76.6 / 30.3	76.3 / 24.5	48.3 / 13.5
FTK	Single-choice	Accuracy	86.1 / 44.3	82.6 / 32.0	83.0 / 34.5	80.8 / 33.8	76.2 / 28.5	77.0 / 28.7	65.4 / 29.3
FTK	Multiple-choice	Accuracy	56.3 / 23.3	52.0 / 17.0	49.5 / 13.8	47.8 / 13.0	42.4 / 12.3	46.2 / 10.8	24.8 / 6.0
FTK	Multiple-choice	Precision	90.3 / 77.5	89.8 / 75.2	90.6 / 75.2	88.4 / 71.8	86.1 / 69.0	86.1 / 67.7	76.1 / 59.5
FTK	Multiple-choice	Recall	90.9 / 78.0	88.3 / 73.5	85.7 / 69.4	86.8 / 70.2	84.9 / 67.6	86.8 / 69.0	74.9 / 58.1
FTK	Multiple-choice	F1	89.1 / 74.8	87.4 / 71.0	86.3 / 68.7	85.8 / 67.6	83.5 / 64.8	84.8 / 65.3	72.9 / 55.0

Table 2: Small-scale LLM results covering all subtasks. Each entry reports full-set / Hard-subset performance (left/right). Bold values denote the highest score for each subtask–metric. Average refers to the mean of all metrics, excluding MAE.

Figure 3: The Overall Average panel shows the mean of the seven task-type scores. The Average rows in Tables 1 and 2 instead show the mean of all 42 eligible metric rows.

Benchmark	Scale	Question Type	Sources	Metrics
TCMD [ 39 ]	3,451	MC	Licensing examinations	Accuracy
TCMBench [ 40 ]	5,473	MC	Licensing examinations	TCMScore, ROUGE, BERTScore
MTCMB [ 41 ]	7,100 (6,000 exam samples)	MC / Cloze / Open QA / Prescription / Diagnosis / IE	Licensing examinations / Classical texts / Clinical records / Guidelines	Accuracy, ROUGE, BLEU, BERTScore
TCMEval-SDT [ 42 ]	300 cases	IE / Pathogenesis / Syndrome / Summary	Internet / Classical texts / Clinical records	Stage-wise qualitative
LingLanMedBench (ours)	25,620 ( ~2,000 per subtask)	SC / MC / Cloze / IE / DTR / DR	TLE / FTK / CPMK / EMR / Classical texts / Master-physician casebooks	Accuracy, Precision, Recall, Char-F1, MAE, Cosine similarity

Table 3: Comparison of major TCM benchmarks by scale, task types, sources, and metrics.

Dom.	Subtask	Metric	Qwen3-14B	Baichuan2-13B-Chat	Lingdan-13B-TCPM	TCMChat	BianCang-7B	BianCang-14B
TLE	Comprehensive	Accuracy	87.6 / 50.0	50.1 / 21.8	44.3 / 20.8	31.6 / 19.5	88.9 / 70.0	91.2 / 73.5
FTK	Single-choice	Accuracy	83.0 / 34.5	60.6 / 28.5	48.4 / 26.3	11.1 / 7.5	78.2 / 43.3	82.6 / 49.3
FTK	Multiple-choice	Accuracy	49.5 / 13.8	23.4 / 12.8	14.3 / 9.5	0.6 / 0.5	36.9 / 19.5	23.7 / 13.0
FTK	Multiple-choice	Precision	90.6 / 75.2	71.8 / 62.0	53.6 / 48.8	58.9 / 48.2	78.9 / 70.1	65.2 / 59.8
FTK	Multiple-choice	Recall	85.7 / 69.4	84.4 / 79.8	56.5 / 56.4	36.3 / 29.1	78.4 / 72.7	53.1 / 48.7
FTK	Multiple-choice	F1	86.3 / 68.7	75.9 / 67.7	52.4 / 49.0	39.9 / 32.4	76.2 / 68.0	55.8 / 50.3

Table S1: TCM LLM results covering all domains and subtasks. Each entry reports full-set / Hard-subset performance (left/right). Bold values denote the highest score for each subtask–metric.

Figure S1: Within each task, items are ranked by decreasing difficulty score (blue curve); the red and green dashed lines show the mean and median, and the orange vertical line marks the top 400 items selected for LingLanMedBench-Hard.

Figure S2: Distribution of items across the 14 subjects in the TLE (TCM Licensing Examination) dataset.

Figure S3: Statistical analysis of entity distribution in IE tasks. (A) Top 10 most frequently mentioned entity types in modern medical records (left, blue) and ancient medical texts (right, orange). Vertical bars show absolute mention counts per category. (B) Distribution of entity mentions per sample for medical records (left, blue) and ancient texts (right, orange).

Traditional Chinese Medicine

See all 17 papers

Lingdan-V2: Large language models for clinically grounded reasoning in traditional Chinese medicine
Oct 10, 2026Rui Hua, Yu Wei, Ziri Jin +15Traditional Chinese Medicine Medical LLMs
School of Computer Science & Technology, Beijing Jiaotong University, Beijing, 100044, China · Tianjin Tasly Digital Chinese Medicine Technology Co., Ltd., Tianjin, 300410, China · Tasly Biopharmaceuticals Co., Ltd., Tianjin, 300410, China +8
TCMQA: A 38K-Question Traditional Chinese Medicine Benchmark with a Licensed-Practitioner Reference
Sep 26, 2026Tzu-Heng Huang, Jet Lin, Eric LinTraditional Chinese Medicine Medical QA
University of Wisconsin-Madison · University of California, Merced · TechTCM
TCMClinicalReason-Bench: Can Language Models Reason from Pathogenesis to Prescription over Real-World Clinical Cases?
Oct 3, 2026Jirui Dai, Chenkai Zhang, Yan Jia +4Traditional Chinese Medicine Clinical Language Model Evaluation
the Department of Computer Science, Johns Hopkins University, Baltimore, USA · the School of Pharmacy, Nanjing University of Chinese Medicine, Nanjing, China · the School of Information Engineering, Huzhou University, Huzhou, China +3

Oct 10, 2026J/K move · Enter open · S save

Lingdan-V2: Large language models for clinically grounded reasoning in traditional Chinese medicine

Rui Hua, Yu Wei, Ziri Jin, Zhuo Sun+14

School of Computer Science & Technology, Beijing Jiaotong University, Beijing, 100044, China · Tianjin Tasly Digital Chinese Medicine Technology Co., Ltd., Tianjin, 300410, China · Tasly Biopharmaceuticals Co., Ltd., Tianjin, 300410, China+8

Traditional Chinese Medicine Medical LLMs Medication Recommendation RL for Language Model Reasoning Clinical Language Model Evaluation

Open paper AskPDF arXiv

Large language models have advanced medical reasoning, but clinically grounded reasoning in traditional Chinese medicine (TCM) remains challenging because prescription decisions require integrating domain knowledge, syndrome differentiation, treatment principles, herbal prescription composition, and dosage determination. We develop a TCM language model family comprising models with 4 billion, 8 billion, and 14 billion parameters. The artificial intelligence contribution is a staged clinical alignment framework integrating large scale continued pretraining on TCM and clinical records, complementary reasoning supervision, and prescription-centered reinforcement learning (RL). We further formulate herbal prescription generation as a structured RL problem with prescription-level feedback, enabling direct optimization of clinical decision outputs beyond knowledge reproduction and generic instruction following. The engineering application is TCM clinical decision support, particularly herbal prescription recommendation. We evaluate the models on a public benchmark containing 25,620 expert-verified items across 13 TCM knowledge and clinical reasoning tasks and on a private retrospective dataset of 4,348 spleen and stomach disease prescription cases. The 14 billion parameter reinforcement aligned model achieves a prescription generation F1 of 37.0 on the public benchmark, compared with 32.3 for DeepSeek-R1 with 671 billion total parameters, and a herb-level F1 of 52.8 on the clinical dataset with improved dosage agreement. Reinforcement post-training consistently improves prescription recommendation across all scales, while physician-source analysis shows that expert heterogeneity affects reward stability. These findings demonstrate a scalable pathway from domain grounding and reasoning supervision to clinically oriented prescription decision support.

Abstract

Traditional Chinese Medicine

Explore similar work

Lingdan-V2: Large language models for clinically grounded reasoning in traditional Chinese medicine

TCMQA: A 38K-Question Traditional Chinese Medicine Benchmark with a Licensed-Practitioner Reference

TCMClinicalReason-Bench: Can Language Models Reason from Pathogenesis to Prescription over Real-World Clinical Cases?