cs.CLSep 26, 2026
SaveTCMQA: A 38K-Question Traditional Chinese Medicine Benchmark with a Licensed-Practitioner Reference
Organizations: University of Wisconsin-Madison · University of California, Merced · TechTCM
Abstract
Medical benchmarks for language models are built almost entirely on Western biomedicine. Traditional Chinese Medicine (TCM) is a separate system, with its own diagnostic framework and its own literature, and it remains largely unmeasured. The few TCM evaluations that exist are small, narrow, and rarely paired with a human reference. We present TCMQA, an open benchmark of 38,279 questions from Chinese TCM licensing examinations, paired with 15,151 responses from 101 licensed practitioners. We evaluate 29 instruction-tuned models from 9 families, spanning 0.27B to 14.8B parameters. Accuracy ranges over 59 points, and no model approaches saturation. Pretraining data predicts TCM ability far better than scale: a 12B Western-pretrained model reaches 39.6%, while a Chinese-pretrained model an eighth its size reaches 60.8%. Nine models exceed the practitioner majority vote of 64.9%, the best by 21.8 points, and all nine come from that same Chinese-pretrained family. Yet difficulty does not transfer between models and practitioners: accuracy is flat across practitioner-rated difficulty, item-level agreement is near zero for all 29 models, and on of items the practitioners are correct where the leading model is wrong. We release the corpus, the practitioner responses, the harness, and per-item model outputs at https://huggingface.co/datasets/TechTCM/TCMQA.
Figures & tables
| # | Domain | Items | Share | Scope |
|---|---|---|---|---|
| 1 | Foundational Theory 中醫基礎理論 | 1,304 | 27.2% | Yin–Yang, Five Phases, visceral manifestation, qi and blood |
| 2 | Classical Texts 中醫經典文獻 | 700 | 14.6% | The canonical medical literature and its interpretive tradition |
| 3 | Diagnosis & Patterns 診斷與辨證學 | 1,309 | 27.3% | The four examinations; pattern differentiation and treatment |
| 4 | Meridians & Acupuncture 經絡與針灸學 | 502 | 10.5% | Channel theory, point location, needling and moxibustion |
| 5 | Materia Medica 中藥本草學 | 1,494 | 31.1% | Herb properties, formula composition, preparation, contraindication |
| 6 | Internal Medicine 中醫內科學 | 1,982 | 41.3% | Internal disorders, incl. gynecology, pediatrics, mental illness |
Table 1: The ten-domain TCM taxonomy, with the domain portfolio of the 4,801-item comparison subset. Items is the number of subset items assigned the domain by at least one of their three raters, and Share is that count as a fraction of 4,801.
| Property | Count | Share |
|---|---|---|
| Total items | 38,279 | 100.0% |
| single-answer | 35,893 | 93.8% |
| multi-answer | 2,386 | 6.2% |
| Items with five options | 38,279 | 100.0% |
| Items with reference explanation | 23,582 | 61.6% |
| Items with source difficulty | 38,217 | 99.8% |
Table 2: Composition of the TCMQA corpus.
| Rating | Responses | Share | Accuracy | Time |
|---|---|---|---|---|
| 1 (easiest) | 3,205 | 21.2% | 74.7% | 20.3 s |
| 2 | 3,611 | 23.8% | 69.7% | 24.5 s |
| 3 | 5,128 | 33.8% | 59.2% | 24.9 s |
| 4 | 1,914 | 12.6% | 49.3% | 24.0 s |
| 5 (hardest) | 1,293 | 8.5% | 40.4% | 23.6 s |
| All | 15,151 | 100.0% | 62.1% | 23.5 s |
Table 3: Practitioner-rated difficulty over all 15,151 responses. Accuracy is the share of responses at each rating that were correct, and Time is the median time spent on the item before submission. Ratings are per response, so an item answered by three practitioners contributes three ratings and may receive a different one from each.
| Model | Params | Full | Single | Multi | Subset | human |
| Gemma2 | ||||||
| Gemma-2-2B-It | 2.61 | 25.09 | 26.44 | 4.78 | 26.74 | |
| Gemma-2-9B-It | 9.24 | 36.93 | 38.72 | 10.10 | 37.78 | |
| Gemma3 | ||||||
| Gemma-3-270M-It | 0.27 | 18.95 | 20.20 | 0.00 | 17.93 | |
| Gemma-3-1B-It | 1.00 | 23.82 | 25.17 | 3.39 | 21.85 | |
Table 4: Full leaderboard. Models are grouped by family and ordered by size within each family. Full is accuracy on all 38,279 items; Single and Multi split that by answer type. Subset is accuracy on the 4,801 items practitioners also answered, and is the difference from their majority vote of , in bold where a model wins. All numbers are exact-match percentages.
Figure 1: Model accuracy by answer type. Overall, single-answer, and multi-answer accuracy for all 29 models, grouped by family. The multi-answer bar is shorter than the single-answer bar for all 29 models, and the gap grows as models get weaker, so the overall score hides this weakness.
Figure 2: Accuracy against model size. Accuracy against parameter count on a log axis, one line per family. The Qwen lines sit in a band of their own, and the gap to Gemma and OLMo grows with size instead of closing. Note also that Qwen2 beats Qwen2.5 at B, B, and B; Qwen2.5 wins only at B, where Qwen2 has no model, so Qwen2.5’s top spot comes from being bigger rather than better.
Figure 3: Precision, recall and F 1 on multi-answer items. Set-overlap metrics on the 2,386 multi-answer items. Precision beats recall for most models. That means they name too few options and treat these questions as single-answer.
| Model | Single EM | Multi EM | P | R | F 1 | |
|---|---|---|---|---|---|---|
| Qwen2.5-14B-Inst. | 79.5 | 55.0 | 92.4 | 90.4 | 90.1 | 3.40 |
| Qwen2-7B-Inst. | 76.7 | 50.3 | 92.5 | 86.9 | 88.3 | 3.25 |
| Qwen3.5-9B | 77.1 | 44.3 | 85.3 | 93.2 | 87.7 | 3.84 |
| Qwen3.5-2B | 53.8 | 4.9 | 81.3 | 42.2 | 51.4 | 1.84 |
| Gemma-3-12B-It | 41.0 | 18.1 | 76.3 | 83.4 | 77.8 | 3.82 |
| OLMo-2-13B-Inst. | 27.3 | 12.1 | 74.2 | 77.2 | 73.6 | 3.61 |
Table 5: Multi-answer decomposition. Results on the 2,386 multi-answer items. is the mean number of options predicted; the answers contain on average. Precision above recall means the model named too few options, recall above precision means too many.
Figure 4: Models against the practitioner reference. Accuracy on the 4,801 shared items: the practitioner majority vote against all 29 models, with the practitioner score as a dashed line. Nine models clear the line and all nine are Qwen. Every Gemma and OLMo falls short, including models larger than the Qwen ones that succeed.
Figure 5: Per-domain accuracy. Accuracy across the ten practitioner-assigned domains: the practitioner majority vote against the best model of each family. The order is the same on every axis. The three Qwen contours sit outside the practitioner contour everywhere, and every Gemma and OLMo contour sits inside it. The model contours are nearly circular while the practitioner contour is lopsided, so the gap is shaped by the human side.
| Practitioners correct | Observed | If independent | Ratio |
| 0 of 3 | 616 | 253.6 | |
| 1 of 3 | 1,067 | 1,266.8 | |
| 2 of 3 | 1,422 | 2,109.6 | |
| 3 of 3 | 1,696 | 1,171.0 | |
| (3 df), Fleiss’ | |||
Table 6: Practitioner agreement against independence. Agreement is far stronger than independent answering would produce. Expected counts come from over the 4,801 items with three raters.
Figure 6: Model accuracy by practitioner consensus. Left: model accuracy by how many of the three practitioners answered correctly. Right: accuracy on the 616 items all three got wrong. The model lines are flat, and the best model does slightly better on the questions every practitioner missed; the dashed practitioner line runs from to by construction. Because practitioner agreement is real rather than noise (Table 6 ), flat lines mean the two difficulties are unrelated.
Figure 7: Accuracy across the difficulty scale. All 29 models plus the practitioner line, over the practitioners’ five-point scale. Practitioner accuracy falls 57 points while almost every model line stays flat: at level 1 the practitioners beat all but two models, and by level 5 they are below Gemma-4-E4B-It. The ratings were given before anyone knew the outcome (Section 2.2 ), so this is separate evidence rather than a restatement of Figure 6 . Level 5 holds only 86 items, so read the last point loosely.
Figure 8: Item-by-item outcomes. Outcomes against the practitioner majority vote for all 29 models, ordered by accuracy; the numbers inside the bars are the two disagreement cells as a share of the 4,801 items. Going from the worst model to the best raises model-only wins from to , but practitioner-only wins fall from to and stop there. The orange band never closes, so a single ranked score hides more than it shows.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Answering an item. The item is presented in its original Chinese with its option list, one item per screen, and the header shows position within the practitioner’s 150-item block and whether the item is single- or multi-answer. Models see the same item text under the protocol of Section 2.3 , so both sides are answering identical questions.
Figure 10: Assigning domain labels. The practitioner tags the item with up to three of the ten domains of Table 1 , plus 其他 (Other) for items the taxonomy does not cover. Multiple labels are allowed because one item can span domains, though of responses use exactly one. These labels produce the per-domain comparison of Figure 5 .
Figure 11: Rating difficulty. A five-point scale, 5 being hardest. The question asks how hard the item was for that rater rather than how hard it is in general, so the ratings are subjective by design. They are collected on the same screen as the answer and before any outcome is revealed, which is what lets Figure 7 treat them as independent evidence.
Figure 12: Flagging and pausing. 問題報錯 flags an item the practitioner believes is defective, and 暫停一下 pauses the block. The flag gives a practitioner somewhere to report a bad item instead of guessing at it, and the pause keeps an interruption from being recorded as a slow or careless response; timing is measured per item, and the median is seconds.
Figure 13: Accuracy by confidence margin. Accuracy grouped by log-probability margin, one panel per family. More confidence really does mean more accuracy in every family, but the useful range varies by close to a factor of ten, so margins from different models are not comparable. The panels have independent -axes, so compare the ranges rather than the slopes. Qwen3.5 is the narrowest, fitting its whole curve into a range where the other families are still climbing.
Figure 14: Overconfidence. Left: accuracy against average confidence on wrong answers, split at the medians. Right: the confidence gap between right and wrong answers, with confidence capped at the 99th percentile. The top-left corner is where models are inaccurate and sure of themselves, and two Gemma3 models sit there. The right panel shows this does not follow from accuracy: Qwen2.5-14B-Inst. and Qwen3.5-9B score similarly but separate right from wrong answers by and .