Organizations: Twinkle AI · APMIC · Open Taiwan Benchmark · National Center for High-performance Computing (NCHC), Hsinchu, Taiwan · MonoLab · Google Developer Expert, Cloud AI · Open Culture Foundation (OCF)
We present OpenTWBench, an open evaluation suite for Taiwan-domain knowledge in large language models, built around two benchmarks. tw-exam-bench contains 255,854 four-option questions across 148 academic subjects, from fifteen years of Taiwan's national examinations published as open data by the Ministry of Examination; answer keys agree across 31,312 duplicated question groups at 99.91%, with all 27 disagreements accounted for. A reviewed second pass withdrew 2,180 items whose stems depend on a figure the text does not carry; all scores use the remaining 253,674. tw-formosa-bench contains 1,696 difficulty-curated questions about everyday and contemporary Taiwan - the Taiwan that licensing examinations never ask about - generated from a community-curated corpus under a discipline in which no model writes a ground truth, and curated against a frozen seven-model pool so that the pool scored at the passing margin (46.9-74.6%, median ~54%); a leave-one-out experiment shows the selection preserves model ranking (near-zero bias). Within six weeks the frontier had cleared that band: of 108 models now on the public board, 22 score above its ceiling, the best 91.6%. We also document how much of a score comes from the apparatus rather than the model. On identical questions, switching answer extraction from pattern matching to an explicit \boxed{} answer moves accuracy from 20.8% to 93.9%; a single placeholder character in a system prompt moved one model from last (58.2%) to first (83.9%) on the same benchmark; doubling the thinking budget from 8k to 16k tokens bought 0.2% while shuffling which questions failed; and a three-judge validation panel proved to be one judge in disguise, two members approving 92-99% of everything shown. We release all datasets, per-item difficulty data, curation manifests and a leaderboard at https://opentwbench.ai
Figures & tables
Pipeline stage
Count
Exam papers seen / parsed
13,812 / 11,621
rejected: incomplete parse / no questions / download failed
2,002 / 188 / 1
Raw items extracted
629,343
Folded as re-printings of an earlier year’s item
265,736
English-language items
70,386
Not four-choice
14,779
Table 1: tw-exam-bench build accounting across all 148 subjects. Rows below the first rule are filtered items; folding of duplicate groups (31,312 groups across the 148 published subjects) is counted separately since those items are re-printings, not defects. The published set is what the dataset ships; the scored set is what every number in this paper is computed on, after the second-pass withdrawal described in Section 3.4 .
Held-out model
n
Full set
Held-out
In-pool mean
Gap
Mistral-Large-675B
1,679
83.9%
75.4%
54.7%
+20.7
gemma-4-31B
1,633
73.9%
61.3%
56.2%
+5.1
Gemma-3-27B
1,599
70.1%
56.3%
56.3%
+0.0
Nemotron-Super-120B
1,643
70.9%
56.1%
57.2%
−1.2
gpt-oss-120b
1,622
69.3%
54.2%
57.1%
−2.9
Llama-3.3-70B
1,505
66.2%
52.9%
54.7%
−1.8
Table 2: Leave-one-out difficulty transfer. Each row selects the band pass≤5/6 using the other six models and measures the held-out model on the resulting n items. “Full set” is the held-out model’s accuracy on all 2,665 grounded items. Selection compresses scores by 8–16%; mid-field models transfer with near-zero bias; stronger models rise above the pool mean, as ability — not selection artifact — predicts.
Model
Accuracy
\boxed{}
Mistral-Large-675B
83.4%
99.9%
gemma-4-31B
82.1%
100.0%
Qwen3.8-27B (anchor, composition-weighted)
80.1%
—
gpt-oss-120b
77.1%
99.9%
Nemotron-Super-120B
76.0%
89.2%
Gemma-3-27B
66.7%
100.0%
Table 3: tw-exam-bench stratified sample (22,021 rows), selected rows; the full board of 107 models is published at https://opentwbench.ai . The second column is the share of responses that contained a \boxed{} at all, and the accuracy is computed on those alone — which is why Llama-3.1-TAIDE-LX-8B, whose knowledge is comparable to the 3–4B models above it, sits last: it wrote the requested format on under a third of items. The anchor row places Qwen’s full-corpus run on the sample’s scale by weighting its per-subject accuracies by the sample’s per-subject counts; it is an estimate on that scale, not a measurement taken on it (its full-corpus figure is 82.2%). Gemma-3-27B and Llama-3.3-70B differ by 0.06 points — a statistical tie. Scoring is row-wise over the 22,021 rows; 102 of the 106 sampled models answered all of them and four runs fell 9–36 rows short.
Model
Accuracy
Mistral-Large-675B
74.6%
gemma-4-31B
59.0%
Nemotron-Super-120B
54.3%
Gemma-3-27B
53.1%
gpt-oss-120b
51.8% †
Llama-3.3-70B
46.9%
Table 4: Frozen-pool scores on tw-formosa-bench (1,696 items). Scores sit at the passing margin by construction (Section 4.2 ). † gpt-oss-120b generated the draft items and its score carries a self-advantage annotation.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Subject
Question (truncated)
Official keys by year
Class
clinical-hematology
下圖是一位健康成年人的骨髓細胞以流式細胞儀（flow cytometry）分…
104: D (lymphocyte); 105: D (monocyte)
figure
clinical-physiology-pathology
這張心電圖最恰當的診斷為何？…
105: A (第一度房室間傳導阻斷); 107: C (第三度房室間傳導阻斷)
figure
critical-care-respiratory
先天性心臟病的兒童經過開心手術後，常有許多術後心臟功能異常，下列心血管壓力監…
103: A (左心室功能異常：右心房壓力正); 103: A (左心室功能異常：右心房壓力：)
typo.
dental-technology
圖中標示星號（*）之肌肉是：…
101: A (顳肌（temporal mu); 101: D (咬肌（masseteric )
figure
dental-technology
圖中，上顎齒列依殘留齒與缺牙區的位置關係為甘迺迪（Kennedy）分類法中之…
102: C (I 類-3 級); 102: D (I 類-4 級)
figure
dentistry
圖中黑色箭頭所指的解剖位置為何？…
111: D (left mandible); 112: A (left maxilla)
figure
Appendix
Table 5: All 27 flagged answer-key conflict groups with manual attribution. figure : stem names a figure the text does not carry; typo. : printings differ only in whitespace or punctuation (detector artifact, not a conflict); revised : option set legitimately rewritten between years; vignette : deictic stem pointing at case context outside the item; true : self-contained stem whose official keys genuinely disagree.
Run
Models
Serving
Dates (2026)
tw-exam-bench full corpus
Qwen3.8-27B
self-hosted vLLM
Aug 22–24
tw-exam-bench budget study
Qwen3.8-27B
self-hosted vLLM
Aug 22
Formosa pool (7 models)
NCHC catalog (see caption)
NCHC API
Aug 23
Formosa selection frozen
—
manifest v2.1
Aug 24
Formosa held-out
Qwen3.8-27B
self-hosted vLLM
Aug 24
Formosa generation
gpt-oss-120b
NCHC API
Aug 23
Appendix
Table 6: Runs, serving paths, and dates. The frozen pool is Mistral-Large-3-675B-Instruct-2512, Llama-3.1-405B-Instruct-FP8, gemma-4-31B-it, NVIDIA-Nemotron-3-Super-120B-A12B, Gemma-3-27B, gpt-oss-120b, and Llama-3.3-70B-Instruct, all served by Taiwan’s National Center for High-performance Computing (NCHC). All runs use Twinkle Eval’s box strategy, temperature 0 , top_p 0.9 , shuffle_options on; generation caps are 16,384 tokens for the full-corpus exam run and 4,096 for all Formosa runs. Datasets, per-item difficulty sidecars, the curation manifest, per-cell parse rates, and the build manifests are published with the benchmark repositories.
Large language models (LLMs) have shown impressive capabilities across diverse tasks, yet their performance on jurisdiction-specific legal reasoning remains underexplored. We present TW-LegalBench that utilizes Taiwanese legal system's rich official corpus open to the public to fill the gap in evaluating LLMs on Taiwanese law, among common-law benchmarks that focus on English sources and civil-law benchmarks focusing on sources of Simplified Chinese. TW-LegalBench comprises three task types: (1) over 16,000 multiple-choice questions (MCQs) across five years of official examinations in 18 professional domains; (2) 117 open-ended essay questions (OEQs) from examinations for legal professionals with official scoring rubrics; and (3) more than 14,000 legal judgment prediction (LJP) instances covering hundreds of crime categories. We evaluate 13 LLMs using accuracy for MCQs, a decomposed LLM-as-Judge framework based on the scoring rubric points for OEQs, and metrics for sentencing accuracy and statute citation for LJP. Our results reveal that top-performing models exceed the passing threshold for qualified lawyers (passing rate: 11%) but fall short of that for judges and prosecutors (passing rate: 1~2%). For LJP, while models demonstrate reasonable verdict type accuracy and sentence prediction capability, they struggle to cite exact legal articles. These findings highlight that reliable legal text generation remains challenging for LLMs, even though their performance on qualification examinations approaches human level.
Fei-Yueh Chen, Chun Huang Lin, Chan Wei Hsu +4
University of Rochester Rochester, NY, USA · National Taiwan University Taipei, Taiwan · NVIDIA Santa Clara, CA, USA
Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve over 90% accuracy on popular benchmarks like MMLU, limiting informed measurement of state-of-the-art LLM capabilities. In response, we introduce Humanity's Last Exam (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be the final closed-ended academic benchmark of its kind with broad subject coverage. HLE consists of 2,500 questions across dozens of subjects, including mathematics, humanities, and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading. Each question has a known solution that is unambiguous and easily verifiable, but cannot be quickly answered via internet retrieval. State-of-the-art LLMs demonstrate low accuracy and calibration on HLE, highlighting a significant gap between current LLM capabilities and the expert human frontier on closed-ended academic questions. To inform research and policymaking upon a clear understanding of model capabilities, we publicly release HLE at https://lastexam.ai.
Medical benchmarks for language models are built almost entirely on Western biomedicine. Traditional Chinese Medicine (TCM) is a separate system, with its own diagnostic framework and its own literature, and it remains largely unmeasured. The few TCM evaluations that exist are small, narrow, and rarely paired with a human reference. We present TCMQA, an open benchmark of 38,279 questions from Chinese TCM licensing examinations, paired with 15,151 responses from 101 licensed practitioners. We evaluate 29 instruction-tuned models from 9 families, spanning 0.27B to 14.8B parameters. Accuracy ranges over 59 points, and no model approaches saturation. Pretraining data predicts TCM ability far better than scale: a 12B Western-pretrained model reaches 39.6%, while a Chinese-pretrained model an eighth its size reaches 60.8%. Nine models exceed the practitioner majority vote of 64.9%, the best by 21.8 points, and all nine come from that same Chinese-pretrained family. Yet difficulty does not transfer between models and practitioners: accuracy is flat across practitioner-rated difficulty, item-level agreement is near zero for all 29 models, and on 8.4% of items the practitioners are correct where the leading model is wrong. We release the corpus, the practitioner responses, the harness, and per-item model outputs at https://huggingface.co/datasets/TechTCM/TCMQA.
Tzu-Heng Huang, Jet Lin, Eric Lin
University of Wisconsin-Madison · University of California, Merced · TechTCM