Organizations: Twinkle AI · APMIC · Open Taiwan Benchmark · National Center for High-performance Computing (NCHC), Hsinchu, Taiwan · MonoLab · Google Developer Expert, Cloud AI · Open Culture Foundation (OCF)
We present OpenTWBench, an open evaluation suite for Taiwan-domain knowledge in large language models, built around two benchmarks. tw-exam-bench contains 255,854 four-option questions across 148 academic subjects, from fifteen years of Taiwan's national examinations published as open data by the Ministry of Examination; answer keys agree across 31,312 duplicated question groups at 99.91%, with all 27 disagreements accounted for. A reviewed second pass withdrew 2,180 items whose stems depend on a figure the text does not carry; all scores use the remaining 253,674. tw-formosa-bench contains 1,696 difficulty-curated questions about everyday and contemporary Taiwan - the Taiwan that licensing examinations never ask about - generated from a community-curated corpus under a discipline in which no model writes a ground truth, and curated against a frozen seven-model pool so that the pool scored at the passing margin (46.9-74.6%, median ~54%); a leave-one-out experiment shows the selection preserves model ranking (near-zero bias). Within six weeks the frontier had cleared that band: of 108 models now on the public board, 22 score above its ceiling, the best 91.6%. We also document how much of a score comes from the apparatus rather than the model. On identical questions, switching answer extraction from pattern matching to an explicit \boxed{} answer moves accuracy from 20.8% to 93.9%; a single placeholder character in a system prompt moved one model from last (58.2%) to first (83.9%) on the same benchmark; doubling the thinking budget from 8k to 16k tokens bought 0.2% while shuffling which questions failed; and a three-judge validation panel proved to be one judge in disguise, two members approving 92-99% of everything shown. We release all datasets, per-item difficulty data, curation manifests and a leaderboard at https://opentwbench.ai
Figures & tables
Pipeline stage
Count
Exam papers seen / parsed
13,812 / 11,621
rejected: incomplete parse / no questions / download failed
2,002 / 188 / 1
Raw items extracted
629,343
Folded as re-printings of an earlier year’s item
265,736
English-language items
70,386
Not four-choice
14,779
Table 1: tw-exam-bench build accounting across all 148 subjects. Rows below the first rule are filtered items; folding of duplicate groups (31,312 groups across the 148 published subjects) is counted separately since those items are re-printings, not defects. The published set is what the dataset ships; the scored set is what every number in this paper is computed on, after the second-pass withdrawal described in Section 3.4 .
Held-out model
n
Full set
Held-out
In-pool mean
Gap
Mistral-Large-675B
1,679
83.9%
75.4%
54.7%
+20.7
gemma-4-31B
1,633
73.9%
61.3%
56.2%
+5.1
Gemma-3-27B
1,599
70.1%
56.3%
56.3%
+0.0
Nemotron-Super-120B
1,643
70.9%
56.1%
57.2%
−1.2
gpt-oss-120b
1,622
69.3%
54.2%
57.1%
−2.9
Llama-3.3-70B
1,505
66.2%
52.9%
54.7%
−1.8
Table 2: Leave-one-out difficulty transfer. Each row selects the band pass≤5/6 using the other six models and measures the held-out model on the resulting n items. “Full set” is the held-out model’s accuracy on all 2,665 grounded items. Selection compresses scores by 8–16%; mid-field models transfer with near-zero bias; stronger models rise above the pool mean, as ability — not selection artifact — predicts.
Model
Accuracy
\boxed{}
Mistral-Large-675B
83.4%
99.9%
gemma-4-31B
82.1%
100.0%
Qwen3.8-27B (anchor, composition-weighted)
80.1%
—
gpt-oss-120b
77.1%
99.9%
Nemotron-Super-120B
76.0%
89.2%
Gemma-3-27B
66.7%
100.0%
Table 3: tw-exam-bench stratified sample (22,021 rows), selected rows; the full board of 107 models is published at https://opentwbench.ai . The second column is the share of responses that contained a \boxed{} at all, and the accuracy is computed on those alone — which is why Llama-3.1-TAIDE-LX-8B, whose knowledge is comparable to the 3–4B models above it, sits last: it wrote the requested format on under a third of items. The anchor row places Qwen’s full-corpus run on the sample’s scale by weighting its per-subject accuracies by the sample’s per-subject counts; it is an estimate on that scale, not a measurement taken on it (its full-corpus figure is 82.2%). Gemma-3-27B and Llama-3.3-70B differ by 0.06 points — a statistical tie. Scoring is row-wise over the 22,021 rows; 102 of the 106 sampled models answered all of them and four runs fell 9–36 rows short.
Model
Accuracy
Mistral-Large-675B
74.6%
gemma-4-31B
59.0%
Nemotron-Super-120B
54.3%
Gemma-3-27B
53.1%
gpt-oss-120b
51.8% †
Llama-3.3-70B
46.9%
Table 4: Frozen-pool scores on tw-formosa-bench (1,696 items). Scores sit at the passing margin by construction (Section 4.2 ). † gpt-oss-120b generated the draft items and its score carries a self-advantage annotation.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Subject
Question (truncated)
Official keys by year
Class
clinical-hematology
下圖是一位健康成年人的骨髓細胞以流式細胞儀（flow cytometry）分…
104: D (lymphocyte); 105: D (monocyte)
figure
clinical-physiology-pathology
這張心電圖最恰當的診斷為何？…
105: A (第一度房室間傳導阻斷); 107: C (第三度房室間傳導阻斷)
figure
critical-care-respiratory
先天性心臟病的兒童經過開心手術後，常有許多術後心臟功能異常，下列心血管壓力監…
103: A (左心室功能異常：右心房壓力正); 103: A (左心室功能異常：右心房壓力：)
typo.
dental-technology
圖中標示星號（*）之肌肉是：…
101: A (顳肌（temporal mu); 101: D (咬肌（masseteric )
figure
dental-technology
圖中，上顎齒列依殘留齒與缺牙區的位置關係為甘迺迪（Kennedy）分類法中之…
102: C (I 類-3 級); 102: D (I 類-4 級)
figure
dentistry
圖中黑色箭頭所指的解剖位置為何？…
111: D (left mandible); 112: A (left maxilla)
figure
Appendix
Table 5: All 27 flagged answer-key conflict groups with manual attribution. figure : stem names a figure the text does not carry; typo. : printings differ only in whitespace or punctuation (detector artifact, not a conflict); revised : option set legitimately rewritten between years; vignette : deictic stem pointing at case context outside the item; true : self-contained stem whose official keys genuinely disagree.
Run
Models
Serving
Dates (2026)
tw-exam-bench full corpus
Qwen3.8-27B
self-hosted vLLM
Aug 22–24
tw-exam-bench budget study
Qwen3.8-27B
self-hosted vLLM
Aug 22
Formosa pool (7 models)
NCHC catalog (see caption)
NCHC API
Aug 23
Formosa selection frozen
—
manifest v2.1
Aug 24
Formosa held-out
Qwen3.8-27B
self-hosted vLLM
Aug 24
Formosa generation
gpt-oss-120b
NCHC API
Aug 23
Appendix
Table 6: Runs, serving paths, and dates. The frozen pool is Mistral-Large-3-675B-Instruct-2512, Llama-3.1-405B-Instruct-FP8, gemma-4-31B-it, NVIDIA-Nemotron-3-Super-120B-A12B, Gemma-3-27B, gpt-oss-120b, and Llama-3.3-70B-Instruct, all served by Taiwan’s National Center for High-performance Computing (NCHC). All runs use Twinkle Eval’s box strategy, temperature 0 , top_p 0.9 , shuffle_options on; generation caps are 16,384 tokens for the full-corpus exam run and 4,096 for all Formosa runs. Datasets, per-item difficulty sidecars, the curation manifest, per-cell parse rates, and the build manifests are published with the benchmark repositories.