KBS: A Korean Benchmark Suite for Foundation Model Evaluation

Yeonkyoung So; Jongmin Kim; Sungmok Jung; Gyuseong Lee; Sangho Kim; Jongyeon Park; Joonhak Lee; Seho Pyo

cs.CLOct 15, 2026

KBS: A Korean Benchmark Suite for Foundation Model Evaluation

Authors: Yeonkyoung So, Jongmin Kim, Sungmok Jung, Gyuseong Lee, Sangho Kim, Jongyeon Park, Joonhak Lee, Seho Pyo,

Organizations: Graduate School of Data Science, Seoul National University · Dept. of Computer Science and Engineering, Seoul National University

Abstract

We present a Korean Benchmark Suite (KBS) comprising Ko-ARC, Ko-GSM8K, Ko-EQ-Bench, Ko-WinoGrande, Ko-LAMBADA, and Ko-IFEval, covering six capabilities across 9,396 items. The suite combines review, localization, direct Korean construction, and redesign for Korean, avoiding a translation-only approach to benchmark adaptation. We evaluate Korean-developed models, multilingual open models, and closed API models under protocols that account for model access, prompt format, and scoring differences. Results show that Korean benchmark scores depend on task construction, scoring rules, prompting, and model post-training. These findings suggest that Korean evaluation should report both final scores and the settings used to obtain them.

Figures & tables

Figure 1: Construction routes of the Korean Benchmark Suite. Translation-based tasks are adapted from English benchmarks through machine translation, correction, localization, and cross-review. Directly constructed tasks are built from Korean sources, while hybrid tasks combine translated items with Korean-specific prompts and automatic checkers.

Figure 2: Examples illustrating why task redesign is needed when adapting English benchmarks to Korean. Each panel contrasts direct transfer with the corresponding redesigned Korean task formulation.

Benchmark	Capability	Items	Construction	English Source
Ko-ARC (easy)	Science QA	2,376	Translated + Refined	ARC-Easy
Ko-ARC (challenge)	Science QA	1,167	Translated + Refined	ARC-Challenge
Ko-GSM8K	Mathematical reasoning	1,319	Translated + Refined	GSM8K
Ko-EQ-Bench	Dialogue-based emotional inference	171	Translated + Refined	EQ-Bench
Ko-WinoGrande	Commonsense/contextual reasoning	1,267	Translated + Refined	WinoGrande
Ko-LAMBADA	Long-context literary understanding	2,255	Newly Constructed	LAMBADA

Table 1: Composition of the Korean Benchmark Suite. Construction type indicates whether the benchmark is translated and refined from an English source, newly constructed from Korean sources, or combines translated items with newly constructed items for Korean-specific constraints. Ko-IFEval consists of 541 translated items and 300 newly constructed items.

Category	Tasks	Models	Main Metrics
Multiple-choice	Ko-ARC, Ko-WinoGrande, Ko-LAMBADA	Open only	acc / acc_norm / acc_npsq
Generation	Ko-GSM8K, Ko-IFEval, Ko-EQ-Bench	Open + closed	exact match / task-specific scores
Generative MCQA	Ko-ARC-gen, Ko-WinoGrande-gen, Ko-LAMBADA-gen	Open + closed	exact match

Table 2: Summary of evaluation formats. Original multiple-choice evaluation is used for open-weight models with log-likelihood access, while closed API models are evaluated using generation-based variants.

Model group	English	DeepL	KBS
Korean-developed	67.29	54.00	59.49
Qwen3.5	51.22	44.14	46.49
Gemma 4	72.56	60.14	67.05

Table 3: ARC accuracy (%) using NPSQ under chat prompting. All versions are restricted to the same 2,376 Easy and 1,167 Challenge items. Scores are averaged equally over the two subsets and then over the models in each group. DeepL denotes the raw machine-translated version before author correction and localization.

Task	Prompt format	acc	acc_norm	acc_npsq
Ko-WinoGrande	plain	61.69	61.06	59.33
Ko-WinoGrande	chat	58.08	57.64	55.73
Ko-LAMBADA	plain	92.49	92.13	87.39
Ko-LAMBADA	chat	81.22	80.06	69.20
Ko-ARC-Easy	plain	67.81	69.04	71.21
Ko-ARC-Easy	chat	64.33	62.04	64.15

Table 4: Mean multiple-choice accuracy of open-weight models under three scoring rules: raw log-likelihood ( acc ), byte-length normalized log-likelihood ( acc_norm ), and NPSQ-based accuracy ( acc_npsq ).

Figure 3: Performance of Qwen3.5 Base (pretrained) and instruction-tuned variants at five paired model sizes under plain prompting. We report NPSQ accuracy for MCQA, strict exact match for Ko-GSM8K, prompt-level strict accuracy for Ko-IFEval, and the EQ-Bench score for Ko-EQ-Bench. Accuracies are expressed as percentages. 35B denotes 35B-A3B.

	Ko-GSM8K		Ko-EQ-Bench
Qwen3.5	Default	Off	Default	Off
0.8B	23.05	23.96	16.77	16.65
2B	43.06	44.50	31.16	30.98
4B	0.15	69.60	0.00	53.11
9B	0.00	77.33	0.00	54.55
27B	0.00	80.67	0.00	72.47

Table 5: Qwen3.5 results under default chat prompting and with thinking disabled (Off). Ko-GSM8K reports strict exact match (%). Ko-EQ-Bench reports the EQ-Bench score. Both conditions use chat templates.

Task	Best open	Best closed	Gap
Ko-ARC-Easy	98.06	99.12	1.06
Ko-ARC-Challenge	96.57	97.77	1.20
Ko-LAMBADA	96.14	98.76	2.62
Ko-WinoGrande	84.85	93.37	8.52

Table 6: Best generative MCQA accuracy (%) among open-weight and closed API models under default chat configurations. Gemma-4-31B-it leads the open-weight group on all four tasks. GPT-5.5 leads the closed group on Ko-ARC-Challenge and Claude Opus 4.7 on the other tasks. Gap denotes the accuracy difference (closed minus open), in percentage points. Qwen non-thinking runs are reported separately in Appendix F .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Original context in English ARC	Localized context in Ko-ARC
Locations	U.S. states, national parks, coastlines, and rivers	Korean or East Asian locations with analogous phenomena (e.g., U.S. Gulf coast $\to$ West Sea / Yellow Sea; Maine solar-noon example $\to$ Baengnyeongdo Island). Globally familiar references, such as the Himalayas, are retained when they are already standard in Korean science education.
Climate and seasonality	Hurricanes; jet-stream examples set in the continental U.S.	Comparable weather and seasonal phenomena in the Northwest Pacific (e.g., hurricanes $\to$ typhoons ; jet-stream examples rewritten using Korean winter-weather patterns; daylight-duration comparisons are rewritten using Korean cities with appropriate latitude differences).
Species and organisms	Black bear; warbler migration; Gulf sturgeon	Ecologically comparable species that are familiar in Korean educational contexts (e.g., Asiatic black bear in Jirisan Mountain; cranes migrating between Siberia or Manchuria and the Korean peninsula; salmon as a river-sea migratory species).
Industry and energy	Kentucky coal-transport context; U.S. 2003 electricity generation mix	Korean settings with comparable industrial or energy-related roles (e.g., Jecheon-si cement freight corridor; Korean 2022 electricity generation mix).
Units and measurements	Imperial units; city-by-city daylight comparisons	Metric units; numerical values adjusted when needed to preserve the scientific relation; city pairs reselected within Korea.
Everyday objects and cultural examples	U.S.-specific household or cultural items used to frame the question	Korean counterparts that make the question natural in Korean while preserving the scientific framing and the correct answer.

Appendix

Table 7: Representative localization rules for Ko-ARC. Each substitution is checked to preserve both the underlying scientific mechanism and the correct answer.

Instruction group	IFEval	Ko-IFEval	Changes
Keywords	4	6	Alphabet-based letter constraints are separated from Korean choseong(초성; leading consonant)-based constraints; a keyword-substitution check is added for Korean prompts.
Length Constraints	4	6	Character-count checks are extended to distinguish whitespace-inclusive and whitespace-exclusive counting, which is necessary for Korean spacing conventions.
Detectable Format	6	7	A multiple-choice formatting check is added.
Language	1	1	No structural change; the required response language can be Korean or English.
Detectable Content	2	2	No structural change.
Combination	2	2	No structural change.

Appendix

Table 8: Instruction-type inventory in the original IFEval and Ko-IFEval. The "Changes" column summarizes Ko-IFEval-specific additions and adaptations.

Dataset	Aligned items	Correction	Localization	Cross-review
Ko-ARC-Easy	2,376	2,056 (86.5%)	643 (27.1%)	72 (3.0%)
Ko-ARC-Challenge	1,172	720 (61.4%)	73 (6.2%)	42 (3.6%)
Ko-GSM8K	1,319	1,291 (97.9%)	830 (62.9%)	358 (27.1%)
Ko-EQ-Bench	171	171 (100.0%)	165 (96.5%)	14 (8.2%)

Appendix

Table 9: Recorded item revisions at each construction stage, reported as counts and percentages of aligned items. Ko-ARC-Challenge statistics cover the 1,172-item review pool before five items were excluded from the final dataset. The separate Ko-IFEval process is not included.

Provider	Model	Size	Hugging Face checkpoint
Korean-developed LLMs
KT	Mi:dm 2.0 Mini	2.3B	K-intelligence/Midm-2.0-Mini-Instruct
	Mi:dm 2.0 Base	11.5B	K-intelligence/Midm-2.0-Base-Instruct
LG	EXAONE-4.0-1.2B	1.2B	LGAI-EXAONE/EXAONE-4.0-1.2B
	EXAONE-4.0-32B	32B	LGAI-EXAONE/EXAONE-4.0-32B
Kakao	Kanana-2-30b-a3b-base-2601	30B-A3B	kakaocorp/kanana-2-30b-a3b-base-2601

Appendix

Table 10: Open-weight models evaluated. Sizes are reported in billions of parameters. For mixture-of-experts models, we also include the active-parameter shorthand reported by the developer, such as 30B-A3B . All checkpoints are accessed through the Hugging Face Hub.

Task	Few-shot	Output type	Max gen. toks	Metric
Ko-ARC-Easy	5	multiple choice	–	acc / acc_norm / acc_npsq
Ko-ARC-Challenge	5	multiple choice	–	acc / acc_norm / acc_npsq
Ko-WinoGrande	0	multiple choice	–	acc / acc_norm / acc_npsq
Ko-LAMBADA	0	multiple choice	–	acc / acc_norm / acc_npsq
Ko-GSM8K	5	generate-until	2048	strict-match / flexible-extract
Ko-IFEval	0	generate-until	1280	prompt-level and instruction-level strict/loose accuracy

Appendix

Table 11: Detailed task configuration. Original multiple-choice tasks are evaluated only for open-weight models with log-likelihood access. Generative multiple-choice variants are used for both open-weight and closed API models. For Ko-EQ-Bench, the maximum generation length is 80 tokens for open-weight models and 1024 tokens for closed API models.

		Ko-WinoGrande			Ko-LAMBADA			Ko-ARC-Easy			Ko-ARC-Challenge
Model	Parameter size	acc	acc _norm	acc _npsq	acc	acc _norm	acc _npsq	acc	acc _norm	acc _npsq	acc	acc _norm	acc _npsq
HyperCLOVAX-SEED-Text- Instruct-0.5B	0.5B	55.72	55.88	54.62	93.30	92.95	87.72	58.71	58.67	61.66	32.39	38.73	38.05
HyperCLOVAX-SEED-Text- Instruct-1.5B	1.5B	56.83	57.06	54.70	90.02	89.22	84.17	60.35	61.45	59.51	34.53	40.62	38.13
A.X 4.0 Light	7B	64.96	63.38	62.75	96.59	96.32	93.13	78.41	78.87	79.46	55.87	59.73	61.27
Mi:dm 2.0 Mini	2.3B	61.25	60.62	58.80	96.76	96.54	93.48	75.55	76.60	78.11	48.33	54.84	58.70
Mi:dm 2.0 Base	11.5B	66.22	65.04	63.69	98.32	98.18	96.72	81.57	82.66	85.23	58.78	64.18	65.55

Appendix

Table 12: Open-weight multiple-choice results under plain prompting. The chat-template subset average uses the same model subset as the chat-template prompting results in Table 13 . The highest score in each metric column is highlighted in red.

		Ko-WinoGrande			Ko-LAMBADA			Ko-ARC-Easy			Ko-ARC-Challenge
Model	Parameter size	acc	acc _norm	acc _npsq	acc	acc _norm	acc _npsq	acc	acc _norm	acc _npsq	acc	acc _norm	acc _npsq
HyperCLOVAX-SEED-Text -Instruct-0.5B	0.5B	55.49	55.09	54.85	83.99	83.19	74.99	55.56	55.60	57.74	31.19	36.59	35.48
HyperCLOVAX-SEED-Text -Instruct-1.5B	1.5B	57.46	56.35	55.09	83.86	83.55	78.76	57.91	59.72	58.88	33.25	40.19	37.45
A.X 4.0 Light	7B	58.72	59.12	57.77	86.56	85.81	78.49	75.46	77.15	77.36	50.64	56.56	56.47
Mi:dm 2.0 Mini	2.3B	59.67	58.49	54.78	83.55	81.82	66.83	72.39	73.70	75.25	45.59	52.19	51.76
Mi:dm 2.0 Base	11.5B	64.33	63.06	60.62	86.43	85.06	68.65	77.53	74.45	79.59	54.67	58.61	61.70

Appendix

Table 13: Open-weight multiple-choice results under chat-template prompting. The highest score in each metric column is highlighted in red.

		Ko-GSM8K		Ko-IFEval				Ko-EQ-Bench
Model	Parameter size	strict -match	flexible -extract	prompt _level _strict _acc	inst _level _strict _acc	prompt _level _loose _acc	inst _level _loose _acc	eqbench	percent _parseable
HyperCLOVAX-SEED-Text-Instruct-0.5B	0.5B	4.09	3.79	12.49	25.99	14.63	29.06	7.90	89.47
HyperCLOVAX-SEED-Text-Instruct-1.5B	1.5B	9.40	7.88	21.64	36.72	23.54	39.44	38.99	92.98
A.X 4.0 Light	7B	78.85	78.92	60.29	74.36	65.16	77.56	54.53	100.00
Mi:dm 2.0 Mini	2.3B	57.85	61.71	19.62	29.13	24.26	32.96	11.93	38.01
Mi:dm 2.0 Base	11.5B	81.73	81.80	28.30	40.70	31.87	43.83	51.01	94.15

Appendix

Table 14: Open-weight generation results under plain prompting. The chat-template subset average uses the same model subset as the chat-template prompting results in Table 15 . The highest score in each metric column is highlighted in red.

		Ko-GSM8K		Ko-IFEval				Ko-EQ-Bench
Model	Parameter size	strict -match	flexible -extract	prompt _level _strict _acc	inst _level _strict _acc	prompt _level _loose _acc	inst _level _loose _acc	eqbench	percent _parseable
HyperCLOVAX-SEED-Text-Instruct-0.5B	0.5B	3.56	6.07	17.48	32.33	18.79	34.49	25.99	95.91
HyperCLOVAX-SEED-Text-Instruct-1.5B	1.5B	12.51	11.90	22.24	36.31	24.38	39.23	41.32	93.57
A.X 4.0 Light	7B	82.18	82.34	73.72	83.07	76.10	85.09	56.04	100.00
Mi:dm 2.0 Mini	2.3B	66.03	67.32	53.98	65.78	57.67	68.43	16.30	46.20
Mi:dm 2.0 Base	11.5B	32.60	84.08	70.15	80.63	74.79	83.76	47.32	78.95

Appendix

Table 15: Open-weight generation results under chat-template prompting. The highest score in each metric column is highlighted in red.

Model	Ko -WinoGrande -gen	Ko -LAMBADA -gen	Ko-ARC -Easy -gen	Ko-ARC -Challenge -gen	Ko-GSM8K		Ko-IFEval				Ko-EQ-Bench
Model	exact _match	exact _match	exact _match	exact _match	strict _match	flexible _extract	prompt _level _strict _acc	inst _level _strict _acc	prompt _level _loose _acc	inst _level _loose _acc	eqbench	percent _parseable
Claude Haiku 4.5	77.82	91.8	97.9	94.17	93.48	93.48	71.7	82.37	81.45	88.5	65.79	98.25
Claude Opus 4.7	93.37	98.76	99.12	97.17	96.06	96.06	79.55	87.25	83.47	89.69	79.23	100
GPT-5-mini	76.95	64.66	97.69	95.89	81.96	94.47	36.98	35.75	37.1	35.82	64.9	99.42
GPT-5.5	91.71	96.63	98.91	97.77	96.29	96.29	81.09	83.69	82.16	84.39	78.71	100

Appendix

Table 16: Closed API model results across all benchmark tasks under chat-style prompting.

		Ko-WinoGrande-gen	Ko-LAMBADA-gen	Ko-ARC-Easy-gen	Ko-ARC-Challenge-gen
Model	Parameter size	exact_match	exact_match	exact_match	exact_match
HyperCLOVAX-SEED-Text-Instruct-0.5B	0.5B	22.18	3.90	66.71	49.44
HyperCLOVAX-SEED-Text-Instruct-1.5B	1.5B	48.38	61.02	76.01	58.35
A.X 4.0 Light	7B	67.25	85.90	93.22	86.38
Mi:dm 2.0 Mini	2.3B	57.70	74.19	84.51	73.09
Mi:dm 2.0 Base	11.5B	68.19	86.65	94.23	87.06

Appendix

Table 17: Generative MCQA results for open-weight models under chat-style prompting. The highest score in each metric column is highlighted in red.

Model	Parameter size	Ko -WinoGrande -gen	Ko -LAMBADA -gen	Ko-ARC -Easy -gen	Ko-ARC -Challenge -gen	Ko-GSM8K		Ko-EQ-Bench
Model	Parameter size	exact _match	exact _match	exact _match	exact _match	strict _match	flexible _extract	eqbench	percent _parseable
Qwen3.5-0.8B	0.8B	50.12	51.71	62.58	50.64	23.96	25.17	16.65	99.42
Qwen3.5-2B	2B	38.20	55.17	81.57	69.75	44.50	45.34	30.98	100.00
Qwen3.5-4B	4B	62.51	81.37	93.52	86.89	69.60	71.04	53.11	100.00
Qwen3.5-9B	9B	68.19	87.01	96.34	91.09	77.33	78.54	54.55	100.00
Qwen3.5-27B	27B	80.58	95.92	98.53	96.06	80.67	82.87	72.47	100.00

Appendix

Table 18: Generative MCQA and generation results for six instruction-tuned Qwen3.5 models with enable_thinking=False , while retaining chat templates.

Task	Model group	English	DeepL	KBS
ARC	Korean-developed	67.29	54.00	59.49
	Qwen3.5	51.22	44.14	46.49
	Gemma 4	72.56	60.14	67.05
GSM8K	Korean-developed	57.52	34.36	44.80
	Qwen3.5	54.11	10.01	11.06
	Gemma 4	82.41	59.84	64.48

Appendix

Table 19: Performance across English, raw DeepL, and KBS variants under chat-style prompting, averaged over the same eight Korean-developed, six Qwen3.5, and four Gemma 4 models. ARC and WinoGrande use NPSQ accuracy, GSM8K uses strict exact match, and EQ-Bench uses its benchmark score. DeepL results are unavailable for WinoGrande and excluded for EQ-Bench because the Korean emotion labels in the translated prompts do not match the English reference labels used by the scorer.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

KMMMU: Evaluation of Massive Multi-discipline Multimodal Understanding in Korean Language and Context
Mar 18, 2026Nahyun Lee, Guijin Son, Hyunwoo Ko +4Multimodal Model Evaluation Multimodal Understanding
Chung-Ang University · HAE-RAE · Seoul National University +2
KoALa-Bench: Evaluating Large Audio Language Models on Korean Speech Understanding and Faithfulness
Mar 30, 2026Jinyoung Kim, Hyeongsoo Lim, Eunseo Seo +4Audio-Language Model Evaluation Speech Processing
Department of Artificial Intelligence, Chung-Ang University · Upstage AI
CoEval: Ranking Language Models for Custom Tasks Without Labeled Data or Trustworthy Benchmarks
Jun 2, 2026Alexander Apartsin, Yehudit ApersteinModel Selection LLM Evaluation
Holon Institute of Technology, Israel · Afeka Tel Aviv Academic College of Engineering, Israel

Mar 18, 2026J/K move · Enter open · S save

KMMMU: Evaluation of Massive Multi-discipline Multimodal Understanding in Korean Language and Context

Nahyun Lee, Guijin Son, Hyunwoo Ko, Chanyoung Kim+3

Chung-Ang University · HAE-RAE · Seoul National University+2

Multimodal Model Evaluation Multimodal Understanding Multilingual VLM Evaluation Cross-Cultural Language Model Evaluation Cultural Knowledge in Language Models

Open paper AskPDF arXiv

Multimodal evaluation is strengthened when benchmarks reflect the languages, institutions, and disciplinary conventions that shape expert visual reasoning. We introduce KMMMU, a native Korean benchmark built from Korean-native examination materials. KMMMU contains 3,466 questions spanning nine disciplines and nine visual modality categories. For targeted evaluation on challenging questions and Korean-specific knowledge, we additionally provide KMMMU-Adv and KMMMU-Local, respectively. Experiments show that the strongest evaluated open-weight model reaches 42.05% on the KMMMU, while the best evaluated proprietary model achieves 52.42% on the KMMMU-Adv. Performance varies substantially across disciplines and answer formats, with Korean-specific questions showing gaps of up to 13.43%. Error analysis suggests that inspected failures often involve convention-to-label mapping, few-shot symbolic induction, localized knowledge recall, and domain-specific standards, not reasoning depth alone. KMMMU provides a testbed for native Korean expert-level multimodal understanding beyond English-centric or translation-based evaluation.

Abstract

LLM Evaluation

Explore similar work

KMMMU: Evaluation of Massive Multi-discipline Multimodal Understanding in Korean Language and Context

KoALa-Bench: Evaluating Large Audio Language Models on Korean Speech Understanding and Faithfulness

CoEval: Ranking Language Models for Custom Tasks Without Labeled Data or Trustworthy Benchmarks