KLEAR: Knowledge-Matched LLM Evaluation in Korean across Application Requirements via Controlled Benchmark Transformation

Seoyeon Kim; Jeonghyun Kang; Hamin Koo; Barom Kang; Heondeuk Lee; Jaehyung Kim

cs.CLNov 6, 2026

KLEAR: Knowledge-Matched LLM Evaluation in Korean across Application Requirements via Controlled Benchmark Transformation

Authors: Seoyeon Kim, Jeonghyun Kang, Hamin Koo, Barom Kang, Heondeuk Lee, Jaehyung Kim

Organizations: Yonsei University · DATUMO INC.

Abstract

Existing Korean benchmarks for Large Language Models (LLMs) evaluate performance across diverse knowledge domains, but their accuracy scores cannot distinguish whether incorrect responses arise from a lack of required knowledge or a failure to apply that knowledge correctly. Our analysis of existing Korean benchmarks reveals that most questions can be answered through factual recall alone, while those demanding greater application also differ in required knowledge, making it difficult to isolate application ability. To address this, we introduce KLEAR, a framework that constructs knowledge-matched contrasts by varying application demand while preserving the required knowledge. During construction, KLEAR identifies the knowledge underlying each source question as an explicit anchor bundle to guide generation of questions requiring more complex application. Generated candidates then undergo automated and human validation, with iterative revision based on evaluation feedback. Experiments demonstrate that the resulting knowledge-matched sets largely satisfy the intended constraints, enabling controlled evaluation across diverse LLMs. These evaluations reveal that increasing application demand exposes failures that persist even when the required knowledge is provided.

Figures & tables

	K		A
	EM	$\kappa$	EM	$\kappa$
w/o silver	68.3	0.67	45.0	0.47
w/ silver	56.7	0.56	69.2	0.69

Table 1: Validation of knowledge (K) and application (A) labels. LLM–human agreement on 60 questions, averaged across two human annotators: exact match (EM; %) and linearly weighted Cohen’s

\kappa

, which penalizes disagreements in proportion to the distance between levels. w/o silver and w/ silver denote the LLM annotation without and with silver reasoning.

	A0	A1	A2	A3	Total
K0	0.0	1.0	0.3	0.0	1.3
K1	2.9	1.2	1.1	0.1	5.2
K2	18.5	4.0	5.5	3.6	31.7
K3	39.6	4.2	5.1	12.7	61.7
Total	61.1	10.4	12.0	16.4	100.0

Table 2: Distribution of knowledge (K) and application (A) demands. Percentage of the 7,340 questions across K and A levels in three Korean benchmarks.

Figure 1: Overview of KLEAR . (a) Seed questions from Korean benchmarks are labeled by knowledge (K) and application (A) demands. (b) Silver reasoning is generated from the seed question and gold answer, then used to derive an anchor knowledge bundle for generating higher-A questions. (c) Generated questions undergo LLM auditing, proxy screening, and human validation, with failed candidates revised using LLM or human feedback. The result is a matched set of questions that share the same knowledge but differ in application demand.

Figure 2: Failure composition by application demand. Distribution of incorrect responses by failure type across three Korean LLMs, with accuracy (%) on the right.

		KMMLU
	CLIcK	Pro	Redux	Total
Source questions	66	53	46	165
A1 $\rightarrow$ A2	29	20	29	78
A1 $\rightarrow$ A3	24	15	12	51
A2 $\rightarrow$ A3	77	55	26	158
Generated questions	130	90	67	287

Table 3: Final dataset statistics. Number of source questions and generated questions by transformation.

Stage	Count	Rate
Automated stage
Candidates generated	1,964	–
All eleven criteria pass	627	31.9%
Proxy answers incorrectly	760	38.7%
Both (pass before revision)	294	15.0%
Candidates revised	1,662	–

Table 4: Stage-wise yield. Indented rates are relative to the preceding unindented count; the overall automated-stage pass rate is relative to all generated candidates.

			KMMLU
	CLIcK		Pro		Redux		All		Matched
Model	A2	A3	A2	A3	A2	A3	A2	A3	A2	A3
K-EXAONE	65.5	46.5	50.0	37.1	34.5	57.9	50.0	45.5	46.7	40.0
A.X K1	72.4	40.6	45.0	27.1	48.3	47.4	56.4	37.3	51.7	21.7
Solar Open	55.2	29.7	30.0	21.4	34.5	36.8	41.0	28.2	38.3	23.3
Aya Expanse 32B	31.0	26.7	25.0	21.4	20.7	18.4	25.6	23.4	33.3	20.0

Table 5: Closed-book accuracy (%) by application level. Matched covers the 30 seeds that have both an A2 and an A3 contrast (40 and 43 questions), averaged over seeds. The numbers of A2 and A3 questions are 29 and 101 for CLIcK, 20 and 70 for KMMLU-Pro, and 29 and 38 for KMMLU-Redux.

Figure 3: Failure composition on the A1–A2–A3 ladder. Incorrect closed-book responses by failure type across three Korean LLMs. A1 corresponds to the seed. Other includes invalid outputs and defective questions.

Figure 4: Effects of knowledge and application interventions. Accuracy changes across closed-book, knowledge (+K), and knowledge and application (+K+A) conditions for A2 (78 questions) and A3 (209 questions). Each bar shows closed-book accuracy, the additional gains from +K and +K+A, and the remaining error rate.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

		KMMLU
	CLIcK	Pro	Redux	Total
Original questions	1,995	2,822	2,587	7,404
Excluded	62	1	1	64
Retained questions	1,933	2,821	2,586	7,340

Appendix

Table 6: Source benchmark statistics. Number of questions in each source benchmark and number retained for the knowledge and application analysis.

Knowledge Demand (K)
K0	No external knowledge required
K1	Everyday common sense or general language knowledge
K2	School-level factual or conceptual knowledge
K3	Specialized professional, institutional, or procedural knowledge
Application Demand (A)
A0	Direct recall; the retrieved fact or rule is itself the answer

Appendix

Table 7: Knowledge and application taxonomy. Definitions of the four levels in each dimension.

	Question (Korean)	English translation	Why this level
K0 No external knowledge	제8회 음식 문화 축제. 기간: 2016년 8월 20일(토)–21일(일). 장소: 인주 시청 앞 광장. 행사 내용: 한국 전통요리 무료 시식 및 체험 행사. 체험을 원하시는 분은 현장에서 직접 신청하시기 바랍니다. 다음 글의 내용과 같은 것을 고르십시오. A) 축제는 올해 처음 열린다. B) 축제는 주말 이틀 동안 진행된다. C) 축제에서 체험을 하려면 예약해야 한다. D) 축제에서 전통 요리를 사 먹을 수 있다.	The 8th Food Culture Festival. Dates: Sat., August 20 – Sun., August 21, 2016. Venue: the plaza in front of Inju City Hall. Events: free tasting of traditional Korean dishes and hands-on activities. Those who wish to take part in the activities should apply on site. Choose the statement that agrees with the text. A) The festival is held for the first time this year. B) The festival runs for two days over a weekend. C) A reservation is required for the activities. D) Traditional dishes can be bought and eaten at the festival.	Everything needed is in the text.
K1 Everyday knowledge	응급환자가 발생했을 때에는 어떤 전화번호로 연락해야 하는가? A) 112 B) 119 C) 118 D) 911	Which telephone number should be called when there is an emergency patient? A) 112 B) 119 C) 118 D) 911	Everyday common sense.
K2 School-level knowledge	한국인의 주식인 쌀이 많이 생산되는 평야가 아닌 곳은? A) 호남평야 B) 나주평야 C) 김포평야 D) 강원평야	Which of the following is not a plain where rice, the staple food of Koreans, is produced in large quantities? A) Honam Plain B) Naju Plain C) Gimpo Plain D) Gangwon Plain	School-level knowledge of geography.
K3 Specialized knowledge	인권 침해 사실이 판단될 경우 국가인권위원회는 어떤 조치를 취하는가? A) 당사자에게 문제를 개선하도록 권고 B) 법적 처벌 C) 당사자를 해고 D) 사건을 무시	What measure does the National Human Rights Commission take when it finds that human rights have been violated? A) Recommends that the party remedy the problem B) Legal punishment C) Dismissal of the party D) Ignoring the case	Knowledge of the powers of a specific institution.

Appendix

Table 8: Examples of knowledge (K) levels. One question from CLIcK per level, with our English translation and the reason for its label. The gold answer is in bold .

	Question (Korean)	English translation	Why this level
A0 Knowledge retrieval	대한민국 헌법재판소에는 몇 명의 헌법재판관이 있는가? A) 7명 B) 8명 C) 9명 D) 10명	How many justices sit on the Constitutional Court of Korea? A) 7 B) 8 C) 9 D) 10	The recalled fact is itself the answer; there is no case to map it onto.
A1 Explicit mapping	이혼 당사자 간의 협의가 잘 이루어지지 않을 때, 자녀의 양육권은 어떻게 결정되는가? A) 가정법원에서 결정한다 B) 자녀의 의사에 따라 결정한다 C) 변호사가 결정한다 D) 이혼한 당사자 간의 협의에 의해 결정한다	When the divorcing parties cannot reach an agreement, how is custody of the children decided? A) The family court decides B) It is decided by the wishes of the children C) A lawyer decides D) It is decided by agreement between the divorced parties	The question states the condition of the rule (the parties fail to agree), so the rule maps onto the case directly.
A2 Inferred mapping	예금액이 많을 경우 어떻게 해야 모두 보호받을 수 있는가? A) 한 은행에 모두 예금한다 B) 여러 금융 기관에 나누어 예금한다 C) 금액을 줄인다 D) 보호받을 수 없다	If one has a large amount of deposits, what should one do so that all of it is protected? A) Deposit all of it in one bank B) Divide it among several financial institutions C) Reduce the amount D) It cannot be protected	One rule applies (deposits are protected up to a limit per institution), but it is not stated; one must infer that a large deposit exceeds the limit and what would satisfy the rule.
A3 Mapping selection	다음 정치관과 관련이 깊은 정책으로 옳은 것은? 임금의 직책은 한 사람의 재상을 논정하는 데 있다 하였으니, 바로 총재를 두고 한 말이다. 총재는 위로는 임금을 받들고 밑으로는 백관을 통솔하여 만민을 다스리는 것이니 직책이 매우 크다. (후략) A) 육조 직계제의 시행 B) 사간원의 독립 C) 의정부 서사제의 시행 D) 집현전의 설치	Which policy is closely related to the following view of government? “It is said that the duty of the king lies in choosing one chancellor. The chancellor serves the king above and leads all officials below in governing the people, so his office is of great weight.” (abridged) A) Direct reporting of the Six Ministries to the king B) Independence of the Office of the Censor-General C) Deliberation by the State Council D) Establishment of the Hall of Worthies	Two institutions both govern how the king and his ministers share power, with opposite effects: direct reporting strengthens the king, and deliberation by the State Council strengthens the chancellors. One must select which of the two the stated view corresponds to.

Appendix

Table 9: Examples of application (A) levels. One question from CLIcK per level, with our English translation and the reason for its label. The gold answer is in bold .

	K	A
Human–Human	0.63 (68.3)	0.60 (56.7)
LLM (with silver)–Human	0.56 (56.7)	0.69 (69.2)
LLM (without silver)–Human	0.67 (68.3)	0.47 (45.0)

Appendix

Table 10: Reliability of K/A annotations. Linearly weighted Cohen’s

\kappa

on 60 questions, with exact agreement (%) in parentheses. LLM–human values are averaged over the two human annotators.

Other errors		650
T0	Missing or invalid answer output	262
T1	Required material (passage, table, or figure) is absent	45
T2	Defective question (wrong gold answer, ambiguous or duplicate options, typo)	342
Knowledge failures		3,422
T3	Recalls a specific fact or value incorrectly	2,425
T4	Binds an entity, statute, or event to the wrong identity or attribute	575

Appendix

Table 11: Failure taxonomy. Definitions of the 13 failure types and the number of incorrect responses with each primary type (

n=5{,}808

). One response fits none of the types and is counted under other errors.

				Failure type (%)
Level	Questions	Accuracy (%)	Incorrect responses	Knowledge	Application	Other
A0	4,482	74.5	3,433	69.8	21.1	9.1
A1	767	88.5	265	41.5	36.6	21.9
A2	884	75.1	661	40.5	41.9	17.5
A3	1,207	60.0	1,449	44.8	44.0	11.2
All	7,340	73.6	5,808	58.9	29.9	11.2

Appendix

Table 12: Failure composition by application level. Share of incorrect responses (%) by primary failure type. Accuracy is averaged over the three models.

ID	Criterion	Passing condition
LLM rubric: validity as a question
L1	Answer correctness	The designated gold answer is correct.
L2	Answer uniqueness	Exactly one option is correct.
L3	Stem determinacy	The stem admits one reading and supplies everything needed to answer.
L4	Bundle conformance	No domain knowledge outside the anchor bundle is required.
L5	Surface form	The text is free of formatting artifacts and orthographic errors.

Appendix

Table 13: Automated and human validation rubrics. A candidate must satisfy every criterion of the LLM rubric (L1–L11) and the human rubric (H1–H5). Annotators judge H1–H3 before the anchor bundle is revealed and H4–H5 after.

Stage	Model	Effort
Anchor knowledge extraction	GPT-5.5	high
Contrast generation	GPT-5.5	high
LLM-based auditing	GPT-5.5	high
Proxy difficulty screening	GPT-5-nano	high
Self-revision	GPT-5.5	high
Human-guided revision	GPT-5.5	high

Appendix

Table 14: Construction configurations. Model and reasoning effort used at each stage of the pipeline.

	A1 $\rightarrow$ A2	A1 $\rightarrow$ A3	A2 $\rightarrow$ A3	All
Seed–target pairs	78	52	165	295
Candidates generated	531	345	1,088	1,964
All eleven criteria pass	167 (31.5%)	126 (36.5%)	334 (30.7%)	627 (31.9%)
Proxy answers incorrectly	137 (25.8%)	125 (36.2%)	498 (45.8%)	760 (38.7%)
Both (pass before revision)	53 (10.0%)	52 (15.1%)	189 (17.4%)	294 (15.0%)
Candidates revised	477	287	898	1,662

Appendix

Table 15: Yield of the automated stage by transformation type. Percentages for the first three candidate rows and the last are shares of the candidates generated; the percentage for self-revision is a share of the candidates revised.

	First	Second	All
Candidates validated	231	70	301
Pass directly	194	58	252
Revised	37	12	49
Pass after revision	23	12	35
Final questions	217	70	287

Appendix

Table 16: Human validation results by batch. Candidates in the first batch are judged by two annotators and those in the second batch by one.

ID	Criterion	Fail	Resolved	New
L1	Answer correctness	1.2	96	34
L2	Answer uniqueness	1.9	100	57
L3	Stem determinacy	1.1	100	37
L4	Bundle conformance	2.2	98	44
L5	Surface form	0.5	89	12
L6	Target-level structure	18.5	67	128

Appendix

Table 17: Failures by criterion. Fail is the share (%) of candidates that do not pass the criterion: before revision for the LLM rubric (1,964 candidates) and at first validation for the human rubric (301 candidates). Resolved is the share (%) of these failures that revision fixes, and New is the number of failures that revision introduces, with self-revision for the LLM rubric and human-guided revision for the human rubric.

ID	Criterion	With	Without
L1	Answer correctness	1.9	6.5
L2	Answer uniqueness	3.7	6.5
L3	Stem determinacy	1.9	3.7
L4	Bundle conformance	1.9	20.4
L5	Surface form	0.9	0.0
L6	Target-level structure	20.4	23.1

Appendix

Table 18: Effect of the anchor bundle. Share (%) of candidates that fail each criterion before revision, when generated with and without the anchor bundle for the same 36 pairs (108 candidates each). Only L4 differs significantly (Fisher’s exact test,

p<0.001

Incorrect models	0	1	2	3
Seeds	12	14	48	24
Seed–target pairs	23	28	81	38
Candidates generated	117	246	549	246
All eleven criteria pass	58.1	43.1	31.3	28.0
Proxy answers incorrectly	12.8	30.1	28.2	39.8
Both (pass before revision)	7.7	11.8	11.8	14.2

Appendix

Table 19: Yield by seed difficulty. A1 seeds grouped by the number of Korean LLMs (out of three) that answer the seed incorrectly. Rates are shares (%) of the candidates generated.

	A2	A3	All
Questions	78	209	287
Source benchmark of the seed
CLIcK	29	101	130
KMMLU-Pro	20	70	90
KMMLU-Redux	29	38	67
Knowledge level

Appendix

Table 20: Composition of KLEAR by application level. Number of questions, except for question length, which is the mean number of characters.

Model	Identifier	Access	Reasoning	Sampling	Max.
K-EXAONE	LGAI-EXAONE/K-EXAONE-236B-A23B	vLLM	Enabled	1.0 / 0.95	28,672
A.X K1	skt/A.X-K1	vLLM	Enabled	0.6 / 0.95	45,056
Solar Open	upstage/Solar-Open-100B	vLLM	High	0.8 / 0.95 †	28,672
Aya Expanse 32B	c4ai-aya-expanse-32b	Cohere	None	0.3	4,000
GLM-4.7	z-ai/glm-4.7	OpenRouter	High	default	32,000
Qwen3.5	qwen/qwen3.5-397b-a17b	OpenRouter	High	default	32,000

Appendix

Table 21: Evaluated models and inference settings. Max. is the limit on output tokens, including reasoning. Sampling is given as temperature / top-

p

; default denotes the default settings of the API. † Top-

k

is set to 50.

Stage	Model	Calls	Input	Output
Construction
Generation	GPT-5.5	1,965	9.3	18.9
Auditing	GPT-5.5	3,626	17.1	27.0
Self-revision	GPT-5.5	1,670	8.0	19.1
Proxy	GPT-5-nano	3,626	1.3	22.5
Evaluation

Appendix

Table 22: Token usage of API calls. Input and output tokens in millions. Evaluation covers the 287 questions under three conditions.

	CLIcK			KMMLU-Pro			KMMLU-Redux			All
Model	C	+K	+K+A	C	+K	+K+A	C	+K	+K+A	C	+K	+K+A
K-EXAONE	50.8	76.9	86.9	40.0	78.9	90.0	47.8	88.1	95.5	46.7	80.1	89.9
A.X K1	47.7	66.9	77.7	31.1	71.1	83.3	47.8	74.6	92.5	42.5	70.0	82.9
Solar Open	35.4	65.4	74.6	23.3	67.8	83.3	35.8	74.6	88.1	31.7	68.3	80.5
Aya Expanse 32B	27.7	41.5	53.8	22.2	40.0	53.3	19.4	35.8	50.7	24.0	39.7	53.0
GLM-4.7	69.2	83.8	86.2	50.0	78.9	92.2	61.2	91.0	94.0	61.3	84.0	89.9

Appendix

Table 23: Accuracy (%) on KLEAR by source benchmark of the seed. C is the Closed condition. The numbers of questions are 130 for CLIcK, 90 for KMMLU-Pro, and 67 for KMMLU-Redux.

	A2			A3
Family	Closed	+K	+K+A	Closed	+K	+K+A
Korean	49.1	86.8	92.7	37.0	67.6	81.3
Multilingual	25.6	47.4	59.0	23.4	36.8	50.7
Open-weight	70.5	92.9	96.8	64.6	81.1	87.6
Proprietary	91.7	99.4	99.4	87.1	91.9	95.7

Appendix

Table 24: Accuracy (%) by model family. Averages over the models in each group: Korean (K-EXAONE, A.X K1, Solar Open), multilingual (Aya Expanse 32B), open-weight (GLM-4.7, Qwen3.5), and proprietary (Claude Sonnet 5.5, GPT-6.1 Sol).

	A2 ( $n=78$ )				A3 ( $n=209$ )
Model	Errors	Fixed	Broken	$p$	Errors	Fixed	Broken	$p$
K-EXAONE	6	6	2	.289	51	33	9	$<$ .001
A.X K1	12	8	3	.227	74	47	15	$<$ .001
Solar Open	13	9	4	.267	78	43	13	$<$ .001
Aya Expanse 32B	41	16	7	.093	132	57	28	.002
GLM-4.7	7	6	2	.289	39	18	5	.011

Appendix

Table 25: Effect of adding application statements. Errors is the number of incorrect responses under +K ; Fixed and Broken are the numbers of responses that become correct and incorrect under +K+A ;

p

is from the exact McNemar test.

	A1			A2
Model	Closed	+K	+K+A	Closed	+K	+K+A
K-EXAONE	39.7	84.5	87.9	32.7	73.8	83.2
A.X K1	43.1	84.5	87.9	30.8	68.2	76.6
Solar Open	27.6	81.0	93.1	28.0	60.7	75.7

Appendix

Table 26: Accuracy (%) on the seeds. The 165 seeds comprise 58 A1 and 107 A2 questions.

Model	Level	$n$	Knowledge	Application	Other
K-EXAONE	A2	39	17.9	66.7	15.4
	A3	114	15.8	74.6	9.6
A.X K1	A2	34	23.5	70.6	5.9
	A3	131	20.6	71.8	7.6
Solar Open	A2	46	32.6	65.2	2.2
	A3	150	21.3	74.0	4.7

Appendix

Table 27: Failure types on KLEAR . Share (%) of each primary failure type among the incorrect Closed responses;

n

is the number of incorrect responses.

	Accuracy		Seeds favoring
Model	A2	A3	A2	A3	$p$
K-EXAONE	46.7	40.0	9	6	.607
A.X K1	51.7	21.7	15	4	.019
Solar Open	38.3	23.3	8	4	.388
Aya Expanse 32B	33.3	20.0	8	4	.388
GLM-4.7	65.0	56.7	7	3	.344

Appendix

Table 28: Matched comparison of A2 and A3 in the Closed condition. Accuracy (%) averaged over the 30 seeds with both contrasts, the number of seeds on which each level is more accurate, and the

p

-value of a two-sided sign test.

	Korean	English translation
Anchor (excerpt)	숯(Charcoal)이나 활성탄을 시료 용기의 상부공간에 두어 휘발성 촉진제 성분을 표면에 붙잡아 포집하는 방식은 흡착법이다. 상부공간법은 시료 위의 기체층인 헤드스페이스 자체를 채취하거나 분석하여 휘발성 성분을 확인하는 방법이다.	Placing charcoal or activated carbon in the headspace of the sample container, so that volatile accelerant components are captured on its surface, is the adsorption method. The headspace method identifies volatile components by collecting or analyzing the headspace itself, the gas layer above the sample.
A2	화재 잔류물이 담긴 밀폐병을 실험실로 옮긴 뒤, 분석자는 병마개 안쪽에 작은 검은 탄소성 알갱이가 든 금속망을 매달았다. 병을 따뜻하게 유지하여 휘발 성분이 시료 위 빈 공간으로 퍼지게 한 후, 금속망만 꺼내 분석했더니 휘발성 촉진제 성분이 확인되었다. 이 촉진제 회수 방법으로 가장 적절한 것은?	After a sealed jar containing fire debris was moved to the laboratory, the analyst suspended a metal mesh holding small black carbonaceous granules inside the lid. The jar was kept warm so that volatile components spread into the empty space above the sample, and when only the mesh was taken out and analyzed, volatile accelerant components were identified. Which method of recovering the accelerant is most appropriate?
A3	화재 잔류물이 든 금속 용기를 온수 욕조에 올려 둔 채 밀봉하였다. 뚜껑 안쪽에는 작은 활성탄 조각을 실로 매달아 잔류물 위 빈 공간에 놓이게 했고, 시간이 지나자 뚜껑 안쪽에는 물방울도 조금 맺혔다. 분석자는 꺼낸 활성탄 조각을 소량의 용매가 든 바이알에 넣은 뒤 그 액을 장비에 주입하였다. 이 절차에서 촉진제 성분을 회수한 방법은 무엇인가?	A metal container holding fire debris was sealed while placed in a hot water bath. A small piece of activated carbon was hung by a thread inside the lid so that it sat in the empty space above the debris, and after some time a few water droplets also formed inside the lid. The analyst placed the removed piece of activated carbon in a vial containing a small amount of solvent and injected the liquid into the instrument. By which method was the accelerant recovered in this procedure?

Appendix

Table 29: Example of a knowledge-matched contrast. An A2 and an A3 contrast from the same seed. The options and the gold answer (A) are the same for both: A) adsorption, B) headspace method, C) solvent extraction, D) steam distillation.

	Korean	English translation
Before	Q-system으로 터널 지보량을 검토할 때 ESR을 선정해야 한다. 다음 지하 암반시설들 중 굴착지보비 값을 가장 크게 잡는 것이 타당한 것은? A) 지하 원자력 발전소 B) 채굴 작업 구간 옆에 마련되어 장비 대기와 버력 적치에 쓰이다가 해당 구간 채굴이 끝나면 되메워질 공동 C) 지하 식품저장 시설 D) 수력발전소의 수로터널	The ESR must be selected when reviewing tunnel support with the Q-system. For which of the following underground rock facilities is it appropriate to set the largest excavation support ratio? A) An underground nuclear power plant B) A cavern prepared beside a mining section, used for equipment standby and muck storage, to be backfilled once mining of the section ends C) An underground food storage facility D) The headrace tunnel of a hydropower plant
Audit (L8)	fail : The gold is isolated by form. B is the only long, detailed situational option, while A, C, and D are short generic facility labels.
After	Q-system으로 터널 지보량을 검토하기 위해 굴착지보비(ESR)를 선정하려고 한다. 다음 지하 암반시설들 중 ESR 값을 가장 크게 잡는 것이 타당한 것은? A) 지하 원자력 발전소에서 운전 설비를 계속 수용하도록 계획된 주설비 공동 B) 채굴 작업 구간 옆에서 장비와 버력을 두었다가 작업 종료 뒤 되메우도록 계획된 보조 공동 C) 지하 식품저장 시설에서 식품 재고를 계속 수용하도록 계획된 저장 공동 D) 수력발전소에서 발전용 용수를 계속 통수하도록 계획된 수로터널	The excavation support ratio (ESR) is to be selected in order to review tunnel support with the Q-system. For which of the following underground rock facilities is it appropriate to set the largest ESR? A) A main cavern in an underground nuclear power plant, planned to keep housing operating equipment B) An auxiliary cavern beside a mining section, planned to hold equipment and muck and to be backfilled after the work ends C) A storage cavern in an underground food storage facility, planned to keep housing food stock D) A headrace tunnel in a hydropower plant, planned to keep conveying water for power generation

Appendix

Table 30: Example of self-revision. An A2 candidate that fails shortcut resistance (L8) and its revision. The gold answer is B in both versions.

	Korean	English translation
Before	은빛군은 군 명칭과 군청을 그대로 두고 있지만, 행정구역 개편 후 현재 관내 주소가 ‘청람광역시 은빛군’으로 표시된다. 다만 일부 행정 안내문에는 아직 ‘백산도 은빛군’이라는 이전 표기가 남아 있다. 은빛군 관내의 통행료를 받지 않는 도로에 설치된 안전표지를 새로 교체하려 한다. 이 안전표지의 도로교통법상 원칙적 설치·관리권자는 누구인가? A) 청람광역시장이다. B) 청람광역시 은빛군수이다. C) 백산도지사이다. D) 백산도 은빛군수이다.	Eunbit County keeps its name and county office, but since an administrative reorganization, addresses within it are written “Eunbit County, Cheongram Metropolitan City.” Some administrative notices, however, still carry the former notation “Eunbit County, Baeksan Province.” A safety sign on a toll-free road in Eunbit County is to be replaced. Under the Road Traffic Act, who is in principle responsible for installing and managing this sign? A) The mayor of Cheongram Metropolitan City. B) The head of Eunbit County, Cheongram Metropolitan City. C) The governor of Baeksan Province. D) The head of Eunbit County, Baeksan Province.
Annotator (H3)	지문이 현재 주소는 “청람광역시 은빛군”이고 “백산도 은빛군”은 “아직 남아 있는 이전 표기”라고 밝혀, 법령 지식 없이 읽기만으로 ‘백산도’가 들어간 C와 D가 지워진다.	The stem states that the current address is “Eunbit County, Cheongram Metropolitan City” and that “Eunbit County, Baeksan Province” is a former notation that still remains, so C and D, which contain “Baeksan Province,” are eliminated by reading alone, without knowledge of the law.
After	은빛군은 군청 명칭과 군수 직함을 계속 사용하지만, 관내 법정 주소는 ‘청람광역시 은빛군’으로 표시된다. 은빛군 안의 통행료를 받지 않는 도로에 설치된 노후 안전표지를 교체하려 한다. 이 안전표지의 도로교통법상 원칙적 설치·관리권자는 누구인가? A) 청람광역시장이다. B) 은빛군수이다. C) 청람광역시경찰청장이다. D) 청람광역시 시설관리공단장이다.	Eunbit County continues to use the name of its county office and the title of county head, but the legal address within it is written “Eunbit County, Cheongram Metropolitan City.” An old safety sign on a toll-free road in Eunbit County is to be replaced. Under the Road Traffic Act, who is in principle responsible for installing and managing this sign? A) The mayor of Cheongram Metropolitan City. B) The head of Eunbit County. C) The commissioner of the Cheongram Metropolitan Police Agency. D) The director of the Cheongram Metropolitan City Facilities Management Corporation.

Appendix

Table 31: Example of human-guided revision. An A2 candidate that fails surface leakage (H3) in human validation and its revision. Place names are fictional. The gold answer is A in both versions.

	Korean	English translation
Closed : B (incorrect)	(모델 추론) 여기서는 용기를 가열하여 증기를 발생시켰고, 그 증기가 뚜껑의 활성탄에 흡착되었다. 따라서 상부공간법을 사용한 것으로 볼 수 있다. 이후 용매로 추출하는 것은 추가적인 과정이지만, 회수 방법 자체는 상부공간법일 것이다.	(Model reasoning) Here the container was heated to generate vapor, and the vapor was adsorbed onto the activated carbon in the lid. It can therefore be regarded as using the headspace method. The later extraction with a solvent is an additional step, but the recovery method itself would be the headspace method.
+K : C (incorrect)	(제공된 지식, 발췌) 흡착법은 휘발성 물질이 흡착제의 표면에 달라붙는 성질을 이용하여 성분을 포집·농축하거나 회수하는 방법이다. 상부공간법은 시료 위의 기체층인 헤드스페이스 자체를 채취하거나 분석하여 휘발성 성분을 확인하는 방법이다.	(Provided knowledge, excerpt) The adsorption method captures, concentrates, or recovers components by using the property of volatile substances to adhere to the surface of an adsorbent. The headspace method identifies volatile components by collecting or analyzing the headspace itself, the gas layer above the sample.
+K+A : A (correct)	(추가된 적용 진술, 발췌) 용기의 상부공간이 언급되더라도 분류 기준은 위치 자체가 아니라 회수 원리이며, 헤드스페이스를 직접 채취·분석하면 상부공간법이고 숯이나 활성탄 같은 흡착제 표면에 휘발성 성분을 붙잡아 포집하면 흡착법이다.	(Added application statement, excerpt) Even when the headspace of the container is mentioned, the criterion for classification is the recovery principle rather than the location itself: directly collecting or analyzing the headspace is the headspace method, whereas capturing volatile components on the surface of an adsorbent such as charcoal or activated carbon is the adsorption method.

Appendix

Table 32: Example of a model failure. Responses of A.X K1 to the A3 contrast of Table 29 (gold answer: A, adsorption) under the three conditions.

	Korean	English translation
H1	한국어 가독성: 문장 자체가 자연스럽고 의미가 명확한가? natural : 한 번에 자연스럽게 읽힌다. minor_issue : 조금 어색하지만 의미는 분명하다. unnatural : 어색함이나 모호함이 이해를 방해한다.	Korean readability: Are the sentences themselves natural and is their meaning clear? natural : Reads naturally at once. minor_issue : Slightly awkward, but the meaning is clear. unnatural : Awkwardness or ambiguity hinders understanding.
H2	출제 인위성: 실제 상황처럼 자연스럽게 읽히는가, 아니면 정답과 선택지를 만들기 위해 조건을 인위적으로 나열한 글처럼 읽히는가? natural : 현실에서 있을 법한 자연스러운 상황으로 읽힌다. noticeable : 인위적인 구성 흔적은 있지만 하나의 상황으로 읽힌다. contrived : 실제 상황보다 문제 풀이를 위한 조건 나열처럼 읽힌다.	Construction artifice: Does the text read naturally like a real situation, or like a list of conditions arranged to produce the answer and the options? natural : Reads as a natural situation that could occur in reality. noticeable : Shows traces of construction but still reads as a single situation. contrived : Reads as a list of conditions for solving the question rather than a real situation.
H3	표면적 정답 누출: 전문 내용을 몰라도 길이·문법·형식만으로 정답을 찾을 수 있는가? sound : 표면 단서로 선택지를 줄이기 어렵다. narrowing : 일부는 지워지지만 둘 이상 남는다. isolating : 표면 단서만으로 정답 하나가 남는다.	Surface answer leakage: Can the answer be found from length, grammar, or format alone, without knowing the subject matter? sound : The options are hard to narrow down with surface cues. narrowing : Some options are eliminated, but two or more remain. isolating : Surface cues alone leave only the gold answer.
H4	정답 근거 추적: 정답을 anchor와 사례만으로 뒷받침할 수 있는가? traceable : 필요한 핵심 근거가 anchor에 있다. partial : 관련 근거는 있지만 anchor 밖 지식이 더 필요하다. out_of_scope : 핵심 근거를 anchor에서 찾을 수 없다.	Traceability of the answer: Can the gold answer be supported by the anchor and the case alone? traceable : The key grounds needed are in the anchor. partial : Relevant grounds exist, but knowledge outside the anchor is also needed. out_of_scope : The key grounds cannot be found in the anchor.
H5 (A2)	목표 구조: 정확히 하나의 mapping이 적용되고, 조건 충족을 사례에서 추론해야 하는가? clear : 하나의 mapping이 적용되고 조건 충족을 사례에서 분명히 추론해야 한다. borderline : 추론은 있으나 조건이 거의 드러나 있거나 mapping 수가 다소 불분명하다. not_met : 직접 적용·조회이거나, 여러 mapping이 필요하거나, 사례로 결정할 수 없다.	Target structure: Does exactly one mapping apply, and must the satisfaction of its condition be inferred from the case? clear : One mapping applies, and whether its condition is met must clearly be inferred from the case. borderline : Inference is involved, but the condition is nearly stated or the number of mappings is somewhat unclear. not_met : The question is direct application or retrieval, requires several mappings, or cannot be decided from the case.
H5 (A3)	목표 구조: 둘 이상의 mapping이 실제로 경쟁하고, 선택에 따라 정답이 달라지는가? clear : 둘 이상의 mapping이 사례에 근거하며, 지배 mapping에 따라 정답이 달라진다. borderline : 두 번째 mapping은 있으나 근거가 약하거나 정답이 달라지는지 불분명하다. not_met : 실질적 경쟁이 없거나, mapping이 달라도 정답이 같거나, 결론을 정할 수 없다.	Target structure: Do two or more mappings genuinely compete, and does the answer change with the choice between them? clear : Two or more mappings are grounded in the case, and the answer depends on which mapping governs. borderline : A second mapping exists, but its grounds are weak or it is unclear whether the answer would change. not_met : There is no real competition, the answer is the same under either mapping, or the conclusion cannot be determined.

Appendix

Table 33: Instructions for human validation. Question and rating levels shown to annotators for each criterion, with our English translation. The first level of each criterion is the passing level. H5 is worded differently for A2 and A3 targets.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,469 papers

Oct 21, 2025

KoSimpleQA: A Korean Factuality Benchmark with an Analysis of Reasoning LLMs

We present

\textbf{Korean SimpleQA (KoSimpleQA)}

, a benchmark for evaluating factuality in large language models (LLMs) with a focus on Korean cultural knowledge. KoSimpleQA is designed to be challenging yet easy to grade, consisting of 938 short, fact-seeking questions with unambiguous answers. We conduct a comprehensive evaluation across a diverse set of open-source LLMs of varying sizes that support Korean, and find that even the strongest model generates correct answer only 31.6% of the time, underscoring the challenging nature of KoSimpleQA. Notably, performance rankings on KoSimpleQA differ substantially from those on the English SimpleQA, highlighting the unique value of our dataset. Furthermore, we observe that reasoning helps mitigate the cross-lingual knowledge gap in LLMs, which refers to disparities in their ability to manifest knowledge across languages. KoSimpleQA can be found at https://github.com/naver-ai/KoSimpleQA.

Donghyeon Ko, Kyubyung Chae, Yeguk Jin +6

Naver Cloud · Graduate School of Data Science, Seoul National University

Multilingual Language Model Evaluation LLM Evaluation Factual Knowledge in Language Models Cultural Knowledge in Language Models

Ask PDF

Oct 15, 2026

KBS: A Korean Benchmark Suite for Foundation Model Evaluation

We present a Korean Benchmark Suite (KBS) comprising Ko-ARC, Ko-GSM8K, Ko-EQ-Bench, Ko-WinoGrande, Ko-LAMBADA, and Ko-IFEval, covering six capabilities across 9,396 items. The suite combines review, localization, direct Korean construction, and redesign for Korean, avoiding a translation-only approach to benchmark adaptation. We evaluate Korean-developed models, multilingual open models, and closed API models under protocols that account for model access, prompt format, and scoring differences. Results show that Korean benchmark scores depend on task construction, scoring rules, prompting, and model post-training. These findings suggest that Korean evaluation should report both final scores and the settings used to obtain them.

Yeonkyoung So, Jongmin Kim, Sungmok Jung +13

Graduate School of Data Science, Seoul National University · Dept. of Computer Science and Engineering, Seoul National University

LLM Evaluation Multilingual Language Model Evaluation

Ask PDF

May 26, 2026

Beyond Questions: Evaluating the Stated Knowledge of LLMs

Parametric knowledge in large language models (LLMs) is a cornerstone of their success, yet remains poorly understood. Existing knowledge benchmarks typically rely on predefined questions (e.g., "What is the birth date of M.L. King?"), evaluating only knowledge that benchmark designers explicitly choose to query, a problematic availability bias. In this paper, we introduce open knowledge evaluation, a new paradigm for benchmarking the stated knowledge of LLMs. Instead of asking narrow questions, it evaluates models on the knowledge they choose to surface in response to open-ended elicitation prompts (e.g., "Tell me everything you know about M.L. King"). This shifts the focus from predefined answer retrieval toward characterizing the knowledge models naturally express. We instantiate this paradigm with BeQu (Beyond Questions), a benchmark of 10,000 entities paired with reference corpora for statement verification. Using BeQu, we evaluate a broad range of language models and analyze the effects of reasoning effort, model scale, prompt format, and knowledge domain. Data and leaderboard are available on this work's GitHub repository and at bequ.scads.ai, the benchmark website.

Luca Giordano, Simon Razniewski

ScaDS.AI Dresden/Leipzig & TU Dresden, Germany

LLM Evaluation Factual Knowledge in Language Models

Ask PDF

Abstract

LLM Evaluation

Explore similar work

KoSimpleQA: A Korean Factuality Benchmark with an Analysis of Reasoning LLMs

KBS: A Korean Benchmark Suite for Foundation Model Evaluation

Beyond Questions: Evaluating the Stated Knowledge of LLMs