cs.CLNov 6, 2026
SaveKLEAR: Knowledge-Matched LLM Evaluation in Korean across Application Requirements via Controlled Benchmark Transformation
Organizations: Yonsei University · DATUMO INC.
Abstract
Existing Korean benchmarks for Large Language Models (LLMs) evaluate performance across diverse knowledge domains, but their accuracy scores cannot distinguish whether incorrect responses arise from a lack of required knowledge or a failure to apply that knowledge correctly. Our analysis of existing Korean benchmarks reveals that most questions can be answered through factual recall alone, while those demanding greater application also differ in required knowledge, making it difficult to isolate application ability. To address this, we introduce KLEAR, a framework that constructs knowledge-matched contrasts by varying application demand while preserving the required knowledge. During construction, KLEAR identifies the knowledge underlying each source question as an explicit anchor bundle to guide generation of questions requiring more complex application. Generated candidates then undergo automated and human validation, with iterative revision based on evaluation feedback. Experiments demonstrate that the resulting knowledge-matched sets largely satisfy the intended constraints, enabling controlled evaluation across diverse LLMs. These evaluations reveal that increasing application demand exposes failures that persist even when the required knowledge is provided.
Figures & tables
| K | A | |||
| EM | EM | |||
| w/o silver | 68.3 | 0.67 | 45.0 | 0.47 |
| w/ silver | 56.7 | 0.56 | 69.2 | 0.69 |
Table 1: Validation of knowledge (K) and application (A) labels. LLM–human agreement on 60 questions, averaged across two human annotators: exact match (EM; %) and linearly weighted Cohen’s , which penalizes disagreements in proportion to the distance between levels. w/o silver and w/ silver denote the LLM annotation without and with silver reasoning.
| A0 | A1 | A2 | A3 | Total | |
| K0 | 0.0 | 1.0 | 0.3 | 0.0 | 1.3 |
| K1 | 2.9 | 1.2 | 1.1 | 0.1 | 5.2 |
| K2 | 18.5 | 4.0 | 5.5 | 3.6 | 31.7 |
| K3 | 39.6 | 4.2 | 5.1 | 12.7 | 61.7 |
| Total | 61.1 | 10.4 | 12.0 | 16.4 | 100.0 |
Table 2: Distribution of knowledge (K) and application (A) demands. Percentage of the 7,340 questions across K and A levels in three Korean benchmarks.
Figure 1: Overview of KLEAR . (a) Seed questions from Korean benchmarks are labeled by knowledge (K) and application (A) demands. (b) Silver reasoning is generated from the seed question and gold answer, then used to derive an anchor knowledge bundle for generating higher-A questions. (c) Generated questions undergo LLM auditing, proxy screening, and human validation, with failed candidates revised using LLM or human feedback. The result is a matched set of questions that share the same knowledge but differ in application demand.
Figure 2: Failure composition by application demand. Distribution of incorrect responses by failure type across three Korean LLMs, with accuracy (%) on the right.
| KMMLU | ||||
| CLIcK | Pro | Redux | Total | |
| Source questions | 66 | 53 | 46 | 165 |
| A1 A2 | 29 | 20 | 29 | 78 |
| A1 A3 | 24 | 15 | 12 | 51 |
| A2 A3 | 77 | 55 | 26 | 158 |
| Generated questions | 130 | 90 | 67 | 287 |
Table 3: Final dataset statistics. Number of source questions and generated questions by transformation.
| Stage | Count | Rate |
| Automated stage | ||
| Candidates generated | 1,964 | – |
| All eleven criteria pass | 627 | 31.9% |
| Proxy answers incorrectly | 760 | 38.7% |
| Both (pass before revision) | 294 | 15.0% |
| Candidates revised | 1,662 | – |
Table 4: Stage-wise yield. Indented rates are relative to the preceding unindented count; the overall automated-stage pass rate is relative to all generated candidates.
| KMMLU | ||||||||||
| CLIcK | Pro | Redux | All | Matched | ||||||
| Model | A2 | A3 | A2 | A3 | A2 | A3 | A2 | A3 | A2 | A3 |
| K-EXAONE | 65.5 | 46.5 | 50.0 | 37.1 | 34.5 | 57.9 | 50.0 | 45.5 | 46.7 | 40.0 |
| A.X K1 | 72.4 | 40.6 | 45.0 | 27.1 | 48.3 | 47.4 | 56.4 | 37.3 | 51.7 | 21.7 |
| Solar Open | 55.2 | 29.7 | 30.0 | 21.4 | 34.5 | 36.8 | 41.0 | 28.2 | 38.3 | 23.3 |
| Aya Expanse 32B | 31.0 | 26.7 | 25.0 | 21.4 | 20.7 | 18.4 | 25.6 | 23.4 | 33.3 | 20.0 |
Table 5: Closed-book accuracy (%) by application level. Matched covers the 30 seeds that have both an A2 and an A3 contrast (40 and 43 questions), averaged over seeds. The numbers of A2 and A3 questions are 29 and 101 for CLIcK, 20 and 70 for KMMLU-Pro, and 29 and 38 for KMMLU-Redux.
Figure 3: Failure composition on the A1–A2–A3 ladder. Incorrect closed-book responses by failure type across three Korean LLMs. A1 corresponds to the seed. Other includes invalid outputs and defective questions.
Figure 4: Effects of knowledge and application interventions. Accuracy changes across closed-book, knowledge (+K), and knowledge and application (+K+A) conditions for A2 (78 questions) and A3 (209 questions). Each bar shows closed-book accuracy, the additional gains from +K and +K+A, and the remaining error rate.
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
| KMMLU | ||||
| CLIcK | Pro | Redux | Total | |
| Original questions | 1,995 | 2,822 | 2,587 | 7,404 |
| Excluded | 62 | 1 | 1 | 64 |
| Retained questions | 1,933 | 2,821 | 2,586 | 7,340 |
Table 6: Source benchmark statistics. Number of questions in each source benchmark and number retained for the knowledge and application analysis.
| Knowledge Demand (K) | |
| K0 | No external knowledge required |
| K1 | Everyday common sense or general language knowledge |
| K2 | School-level factual or conceptual knowledge |
| K3 | Specialized professional, institutional, or procedural knowledge |
| Application Demand (A) | |
| A0 | Direct recall; the retrieved fact or rule is itself the answer |
Table 7: Knowledge and application taxonomy. Definitions of the four levels in each dimension.
| Question (Korean) | English translation | Why this level | |
| K0 No external knowledge | 제8회 음식 문화 축제. 기간: 2016년 8월 20일(토)–21일(일). 장소: 인주 시청 앞 광장. 행사 내용: 한국 전통요리 무료 시식 및 체험 행사. 체험을 원하시는 분은 현장에서 직접 신청하시기 바랍니다. 다음 글의 내용과 같은 것을 고르십시오. A) 축제는 올해 처음 열린다. B) 축제는 주말 이틀 동안 진행된다. C) 축제에서 체험을 하려면 예약해야 한다. D) 축제에서 전통 요리를 사 먹을 수 있다. | The 8th Food Culture Festival. Dates: Sat., August 20 – Sun., August 21, 2016. Venue: the plaza in front of Inju City Hall. Events: free tasting of traditional Korean dishes and hands-on activities. Those who wish to take part in the activities should apply on site. Choose the statement that agrees with the text. A) The festival is held for the first time this year. B) The festival runs for two days over a weekend. C) A reservation is required for the activities. D) Traditional dishes can be bought and eaten at the festival. | Everything needed is in the text. |
| K1 Everyday knowledge | 응급환자가 발생했을 때에는 어떤 전화번호로 연락해야 하는가? A) 112 B) 119 C) 118 D) 911 | Which telephone number should be called when there is an emergency patient? A) 112 B) 119 C) 118 D) 911 | Everyday common sense. |
| K2 School-level knowledge | 한국인의 주식인 쌀이 많이 생산되는 평야가 아닌 곳은? A) 호남평야 B) 나주평야 C) 김포평야 D) 강원평야 | Which of the following is not a plain where rice, the staple food of Koreans, is produced in large quantities? A) Honam Plain B) Naju Plain C) Gimpo Plain D) Gangwon Plain | School-level knowledge of geography. |
| K3 Specialized knowledge | 인권 침해 사실이 판단될 경우 국가인권위원회는 어떤 조치를 취하는가? A) 당사자에게 문제를 개선하도록 권고 B) 법적 처벌 C) 당사자를 해고 D) 사건을 무시 | What measure does the National Human Rights Commission take when it finds that human rights have been violated? A) Recommends that the party remedy the problem B) Legal punishment C) Dismissal of the party D) Ignoring the case | Knowledge of the powers of a specific institution. |
Table 8: Examples of knowledge (K) levels. One question from CLIcK per level, with our English translation and the reason for its label. The gold answer is in bold .
| Question (Korean) | English translation | Why this level | |
| A0 Knowledge retrieval | 대한민국 헌법재판소에는 몇 명의 헌법재판관이 있는가? A) 7명 B) 8명 C) 9명 D) 10명 | How many justices sit on the Constitutional Court of Korea? A) 7 B) 8 C) 9 D) 10 | The recalled fact is itself the answer; there is no case to map it onto. |
| A1 Explicit mapping | 이혼 당사자 간의 협의가 잘 이루어지지 않을 때, 자녀의 양육권은 어떻게 결정되는가? A) 가정법원에서 결정한다 B) 자녀의 의사에 따라 결정한다 C) 변호사가 결정한다 D) 이혼한 당사자 간의 협의에 의해 결정한다 | When the divorcing parties cannot reach an agreement, how is custody of the children decided? A) The family court decides B) It is decided by the wishes of the children C) A lawyer decides D) It is decided by agreement between the divorced parties | The question states the condition of the rule (the parties fail to agree), so the rule maps onto the case directly. |
| A2 Inferred mapping | 예금액이 많을 경우 어떻게 해야 모두 보호받을 수 있는가? A) 한 은행에 모두 예금한다 B) 여러 금융 기관에 나누어 예금한다 C) 금액을 줄인다 D) 보호받을 수 없다 | If one has a large amount of deposits, what should one do so that all of it is protected? A) Deposit all of it in one bank B) Divide it among several financial institutions C) Reduce the amount D) It cannot be protected | One rule applies (deposits are protected up to a limit per institution), but it is not stated; one must infer that a large deposit exceeds the limit and what would satisfy the rule. |
| A3 Mapping selection | 다음 정치관과 관련이 깊은 정책으로 옳은 것은? 임금의 직책은 한 사람의 재상을 논정하는 데 있다 하였으니, 바로 총재를 두고 한 말이다. 총재는 위로는 임금을 받들고 밑으로는 백관을 통솔하여 만민을 다스리는 것이니 직책이 매우 크다. (후략) A) 육조 직계제의 시행 B) 사간원의 독립 C) 의정부 서사제의 시행 D) 집현전의 설치 | Which policy is closely related to the following view of government? “It is said that the duty of the king lies in choosing one chancellor. The chancellor serves the king above and leads all officials below in governing the people, so his office is of great weight.” (abridged) A) Direct reporting of the Six Ministries to the king B) Independence of the Office of the Censor-General C) Deliberation by the State Council D) Establishment of the Hall of Worthies | Two institutions both govern how the king and his ministers share power, with opposite effects: direct reporting strengthens the king, and deliberation by the State Council strengthens the chancellors. One must select which of the two the stated view corresponds to. |
Table 9: Examples of application (A) levels. One question from CLIcK per level, with our English translation and the reason for its label. The gold answer is in bold .
| K | A | |
| Human–Human | 0.63 (68.3) | 0.60 (56.7) |
| LLM (with silver)–Human | 0.56 (56.7) | 0.69 (69.2) |
| LLM (without silver)–Human | 0.67 (68.3) | 0.47 (45.0) |
Table 10: Reliability of K/A annotations. Linearly weighted Cohen’s on 60 questions, with exact agreement (%) in parentheses. LLM–human values are averaged over the two human annotators.
| Other errors | 650 | |
| T0 | Missing or invalid answer output | 262 |
| T1 | Required material (passage, table, or figure) is absent | 45 |
| T2 | Defective question (wrong gold answer, ambiguous or duplicate options, typo) | 342 |
| Knowledge failures | 3,422 | |
| T3 | Recalls a specific fact or value incorrectly | 2,425 |
| T4 | Binds an entity, statute, or event to the wrong identity or attribute | 575 |
Table 11: Failure taxonomy. Definitions of the 13 failure types and the number of incorrect responses with each primary type ( ). One response fits none of the types and is counted under other errors.
| Failure type (%) | ||||||
| Level | Questions | Accuracy (%) | Incorrect responses | Knowledge | Application | Other |
| A0 | 4,482 | 74.5 | 3,433 | 69.8 | 21.1 | 9.1 |
| A1 | 767 | 88.5 | 265 | 41.5 | 36.6 | 21.9 |
| A2 | 884 | 75.1 | 661 | 40.5 | 41.9 | 17.5 |
| A3 | 1,207 | 60.0 | 1,449 | 44.8 | 44.0 | 11.2 |
| All | 7,340 | 73.6 | 5,808 | 58.9 | 29.9 | 11.2 |
Table 12: Failure composition by application level. Share of incorrect responses (%) by primary failure type. Accuracy is averaged over the three models.
| ID | Criterion | Passing condition |
| LLM rubric: validity as a question | ||
| L1 | Answer correctness | The designated gold answer is correct. |
| L2 | Answer uniqueness | Exactly one option is correct. |
| L3 | Stem determinacy | The stem admits one reading and supplies everything needed to answer. |
| L4 | Bundle conformance | No domain knowledge outside the anchor bundle is required. |
| L5 | Surface form | The text is free of formatting artifacts and orthographic errors. |
Table 13: Automated and human validation rubrics. A candidate must satisfy every criterion of the LLM rubric (L1–L11) and the human rubric (H1–H5). Annotators judge H1–H3 before the anchor bundle is revealed and H4–H5 after.
| Stage | Model | Effort |
| Anchor knowledge extraction | GPT-5.5 | high |
| Contrast generation | GPT-5.5 | high |
| LLM-based auditing | GPT-5.5 | high |
| Proxy difficulty screening | GPT-5-nano | high |
| Self-revision | GPT-5.5 | high |
| Human-guided revision | GPT-5.5 | high |
Table 14: Construction configurations. Model and reasoning effort used at each stage of the pipeline.
| A1 A2 | A1 A3 | A2 A3 | All | |
| Seed–target pairs | 78 | 52 | 165 | 295 |
| Candidates generated | 531 | 345 | 1,088 | 1,964 |
| All eleven criteria pass | 167 (31.5%) | 126 (36.5%) | 334 (30.7%) | 627 (31.9%) |
| Proxy answers incorrectly | 137 (25.8%) | 125 (36.2%) | 498 (45.8%) | 760 (38.7%) |
| Both (pass before revision) | 53 (10.0%) | 52 (15.1%) | 189 (17.4%) | 294 (15.0%) |
| Candidates revised | 477 | 287 | 898 | 1,662 |
Table 15: Yield of the automated stage by transformation type. Percentages for the first three candidate rows and the last are shares of the candidates generated; the percentage for self-revision is a share of the candidates revised.
| First | Second | All | |
| Candidates validated | 231 | 70 | 301 |
| Pass directly | 194 | 58 | 252 |
| Revised | 37 | 12 | 49 |
| Pass after revision | 23 | 12 | 35 |
| Final questions | 217 | 70 | 287 |
Table 16: Human validation results by batch. Candidates in the first batch are judged by two annotators and those in the second batch by one.
| ID | Criterion | Fail | Resolved | New |
| L1 | Answer correctness | 1.2 | 96 | 34 |
| L2 | Answer uniqueness | 1.9 | 100 | 57 |
| L3 | Stem determinacy | 1.1 | 100 | 37 |
| L4 | Bundle conformance | 2.2 | 98 | 44 |
| L5 | Surface form | 0.5 | 89 | 12 |
| L6 | Target-level structure | 18.5 | 67 | 128 |
Table 17: Failures by criterion. Fail is the share (%) of candidates that do not pass the criterion: before revision for the LLM rubric (1,964 candidates) and at first validation for the human rubric (301 candidates). Resolved is the share (%) of these failures that revision fixes, and New is the number of failures that revision introduces, with self-revision for the LLM rubric and human-guided revision for the human rubric.
| ID | Criterion | With | Without |
| L1 | Answer correctness | 1.9 | 6.5 |
| L2 | Answer uniqueness | 3.7 | 6.5 |
| L3 | Stem determinacy | 1.9 | 3.7 |
| L4 | Bundle conformance | 1.9 | 20.4 |
| L5 | Surface form | 0.9 | 0.0 |
| L6 | Target-level structure | 20.4 | 23.1 |
Table 18: Effect of the anchor bundle. Share (%) of candidates that fail each criterion before revision, when generated with and without the anchor bundle for the same 36 pairs (108 candidates each). Only L4 differs significantly (Fisher’s exact test, ).
| Incorrect models | 0 | 1 | 2 | 3 |
| Seeds | 12 | 14 | 48 | 24 |
| Seed–target pairs | 23 | 28 | 81 | 38 |
| Candidates generated | 117 | 246 | 549 | 246 |
| All eleven criteria pass | 58.1 | 43.1 | 31.3 | 28.0 |
| Proxy answers incorrectly | 12.8 | 30.1 | 28.2 | 39.8 |
| Both (pass before revision) | 7.7 | 11.8 | 11.8 | 14.2 |
Table 19: Yield by seed difficulty. A1 seeds grouped by the number of Korean LLMs (out of three) that answer the seed incorrectly. Rates are shares (%) of the candidates generated.
| A2 | A3 | All | |
| Questions | 78 | 209 | 287 |
| Source benchmark of the seed | |||
| CLIcK | 29 | 101 | 130 |
| KMMLU-Pro | 20 | 70 | 90 |
| KMMLU-Redux | 29 | 38 | 67 |
| Knowledge level | |||
Table 20: Composition of KLEAR by application level. Number of questions, except for question length, which is the mean number of characters.
| Model | Identifier | Access | Reasoning | Sampling | Max. |
| K-EXAONE | LGAI-EXAONE/K-EXAONE-236B-A23B | vLLM | Enabled | 1.0 / 0.95 | 28,672 |
| A.X K1 | skt/A.X-K1 | vLLM | Enabled | 0.6 / 0.95 | 45,056 |
| Solar Open | upstage/Solar-Open-100B | vLLM | High | 0.8 / 0.95 † | 28,672 |
| Aya Expanse 32B | c4ai-aya-expanse-32b | Cohere | None | 0.3 | 4,000 |
| GLM-4.7 | z-ai/glm-4.7 | OpenRouter | High | default | 32,000 |
| Qwen3.5 | qwen/qwen3.5-397b-a17b | OpenRouter | High | default | 32,000 |
Table 21: Evaluated models and inference settings. Max. is the limit on output tokens, including reasoning. Sampling is given as temperature / top- ; default denotes the default settings of the API. † Top- is set to 50.
| Stage | Model | Calls | Input | Output |
| Construction | ||||
| Generation | GPT-5.5 | 1,965 | 9.3 | 18.9 |
| Auditing | GPT-5.5 | 3,626 | 17.1 | 27.0 |
| Self-revision | GPT-5.5 | 1,670 | 8.0 | 19.1 |
| Proxy | GPT-5-nano | 3,626 | 1.3 | 22.5 |
| Evaluation | ||||
Table 22: Token usage of API calls. Input and output tokens in millions. Evaluation covers the 287 questions under three conditions.
| CLIcK | KMMLU-Pro | KMMLU-Redux | All | |||||||||
| Model | C | +K | +K+A | C | +K | +K+A | C | +K | +K+A | C | +K | +K+A |
| K-EXAONE | 50.8 | 76.9 | 86.9 | 40.0 | 78.9 | 90.0 | 47.8 | 88.1 | 95.5 | 46.7 | 80.1 | 89.9 |
| A.X K1 | 47.7 | 66.9 | 77.7 | 31.1 | 71.1 | 83.3 | 47.8 | 74.6 | 92.5 | 42.5 | 70.0 | 82.9 |
| Solar Open | 35.4 | 65.4 | 74.6 | 23.3 | 67.8 | 83.3 | 35.8 | 74.6 | 88.1 | 31.7 | 68.3 | 80.5 |
| Aya Expanse 32B | 27.7 | 41.5 | 53.8 | 22.2 | 40.0 | 53.3 | 19.4 | 35.8 | 50.7 | 24.0 | 39.7 | 53.0 |
| GLM-4.7 | 69.2 | 83.8 | 86.2 | 50.0 | 78.9 | 92.2 | 61.2 | 91.0 | 94.0 | 61.3 | 84.0 | 89.9 |
Table 23: Accuracy (%) on KLEAR by source benchmark of the seed. C is the Closed condition. The numbers of questions are 130 for CLIcK, 90 for KMMLU-Pro, and 67 for KMMLU-Redux.
| A2 | A3 | |||||
| Family | Closed | +K | +K+A | Closed | +K | +K+A |
| Korean | 49.1 | 86.8 | 92.7 | 37.0 | 67.6 | 81.3 |
| Multilingual | 25.6 | 47.4 | 59.0 | 23.4 | 36.8 | 50.7 |
| Open-weight | 70.5 | 92.9 | 96.8 | 64.6 | 81.1 | 87.6 |
| Proprietary | 91.7 | 99.4 | 99.4 | 87.1 | 91.9 | 95.7 |
Table 24: Accuracy (%) by model family. Averages over the models in each group: Korean (K-EXAONE, A.X K1, Solar Open), multilingual (Aya Expanse 32B), open-weight (GLM-4.7, Qwen3.5), and proprietary (Claude Sonnet 5.5, GPT-6.1 Sol).
| A2 ( ) | A3 ( ) | |||||||
| Model | Errors | Fixed | Broken | Errors | Fixed | Broken | ||
| K-EXAONE | 6 | 6 | 2 | .289 | 51 | 33 | 9 | .001 |
| A.X K1 | 12 | 8 | 3 | .227 | 74 | 47 | 15 | .001 |
| Solar Open | 13 | 9 | 4 | .267 | 78 | 43 | 13 | .001 |
| Aya Expanse 32B | 41 | 16 | 7 | .093 | 132 | 57 | 28 | .002 |
| GLM-4.7 | 7 | 6 | 2 | .289 | 39 | 18 | 5 | .011 |
Table 25: Effect of adding application statements. Errors is the number of incorrect responses under +K ; Fixed and Broken are the numbers of responses that become correct and incorrect under +K+A ; is from the exact McNemar test.
| A1 | A2 | |||||
| Model | Closed | +K | +K+A | Closed | +K | +K+A |
| K-EXAONE | 39.7 | 84.5 | 87.9 | 32.7 | 73.8 | 83.2 |
| A.X K1 | 43.1 | 84.5 | 87.9 | 30.8 | 68.2 | 76.6 |
| Solar Open | 27.6 | 81.0 | 93.1 | 28.0 | 60.7 | 75.7 |
Table 26: Accuracy (%) on the seeds. The 165 seeds comprise 58 A1 and 107 A2 questions.
| Model | Level | Knowledge | Application | Other | |
| K-EXAONE | A2 | 39 | 17.9 | 66.7 | 15.4 |
| A3 | 114 | 15.8 | 74.6 | 9.6 | |
| A.X K1 | A2 | 34 | 23.5 | 70.6 | 5.9 |
| A3 | 131 | 20.6 | 71.8 | 7.6 | |
| Solar Open | A2 | 46 | 32.6 | 65.2 | 2.2 |
| A3 | 150 | 21.3 | 74.0 | 4.7 |
Table 27: Failure types on KLEAR . Share (%) of each primary failure type among the incorrect Closed responses; is the number of incorrect responses.
| Accuracy | Seeds favoring | ||||
| Model | A2 | A3 | A2 | A3 | |
| K-EXAONE | 46.7 | 40.0 | 9 | 6 | .607 |
| A.X K1 | 51.7 | 21.7 | 15 | 4 | .019 |
| Solar Open | 38.3 | 23.3 | 8 | 4 | .388 |
| Aya Expanse 32B | 33.3 | 20.0 | 8 | 4 | .388 |
| GLM-4.7 | 65.0 | 56.7 | 7 | 3 | .344 |
Table 28: Matched comparison of A2 and A3 in the Closed condition. Accuracy (%) averaged over the 30 seeds with both contrasts, the number of seeds on which each level is more accurate, and the -value of a two-sided sign test.
| Korean | English translation | |
| Anchor (excerpt) | 숯(Charcoal)이나 활성탄을 시료 용기의 상부공간에 두어 휘발성 촉진제 성분을 표면에 붙잡아 포집하는 방식은 흡착법이다. 상부공간법은 시료 위의 기체층인 헤드스페이스 자체를 채취하거나 분석하여 휘발성 성분을 확인하는 방법이다. | Placing charcoal or activated carbon in the headspace of the sample container, so that volatile accelerant components are captured on its surface, is the adsorption method. The headspace method identifies volatile components by collecting or analyzing the headspace itself, the gas layer above the sample. |
| A2 | 화재 잔류물이 담긴 밀폐병을 실험실로 옮긴 뒤, 분석자는 병마개 안쪽에 작은 검은 탄소성 알갱이가 든 금속망을 매달았다. 병을 따뜻하게 유지하여 휘발 성분이 시료 위 빈 공간으로 퍼지게 한 후, 금속망만 꺼내 분석했더니 휘발성 촉진제 성분이 확인되었다. 이 촉진제 회수 방법으로 가장 적절한 것은? | After a sealed jar containing fire debris was moved to the laboratory, the analyst suspended a metal mesh holding small black carbonaceous granules inside the lid. The jar was kept warm so that volatile components spread into the empty space above the sample, and when only the mesh was taken out and analyzed, volatile accelerant components were identified. Which method of recovering the accelerant is most appropriate? |
| A3 | 화재 잔류물이 든 금속 용기를 온수 욕조에 올려 둔 채 밀봉하였다. 뚜껑 안쪽에는 작은 활성탄 조각을 실로 매달아 잔류물 위 빈 공간에 놓이게 했고, 시간이 지나자 뚜껑 안쪽에는 물방울도 조금 맺혔다. 분석자는 꺼낸 활성탄 조각을 소량의 용매가 든 바이알에 넣은 뒤 그 액을 장비에 주입하였다. 이 절차에서 촉진제 성분을 회수한 방법은 무엇인가? | A metal container holding fire debris was sealed while placed in a hot water bath. A small piece of activated carbon was hung by a thread inside the lid so that it sat in the empty space above the debris, and after some time a few water droplets also formed inside the lid. The analyst placed the removed piece of activated carbon in a vial containing a small amount of solvent and injected the liquid into the instrument. By which method was the accelerant recovered in this procedure? |
Table 29: Example of a knowledge-matched contrast. An A2 and an A3 contrast from the same seed. The options and the gold answer (A) are the same for both: A) adsorption, B) headspace method, C) solvent extraction, D) steam distillation.
| Korean | English translation | |
| Before | Q-system으로 터널 지보량을 검토할 때 ESR을 선정해야 한다. 다음 지하 암반시설들 중 굴착지보비 값을 가장 크게 잡는 것이 타당한 것은? A) 지하 원자력 발전소 B) 채굴 작업 구간 옆에 마련되어 장비 대기와 버력 적치에 쓰이다가 해당 구간 채굴이 끝나면 되메워질 공동 C) 지하 식품저장 시설 D) 수력발전소의 수로터널 | The ESR must be selected when reviewing tunnel support with the Q-system. For which of the following underground rock facilities is it appropriate to set the largest excavation support ratio? A) An underground nuclear power plant B) A cavern prepared beside a mining section, used for equipment standby and muck storage, to be backfilled once mining of the section ends C) An underground food storage facility D) The headrace tunnel of a hydropower plant |
| Audit (L8) | fail : The gold is isolated by form. B is the only long, detailed situational option, while A, C, and D are short generic facility labels. | |
| After | Q-system으로 터널 지보량을 검토하기 위해 굴착지보비(ESR)를 선정하려고 한다. 다음 지하 암반시설들 중 ESR 값을 가장 크게 잡는 것이 타당한 것은? A) 지하 원자력 발전소에서 운전 설비를 계속 수용하도록 계획된 주설비 공동 B) 채굴 작업 구간 옆에서 장비와 버력을 두었다가 작업 종료 뒤 되메우도록 계획된 보조 공동 C) 지하 식품저장 시설에서 식품 재고를 계속 수용하도록 계획된 저장 공동 D) 수력발전소에서 발전용 용수를 계속 통수하도록 계획된 수로터널 | The excavation support ratio (ESR) is to be selected in order to review tunnel support with the Q-system. For which of the following underground rock facilities is it appropriate to set the largest ESR? A) A main cavern in an underground nuclear power plant, planned to keep housing operating equipment B) An auxiliary cavern beside a mining section, planned to hold equipment and muck and to be backfilled after the work ends C) A storage cavern in an underground food storage facility, planned to keep housing food stock D) A headrace tunnel in a hydropower plant, planned to keep conveying water for power generation |
Table 30: Example of self-revision. An A2 candidate that fails shortcut resistance (L8) and its revision. The gold answer is B in both versions.
| Korean | English translation | |
| Before | 은빛군은 군 명칭과 군청을 그대로 두고 있지만, 행정구역 개편 후 현재 관내 주소가 ‘청람광역시 은빛군’으로 표시된다. 다만 일부 행정 안내문에는 아직 ‘백산도 은빛군’이라는 이전 표기가 남아 있다. 은빛군 관내의 통행료를 받지 않는 도로에 설치된 안전표지를 새로 교체하려 한다. 이 안전표지의 도로교통법상 원칙적 설치·관리권자는 누구인가? A) 청람광역시장이다. B) 청람광역시 은빛군수이다. C) 백산도지사이다. D) 백산도 은빛군수이다. | Eunbit County keeps its name and county office, but since an administrative reorganization, addresses within it are written “Eunbit County, Cheongram Metropolitan City.” Some administrative notices, however, still carry the former notation “Eunbit County, Baeksan Province.” A safety sign on a toll-free road in Eunbit County is to be replaced. Under the Road Traffic Act, who is in principle responsible for installing and managing this sign? A) The mayor of Cheongram Metropolitan City. B) The head of Eunbit County, Cheongram Metropolitan City. C) The governor of Baeksan Province. D) The head of Eunbit County, Baeksan Province. |
| Annotator (H3) | 지문이 현재 주소는 “청람광역시 은빛군”이고 “백산도 은빛군”은 “아직 남아 있는 이전 표기”라고 밝혀, 법령 지식 없이 읽기만으로 ‘백산도’가 들어간 C와 D가 지워진다. | The stem states that the current address is “Eunbit County, Cheongram Metropolitan City” and that “Eunbit County, Baeksan Province” is a former notation that still remains, so C and D, which contain “Baeksan Province,” are eliminated by reading alone, without knowledge of the law. |
| After | 은빛군은 군청 명칭과 군수 직함을 계속 사용하지만, 관내 법정 주소는 ‘청람광역시 은빛군’으로 표시된다. 은빛군 안의 통행료를 받지 않는 도로에 설치된 노후 안전표지를 교체하려 한다. 이 안전표지의 도로교통법상 원칙적 설치·관리권자는 누구인가? A) 청람광역시장이다. B) 은빛군수이다. C) 청람광역시경찰청장이다. D) 청람광역시 시설관리공단장이다. | Eunbit County continues to use the name of its county office and the title of county head, but the legal address within it is written “Eunbit County, Cheongram Metropolitan City.” An old safety sign on a toll-free road in Eunbit County is to be replaced. Under the Road Traffic Act, who is in principle responsible for installing and managing this sign? A) The mayor of Cheongram Metropolitan City. B) The head of Eunbit County. C) The commissioner of the Cheongram Metropolitan Police Agency. D) The director of the Cheongram Metropolitan City Facilities Management Corporation. |
Table 31: Example of human-guided revision. An A2 candidate that fails surface leakage (H3) in human validation and its revision. Place names are fictional. The gold answer is A in both versions.
| Korean | English translation | |
| Closed : B (incorrect) | (모델 추론) 여기서는 용기를 가열하여 증기를 발생시켰고, 그 증기가 뚜껑의 활성탄에 흡착되었다. 따라서 상부공간법을 사용한 것으로 볼 수 있다. 이후 용매로 추출하는 것은 추가적인 과정이지만, 회수 방법 자체는 상부공간법일 것이다. | (Model reasoning) Here the container was heated to generate vapor, and the vapor was adsorbed onto the activated carbon in the lid. It can therefore be regarded as using the headspace method. The later extraction with a solvent is an additional step, but the recovery method itself would be the headspace method. |
| +K : C (incorrect) | (제공된 지식, 발췌) 흡착법은 휘발성 물질이 흡착제의 표면에 달라붙는 성질을 이용하여 성분을 포집·농축하거나 회수하는 방법이다. 상부공간법은 시료 위의 기체층인 헤드스페이스 자체를 채취하거나 분석하여 휘발성 성분을 확인하는 방법이다. | (Provided knowledge, excerpt) The adsorption method captures, concentrates, or recovers components by using the property of volatile substances to adhere to the surface of an adsorbent. The headspace method identifies volatile components by collecting or analyzing the headspace itself, the gas layer above the sample. |
| +K+A : A (correct) | (추가된 적용 진술, 발췌) 용기의 상부공간이 언급되더라도 분류 기준은 위치 자체가 아니라 회수 원리이며, 헤드스페이스를 직접 채취·분석하면 상부공간법이고 숯이나 활성탄 같은 흡착제 표면에 휘발성 성분을 붙잡아 포집하면 흡착법이다. | (Added application statement, excerpt) Even when the headspace of the container is mentioned, the criterion for classification is the recovery principle rather than the location itself: directly collecting or analyzing the headspace is the headspace method, whereas capturing volatile components on the surface of an adsorbent such as charcoal or activated carbon is the adsorption method. |
Table 32: Example of a model failure. Responses of A.X K1 to the A3 contrast of Table 29 (gold answer: A, adsorption) under the three conditions.
| Korean | English translation | |
| H1 | 한국어 가독성: 문장 자체가 자연스럽고 의미가 명확한가? natural : 한 번에 자연스럽게 읽힌다. minor_issue : 조금 어색하지만 의미는 분명하다. unnatural : 어색함이나 모호함이 이해를 방해한다. | Korean readability: Are the sentences themselves natural and is their meaning clear? natural : Reads naturally at once. minor_issue : Slightly awkward, but the meaning is clear. unnatural : Awkwardness or ambiguity hinders understanding. |
| H2 | 출제 인위성: 실제 상황처럼 자연스럽게 읽히는가, 아니면 정답과 선택지를 만들기 위해 조건을 인위적으로 나열한 글처럼 읽히는가? natural : 현실에서 있을 법한 자연스러운 상황으로 읽힌다. noticeable : 인위적인 구성 흔적은 있지만 하나의 상황으로 읽힌다. contrived : 실제 상황보다 문제 풀이를 위한 조건 나열처럼 읽힌다. | Construction artifice: Does the text read naturally like a real situation, or like a list of conditions arranged to produce the answer and the options? natural : Reads as a natural situation that could occur in reality. noticeable : Shows traces of construction but still reads as a single situation. contrived : Reads as a list of conditions for solving the question rather than a real situation. |
| H3 | 표면적 정답 누출: 전문 내용을 몰라도 길이·문법·형식만으로 정답을 찾을 수 있는가? sound : 표면 단서로 선택지를 줄이기 어렵다. narrowing : 일부는 지워지지만 둘 이상 남는다. isolating : 표면 단서만으로 정답 하나가 남는다. | Surface answer leakage: Can the answer be found from length, grammar, or format alone, without knowing the subject matter? sound : The options are hard to narrow down with surface cues. narrowing : Some options are eliminated, but two or more remain. isolating : Surface cues alone leave only the gold answer. |
| H4 | 정답 근거 추적: 정답을 anchor와 사례만으로 뒷받침할 수 있는가? traceable : 필요한 핵심 근거가 anchor에 있다. partial : 관련 근거는 있지만 anchor 밖 지식이 더 필요하다. out_of_scope : 핵심 근거를 anchor에서 찾을 수 없다. | Traceability of the answer: Can the gold answer be supported by the anchor and the case alone? traceable : The key grounds needed are in the anchor. partial : Relevant grounds exist, but knowledge outside the anchor is also needed. out_of_scope : The key grounds cannot be found in the anchor. |
| H5 (A2) | 목표 구조: 정확히 하나의 mapping이 적용되고, 조건 충족을 사례에서 추론해야 하는가? clear : 하나의 mapping이 적용되고 조건 충족을 사례에서 분명히 추론해야 한다. borderline : 추론은 있으나 조건이 거의 드러나 있거나 mapping 수가 다소 불분명하다. not_met : 직접 적용·조회이거나, 여러 mapping이 필요하거나, 사례로 결정할 수 없다. | Target structure: Does exactly one mapping apply, and must the satisfaction of its condition be inferred from the case? clear : One mapping applies, and whether its condition is met must clearly be inferred from the case. borderline : Inference is involved, but the condition is nearly stated or the number of mappings is somewhat unclear. not_met : The question is direct application or retrieval, requires several mappings, or cannot be decided from the case. |
| H5 (A3) | 목표 구조: 둘 이상의 mapping이 실제로 경쟁하고, 선택에 따라 정답이 달라지는가? clear : 둘 이상의 mapping이 사례에 근거하며, 지배 mapping에 따라 정답이 달라진다. borderline : 두 번째 mapping은 있으나 근거가 약하거나 정답이 달라지는지 불분명하다. not_met : 실질적 경쟁이 없거나, mapping이 달라도 정답이 같거나, 결론을 정할 수 없다. | Target structure: Do two or more mappings genuinely compete, and does the answer change with the choice between them? clear : Two or more mappings are grounded in the case, and the answer depends on which mapping governs. borderline : A second mapping exists, but its grounds are weak or it is unclear whether the answer would change. not_met : There is no real competition, the answer is the same under either mapping, or the conclusion cannot be determined. |
Table 33: Instructions for human validation. Question and rating levels shown to annotators for each criterion, with our English translation. The first level of each criterion is the passing level. H5 is worded differently for A2 and A3 targets.
More in this topic
LLM Evaluation
Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.
See all 2,469 papers