Same Rule, Different Verdicts: Rubric Language and Clause Structure in LLM Judges for Korean Support Dialogues
Organizations: CEO Business School
Abstract
Services that operate in Korean often write their evaluation rubrics in Korean and hand them to an LLM judge. We test whether a judge's verdict depends on the language of the rubric and on how a rubric item is phrased. We write 12 criteria for support dialogues in four clause structures (affirmative, negated, conditional, exception) such that two pairs of structures have the same meaning, and we give each item in Korean and in English. Six low-cost models from three providers, plus one reference model, judge 60 translated ESConv dialogues and 144 constructed boundary cases under every combination (51,744 requests across the seven models). The analysis plan was fixed and committed before data collection. Cross-provider agreement is lower with Korean rubrics than with English rubrics (Cohen's kappa 0.687 vs. 0.745; difference -0.058, 95% CI [-0.072,-0.045]). Disagreement concentrates in negated, conditional, and exception items (odds ratio 5.6 relative to affirmative items). A single model changes its verdict between two items with the same meaning in 11-15% of cases. Two hypotheses are not supported: models from the same provider do not agree more than models from different providers, and adding paired examples does not reduce the language gap. We measure consistency only; no human labels are used, and all materials were drafted by an LLM.
Figures & tables
| Set | Rubric | POS | NEG | COND | EXC |
|---|---|---|---|---|---|
| A | Korean | .752 | .621 | .675 | .696 |
| A | English | .766 | .676 | .736 | .747 |
| B | Korean | .827 | .505 | .636 | .597 |
| B | English | .892 | .784 | .729 | .729 |
| Structure flip (%) | Language | Self-consist. | Literal reading (%) | |||
|---|---|---|---|---|---|---|
| Model | Korean | English | flip (%) | (Fleiss ) | Korean | English |
| Claude Sonnet 5.5 | 3.8 | 5.6 | 7.9 | .971 | 78 | 65 |
| Claude Haiku 4.5 | 15.6 | 10.1 | 15.2 | .896 | 60 | 58 |
| GPT-5.6 Luna | 14.6 | 7.3 | 12.7 | .855 | 63 | 62 |
| GPT-5.4 nano | 24.3 | 9.7 | 20.9 | .738 | 88 | 92 |
| Gemini 3.5 Flash-Lite | 11.5 | 5.2 | 15.4 | .937 | 43 | 47 |
| POS | NEG | |||
|---|---|---|---|---|
| Criterion | Ko | En | Ko | En |
| C01 Question before advice | 1.00 | 1.00 | .45 | 1.00 |
| C06 Stating client’s choice | .88 | 1.00 | .44 | 1.00 |
| C07 Recommending referral | .83 | .85 | .11 | .92 |
| C08 Closing summary | .62 | .80 | .05 | .58 |
| C11 Reason for optimism | .67 | 1.00 | .41 | .67 |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Item ID | Full item text |
|---|---|
| C01-POS-ko | 상담자는 첫 조언을 하기 전에 내담자의 상황을 묻는 질문을 한 번 이상 한다. |
| C01-POS-en | The counselor asks at least one question about the client’s situation before giving the first piece of advice. |
| C01-NEG-ko | 상담자는 내담자의 상황을 묻는 질문을 한 번도 하지 않은 채 첫 조언을 하지 않는다. |
| C01-NEG-en | The counselor does not give the first piece of advice without having asked at least one question about the client’s situation. |
| C01-COND-ko | 내담자가 조언을 요청한 경우가 아니라면, 상담자는 첫 조언을 하기 전에 내담자의 상황을 묻는 질문을 한 번 이상 한다. |
| C01-COND-en | If the client has not asked for advice, the counselor asks at least one question about the client’s situation before giving the first piece of advice. |
| Model / role | Requested model ID | Call parameters |
|---|---|---|
| claude-sonnet confirmatory | claude-sonnet-5-5 | {"output_config": {"effort": "low"}, "thinking": {"type": "between_tools"}} |
| claude-haiku confirmatory | claude-haiku-4-5-20251001 | {"thinking": {"type": "disabled"}} |
| claude-opus reference | claude-opus-5-5 | {"output_config": {"effort": "low"}, "thinking": {"type": "adaptive"}} |
| gpt-luna confirmatory | gpt-5.6-luna | {"reasoning_effort": "none"} |
| gpt-nano confirmatory | gpt-5.4-nano-2026-03-17 | {"reasoning_effort": "none"} |
| gemini-fl35 confirmatory | gemini-3.5-flash-lite | {"thinking_config": {"thinking_level": "minimal"}} |
| Model | Reason for the reasoning setting |
|---|---|
| claude-sonnet | between_tools disables up-front thinking in these tool-free requests; low effort minimizes the remaining effort setting. See provider documentation . |
| claude-haiku | disabled turns off optional thinking, keeping the condition free of explicit reasoning blocks. |
| claude-opus | Adaptive thinking cannot be disabled for this reference condition; low is the lowest supported effort used. |
| gpt-luna | none was chosen to disable reasoning and was accepted in the pilot. |
| gpt-nano | none was chosen to disable reasoning and was accepted in the pilot. |
| gemini-fl35 | minimal was the lowest supported thinking level selected and accepted in the pilot; it does not guarantee zero thinking tokens. |
| Block | Per model | Models | All models |
|---|---|---|---|
| B-MAIN-KO | 2304 | 7 | 16128 |
| A-MAIN | 480 | 7 | 3360 |
| B-BASELINE | 1152 | 7 | 8064 |
| B-REPEAT | 1152 | 7 | 8064 |
| B-MAIN-EN | 2304 | 7 | 16128 |
| Total | 7392 | 7 | 51744 |
| Internal block name | Meaning |
|---|---|
| B-MAIN-KO | Main boundary-case judgments on Korean cases, with and without examples. |
| A-MAIN | Main judgments on translated Korean dialogues, without examples. |
| B-BASELINE | Verdict-only judgments on Korean boundary cases, without examples. |
| B-REPEAT | Two additional repetitions on the fixed subset of 36 Korean boundary cases. |
| B-MAIN-EN | Auxiliary judgments on English boundary cases, with and without examples. |
| Model | Block | Re- quests | Failed requests | Request rate | Judg- ments | Failed judgments | Judgment rate |
|---|---|---|---|---|---|---|---|
| claude-opus | A-MAIN | 480 | 57 | 0.11875 | 5760 | 684 | 0.11875 |
| claude-opus | B-BASELINE | 1152 | 394 | 0.342014 | 1152 | 394 | 0.342014 |
| claude-opus | B-MAIN-EN | 2304 | 50 | 0.0217014 | 2304 | 50 | 0.0217014 |
| claude-opus | B-MAIN-KO | 2304 | 56 | 0.0243056 | 2304 | 56 | 0.0243056 |
| claude-opus | B-REPEAT | 1152 | 15 | 0.0130208 | 1152 | 15 | 0.0130208 |
| claude-sonnet | A-MAIN | 480 | 1 | 0.00208333 | 5760 | 12 | 0.00208333 |
| Scope | Block | Original stop reason | Count | Output limit |
|---|---|---|---|---|
| all_failed_attempts | A-MAIN | end_turn | 1 | No |
| all_failed_attempts | A-MAIN | max_tokens | 136 | Yes |
| all_failed_attempts | B-BASELINE | end_turn | 3 | No |
| all_failed_attempts | B-BASELINE | max_tokens | 823 | Yes |
| all_failed_attempts | B-MAIN-EN | max_tokens | 132 | Yes |
| all_failed_attempts | B-MAIN-KO | end_turn | 1 | No |
| Item ID | Only model reported mismatching | Current confirmatory flag |
|---|---|---|
| C04-NEG-en | gpt-nano | true |
| C10-NEG-ko | gpt-nano | true |
| C11-NEG-en | gpt-nano | true |
| Scope | Metric | Estimate | CI low | CI high | p | Units |
|---|---|---|---|---|---|---|
| A | H1 | -0.0462846 | -0.0614243 | -0.0313076 | <0.0001 | 60 |
| A | H3 | 0.00268113 | -0.00481183 | 0.0101819 | 0.4762 | 60 |
| A | cross_en | 0.732999 | 0.712587 | 0.754151 | <0.0001 | 60 |
| A | cross_ko | 0.686714 | 0.665477 | 0.707502 | <0.0001 | 60 |
| A | examples_kappa_difference | – | – | – | – | 60 |
| A | within_en | 0.729622 | 0.706266 | 0.75341 | <0.0001 | 60 |
| Model | Block | Re- quests | Failed requests | Request rate | Judg- ments | Failed judgments | Judgment rate |
|---|---|---|---|---|---|---|---|
| claude-haiku | A-MAIN | 480 | 0 | 0 | 5760 | 0 | 0 |
| claude-haiku | B-BASELINE | 1152 | 0 | 0 | 1152 | 0 | 0 |
| claude-haiku | B-MAIN-EN | 2304 | 0 | 0 | 2304 | 0 | 0 |
| claude-haiku | B-MAIN-KO | 2304 | 0 | 0 | 2304 | 0 | 0 |
| claude-haiku | B-REPEAT | 1152 | 0 | 0 | 1152 | 0 | 0 |
| claude-opus | A-MAIN | 480 | 57 | 0.11875 | 5760 | 684 | 0.11875 |
LLM-as-a-Judge
Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.
See all 605 papers