Khmer PFGen-Bench: Building a Judge-Free Benchmark for Open-Ended Generation in a Low-Resource Language
Organizations: Preferred Networks, Inc.
Abstract
Benchmarks for open-ended generation often rely on large language model (LLM) judges, whose reliability in low-resource languages is uncertain. We introduce Khmer PFGen-Bench, to our knowledge the first judge-free benchmark for multi-sentence answers to questions composed directly in Khmer, a low-resource language. Khmer PFGen-Bench adapts the scorer of PFGen-Bench, a benchmark for Japanese that scores responses with keyword rules and n-gram statistics from LLM-generated references. This scoring requires references that are both valid and varied, which LLMs may not produce in Khmer, so we examine how to build such references. We find that post-trained models can give more valid but less varied references than their base variants, and that mixing model families increases variety for the same number of references. Using references from post-trained models in nine families, the benchmark yields model rankings that are stable across reference sets and consistent with a native speaker's ratings. These findings offer guidance for building judge-free benchmarks in other low-resource languages.
Figures & tables
| ID | Category | Question |
|---|---|---|
| Q01 | Khmer language | \khmerfont \XeTeXlinebreaklocale "km" ចូររៀបរាប់អំពីពាក្យក្លាយក្នុងភាសាខ្មែរ។ Describe derived words in Khmer. |
| Q30 | Geography | \khmerfont \XeTeXlinebreaklocale "km" ហេតុអ្វីបានជាបឹងទន្លេសាបមានទំហំខុសគ្នារវាងរដូវប្រាំងនិងរដូវវស្សា? Why does the size of Tonle Sap Lake differ between the dry and rainy seasons? |
| Q11 | Physics | \khmerfont \XeTeXlinebreaklocale "km" តើកម្តៅនិងសីតុណ្ហភាពខុសគ្នាដូចម្តេច? How do heat and temperature differ? |
| Q46 | Health | \khmerfont \XeTeXlinebreaklocale "km" ហេតុអ្វីបានជាភាគល្អិតនិចល PM2.5 មានគ្រោះថ្នាក់ជាង PM10? Why is PM2.5 particulate matter more harmful than PM10? |
| Generator | Variant | Acceptance | Validity | Yield |
|---|---|---|---|---|
| Japanese | ||||
| Gemma-4-31B | Base | 80% | 65% | 52% |
| Post | 90% | 75% | 68% | |
| Qwen3-30B-A3B | Base | 65% | 45% | 29% |
| Post | 75% | 50% | 38% | |
| Khmer | ||||
| Coverage | Distinct-3 | ||||
| Generator | Base | Post | Base | Post | |
| Japanese | |||||
| Gemma-4-31B | 39 | 76% | 48% | 31% | 14% |
| Qwen3-30B-A3B | 36 | 71% | 60% | 37% | 24% |
| Khmer | |||||
| Gemma-4-31B | 20 | 58% | 48% | 54% | 20% |
| Per family | Coverage | Rank corr. | Questions | ||
|---|---|---|---|---|---|
| 40 | 1 | 40 | 64–76% | 0.96–0.99 | 36–50 |
| 2 | 20 | 80–84% | 0.98–1.00 | 32–47 | |
| 4 | 10 | 87–89% | 0.95–1.00 | 30–40 | |
| 80 | 4 | 20 | 89–91% | 0.97–1.00 | 30–39 |
| 160 | 4 | 40 | 93–94% | 0.97–1.00 | 31–38 |
| All | 9 | 94% | 1.00 | 50 |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Reference generation: conversation turns | Model evaluation: one user message |
| User: <example question> \khmerfont \XeTeXlinebreaklocale "km" ឆ្លើយខ្លីៗ ២-៣ ប្រយោគ។ [Answer briefly in 2–3 sentences.] Assistant: <example gold answer> [Repeat for four examples.] User: <target question> \khmerfont \XeTeXlinebreaklocale "km" ឆ្លើយខ្លីៗ ២-៣ ប្រយោគ។ | \khmerfont \XeTeXlinebreaklocale "km" ចូរឆ្លើយសំណួរក្នុងមួយបន្ទាត់ ដោយប្រើរបៀបសរសេរ និងប្រវែងប្រហាក់ប្រហែលឧទាហរណ៍។ [Answer the question in one line, using a style and length similar to the examples.] \khmerfont \XeTeXlinebreaklocale "km" ## ឧទាហរណ៍ [Examples] Q: <example question> A: <example gold answer> [Repeat for eight examples.] \khmerfont \XeTeXlinebreaklocale "km" ## សំណួរ [Question] Q: <target question> |
| Stage | Remaining |
|---|---|
| Candidates | 29,321 |
| Format | 27,986 |
| Duplication | 26,212 |
| Length | 25,514 |
| Keywords | 18,274 |
| Generator | Candidates | Retained | Acceptance |
|---|---|---|---|
| DeepSeek-V4-Pro | 3,060 | 2,371 | 77% |
| Llama-4-Maverick | 3,650 | 1,691 | 46% |
| Gemma-4-31B | 2,910 | 1,693 | 58% |
| Qwen3.5-397B | 3,060 | 2,074 | 68% |
| GLM-5.3-flash | 3,341 | 2,150 | 64% |
| GPT-5.6-sol | 3,580 | 2,014 | 56% |
| Model | Role | Family | Failures |
| DeepSeek-V4-Pro | generator | DeepSeek | 0 |
| Llama-4-Maverick-17B-128E-Instruct-FP8 | generator | Llama | 0 |
| Gemma-4-31B-it | generator | Gemma | 0 |
| Qwen3.5-397B-A17B | generator | Qwen | 0 |
| GLM-5.3-flash | generator | GLM | 13 |
| GPT-5.6-sol | generator | GPT | 0 |
| Category | ID | Question | English translation |
|---|---|---|---|
| Khmer language | Q01 | \khmerfont \XeTeXlinebreaklocale "km" ចូររៀបរាប់អំពីពាក្យក្លាយក្នុងភាសាខ្មែរ។ | Describe derived words in Khmer. |
| Q02 | \khmerfont \XeTeXlinebreaklocale "km" ចូររៀបរាប់កាព្យមាត្រនិងចំណាប់ចួននៃកំណាព្យបទភុជង្គលីលា។ | Describe the meter and rhyme scheme of Phujong Lilea verse. | |
| Q03 | \khmerfont \XeTeXlinebreaklocale "km" តើរឿងព្រះអាទិត្យថ្មីរះលើផែនដីចាស់និយាយអំពីអ្វី? | What is the novel A New Sun Rises over the Old Land about? | |
| Q04 | \khmerfont \XeTeXlinebreaklocale "km" ចូរបង្ហាញភាពខុសគ្នារវាងសុភាសិតនិងពាក្យស្លោក។ | Explain the difference between proverbs and slogans. | |
| Q05 | \khmerfont \XeTeXlinebreaklocale "km" ចូររៀបរាប់អំពីលក្ខណៈសម្គាល់របស់អក្សរសិល្ប៍ខ្មែរបែបពុទ្ធនិយម។ | Describe the distinctive features of Buddhist Khmer literature. | |
| Mathematics | Q06 | \khmerfont \XeTeXlinebreaklocale "km" តើគេអាចសម្គាល់ថាត្រីកោណសាមញ្ញពីរប៉ុនគ្នា តាមលក្ខខណ្ឌអ្វីខ្លះ? | By which criteria can two general triangles be identified as congruent? |
| Category | ID | Question | English translation |
|---|---|---|---|
| History | Q26 | \khmerfont \XeTeXlinebreaklocale "km" ដូចម្តេចដែលហៅថាសមាជជាតិ? | What is the National Congress? |
| Q27 | \khmerfont \XeTeXlinebreaklocale "km" ក្នុងសម័យកម្ពុជាប្រជាធិបតេយ្យ តើពាក្យ « \khmerfont \XeTeXlinebreaklocale "km" អង្គការ» \khmerfont \XeTeXlinebreaklocale "km" មានន័យដូចម្តេច? | In the Democratic Kampuchea period, what did the word Angkar (organization) mean? | |
| Q28 | \khmerfont \XeTeXlinebreaklocale "km" តើកិច្ចព្រមព្រៀងទីក្រុងប៉ារីសឆ្នាំ១៩៩១មានគោលបំណងសំខាន់ៗអ្វីខ្លះ? | What were the main objectives of the 1991 Paris Peace Agreements? | |
| Geography | Q29 | \khmerfont \XeTeXlinebreaklocale "km" តើសណ្ឋានដីប្រទេសកម្ពុជាចែកចេញជាតំបន់អ្វីខ្លះ? | Into which regions is the topography of Cambodia divided? |
| Q30 | \khmerfont \XeTeXlinebreaklocale "km" ហេតុអ្វីបានជាបឹងទន្លេសាបមានទំហំខុសគ្នារវាងរដូវប្រាំងនិងរដូវវស្សា? | Why does the size of Tonle Sap Lake differ between the dry and rainy seasons? | |
| Q31 | \khmerfont \XeTeXlinebreaklocale "km" តើទន្លេមេគង្គហូរឆ្លងកាត់រាជធានីខេត្តណាខ្លះនៃប្រទេសកម្ពុជា? | Through which of Cambodia’s capital and provinces does the Mekong River flow? |
| Rank | Model | Score | Self-incl. | F | T | H | Generator |
|---|---|---|---|---|---|---|---|
| 1 | Gemini-4-argon | 0.875 | 0.875 | 0.95 | 0.92 | 0.75 | — |
| 2 | Claude-opus-5.5 | 0.853 | 0.853 | 0.97 | 0.92 | 0.67 | — |
| 3 | Gemini-3.8-flash | 0.846 | 0.887 | 0.92 | 0.92 | 0.70 | ✓ |
| 4 | GPT-6-sol | 0.837 | 0.837 | 0.94 | 0.96 | 0.61 | — |
| 5 | GPT-5.6-sol | 0.832 | 0.857 | 0.92 | 0.96 | 0.62 | ✓ |
| 6 | Claude-opus-5 | 0.815 | 0.859 | 0.92 | 0.89 | 0.63 | ✓ |
| Item | Axis | Scale |
|---|---|---|
| Is the Khmer well formed? | F | 0: broken 1: readable but unnatural or with errors 2: natural |
| Is the content correct? | T | 0: errors or fabrication 1: imprecise details 2: correct (“unsure” allowed, treated as missing) |
| Does it answer the question? | H | 0: no 1: partly 2: contains the needed information |
| Overall | Score | Rank among the six answers to the question (1 = best; ties allowed) |
Language Model Generation Evaluation
Language model generation evaluation measures the quality, reliability, diversity, and task fit of outputs produced by language models.
See all 420 papers