cs.CLNov 6, 2026

Khmer PFGen-Bench: Building a Judge-Free Benchmark for Open-Ended Generation in a Low-Resource Language

Authors: Mengsay Loem, Kentaro Imajo, Ryota Ozaki, Masanori Hirano

Organizations: Preferred Networks, Inc.

Abstract

Benchmarks for open-ended generation often rely on large language model (LLM) judges, whose reliability in low-resource languages is uncertain. We introduce Khmer PFGen-Bench, to our knowledge the first judge-free benchmark for multi-sentence answers to questions composed directly in Khmer, a low-resource language. Khmer PFGen-Bench adapts the scorer of PFGen-Bench, a benchmark for Japanese that scores responses with keyword rules and n-gram statistics from LLM-generated references. This scoring requires references that are both valid and varied, which LLMs may not produce in Khmer, so we examine how to build such references. We find that post-trained models can give more valid but less varied references than their base variants, and that mixing model families increases variety for the same number of references. Using references from post-trained models in nine families, the benchmark yields model rankings that are stable across reference sets and consistent with a native speaker's ratings. These findings offer guidance for building judge-free benchmarks in other low-resource languages.

Figures & tables

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. BURMESE-SAN: Burmese NLP Benchmark for Evaluating Large Language Models

    Feb 21, 2026Thura Aung, Jann Railey Montalan, Jian Gang Ngui +1LLM EvaluationLow-Resource Language Processing

  2. sk-bench: A Native-First Benchmark for Evaluating Large Language Models in Slovak

    Oct 6, 2026Marek Šuppa, Ivan Vykopal, Andrej Ridzik +9LLM EvaluationMultilingual Language Model Evaluation

  3. Is Jev a Good Multilingual Judge?

    Nov 11, 2026Wei Liu, Yi Fan, Xiyan FuLLM-as-a-JudgeMultilingual Language Model Evaluation