Accurate evaluation is central to the large language model (LLM) ecosystem, guiding model selection and downstream adoption across diverse use cases. In practice, however, evaluating generative outputs typically relies on rigid lexical methods to extract and assess answers, which can conflate a model's true problem-solving ability with its compliance with predefined formatting guidelines. While recent LLM-as-a-Judge approaches mitigate this issue by assessing semantic correctness rather than strict structural conformity, they also introduce substantial computational overhead, making evaluation costly. In this work, we first systematically investigate the limitations of lexical evaluation through a large-scale empirical study spanning 36 models and 15 downstream tasks, demonstrating that such methods correlate poorly with human judgments. To address this limitation, we introduce BERT-as-a-Judge, an encoder-driven approach for assessing answer correctness in reference-based generative settings, robust to variations in output phrasing, and requiring only lightweight training on synthetically annotated question-candidate-reference triplets. We show that it consistently outperforms the lexical baseline while matching the performance of much larger LLM judges, providing a compelling trade-off between the two and enabling reliable, scalable evaluation. Finally, through extensive experimentation, we provide detailed insights into BERT-as-a-Judge's performance to offer practical guidance for practitioners, and release all project artifacts to foster downstream adoption.
Figures & tables
Figure 1: Comparison between regex-based (lexical) evaluation and BERT-as-a-Judge. Top: illustration of both approaches with simple examples. Bottom: model rankings for four similarly sized models from different families, computed via task-wise Borda count.
Figure 2: Quantification of regex parsing failures. Values represent the failure rate, defined as the percentage of instances with unparsable outputs. Results are shown for the Llama-3, Gemma-3, and Qwen-3 model families and aggregated by task category.
Family
Size
Multiple-Choice
Context Extraction
Open-Form Math
Δ Accuracy
Δ Rank
Δ Accuracy
Δ Rank
Δ Accuracy
Δ Rank
Llama-3
1B
-0.9 (-2.6/+1.7)
↑ 4.8
-23.9 (-18.1/-5.8)
↓ 1.2
-11.9 (-11.1/-0.8)
↓ 0.8
3B
-13.1 (-8.5/-4.6)
↓ 0.5
-27.5 (-15.6/-11.9)
↓ 0.4
-7.0 (-2.6/-4.4)
↑ 1.7
8B
-23.3 (-0.9/-22.3)
↓ 6.4
-21.0 (-4.0/-17.0)
↑ 3.5
-5.4 (-0.8/-4.6)
↑ 4.2
70B
-1.1 (-1.1/-0.1)
↑ 1.3
-18.4 (-0.1/-18.3)
↑ 6.5
-30.4 (-29.0/-1.4)
↓ 13.3
Gemma-3
1B
-0.1 (-0.7/+0.5)
↑ 5.7
-12.3 (-0.1/-12.2)
↑ 4.6
-11.3 (-6.8/-4.5)
↓ 1.2
Table 1: Impact of regex-based evaluation on measured model performance. The “ Δ Accuracy” columns report the difference between regex-based and ground-truth (synthetic-label) accuracy, with values in parentheses indicating the contributions from parsing failures and post-parsing matching, respectively. The “ Δ Rank” columns show the corresponding average change in ranking across the 36 evaluated models, where greener values indicate upward rank changes and redder values indicate downward rank changes.
Task
Regex
Qwen-3
Gemma-3
JudgeLM
BLEURT
BERTJudge
0.6B
0.6B (FT)
14B
12B
7B
13B
33B
Base
Large
Multiple-Choice
ARC-Challenge
89.0
50.2
98.1
99.1
99.4
59.1
60.8
65.4
23.8
23.8
99.4
ARC-Easy
88.2
54.0
99.1
98.9
99.7
53.2
53.8
59.5
15.1
15.1
99.7
MMLU
88.1
50.2
94.3
98.6
98.3
57.4
60.6
66.6
37.9
38.0
98.5
GPQA
68.0
65.9
82.0
95.3
92.5
64.9
70.6
73.5
68.0
68.1
93.5
Table 2: Accuracy of evaluation methods against ground-truth labels across tasks, averaged over models. Dashed lines separate test-only tasks from tasks with an available training split. Qwen-3 and Gemma-3 models use direct assessment (S method). “FT” denotes the fine-tuned Qwen-3 0.6B model. Bold indicates the two highest accuracies for each task.
Size
Multiple-
Context
Open-Form
Choice
Extraction
Math
ID
OOD
ID
OOD
ID
OOD
Ministral-3
3B
97.0
96.9
83.7
82.5
81.4
86.5
8B
97.9
97.8
87.9
87.1
83.5
84.8
14B
98.2
98.3
89.1
88.6
83.5
82.6
Table 3: Assessment accuracy on out-of-domain models. “ID” denotes training on all model outputs, while “OOD” excludes specific models from the training mixture. Results are aggregated by task category.
Figure 3: Compute-aware comparison between BERT-as-a-Judge and LLM judges from the Qwen-3 and Gemma-3 families under S and L prompting strategies.
Figure 4: BERT-as-a-Judge evaluation quality across different training budgets.
Table 8
Task
Test Set (Form.)
Test Set (Free)
Regex
BERTJ. (Form.)
BERTJ. (Free)
Regex
BERTJ. (Form.)
BERTJ. (Free)
Multiple-Choice
88.8
97.7
97.4
–
94.0
97.6
Context Extraction
73.0
89.2
85.8
–
84.3
91.6
Open-Form Math
87.3
93.9
93.7
–
93.1
93.5
Table 6: Encoder’s robustness to answer formatting. “Test Set (Form.)” denotes test sets with formatted answers, while “Test Set (Free)” contains unformatted answers. “Regex” columns show regex-based results, and “BERT-J. (Form.)” and “BERT-J. (Free)” report accuracies for BERT-as-a-Judge encoders trained on formatted and unformatted answers, respectively. Regex results are omitted for the free-format test set, as answers cannot be reliably parsed.
Figure 5: Effect of score thresholding on BERT-as-a-Judge downstream assessment accuracy across the three task categories, averaged over all models.
Figure 6: Calibration analysis of BERT-as-a-Judge’s soft probabilities. The blue curve corresponds to the original (unscaled) soft probabilities, while the orange curve shows the probabilities after temperature scaling ( τ =1.75). The grey dashed line represents perfect calibration, where predicted probabilities match the true likelihood of the label being correct.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Task Category
Generation Prompt
Multiple-Choice
Answer the following multiple-choice question. Question: {question} Choices: A) {choice_1} B) {choice_2} C) {choice_3} D) {choice_4} [...]
Context Extraction
Answer the question based on the provided context. Context: {context} Question: {question}
Open-Form Math
{question}
Appendix
Table 7: Base generation prompts for each task category.
Task Category
Formatting Instruction
Generation Suffix
Multiple-Choice
Free
None
Soft
Conclude your response with "Final answer: X", where X is the letter of the correct choice.
Strict
Respond only with the exact format "Final answer: X", where X is the letter of the correct choice.
Context Extraction
Free
None
Soft
Conclude your response with "Final answer: X", where X is the exact span from the context that answers the question.
Strict
Respond only with the exact format "Final answer: X", where X is the exact span from the context that answers the question.
Appendix
Table 8: Generation suffixes used across task categories for all formatting strategies.
Assessment Prompt
You are an expert evaluator. Your task is to determine whether the CANDIDATE response correctly answers the QUESTION. Judge the CANDIDATE as correct only if its final answer, disregarding any intermediate reasoning or explanation, is semantically equivalent to the REFERENCE with respect to the QUESTION. Base your judgment solely on the information given. Do not rely on external knowledge. [QUESTION starts here] {question} [QUESTION ends here] [REFERENCE starts here] {reference} [REFERENCE ends here] [CANDIDATE starts here] {candidate} [CANDIDATE ends here] Conclude your response with exactly one of the following: - "Final answer: True" if the CANDIDATE is correct - "Final answer: False" if the CANDIDATE is incorrect
Appendix
Table 9: Answer assessment prompt for LLM judges, allowing intermediate token generation before the final judgment. This involves Nemotron-Super-v1.5 for label generation ( § 2.2 ) and generative judges evaluated under inference budget L ( Figure 3 ).
Assessment Prompt
You are an expert evaluator. Your task is to determine whether the CANDIDATE response correctly answers the QUESTION. Judge the CANDIDATE as correct only if its final answer, disregarding any intermediate reasoning or explanation, is semantically equivalent to the REFERENCE with respect to the QUESTION. Base your judgment solely on the information given. Do not rely on external knowledge. [QUESTION starts here] {question} [QUESTION ends here] [REFERENCE starts here] {reference} [REFERENCE ends here] [CANDIDATE starts here] {candidate} [CANDIDATE ends here] Respond with exactly one of the following strings (add no additional text): - "Final answer: True" if the CANDIDATE is correct - "Final answer: False" if the CANDIDATE is incorrect
Appendix
Table 10: Prompt used for direct assessment by LLM judges, applied to generative judges evaluated under inference budget S ( Figure 3 ).
Task
Log-lik.
Generative
Strict
Soft
Free
Multiple-Choice
ARC-Challenge
46.3
74.4
76.2
75.0
ARC-Easy
65.0
84.6
84.9
84.4
GPQA
25.9
32.3
31.8
28.8
MMLU
39.9
59.4
62.0
60.5
Appendix
Table 11: Comparison of evaluation modes across all benchmarks. Results are averaged over all models, and bold values indicate the best performance for each task.
Task
Accuracy (%)
Multiple-Choice
ARC-Challenge
96.4
ARC-Easy
96.9
MMLU
96.4
GPQA
97.3
MMLU-Pro
97.5
Appendix
Table 12: Accuracy between human and synthetic labels across task categories.
Figure 7: Sensitivity of the AH estimate to variations in AS and ρ .
Language
Regex
Gemma-3 (12B)
Qwen-3 (14B)
BERTJudge
English
Multilingual
ar
89.3
97.3
97.6
97.2
97.8
de
87.9
96.9
96.8
96.5
97.5
en
89.8
97.4
97.4
97.1
97.7
es
89.7
97.2
97.2
96.3
97.7
fr
88.4
96.8
96.8
96.2
97.6
Appendix
Table 14: Assessment accuracy across 12 languages of the multilingual MMLU benchmark. BERT-as-a-Judge results are reported for both the English-only fine-tuning setting and after multilingual adaptation. The highest accuracy in each row is shown in bold.
Family
Size
ARC
ARC
GPQA
MMLU
MMLU
Truthful
Challenge
Easy
Pro
QA
Apertus
8B
0.2
0.1
4.9
0.8
4.5
0.0
70B
0.1
0.1
21.2
2.6
12.3
1.5
EuroLLM
1.7B
97.6
98.7
83.0
94.1
69.9
66.3
9B
0.0
0.0
11.8
1.2
11.8
0.0
22B
0.1
0.0
7.4
0.4
3.0
0.0
Appendix
Table 15: Parsing failure rates on multiple-choice benchmarks.
Family
Size
CoQA
DROP
HotpotQA
SQuAD-v2
Apertus
8B
0.0
0.2
0.1
0.1
70B
0.0
0.3
0.0
0.1
EuroLLM
1.7B
41.4
39.7
16.3
38.6
9B
0.0
0.0
0.0
0.0
22B
0.2
0.0
0.3
0.0
Falcon-3
1B
20.6
6.2
3.9
5.2
Appendix
Table 16: Parsing failure rates on context extraction benchmarks.
Family
Size
AIME24
AIME25
ASDiv
GSM8K
Math
Apertus
8B
50.0
46.7
0.9
0.8
13.3
70B
26.7
36.7
1.3
0.4
15.5
EuroLLM
1.7B
70.0
76.7
5.3
6.3
54.1
9B
63.3
50.0
16.9
16.2
43.7
22B
33.3
30.0
2.5
6.8
11.7
Falcon-3
1B
83.3
90.0
69.0
59.9
77.7
Appendix
Table 17: Parsing failure rates on open-form math benchmarks.
Family
Size
CoQA
DROP
HotpotQA
SQuAD-v2
Δ Accuracy
Δ Rank
Δ Accuracy
Δ Rank
Δ Accuracy
Δ Rank
Δ Accuracy
Δ Rank
Apertus
8B
-30.2 (-0.0/-30.2)
↓ 3.0
-21.8 (-0.0/-21.8)
↑ 1.5
-20.6 (-0.1/-20.5)
↑ 2.0
-19.2 (-0.0/-19.2)
↑ 10.0
70B
-41.4 (-0.0/-41.4)
↓ 7.5
-21.9 (-0.1/-21.9)
↑ 3.0
-22.7 (-0.0/-22.7)
↓ 3.0
-25.3 (-0.0/-25.3)
↓ 1.0
EuroLLM
1.7B
-49.6 (-30.6/-19.0)
↓ 8.0
-19.5 (-11.6/-7.9)
↓ 1.0
-37.2 (-9.9/-27.3)
↓ 6.0
-30.0 (-16.6/-13.4)
↓ 7.0
9B
-27.6 (-0.0/-27.6)
↓ 1.0
-18.9 (-0.0/-18.9)
↑ 1.0
-15.9 (-0.0/-15.9)
↑ 5.0
-22.6 (-0.0/-22.6)
↑ 6.0
22B
-24.8 (-0.2/-24.6)
↑ 1.0
-25.4 (-0.0/-25.4)
0.0
-16.3 (-0.2/-16.1)
↑ 1.0
-29.2 (-0.0/-29.2)
0.0
Appendix
Table 18: Effect of regex-based evaluation on performance measurement for context extraction benchmarks.
Family
Size
ARC-Challenge
ARC-Easy
GPQA
MMLU
MMLU-Pro
TruthfulQA
Δ Accuracy
Δ Rank
Δ Accuracy
Δ Rank
Δ Accuracy
Δ Rank
Δ Accuracy
Δ Rank
Δ Accuracy
Δ Rank
Δ Accuracy
Δ Rank
Apertus
8B
0.0 (-0.1/+0.1)
↑ 3.0
0.0 (-0.0/0.0)
↑ 6.0
+5.8 (-0.0/+5.8)
↑ 16.0
-0.9 (-0.1/-0.8)
↑ 4.0
+0.9 (-0.5/+1.4)
↑ 4.0
-0.5 (-0.0/-0.5)
0.0
70B
-1.8 (-0.1/-1.7)
↑ 3.0
-1.5 (-0.0/-1.5)
↑ 2.0
-1.3 (-1.3/0.0)
↑ 8.5
-3.5 (-0.4/-3.1)
↑ 3.0
-2.9 (-1.2/-1.7)
↑ 1.0
-3.3 (-0.5/-2.8)
↓ 2.0
EuroLLM
1.7B
-29.8 (-28.2/-1.5)
↓ 1.0
-29.2 (-28.4/-0.8)
↓ 1.0
-28.3 (-19.2/-9.2)
↓ 17.5
-27.4 (-24.6/-2.9)
↓ 1.0
-22.4 (-9.3/-13.1)
↓ 7.0
-29.0 (-16.3/-12.7)
↓ 5.0
9B
-0.3 (-0.0/-0.3)
↑ 4.0
-0.0 (-0.0/-0.0)
↑ 6.0
-8.9 (-2.7/-6.3)
↑ 0.5
-1.6 (-0.5/-1.2)
↑ 2.0
-8.2 (-2.7/-5.5)
0.0
+0.2 (-0.0/+0.2)
↑ 3.0
22B
-8.6 (-0.1/-8.5)
↓ 6.0
-7.7 (-0.0/-7.7)
↓ 8.0
-9.4 (-1.1/-8.3)
↓ 2.5
-12.7 (-0.1/-12.6)
↓ 8.0
-10.0 (-0.7/-9.3)
↓ 2.0
-6.2 (-0.0/-6.2)
↓ 1.0
Appendix
Table 19: Effect of regex-based evaluation on performance measurement for multiple-choice benchmarks.
Family
Size
AIME24
AIME25
ASDiv
GSM8K
Math
Δ Accuracy
Δ Rank
Δ Accuracy
Δ Rank
Δ Accuracy
Δ Rank
Δ Accuracy
Δ Rank
Δ Accuracy
Δ Rank
Apertus
8B
0.0 (-0.0/0.0)
↑ 4.0
0.0 (-0.0/0.0)
↑ 5.0
-5.1 (-0.1/-5.0)
↑ 3.0
-0.4 (-0.2/-0.2)
↑ 3.0
-10.2 (-1.2/-9.0)
↑ 6.0
70B
0.0 (-0.0/0.0)
↑ 4.0
0.0 (-0.0/0.0)
↑ 5.0
-5.8 (-0.7/-5.1)
↑ 1.0
-0.3 (-0.0/-0.3)
↑ 2.0
-13.3 (-1.3/-12.0)
↑ 7.0
EuroLLM
1.7B
0.0 (-0.0/0.0)
↑ 4.0
0.0 (-0.0/0.0)
↑ 5.0
-1.7 (-0.9/-0.8)
↑ 1.0
-0.8 (-0.2/-0.7)
↑ 1.0
-2.7 (-2.0/-0.7)
0.0
9B
-10.0 (-6.7/-3.3)
↓ 7.5
-3.3 (-3.3/0.0)
↓ 3.0
-29.5 (-11.9/-17.6)
↓ 4.0
-27.5 (-10.7/-16.8)
↓ 6.0
-30.2 (-15.5/-14.7)
↓ 1.0
22B
-20.0 (-3.3/-16.7)
↓ 3.0
-6.7 (-0.0/-6.7)
↓ 7.5
-32.3 (-1.9/-30.4)
↓ 8.0
-29.7 (-5.4/-24.3)
↓ 8.0
-38.6 (-4.2/-34.4)
↓ 7.5
Appendix
Table 20: Effect of regex-based evaluation on performance measurement for open-form math benchmarks.
Family
Size
ARC
ARC
GPQA
MMLU
MMLU
Truthful
Challenge
Easy
Pro
QA
Apertus
8B
99.4
99.6
92.4
98.2
96.7
99.1
70B
99.5
99.7
95.1
98.4
96.6
99.0
EuroLLM
1.7B
99.5
99.7
92.4
97.0
88.5
83.0
9B
99.9
100.0
94.9
99.3
97.5
99.8
22B
99.7
99.9
94.4
98.4
97.2
99.9
Appendix
Table 21: BERT-as-a-Judge assessment accuracy on multiple-choice benchmarks.
Family
Size
ARC
ARC
GPQA
MMLU
MMLU
Truthful
Challenge
Easy
Pro
QA
Apertus
8B
99.0
99.2
91.5
96.4
96.2
97.8
70B
97.7
97.9
93.8
94.9
94.5
96.2
EuroLLM
1.7B
70.2
70.8
71.7
72.6
77.6
71.0
9B
99.5
99.9
89.7
98.0
91.3
99.8
22B
91.2
92.1
85.3
85.8
87.3
93.5
Appendix
Table 22: Regex assessment accuracy on multiple-choice benchmarks.
Family
Size
CoQA
DROP
HotpotQA
SQuAD-v2
Apertus
8B
90.2
89.1
91.9
89.9
70B
87.4
89.8
91.8
89.0
EuroLLM
1.7B
91.4
91.3
92.1
91.2
9B
90.0
89.6
93.5
89.6
22B
90.4
89.2
93.3
89.4
Falcon-3
1B
90.4
90.2
90.4
91.5
Appendix
Table 23: BERT-as-a-Judge assessment accuracy on context extraction benchmarks.
Family
Size
CoQA
DROP
HotpotQA
SQuAD-v2
Apertus
8B
67.0
75.7
73.7
78.3
70B
56.6
76.3
72.5
72.8
EuroLLM
1.7B
49.2
80.0
60.5
69.0
9B
70.8
79.9
80.5
76.1
22B
72.4
73.0
80.2
69.4
Falcon-3
1B
55.4
85.1
77.5
78.8
Appendix
Table 24: Regex assessment accuracy on context extraction benchmarks.
Family
Size
AIME24
AIME25
ASDiv
GSM8K
Math
Apertus
8B
100.0
100.0
95.4
98.6
93.3
70B
100.0
100.0
94.5
99.1
92.6
EuroLLM
1.7B
100.0
100.0
97.8
97.6
97.0
9B
90.0
96.7
95.4
98.6
91.8
22B
80.0
93.3
95.1
98.9
90.2
Falcon-3
1B
96.7
96.7
95.6
97.9
90.6
Appendix
Table 25: BERT-as-a-Judge assessment accuracy on open-form math benchmarks.
Family
Size
AIME24
AIME25
ASDiv
GSM8K
Math
Apertus
8B
100.0
100.0
94.0
99.2
88.4
70B
100.0
100.0
92.9
99.7
85.0
EuroLLM
1.7B
100.0
100.0
96.4
97.5
96.5
9B
90.0
96.7
70.2
72.5
69.4
22B
80.0
93.3
66.9
70.3
61.1
Falcon-3
1B
96.7
96.7
48.6
63.3
77.0
Appendix
Table 26: Regex assessment accuracy on open-form math benchmarks.
Question
Answer the question based on the provided context. Context: A psychological identity relates to self-image (one’s mental model of oneself), self-esteem, and individuality. Consequently, Weinreich gives the definition "A person’s identity is defined as the totality of one’s self-construal, in which how one construes oneself in the present expresses the continuity between how one construes oneself as one was in the past and how one construes oneself as one aspires to be in the future"; this allows for definitions of aspects of identity, such as: "One’s ethnic identity is defined as that part of the totality of one’s self-construal made up of those dimensions that express the continuity between one’s construal of past ancestry and one’s future aspirations in relation to ethnicity" (Weinreich, 1986a). The description or representation of individual and group identity is a central task for psychologists, sociologists and anthropologists and those of other disciplines where "identity" needs to be mapped and defined. How should one describe the identity of another, in ways which encompass both their idiosyncratic qualities and their group memberships or identifications, both of which can shift according to circumstance? Following on from the work of Kelly, Erikson, Tajfel and others Weinreich’s Identity Structure Analysis (ISA), is "a structural representation of the individual’s existential experience, in which the relationships between self and other agents are organised in relatively stable structures over time … with the emphasis on the socio-cultural milieu in which self relates to other agents and institutions" (Weinreich and Saunderson, (eds) 2003, p1). Using constructs drawn from the salient discourses of the individual, the group and cultural norms, the practical operationalisation of ISA provides a methodology that maps how these are used by the individual, applied across time and milieus by the "situated self" to appraise self and other agents and institutions (for example, resulting in the individual’s evaluation of self and significant others and institutions).[citation needed] Question: What is the ISA? Conclude your response with "Final answer: X", where X is the exact span from the context that answers the question.
Reference
Identity Structure Analysis (ISA)
Candidate
The ISA, or Identity Structure Analysis, is a methodology developed by Weinreich that maps how individuals perceive themselves and others in relation to their social environment, including their own group memberships and identifications, as well as the cultural norms and discourses that shape these perceptions.
Appendix
Table 27: Example 488 from the CoQA dataset, along with the answer from Llama-3 1B
Question
We have a triangle △ABC such that AB=6,BC=8, and CA=10. If AD is an angle bisector such that D is on BC, then find the value of AD2. Conclude your response with "Final answer: X", where X is the computed solution.
Reference
45
Candidate
To solve for AD2 in △ABC where AB=6, BC=8, and CA=10, and AD is the angle bisector of ∠BAC with D on BC, we can use the Angle Bisector Theorem and the Angle Bisector Length Formula. ### Step-by-Step Solution: 1. Angle Bisector Theorem: The Angle Bisector Theorem states that the angle bisector divides the opposite side into segments proportional to the adjacent sides. Therefore, if D divides BC into segments BD and DC, then: DCBD=ACAB=106=53 [...] 4. Calculate AD2:AD2=6⋅10(1−(6+10)282)AD2=60(1−16264)AD2=60(1−25664)AD2=60(1−41)AD2=60(43)AD2=60⋅0.75AD2=45 Final answer: 45
Appendix
Table 28: Example 2070 from the MATH dataset, along with the answer from Falcon-3 7B
LLM judges are increasingly being used to evaluate open-ended model responses, often in no-reference settings where a ground-truth answer is unavailable. However, can they reliably assess in such evaluation setups? We explore this question in this paper through a two stage pipeline with a) calibration experiments that assess the judge model's knowledge of the task it is evaluating, and b) sensitivity experiments that assess how the judge model's performance is impacted by the presence and positioning of the reference answer in the prompt. Across experiments covering three languages, we show that the judge models we evaluated tend to over-credit incorrect answers in the absence of a reference answer, and adding reference answer information to the prompt flips the judge model's correct/incorrect decisions by as much as 85% in some experimental settings. Comparison with a subset of human annotations shows that these reference-driven changes generally align with human judgments. Our results emphasize the need for calibrating the LLM judges with a sample with reference-aware evaluation before using them in reference-free setups reliably, and our methodology provides a blueprint for researchers and practitioners in doing such calibration of LLM judges for other tasks.
Chalamalasetti Kranti, Sowmya Vajjala
Computational Linguistics, Department of Linguistics, University of Potsdam, Germany · National Research Council, Ottawa, Canada
Large language models (LLMs) are widely used as reference-free evaluators via prompting, but this "LLM-as-a-Judge" paradigm is costly, opaque, and sensitive to prompt design. In this work, we investigate whether smaller models can serve as efficient evaluators by leveraging internal representations instead of surface generation. We uncover a consistent empirical pattern: small LMs, despite with weak generative ability, encode rich evaluative signals in their hidden states. This motivates us to propose the Semantic Capacity Asymmetry Hypothesis: evaluation requires significantly less semantic capacity than generation and can be grounded in intermediate representations, suggesting that evaluation does not necessarily need to rely on large-scale generative models but can instead leverage latent features from smaller ones. Our findings motivate a paradigm shift from LLM-as-a-Judge to Representation-as-a-Judge, a decoding-free evaluation strategy that probes internal model structure rather than relying on prompted output. We instantiate this paradigm through INSPECTOR, a probing-based framework that predicts aspect-level evaluation scores from small model representations. Experiments on reasoning benchmarks (GSM8K, MATH, GPQA) show that INSPECTOR substantially outperforms prompting-based small LMs and closely approximates full LLM judges, while offering a more efficient, reliable, and interpretable alternative for scalable evaluation. The code and data are available at: https://github.com/zhuochunli/Representation-as-a-judge
Zhuochun Li, Yong Zhang, Ming Li +8
Ping An Technology (Shenzhen) Co., Ltd. · University of Pittsburgh · University of Maryland, College Park +1
Large language models (LLMs) are increasingly applied as automatic evaluators for natural language generation assessment often using pairwise comparative judgements. Existing approaches typically rely on single judges or aggregate multiple judges assuming equal reliability. In practice, LLM judges vary substantially in performance across tasks and evaluation aspects, and their judgment probabilities may be biased and inconsistent. Furthermore, human-labelled supervision for judge calibration may be unavailable. We first empirically demonstrate that inconsistencies in LLM comparison probabilities exist and show that it limits the effectiveness of direct probability-based ranking. To address this, we study the LLM-asa-jury setting and propose BT-sigma, a judge-aware extension of the Bradley-Terry model that introduces a discriminator parameter for each judge to jointly infer item rankings and judge reliability from pairwise comparisons alone. Experiments on benchmark NLG evaluation datasets show that BT-sigma consistently outperforms averaging-based aggregation methods, and that the learned discriminators strongly correlate with independent measures of the cycle consistency of LLM judgments. Further analysis reveals that BT-sigma can be interpreted as an unsupervised calibration mechanism that improves aggregation by modelling judge reliability.
Mengjie Qian, Guangzhi Sun, Mark J. F. Gales +1
Department of Engineering, University of Cambridge, UK.