Benchmarking Code LLM-as-a-Judge via Constrained Perturbation and Calibration

Ruiqi Wang; Xinchen Wang; Cuiyun Gao; Chun Yong Chong; Xin Xia; Qing Liao

cs.SEDec 23, 2025

Benchmarking Code LLM-as-a-Judge via Constrained Perturbation and Calibration

Authors: Ruiqi Wang, Xinchen Wang, Cuiyun Gao, Chun Yong Chong, Xin Xia, Qing Liao

Organizations: Harbin Institute of Technology, Shenzhen, China · Monash University Malaysia, Malaysia · Zhejiang University, China

Abstract

Code from LLM-assisted development may contain defects like API hallucinations, motivating the development of code evaluation metrics to assess such code before deployment. As conventional rule-based metrics merely score code based on similarity to reference programs, and execution-based metrics require expensive manual preparation for tests, researchers have developed LLM-as-a-judge metrics for both behavioral and nonbehavioral code evaluation, and curated multiple benchmarks to validate their effectiveness. However, such benchmarks suffer from critical limitations: Some feature coarse-grained binary labels of passing/failing unit tests, unable to distinguish between fundamental and minor flaws; others utilize fine-grained criteria with manually annotated scores as ground-truth, but such criteria are often vaguely defined, introducing subjectivity and unreliability in human scores. Furthermore, they often directly prompt LLMs to generate code for evaluation, which may lead to unbalanced code quality distributions, failing to test the metrics' discriminative capability across varying quality levels. To curate a diverse benchmark with a balanced program quality distribution and streamline manual annotation, we propose AXIOM, a novel perturbation-based framework. It reframes program scores as the refinement effort needed for deployment, with two stages: (1) Quality-constrained semantic perturbation, prompting LLMs to apply sequences of predefined perturbation rules to existing programs, precisely controlling each program's target quality to achieve a balanced distribution. (2) Hybrid quality calibration, selecting semantically distinct programs for benchmark diversity, and performing streamlined manual annotation with multisource context. With AXIOM, we curate AXIOMBENCH, a large-scale multilingual benchmark with 1,962 code evaluation problems in a more balanced score distribution...

Figures & tables

Fig. 1 : Overview of Axiom . Rule(s) refer to the perturbation rules indicating how the perturbation should be performed, and SC refers to the score ceiling of each rule, as described in Section III-B1 .

$c(u)$	Category	#	Examples ( $\mathcal{I}_{u}$ )
5	Alternatives	5	• Refactor to functional style (e.g., streams, lambdas). • Replace primitive structures with standard library equivalents. • Shorten code using ternary operators or comprehensions.
4	Readability	10	• Replace descriptive names with generic single-letter variables. • Introduce unreachable code. • Introduce magic numbers.
3	Design	9	• Create “Long Methods” by inlining helper functions. • Introduce duplicated code by inlining function calls. • Increase nesting depth of conditional logic.
2	Minor Bugs	11	• Introduce off-by-one errors. • Swap binary operators (e.g., $a+b\rightarrow a-b$ ). • Replace deep copies with shallow ones.
1	Major Defects	10	• Remove recursion base cases. • Replace algorithms with plausible but wrong ones. • Remove essential input validation logic.

TABLE I: Summary of Perturbation Rules by score ceiling

c(u)

. The rule set contains 45 rules total.

Benchmark	EX	#Prog.	#Req.	$\kappa$	$S$
AxiomBench-Full	✗	1,962	388	96.7	0.997
AxiomBench-Rectifiable	✗	1,632	373	96.7	0.995
Wang et al. [ 10 ]	✗	300	100	82.2	0.950
CodeVisionary [ 14 ]	✗	363	121	80.0	0.675
CodeJudge-Eval [ 12 ]	✓	1,860	457	-	0.785
CodeJudgeBench [ 11 ]	✓	10,704	5,352	-	1.000

TABLE II : Benchmark statistics. Rectifiable refers to AxiomBench without 0-point programs, used in RQ1. Full refers to the complete AxiomBench , used in other experiments. #Prog. / #Req. means the number of programs/requirements.

\kappa

on our benchmark is computed on 10% data (196 programs). Execution-based benchmarks (EX marked as ✓) only evaluate code on functionality, and due to their absence of manual annotation,

\kappa

is inapplicable to them.

Metric	$\rho$	$\tau$	$\alpha$	EMD	Mean
Part I: AxiomBench-Full
Rule-Based Metrics
ChrF++	46.18	37.48	44.57	0.308	2.703
CodeBLEU	43.92	35.77	42.12	0.326	2.477
Edit Similarity	43.78	35.43	42.30	0.437	2.098
LLM-as-a-Judge: DeepSeek-V3.1

TABLE III : Performance of each metric. The best result in each column is underlined and marked bold. Results at most 3% worse than it are underlined.

Fig. 2 : Plots of Matthews correlation coefficient (MCC) of each metric on different subtasks.

Metric	Consistency		Performance (Mean ± Std.)
Metric	$\alpha_{\text{Run}}$	ICC ${}_{\text{Run}}$	$\rho_{\text{GT}}$	$\alpha_{\text{GT}}$
LLM-as-a-Judge: Gemini-2.5-Flash
ICE-Score	86.61	85.23	39.92 ± 2.40	16.84 ± 2.57
CodeJudge	73.89	77.89	29.82 ± 1.42	-17.71 ± 2.17
CodeVisionary	51.66	52.09	34.36 ± 1.66	3.25 ± 3.66

TABLE IV : Metric consistency. Run metrics are computed across five runs to measure sample-level consistency. GT metrics are computed for each run against the ground-truth scores to measure variance in dataset-level performance.

Fig. 3 : Illustrative example in AxiomBench .

LLM-as-a-Judge

Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.

See all 555 papers

Comparing Developer and LLM Biases in Code Evaluation
Mar 25, 2026Aditya Mittal, Ryan Shar, Zichu Wu +6Bias in LLM-as-a-Judge Language Model Bias Evaluation
Carnegie Mellon University
Conv-to-Bench: Evaluating Language Models Via User-Assistant Dialogues In Code Tasks
Apr 7, 2026Victor M. dos Santos, Andre C. Castro, Samuel L. de S. Toledo +5LLM Evaluation Synthetic Benchmark Generation
Institute of Mathematics and Computer Science, University of São Paulo, Brazil · HUG Labs, Brazil · Institute of Informatics, Federal University of Goiás, Brazil +1
Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility
Jan 18, 2025Jialun Cao, Yuk-Kit Chan, Zixuan Ling +12Benchmark Construction Benchmark Design
The Hong Kong University of Science and Technology · Guangzhou HKUST Fok Ying Tung Research Institute · The Chinese University of Hong Kong +6

Mar 25, 2026J/K move · Enter open · S save

Comparing Developer and LLM Biases in Code Evaluation

Aditya Mittal, Ryan Shar, Zichu Wu, Shyam Agarwal+5

Carnegie Mellon University

Bias in LLM-as-a-Judge Language Model Bias Evaluation Code Generation Evaluation

Open paper AskPDF arXiv

As LLMs are increasingly used as judges in code applications, they should be evaluated in realistic interactive settings that capture partial context and ambiguous intent. We present TRACE (Tool for Rubric Analysis in Code Evaluation), a framework that evaluates LLM judges' ability to predict human preferences and automatically extracts rubric items to reveal systematic biases in how humans and models weigh each item. Across three modalities -- chat-based programming, IDE autocompletion, and instructed code editing -- we use TRACE to measure how well LLM judges align with developer preferences. Among 13 different models, the best judges underperform human annotators by 12-23%. TRACE identifies 35 significant sources of misalignment between humans and judges across interaction modalities, the majority of which correspond to existing software engineering code quality criteria. For example, in chat-based coding, judges are biased towards longer code explanations while humans prefer shorter ones. We find significant misalignment on the majority of existing code quality dimensions, showing alignment gaps between LLM judges and human preference in realistic coding applications.

Abstract

LLM-as-a-Judge

Explore similar work

Comparing Developer and LLM Biases in Code Evaluation

Conv-to-Bench: Evaluating Language Models Via User-Assistant Dialogues In Code Tasks

Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility