Code from LLM-assisted development may contain defects like API hallucinations, motivating the development of code evaluation metrics to assess such code before deployment. As conventional rule-based metrics merely score code based on similarity to reference programs, and execution-based metrics require expensive manual preparation for tests, researchers have developed LLM-as-a-judge metrics for both behavioral and nonbehavioral code evaluation, and curated multiple benchmarks to validate their effectiveness. However, such benchmarks suffer from critical limitations: Some feature coarse-grained binary labels of passing/failing unit tests, unable to distinguish between fundamental and minor flaws; others utilize fine-grained criteria with manually annotated scores as ground-truth, but such criteria are often vaguely defined, introducing subjectivity and unreliability in human scores. Furthermore, they often directly prompt LLMs to generate code for evaluation, which may lead to unbalanced code quality distributions, failing to test the metrics' discriminative capability across varying quality levels. To curate a diverse benchmark with a balanced program quality distribution and streamline manual annotation, we propose AXIOM, a novel perturbation-based framework. It reframes program scores as the refinement effort needed for deployment, with two stages: (1) Quality-constrained semantic perturbation, prompting LLMs to apply sequences of predefined perturbation rules to existing programs, precisely controlling each program's target quality to achieve a balanced distribution. (2) Hybrid quality calibration, selecting semantically distinct programs for benchmark diversity, and performing streamlined manual annotation with multisource context. With AXIOM, we curate AXIOMBENCH, a large-scale multilingual benchmark with 1,962 code evaluation problems in a more balanced score distribution...
Figures & tables
Fig. 1 : Overview of Axiom . Rule(s) refer to the perturbation rules indicating how the perturbation should be performed, and SC refers to the score ceiling of each rule, as described in Section III-B1 .
c(u)
Category
#
Examples ( Iu )
5
Alternatives
5
• Refactor to functional style (e.g., streams, lambdas). • Replace primitive structures with standard library equivalents. • Shorten code using ternary operators or comprehensions.
• Create “Long Methods” by inlining helper functions. • Introduce duplicated code by inlining function calls. • Increase nesting depth of conditional logic.
2
Minor Bugs
11
• Introduce off-by-one errors. • Swap binary operators (e.g., a+b→a−b ). • Replace deep copies with shallow ones.
1
Major Defects
10
• Remove recursion base cases. • Replace algorithms with plausible but wrong ones. • Remove essential input validation logic.
TABLE I: Summary of Perturbation Rules by score ceiling c(u) . The rule set contains 45 rules total.
Benchmark
EX
#Prog.
#Req.
κ
S
AxiomBench-Full
✗
1,962
388
96.7
0.997
AxiomBench-Rectifiable
1,632
373
0.995
Wang et al. [ 10 ]
✗
300
100
82.2
0.950
CodeVisionary [ 14 ]
✗
363
121
80.0
0.675
CodeJudge-Eval [ 12 ]
✓
1,860
457
-
0.785
CodeJudgeBench [ 11 ]
✓
10,704
5,352
-
1.000
TABLE II : Benchmark statistics. Rectifiable refers to AxiomBench without 0-point programs, used in RQ1. Full refers to the complete AxiomBench , used in other experiments. #Prog. / #Req. means the number of programs/requirements. κ on our benchmark is computed on 10% data (196 programs). Execution-based benchmarks (EX marked as ✓) only evaluate code on functionality, and due to their absence of manual annotation, κ is inapplicable to them.
Metric
ρ
τ
α
EMD
Mean
Part I: AxiomBench-Full
Rule-Based Metrics
ChrF++
46.18
37.48
44.57
0.308
2.703
CodeBLEU
43.92
35.77
42.12
0.326
2.477
Edit Similarity
43.78
35.43
42.30
0.437
2.098
LLM-as-a-Judge: DeepSeek-V3.1
TABLE III : Performance of each metric. The best result in each column is underlined and marked bold. Results at most 3% worse than it are underlined.
Fig. 2 : Plots of Matthews correlation coefficient (MCC) of each metric on different subtasks.
Metric
Consistency
Performance (Mean ± Std.)
αRun
ICC Run
ρGT
αGT
LLM-as-a-Judge: Gemini-2.5-Flash
ICE-Score
86.61
85.23
39.92 ± 2.40
16.84 ± 2.57
CodeJudge
73.89
77.89
29.82 ± 1.42
-17.71 ± 2.17
CodeVisionary
51.66
52.09
34.36 ± 1.66
3.25 ± 3.66
TABLE IV : Metric consistency. Run metrics are computed across five runs to measure sample-level consistency. GT metrics are computed for each run against the ground-truth scores to measure variance in dataset-level performance.
Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.
As LLMs are increasingly used as judges in code applications, they should be evaluated in realistic interactive settings that capture partial context and ambiguous intent. We present TRACE (Tool for Rubric Analysis in Code Evaluation), a framework that evaluates LLM judges' ability to predict human preferences and automatically extracts rubric items to reveal systematic biases in how humans and models weigh each item. Across three modalities -- chat-based programming, IDE autocompletion, and instructed code editing -- we use TRACE to measure how well LLM judges align with developer preferences. Among 13 different models, the best judges underperform human annotators by 12-23%. TRACE identifies 35 significant sources of misalignment between humans and judges across interaction modalities, the majority of which correspond to existing software engineering code quality criteria. For example, in chat-based coding, judges are biased towards longer code explanations while humans prefer shorter ones. We find significant misalignment on the majority of existing code quality dimensions, showing alignment gaps between LLM judges and human preference in realistic coding applications.
The rapid advancement of Large Language Models (LLMs) has outpaced the scalability of traditional evaluation benchmarks, which remain heavily dependent on labor-intensive expert curation. We address this bottleneck with Conv-to-Bench, a multi-stage framework that automatically transforms authentic multi-turn user-assistant dialogues into structured, verifiable requirement checklists. By leveraging the "instructional evolution" found in real-world conversational logs, our approach deconstructs fragmented user intent into consolidated instructions and binary evaluation criteria. Applied to the programming domain, Conv-to-Bench produces evaluation sets that demonstrate near-perfect alignment with human-authored standards like BigCodeBench, achieving Spearman correlations of up to ρ = 1.000 with significantly lower computational overhead. Validation of the LLM-as-a-judge framework further confirms its reliability, with the primary evaluator achieving substantial agreement with human-verified ground truth (κ = 0.705). Our comprehensive ablation studies reveal that while multi-turn interactions capture the iterative evolution of user intent, instruction-centric extraction provides a more robust foundation. Ultimately, Conv-to-Bench provides a scalable, cost-effective paradigm for maintaining high-fidelity evaluation standards as user-centric AI applications continue to diversify.
Victor M. dos Santos, Andre C. Castro, Samuel L. de S. Toledo +5
Institute of Mathematics and Computer Science, University of São Paulo, Brazil · HUG Labs, Brazil · Institute of Informatics, Federal University of Goiás, Brazil +1
Code-related benchmarks play a critical role in evaluating large language models (LLMs), yet their quality fundamentally shapes how the community interprets model capabilities. In the past few years, awareness of benchmark quality has grown. Yet, after a decade-scale (2014-2025) survey over 672 code benchmarks, we observed a lag between growing awareness and actual practice. For example, in 2025 alone, the number of benchmarks that ignore code coverage when providing test cases nearly matches the total count accumulated across the previous ten years. In response, we take a clear position: Code benchmarks must prioritize rigor in benchmark construction, reliability in evaluation, and reproducibility in release. To operationalize this position, we introduce a code benchmark guideline HOW2BENCH with 55 checklists. Finally, our further human study also exposed that the current issues not only stem from the significant effort required, but also from a lack of awareness regarding their importance.
Jialun Cao, Yuk-Kit Chan, Zixuan Ling +12
The Hong Kong University of Science and Technology · Guangzhou HKUST Fok Ying Tung Research Institute · The Chinese University of Hong Kong +6