Code Generation Evaluation

Follow this topic Ask about this topic

Code generation evaluation studies how well AI systems produce programs that work and satisfy the requirements developers intended. Common approaches include executable unit tests and pass@k benchmarks, repository-level issue resolution, and tests of generated applications through browsers or other interactive environments. Researchers also assess qualities beyond passing tests, such as security, maintainability, performance, instruction adherence, and fidelity to domain-specific behavior. A central challenge is that test success can overstate real-world reliability, while stochastic outputs, benchmark contamination, and imperfect automated judges make comparisons difficult.

Latest papers

215 papers31 papers in the last four weeks, up 63% on the four weeks before. 0.2% of all new papers.

CriterAlign: Criterion-Centric Rationale Alignment for Code Preference Judging
May 19, 2026Zhenyu Li, Aleksandar Cvejic, Zehui Chen +1Pairwise Preference Learning Human Preference Evaluation
KAUST · USTC
WebGameBench: Requirement-to-Application Evaluation for Coding Agents via Browser-Native Games
May 17, 2026Wenyu Zhang, Guoliang You, Tianlun +8Web Application Generation AI Coding Agents
1Baidu · University of Science and Technology of China
When Retrieval Hurts Code Completion: A Diagnostic Study of Stale Repository Context
May 14, 2026Haojun Weng, Qianqian Yang, Hao Fu +2Retrieval-Augmented Generation Retrieval Robustness
Independent Researcher, California, USA · Independent Researcher, Beijing, China
The Readability Spectrum: Patterns, Issues, and Prompt Effects in LLM-Generated Code
May 13, 2026Hengzhi Ye, Fengyuan Ran, Weiwei Xu +1LLM Prompting Code Generation Evaluation
Peking university · Beijing, Beijing, China · Wuhan University +1
PerfCodeBench: Benchmarking LLMs for System-Level High-Performance Code Optimization
May 13, 2026Huihao Jing, Wenbin Hu, Shaojin Chen +5Benchmark Design High-Performance Computing
HKUST · NYU · SWUPL
Vision2Code: A Multi-Domain Benchmark for Evaluating Image-to-Code Generation
May 11, 2026Ajay Vikram Periasami, Junlin Wang, Bhuwan DhingraVLM Evaluation Image-to-Code Generation
Duke University
Instruction Adherence in Coding Agent Configuration Files: A Factorial Study of Four File-Structure Variables
May 11, 2026Damon McMillanCoding Agents LLM Agent Evaluation
HxAI Australia
SmartEval: A Benchmark for Evaluating LLM-Generated Smart Contracts from Natural Language Specifications
May 10, 2026Abhinav Goel, Agostino Capponi, Alfio Gliozzo +1Code Generation Code Generation Evaluation
Columbia University New York, NY, USA · IBM T.J. Watson Research Center Yorktown Heights, NY, USA
Your Simulation Runs but Solves the Wrong Physics: PDE-Grounded Intent Verification for LLM-Generated Multiphysics Simulation Code
May 10, 2026Zhenghan Song, Yulong Liu, Cheng Wan +4LLM Self-Correction Scientific Code Generation
Cornell University · Columbia University · Harvard University & Nanyang Technological University
Semantic Voting: Execution-Grounded Consensus for LLM Code Generation
May 9, 2026Shan Jiang, Zijian Yi, Chenguang ZhuExecution-Guided Code Generation Code Generation Evaluation
The University of Texas at Austin
Mage: Multi-Axis Evaluation of LLM-Generated Executable Game Scenes Beyond Compile-Pass Rate
May 8, 2026Hugh Xuechen Liu, Kıvanç TatarCode Generation Code Generation Evaluation
Chalmers University of Technology and University of Gothenburg
Delulu: A Verified Multi-Lingual Benchmark for Code Hallucination Detection in Fill-in-the-Middle Tasks
May 7, 2026Mahdi Erfanian, Nelson Daniel Troncoso, Aashna Garg +4Language Model Error Detection Language Model Generation Evaluation
University of Illinois Chicago · Microsoft
Correct Code, Vulnerable Dependencies: A Large Scale Measurement Study of LLM-Specified Library Versions
May 7, 2026Chengjie Wang, Jingzheng Wu, Xiang Ling +2Software Engineering LLM Security
Intelligent Software Research Center, Institute of Software, Chinese Academy of Sciences, Beijing, China · Key Laboratory of System Software (Chinese Academy of Sciences), Beijing, China
The Single-File Test: A Longitudinal Public-Interface Evaluation of First-Output LLM Web Generation with Social Reach Tracking
May 6, 2026Diego Cabezas PalaciosCode Generation Code Generation Evaluation
Beyond BLEU: A Semantic Evaluation Method for Code Translation
May 6, 2026Julius Näumann, Sven Keidel, Amir Molzam Sharifloo +1Code Translation Code Generation Evaluation
∗TU Darmstadt · †Hessian Center for Artificial Intelligence, Darmstadt, Germany · ‡National Research Center for Applied Cybersecurity ATHENE
Bridging Generation and Training: A Systematic Review of Quality Issues in LLMs for Code
May 6, 2026Kaifeng He, Xiaojun Zhang, Peiliang Cai +7Training Data Curation Code Generation Evaluation
Sun Yat-sen University, China · Nanyang Technological University, Singapore · Tongji University, China +1
Measuring and Mitigating Bias in Code Generated by Large Language Models
May 5, 2026Yuxi Chen, Yutian Tang, Timothy StorerAlgorithmic Bias Language Model Bias Evaluation
School of Computing Science, University of Glasgow, United Kingdom.
MolViBench: Evaluating LLMs on Molecular Vibe Coding
May 4, 2026Jiatong Li, Yuxuan Ren, Weida Wang +4AI-Assisted Scientific Research Scientific Code Generation
The Hong Kong Polytechnic University · National University of Singapore · Fudan University
Social Bias in LLM-Generated Code: Benchmark and Mitigation
May 1, 2026Fazle Rabbi, Lin Ling, Song Wang +1Social Bias in Language Models Agentic Code Generation
Concordia University, Montreal, Canada · Lassonde School of Engineering, York University, Toronto, Canada
Defective Task Descriptions in LLM-Based Code Generation: Detection and Analysis
Apr 27, 2026Amal Akli, Mike Papadakis, Maxime Cordy +1Code Generation Evaluation Language Model Robustness
University of Luxembourg Luxembourg
Characterizing Paraphrase-Induced Failures in Lean 4 Autoformalization
Apr 25, 2026William Feng, Ethan Lou, Aryan SharmaLean Theorem Proving Code Generation Evaluation
Yale University, New Haven, Connecticut, USA.
BLAST: Benchmarking LLMs with ASP-based Structured Testing
Apr 24, 2026Manuel Alejandro Borroto Santana, Erica Coppolillo, Francesco Calimeri +3Logic Programming LLM Evaluation
University of Calabria, Rende, Italy · ICAR-CNR, Rende, Italy · Artificial Intelligence LAB, DeMaCS, University of Calabria, Rende, Italy +1
From If-Statements to ML Pipelines: Revisiting Bias in Code-Generation
Apr 23, 2026Minh Duc Bui, Xenia Heilmann, Mattia Cerrato +2Algorithmic Bias Code Generation Evaluation
Johannes Gutenberg University Mainz, Germany · Universidad Iberoamericana, Ciudad de Mexico · University of Colorado Boulder, USA
Co-Located Tests, Better AI Code: How Test Syntax Structure Affects Foundation Model Code Generation
Apr 20, 2026Éric JacopinCode Generation Code Generation Evaluation
Cosmic AI
Playing Psychic: Using Thought Trees to Predict Reasoning Models Accuracy on Coding Tasks
Apr 18, 2026Jiaxin Fang, Runyuan He, Sahil Bhatia +2Structured Reasoning Code Generation Evaluation
University of California, Berkeley
Bias in the Loop: Auditing LLM-as-a-Judge for Software Engineering
Apr 18, 2026Zixiao Zhao, Amirreza Esmaeili, Fatemeh FardSoftware Engineering LLM-as-a-Judge
University of British Columbia, Canada
QuantCode-Bench: A Benchmark for Evaluating the Ability of Large Language Models to Generate Executable Algorithmic Trading Strategies
Apr 16, 2026Alexey Khoroshilov, Alexey Chernysh, Orkhan Ekhtibarov +2Agentic Code Generation Code Generation Evaluation
Lime
Beyond Output Correctness: Benchmarking and Evaluating Large Language Model Reasoning in Coding Tasks
Apr 14, 2026Yuangang Li, Justin Tian Jin Chen, Ethan Yu +2Language Model Reasoning Evaluation LLM Evaluation
University of California, Irvine
Is Vibe Coding the Future? An Empirical Assessment of LLM Generated Codes for Construction Safety
Apr 14, 2026S M Jamil UddinConstruction Safety LLM Code Generation
Assistant Professor, Department of Construction Management, Colorado State University
Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios
Apr 8, 2026Ruida Hu, Xinchen Wang, Chao Peng +2Code Generation Code Generation Evaluation
Harbin Institute of Technology, Shenzhen, China · Independent Researcher, China · Singapore Management University, Singapore

May 19, 2026J/K move · Enter open · S save

CriterAlign: Criterion-Centric Rationale Alignment for Code Preference Judging

Zhenyu Li, Aleksandar Cvejic, Zehui Chen, Peter Wonka

KAUST · USTC

Pairwise Preference Learning Human Preference Evaluation Pairwise Comparison LLM-as-a-Judge Pairwise Preference Evaluation Pairwise Ranking+4

Open paper AskPDF arXiv

Pairwise human preference prediction is central to evaluating code-generation systems, where quality often depends on task-specific trade-offs beyond functional correctness. While rubric-based LLM judges improve interpretability by decomposing evaluation into explicit criteria, most existing pipelines remain pointwise: they score each response independently and derive preferences by comparing aggregated scores. We show that this design is poorly matched to pairwise code preference prediction and can underperform a strong monolithic judge. We propose CriterAlign, a criterion-centric framework that adapts rubric-based judging to pairwise preference evaluation through direct criterion-level pairwise judgments, tie-driven criterion refinement, swap-consistency filtering, and final pairwise synthesis. We further introduce Human-Preference-Aligned Guidance (HPAG), synthesized offline from training examples by extracting recurring rationale gaps between human preferences and monolithic judge predictions, and injected into the criterion generator, criterion judge, and final judge. On BigCodeReward, CriterAlign improves a Qwen2.5-VL-32B monolithic judge from 60.4% to 66.3% accuracy, with ablations confirming the contributions of pairwise criterion design and HPAG.