Code Generation Benchmarks

Follow this topic Ask about this topic

Momentum

15 papers in the last four weeks, up 275% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 104

All topics

Beyond Pass Rate: A Multilingual, Execution-Grounded Evaluation of Open Code LLMs
Jun 7, 2026Sayed Erfan ArefinCode Generation Evaluation Code Language Models
Department of Computer Science University of Dayton Dayton, OH 45469, USA
Asuka-Bench: Benchmarking Code Agents on Underspecified User Intent and Multi-Round Refinement
Jun 4, 2026Xin Wang, Liangtai Sun, Yaoming Zhu +8Computer-Use Agent Benchmarks Language Model Generation Evaluation
University of Science and Technology of China · 2Independent researchers
CodegenBench: Can LLMs Write Efficient Code Across Architectures?
Jun 1, 2026Jie Li, Wenzhao Wu, Junqi Hu +5High-Performance Computing Code Generation
1Sun Yat-sen University · 2National Supercomputing Center in Wuxi · 3National Supercomputing Center in Shenzhen +2
WorldCoder-Bench: Benchmarking Physically Grounded 3D World Synthesis
Jun 1, 2026Shuo Lu, Yinuo Xu, Kecheng Yu +8LLM-Based Program Synthesis Synthetic Benchmark Generation
NLPR & MAIS, CASIA · Huawei Noah’s Ark Lab
BenchEvolver: Frontier Task Synthesis via Solution-Centric Evolution
May 31, 2026Yangzhen Wu, Aaron J. Li, Wenjie Ma +10Synthetic Benchmark Generation Code Generation Evaluation
University of California, Berkeley · Institute for Interdisciplinary Information Sciences, Tsinghua University
3DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Code
May 31, 2026Yipeng Gao, Lei Shu, Genzhi Ye +5VLM Evaluation 3D Asset Generation
Google DeepMind · University of Southern California · Google Research
Knowledge boundary probing and demand-guided intervention for LLM-based power system code generation
May 29, 2026Hui Wu, Xiaoyang Wang, Zhong FanAutomated Program Repair Code Generation
Department of Engineering, University of Exeter · Department of Computer Science, University of Exeter
Benchmarking Multimodal LLMs on Code Generation for Complex Interactive Webpages
May 29, 2026Fan Wu, Lishuai Dong, Cuiyun Gao +4Web Application Generation Code Generation
Computer Science and Technolog, Harbin Institute of Technology. · Institute of Information Engineering, Chinese Academy of Sciences.
CodeGolf Bench: A Multi-Language Benchmark for Evaluating Concise Code Generation Capabilities of Large Language Models
May 28, 2026Vedant PadwalLanguage Model Generation Evaluation Code Generation
Independent
Converted, Not Equivalent: Benchmarking Codebase Conversion via Observational Equivalence
May 27, 2026Linxin Song, Jiefeng Chen, Yue Huang +5Code Generation Evaluation Code Generation Benchmarks
University of Southern California · Google Cloud AI Research · University of Notre Dame +1
Do LLMs Favor Their Providers? Measuring Vertical Integration Bias in Code Generation
May 27, 2026Melih Catal, Alex Wolf, Tiago Ferreiro Matos +2Agentic Code Generation Algorithmic Bias
University of Zurich · University of Mannheim
GUI Agents for Continual Game Generation
May 27, 2026Yixu Huang, Bo Li, Na Li +8GUI Agents Code Generation
αFudan University · βXiaohongshu Inc. · γTongji University +2
VeriScale: Adversarial Test-Suite Scaling for Verifiable Code Generation
May 21, 2026Yifan Bai, Xiaoyang Liu, Zihao Mou +7Code Generation Synthetic Benchmark Generation
School of Mathematical Sciences, Shanghai Jiao Tong University · School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen · School of Mathematics, Jilin University +5
Stdlib or Third-Party? Empirical Performance and Correctness of LLM-Assisted Zero-Dependency Python Libraries
May 20, 2026Peng Ding, Rick StevensCode Generation LLM-Based Program Synthesis
University of Chicago · Argonne National Laboratory
CriterAlign: Criterion-Centric Rationale Alignment for Code Preference Judging
May 19, 2026Zhenyu Li, Aleksandar Cvejic, Zehui Chen +1Pairwise Preference Learning Human Preference Evaluation
KAUST · USTC
PRISM: A Benchmark for Programmatic Spatial-Temporal Reasoning
May 19, 2026Qiran Zhang, Yuheng Wang, Runde Yang +9Spatial Reasoning Code Generation Benchmarks
School of Artificial Intelligence, Shanghai Jiao Tong University
Rule2DRC: Benchmarking LLM Agents for DRC Script Synthesis with Execution-Guided Test Generation
May 15, 2026Jinuk Kim, Junsoo Byun, Donghwi Hwang +2Electronic Design Automation LLM Agent Evaluation
Department of Computer Science and Engineering, Seoul National University · Neural Processing Research Center · Samsung Electronics Co., Ltd.
PerfCodeBench: Benchmarking LLMs for System-Level High-Performance Code Optimization
May 13, 2026Huihao Jing, Wenbin Hu, Shaojin Chen +5Benchmark Design High-Performance Computing
HKUST · NYU · SWUPL
ShapeCodeBench: A Renewable Benchmark for Perception-to-Program Reconstruction of Synthetic Shape Scenes
May 12, 2026Shivam KumarSynthetic Benchmark Generation Code Generation Benchmarks
Independent Researcher
Vision2Code: A Multi-Domain Benchmark for Evaluating Image-to-Code Generation
May 11, 2026Ajay Vikram Periasami, Junlin Wang, Bhuwan DhingraVLM Evaluation Code Generation
Duke University
BenchCAD: A Comprehensive, Industry-Standard Benchmark for Programmatic CAD
May 11, 2026Haozhe Zhang, Kaichen Liu, Miaomiao Chen +4Benchmark Construction Parametric CAD Modeling
University of Virginia · University of California, San Diego · Rice University
PDEAgent-Bench: A Multi-Metric, Multi-Library Benchmark for PDE Solver Generation
May 10, 2026Zhen Hang, Yushan Yashengjiang, Junhui Li +21PDE Solving Benchmark Design
University of Science and Technology of China · Beijing University of Posts and Telecommunications · Tencent +10
VeriContest: A Competitive-Programming Benchmark for Verifiable Code Generation
May 8, 2026Zichen Xie, Mrigank Pawagi, Yuxin Liu +5Code Generation Formal Verification
University of Virginia · 2Indian Institute of Science · 3Rice University +1
Mage: Multi-Axis Evaluation of LLM-Generated Executable Game Scenes Beyond Compile-Pass Rate
May 8, 2026Hugh Xuechen Liu, Kıvanç TatarCode Generation Code Generation Evaluation
Chalmers University of Technology and University of Gothenburg
Constraint Decay: The Fragility of LLM Agents in Backend Code Generation
May 7, 2026Francesco Dente, Dario Satriani, Paolo PapottiAI Coding Agents Code Generation
EURECOM, France · University of Basilicata, Italy
BUILD-AND-FIND: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases
May 7, 2026Jhen-Ke LinBenchmark Construction LLM Agent Evaluation
National Yang Ming Chiao Tung University
KernelBenchX: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernels
May 6, 2026Han Wang, Jintao Zhang, Kai Jiang +3GPU Kernel Optimization GPU Acceleration
∗Equal contribution.
MolViBench: Evaluating LLMs on Molecular Vibe Coding
May 4, 2026Jiatong Li, Yuxuan Ren, Weida Wang +4AI-Assisted Scientific Research Scientific Code Generation
The Hong Kong Polytechnic University · National University of Singapore · Fudan University
RuC: HDL-Agnostic Rule Completion Benchmark Generation
Apr 30, 2026Arnau Ayguadé Domingo, Miquel Alberti-Binimelis, Cristian Gutierrez-Gomez +5LLM Evaluation LLM Prompting
Barcelona Supercomputing Center · Universitat Polit`ecnica de Catalunya
ClassEval-Pro: A Cross-Domain Benchmark for Class-Level Code Generation
Apr 29, 2026Yeheng Chen, Chaoxiang Xie, Yuling Shi +4Code Generation Synthetic Benchmark Generation
Shanghai Jiao Tong University China

Jun 7, 2026, cs.AIJ/K move · Enter open · S save

Beyond Pass Rate: A Multilingual, Execution-Grounded Evaluation of Open Code LLMs

Sayed Erfan Arefin

Department of Computer Science University of Dayton Dayton, OH 45469, USA

Code Generation Evaluation Code Language Models Code Generation Benchmarks Multilingual Code Generation

Open paper AskPDF arXiv

Code generation models are typically compared using compact execution benchmarks and aggregate pass rates, but such summaries obscure how performance varies across programming languages, problem families, and failure modes. We present a large-scale, execution-grounded evaluation of 9 openly accessible LLMs specialized for coding on 2,707 free LeetCode problems across 12 programming languages. Our corpus contains 325,343 problem-model-language jobs, each linked to prompt metadata, extracted code, LeetCode execution outcomes, and static-analysis signals. The results show that current open models remain far from the human acceptance reference: the best model, Yi-Coder-9B-Chat, reaches 23.64% mean correctness, compared with a 57.2% human acceptance baseline. Rankings are also slice-dependent: Qwen2.5-Coder-14B-Instruct is strongest on hard problems and distinct-problem coverage, while Gemma-2-27B-IT achieves the highest all-language lint pass rate. Failure analysis shows that compile errors account for 63.25% of non-accepted best submissions, indicating that many failures occur before semantic correctness can be tested. Static quality further diverges from functional correctness. Together, these findings show that multilingual, artifact-preserving evaluation reveals tradeoffs hidden by single-language or single-metric leaderboards.

Momentum

Related topics

Latest papers 104