Code Generation Benchmarks

Follow this topic Ask about this topic

Momentum

15 papers in the last four weeks, up 275% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 104

All topics

Beyond Pass Rate: A Multilingual, Execution-Grounded Evaluation of Open Code LLMs
Jun 7, 2026Sayed Erfan ArefinCode Generation Evaluation Code Language Models
Department of Computer Science University of Dayton Dayton, OH 45469, USA
Asuka-Bench: Benchmarking Code Agents on Underspecified User Intent and Multi-Round Refinement
Jun 4, 2026Xin Wang, Liangtai Sun, Yaoming Zhu +8Computer-Use Agent Benchmarks Language Model Generation Evaluation
University of Science and Technology of China · 2Independent researchers
CodegenBench: Can LLMs Write Efficient Code Across Architectures?
Jun 1, 2026Jie Li, Wenzhao Wu, Junqi Hu +5High-Performance Computing Code Generation
1Sun Yat-sen University · 2National Supercomputing Center in Wuxi · 3National Supercomputing Center in Shenzhen +2
WorldCoder-Bench: Benchmarking Physically Grounded 3D World Synthesis
Jun 1, 2026Shuo Lu, Yinuo Xu, Kecheng Yu +8LLM-Based Program Synthesis Synthetic Benchmark Generation
NLPR & MAIS, CASIA · Huawei Noah’s Ark Lab
BenchEvolver: Frontier Task Synthesis via Solution-Centric Evolution
May 31, 2026Yangzhen Wu, Aaron J. Li, Wenjie Ma +10Synthetic Benchmark Generation Code Generation Evaluation
University of California, Berkeley · Institute for Interdisciplinary Information Sciences, Tsinghua University
3DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Code
May 31, 2026Yipeng Gao, Lei Shu, Genzhi Ye +5VLM Evaluation 3D Asset Generation
Google DeepMind · University of Southern California · Google Research
Knowledge boundary probing and demand-guided intervention for LLM-based power system code generation
May 29, 2026Hui Wu, Xiaoyang Wang, Zhong FanAutomated Program Repair Code Generation
Department of Engineering, University of Exeter · Department of Computer Science, University of Exeter
Benchmarking Multimodal LLMs on Code Generation for Complex Interactive Webpages
May 29, 2026Fan Wu, Lishuai Dong, Cuiyun Gao +4Web Application Generation Code Generation
Computer Science and Technolog, Harbin Institute of Technology. · Institute of Information Engineering, Chinese Academy of Sciences.
CodeGolf Bench: A Multi-Language Benchmark for Evaluating Concise Code Generation Capabilities of Large Language Models
May 28, 2026Vedant PadwalLanguage Model Generation Evaluation Code Generation
Independent
Converted, Not Equivalent: Benchmarking Codebase Conversion via Observational Equivalence
May 27, 2026Linxin Song, Jiefeng Chen, Yue Huang +5Code Generation Evaluation Code Generation Benchmarks
University of Southern California · Google Cloud AI Research · University of Notre Dame +1
Do LLMs Favor Their Providers? Measuring Vertical Integration Bias in Code Generation
May 27, 2026Melih Catal, Alex Wolf, Tiago Ferreiro Matos +2Agentic Code Generation Algorithmic Bias
University of Zurich · University of Mannheim
GUI Agents for Continual Game Generation
May 27, 2026Yixu Huang, Bo Li, Na Li +8GUI Agents Code Generation
αFudan University · βXiaohongshu Inc. · γTongji University +2
VeriScale: Adversarial Test-Suite Scaling for Verifiable Code Generation
May 21, 2026Yifan Bai, Xiaoyang Liu, Zihao Mou +7Code Generation Synthetic Benchmark Generation
School of Mathematical Sciences, Shanghai Jiao Tong University · School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen · School of Mathematics, Jilin University +5
Stdlib or Third-Party? Empirical Performance and Correctness of LLM-Assisted Zero-Dependency Python Libraries
May 20, 2026Peng Ding, Rick StevensCode Generation LLM-Based Program Synthesis
University of Chicago · Argonne National Laboratory
CriterAlign: Criterion-Centric Rationale Alignment for Code Preference Judging
May 19, 2026Zhenyu Li, Aleksandar Cvejic, Zehui Chen +1Pairwise Preference Learning Human Preference Evaluation
KAUST · USTC
PRISM: A Benchmark for Programmatic Spatial-Temporal Reasoning
May 19, 2026Qiran Zhang, Yuheng Wang, Runde Yang +9Spatial Reasoning Code Generation Benchmarks
School of Artificial Intelligence, Shanghai Jiao Tong University
Rule2DRC: Benchmarking LLM Agents for DRC Script Synthesis with Execution-Guided Test Generation
May 15, 2026Jinuk Kim, Junsoo Byun, Donghwi Hwang +2Electronic Design Automation LLM Agent Evaluation
Department of Computer Science and Engineering, Seoul National University · Neural Processing Research Center · Samsung Electronics Co., Ltd.
PerfCodeBench: Benchmarking LLMs for System-Level High-Performance Code Optimization
May 13, 2026Huihao Jing, Wenbin Hu, Shaojin Chen +5Benchmark Design High-Performance Computing
HKUST · NYU · SWUPL
ShapeCodeBench: A Renewable Benchmark for Perception-to-Program Reconstruction of Synthetic Shape Scenes
May 12, 2026Shivam KumarSynthetic Benchmark Generation Code Generation Benchmarks
Independent Researcher
Vision2Code: A Multi-Domain Benchmark for Evaluating Image-to-Code Generation
May 11, 2026Ajay Vikram Periasami, Junlin Wang, Bhuwan DhingraVLM Evaluation Code Generation
Duke University
BenchCAD: A Comprehensive, Industry-Standard Benchmark for Programmatic CAD
May 11, 2026Haozhe Zhang, Kaichen Liu, Miaomiao Chen +4Benchmark Construction Parametric CAD Modeling
University of Virginia · University of California, San Diego · Rice University
PDEAgent-Bench: A Multi-Metric, Multi-Library Benchmark for PDE Solver Generation
May 10, 2026Zhen Hang, Yushan Yashengjiang, Junhui Li +21PDE Solving Benchmark Design
University of Science and Technology of China · Beijing University of Posts and Telecommunications · Tencent +10
VeriContest: A Competitive-Programming Benchmark for Verifiable Code Generation
May 8, 2026Zichen Xie, Mrigank Pawagi, Yuxin Liu +5Code Generation Formal Verification
University of Virginia · 2Indian Institute of Science · 3Rice University +1
Mage: Multi-Axis Evaluation of LLM-Generated Executable Game Scenes Beyond Compile-Pass Rate
May 8, 2026Hugh Xuechen Liu, Kıvanç TatarCode Generation Code Generation Evaluation
Chalmers University of Technology and University of Gothenburg
Constraint Decay: The Fragility of LLM Agents in Backend Code Generation
May 7, 2026Francesco Dente, Dario Satriani, Paolo PapottiAI Coding Agents Code Generation
EURECOM, France · University of Basilicata, Italy
BUILD-AND-FIND: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases
May 7, 2026Jhen-Ke LinBenchmark Construction LLM Agent Evaluation
National Yang Ming Chiao Tung University
KernelBenchX: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernels
May 6, 2026Han Wang, Jintao Zhang, Kai Jiang +3GPU Kernel Optimization GPU Acceleration
∗Equal contribution.
MolViBench: Evaluating LLMs on Molecular Vibe Coding
May 4, 2026Jiatong Li, Yuxuan Ren, Weida Wang +4AI-Assisted Scientific Research Scientific Code Generation
The Hong Kong Polytechnic University · National University of Singapore · Fudan University
RuC: HDL-Agnostic Rule Completion Benchmark Generation
Apr 30, 2026Arnau Ayguadé Domingo, Miquel Alberti-Binimelis, Cristian Gutierrez-Gomez +5LLM Evaluation LLM Prompting
Barcelona Supercomputing Center · Universitat Polit`ecnica de Catalunya
ClassEval-Pro: A Cross-Domain Benchmark for Class-Level Code Generation
Apr 29, 2026Yeheng Chen, Chaoxiang Xie, Yuling Shi +4Code Generation Synthetic Benchmark Generation
Shanghai Jiao Tong University China
Context-Augmented Code Generation: How Product Context Improves AI Coding Agent Decision Compliance by 49%
Apr 27, 2026Drew Dillon, Kasyap VaranasiRetrieval-Augmented Generation AI Coding Agents
Brief
BLAST: Benchmarking LLMs with ASP-based Structured Testing
Apr 24, 2026Manuel Alejandro Borroto Santana, Erica Coppolillo, Francesco Calimeri +3Logic Programming LLM Evaluation
University of Calabria, Rende, Italy · ICAR-CNR, Rende, Italy · Artificial Intelligence LAB, DeMaCS, University of Calabria, Rende, Italy +1
WebCompass: Towards Multimodal Web Coding Evaluation for Code Language Models
Apr 20, 2026Xinping Lei, Xinyu Che, Junqi Xiong +16LLM Agent Evaluation Code Language Models
Nanjing University · Kuaishou Technology
Playing Psychic: Using Thought Trees to Predict Reasoning Models Accuracy on Coding Tasks
Apr 18, 2026Jiaxin Fang, Runyuan He, Sahil Bhatia +2Structured Reasoning Code Generation Evaluation
University of California, Berkeley
ACES: Who Tests the Tests? Leave-One-Out AUC Consistency for Code Generation
Apr 5, 2026Hui Sun, Yun-Ji Zhang, Zheng Xie +4Language Model Generation Evaluation Code Generation Evaluation
National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China
VectorGym: A Multi-Task Benchmark for SVG Code Generation, Sketching and Editing
Feb 22, 2026Joan Rodriguez, Haotian Zhang, Abhay Puri +14VLM Evaluation Code Generation
Mila, Quebec AI Institute · ÉTS Montréal · QuiverAI +11
1D-Bench: A Benchmark for Iterative UI Code Generation with Visual Feedback in Real-World
Feb 20, 2026Qiao Xu, Yipeng Yu, Chengxiao Feng +1Benchmark Design Execution-Guided Code Generation
Taobao & Tmall Group of Alibaba
HardSecBench: Benchmarking the Security Awareness of LLMs for Hardware Code Generation
Jan 20, 2026Qirui Chen, Jingxian Shuai, Shuangwu Chen +8LLM Security Secure Code Generation
University of Science and Technology of China, China · Xi’an Jiaotong University, China · Nanjing University, China +1
Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks
Dec 2, 2025Songwen Zhao, Danqing Wang, Kexun Zhang +3AI Coding Agents Software Security
Carnegie Mellon University, Language Technologies Institute · Columbia University · Johns Hopkins University +1
LiveOIBench: Can Large Language Models Outperform Human Contestants in Informatics Olympiads?
Oct 10, 2025Kaijian Zou, Aaron Xiong, Yunxiang Zhang +6LLM Evaluation Competitive Programming
University of Michigan, Ann Arbor.
Benchmarking Web API Integration Code Generation
Sep 24, 2025Daniel Maninger, Leon Chemnitz, Amir Molzam Sharifloo +2Code Generation Code Generation Evaluation
Technische Universität Darmstadt, Germany · Hessian Center for Artificial Intelligence (hessian.AI), Germany · Pariton AI, Germany +1
BigO(Bench): Can LLMs Generate Code with Controlled Time and Space Complexity?
Mar 19, 2025Pierre Chambon, Baptiste Roziere, Benoit Sagot +1Benchmark Design Code Generation
FAIR at Meta · Inria · Work done at Meta, now working at Mistral AI
An Agentic Evaluation Framework for AI-Generated Scientific Code in PETSc
Date pendingHong Zhang, Barry Smith, Satish Balay +4High-Performance Computing Code Generation
Argonne National Laboratory, Lemont, IL, USA
PhysCodeBench: Benchmarking Physics-Aware Symbolic Simulation of 3D Scenes via Self-Corrective Multi-Agent Refinement
Date pendingTianyidan Xie, Peiyu Wang, Hu Jiaxin +9LLM-Based Program Synthesis Scientific Code Generation
Nanjing University · Skywork AI · Jilin University +2

Jun 7, 2026, cs.AIJ/K move · Enter open · S save

Beyond Pass Rate: A Multilingual, Execution-Grounded Evaluation of Open Code LLMs

Sayed Erfan Arefin

Department of Computer Science University of Dayton Dayton, OH 45469, USA

Code Generation Evaluation Code Language Models Code Generation Benchmarks Multilingual Code Generation

Open paper AskPDF arXiv

Code generation models are typically compared using compact execution benchmarks and aggregate pass rates, but such summaries obscure how performance varies across programming languages, problem families, and failure modes. We present a large-scale, execution-grounded evaluation of 9 openly accessible LLMs specialized for coding on 2,707 free LeetCode problems across 12 programming languages. Our corpus contains 325,343 problem-model-language jobs, each linked to prompt metadata, extracted code, LeetCode execution outcomes, and static-analysis signals. The results show that current open models remain far from the human acceptance reference: the best model, Yi-Coder-9B-Chat, reaches 23.64% mean correctness, compared with a 57.2% human acceptance baseline. Rankings are also slice-dependent: Qwen2.5-Coder-14B-Instruct is strongest on hard problems and distinct-problem coverage, while Gemma-2-27B-IT achieves the highest all-language lint pass rate. Failure analysis shows that compile errors account for 63.25% of non-accepted best submissions, indicating that many failures occur before semantic correctness can be tested. Static quality further diverges from functional correctness. Together, these findings show that multilingual, artifact-preserving evaluation reveals tradeoffs hidden by single-language or single-metric leaderboards.

Momentum

Related topics

Latest papers 104