Code Generation Benchmarks

Follow this topic Ask about this topic

Code generation benchmarks test whether AI systems can turn instructions, examples, or visual inputs into working programs, from short algorithm problems to repository-scale changes. They use executable and hidden tests, static analysis, formal verification, browser or rendering checks, and sometimes human judgments to assess correctness, visual fidelity, security, efficiency, and maintainability. Current benchmarks cover competitive programming, software agents, scientific and hardware code, and multimodal tasks such as generating interactive webpages or 3D scenes. A central challenge is that passing tests or compiling does not guarantee that code meets its intended behavior or quality requirements, while benchmark coverage, test reliability, and contamination can also distort measured capability.

Latest papers

154 papers24 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

Beyond Output Correctness: Benchmarking and Evaluating Large Language Model Reasoning in Coding Tasks
Apr 14, 2026Yuangang Li, Justin Tian Jin Chen, Ethan Yu +2Language Model Reasoning Evaluation LLM Evaluation
University of California, Irvine
DuET: Dual Execution for Test Output Prediction with Generated Code and Pseudocode
Apr 13, 2026Hojae Han, Jaejin Kim, Seung-won Hwang +2Execution-Guided Code Generation Code Generation Benchmarks
ETRI · Seoul National University · SNU-LG AI Research Center +1
Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios
Apr 8, 2026Ruida Hu, Xinchen Wang, Chao Peng +2Code Generation Code Generation Evaluation
Harbin Institute of Technology, Shenzhen, China · Independent Researcher, China · Singapore Management University, Singapore
ACES: Who Tests the Tests? Leave-One-Out AUC Consistency for Code Generation
Apr 5, 2026Hui Sun, Yun-Ji Zhang, Zheng Xie +4Language Model Generation Evaluation Code Generation Evaluation
National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China
IndustryCode: A Benchmark for Industry Code Generation
Apr 3, 2026Puyu Zeng, Zhaoxi Wang, Zhixu Duan +7Code Generation Code Generation Benchmarks
Shanghai Jiao Tong University, Shanghai, China · Alibaba Group, Hangzhou, China · Tsinghua University, Beijing, China
VoxelCodeBench: Benchmarking 3D World Modeling Through Code Generation
Apr 2, 2026Yan Zheng, Florian BordesCode Generation Benchmarks 3D Spatial Reasoning
FAIR at Meta · The University of Texas at Austin
Benchmarking PhD-Level Coding in 3D Geometric Computer Vision
Mar 31, 2026Wenyi Li, Renkai Luo, Yue Yu +5Code Generation Benchmarks Code Generation Evaluation
AIR, Tsinghua University · Qiuzhen College, Tsinghua University · BAAI +3
Needle in the Repo: A Benchmark for Maintainability in AI-Generated Repository Edits
Mar 29, 2026Haichao Zhu, Qian Zhang, Jiyuan Wang +2Software Engineering AI Coding Agents
Independent Researcher · UC Riverside · Tulane University
RealChart2Code: Advancing Chart-to-Code Generation with Real Data and Multi-Task Evaluation
Mar 26, 2026Jiajun Zhang, Yuying Li, Zhixun Li +13Chart-to-Code Generation Code Generation
USTC · CASIA · THU +6
A Judge Agent Closes the Reliability Gap in AI-Generated Scientific Simulation
Mar 26, 2026Chengshuai YangScientific Code Generation LLM Reliability
NextGen PlatformAI C Corp, USA
ReCUBE: Evaluating Repository-Level Context Utilization in Code Generation
Mar 26, 2026Jiseung Hong, Benjamin G. Ascoli, Jinho D. ChoiRepository-Level Code Generation Code Generation Evaluation
Language Technologies Institute, Carnegie Mellon University · Computer Science, Emory University
QuanBench+: A Unified Multi-Framework Benchmark for LLM-Based Quantum Code Generation
Mar 25, 2026Ali Slim, Haydar Hamieh, Jawad Kotaich +5Code Generation Benchmarks Code Generation Evaluation
American University of Beirut · King Abdullah University of Science and Technology
SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks
Mar 25, 2026Gabriel Orlanski, Devjeet Roy, Alexander Yun +7AI Coding Agents Coding Agent Benchmarks
University of Wisconsin–Madison · Washington State University · MIT
An Agentic Evaluation Framework for AI-Generated Scientific Code in PETSc
Mar 16, 2026Hong Zhang, Barry Smith, Satish Balay +4High-Performance Computing Code Generation
Argonne National Laboratory, Lemont, IL, USA
Coding with Eyes: Visual Feedback Unlocks Reliable GUI Code Generating and Debugging
Mar 14, 2026Zhilin Liu, Ye Huang, Ting Xie +3Code Generation Code Generation Benchmarks
Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China
CreativeBench: Benchmarking and Enhancing Machine Creativity via Self-Evolving Challenges
Mar 12, 2026Zi-Han Wang, Lam Nguyen, Zhengyang Zhao +4Computational Creativity Creativity Assessment
Southern University of Science and Technology · Tsinghua University · Xi’an Jiaotong University +3
Breaking Training Bottlenecks: Effective and Stable Reinforcement Learning for Coding Models
Mar 8, 2026Zongqian Li, Shaohan Huang, Zewen Chi +5RL for Code Generation LLM Code Generation
Microsoft Research · University of Cambridge · Princeton University
Vibe Code Bench: Evaluating AI Models on End-to-End Web Application Development
Mar 4, 2026Hung Tran, Langston Nashold, Rayan Krishnan +2Web Application Generation Code Generation Evaluation
Vals AI San Francisco, California, USA · Massachusetts Institute of Technology Cambridge, Massachusetts, USA
CodeTaste: Can LLMs Generate Human-Level Code Refactorings?
Mar 4, 2026Alex Thillen, Niels Mündler, Veselin Raychev +1Automated Code Refactoring Automated Software Engineering
Department of Computer Science, ETH Zurich · LogicStar.ai, Zurich, Switzerland
CONCUR: Benchmarking LLMs for Concurrent Code Generation
Mar 4, 2026Jue Huang, Tarek Mahmud, Corina Pasareanu +1Code Generation Evaluation Code Generation Benchmarks
The University of Queensland · Texas A&M University - Kingsville · Carnegie Mellon University
BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing?
Mar 3, 2026Guoxin Chen, Fanzhe Meng, Jiale Zhao +12Retrieval-Augmented Code Generation Software Engineering Benchmarks
Steerable Instruction Following Coding Data Synthesis with Actor-Parametric Schema Co-Evolution
Feb 27, 2026Tinglin Huang, Bo Chen, Xiao Zhang +2Synthetic Data Generation for Language Models Code Generation
Yale University · ByteDance Seed
ManiBench: A Benchmark for Testing Visual-Logic Drift and Syntactic Hallucinations in Manim Code Generation
Feb 24, 2026Nabin OliCitation Hallucination Code Generation
Sunway College Kathmandu, Birmingham City University
VectorGym: A Multi-Task Benchmark for SVG Code Generation, Sketching and Editing
Feb 22, 2026Joan Rodriguez, Haotian Zhang, Abhay Puri +14VLM Evaluation Code Generation
Mila, Quebec AI Institute · ÉTS Montréal · QuiverAI +11
1D-Bench: A Benchmark for Iterative UI Code Generation with Visual Feedback in Real-World
Feb 20, 2026Qiao Xu, Yipeng Yu, Chengxiao Feng +1Benchmark Design Execution-Guided Code Generation
Taobao & Tmall Group of Alibaba
SecCodeBench-V2 Technical Report
Feb 17, 2026Longfei Chen, Ji Zhao, Lanxiao Cui +24Secure Code Generation Code Generation Benchmarks
TSINGHUA UNIVERSITY, BEIJING, CHINA · ALIBABA GROUP, HANGZHOU, CHINA
HardSecBench: Benchmarking the Security Awareness of LLMs for Hardware Code Generation
Jan 20, 2026Qirui Chen, Jingxian Shuai, Shuangwu Chen +8LLM Security Secure Code Generation
University of Science and Technology of China, China · Xi’an Jiaotong University, China · Nanjing University, China +1
Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks
Dec 2, 2025Songwen Zhao, Danqing Wang, Kexun Zhang +3AI Coding Agents Software Security
Carnegie Mellon University, Language Technologies Institute · Columbia University · Johns Hopkins University +1
LiveOIBench: Can Large Language Models Outperform Human Contestants in Informatics Olympiads?
Oct 10, 2025Kaijian Zou, Aaron Xiong, Yunxiang Zhang +6LLM Evaluation Competitive Programming
University of Michigan, Ann Arbor.
Benchmarking Web API Integration Code Generation
Sep 24, 2025Daniel Maninger, Leon Chemnitz, Amir Molzam Sharifloo +2Code Generation Code Generation Evaluation
Technische Universität Darmstadt, Germany · Hessian Center for Artificial Intelligence (hessian.AI), Germany · Pariton AI, Germany +1

Apr 14, 2026J/K move · Enter open · S save

Beyond Output Correctness: Benchmarking and Evaluating Large Language Model Reasoning in Coding Tasks

Yuangang Li, Justin Tian Jin Chen, Ethan Yu, David Hong+1

University of California, Irvine

Language Model Reasoning Evaluation LLM Evaluation LLM Reasoning Code Generation Evaluation Code Generation Benchmarks

Open paper AskPDF arXiv

Large language models (LLMs) increasingly rely on explicit reasoning to solve coding tasks, yet evaluating the quality of this reasoning remains challenging. Existing reasoning evaluators are not designed for coding, and current benchmarks focus primarily on code generation, leaving other coding tasks largely unexplored. We introduce CodeRQ-Bench, the first benchmark for evaluating LLM reasoning quality across three coding task categories: generation, summarization, and classification. Using this benchmark, we analyze 1,069 mismatch cases from existing evaluators, identify five recurring limitations, and derive four design insights for reasoning evaluation in coding tasks. Guided by these insights, we propose VERA, a two-stage evaluator that combines evidence-grounded verification with ambiguity-aware score correction. Experiments on CodeRQ-Bench show that VERA consistently outperforms strong baselines across four datasets, improving AUCROC by up to 0.26 and AUPRC by up to 0.21. We release CodeRQ-Bench at https://github.com/MrLYG/CodeRQ-Bench, supporting future investigations.