Code Generation Benchmarks

Follow this topic Ask about this topic

Code generation benchmarks test whether AI systems can turn instructions, examples, or visual inputs into working programs, from short algorithm problems to repository-scale changes. They use executable and hidden tests, static analysis, formal verification, browser or rendering checks, and sometimes human judgments to assess correctness, visual fidelity, security, efficiency, and maintainability. Current benchmarks cover competitive programming, software agents, scientific and hardware code, and multimodal tasks such as generating interactive webpages or 3D scenes. A central challenge is that passing tests or compiling does not guarantee that code meets its intended behavior or quality requirements, while benchmark coverage, test reliability, and contamination can also distort measured capability.

Latest papers

154 papers24 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

How Stable Are LLM-Generated Proofs Under Program Refactoring? A Stress Test for Verifiable Code Generation
Oct 26, 2026Aravind Mandiga, Guoming LiCode Generation Evaluation Formal Verification
University of Georgia, Athens, USA
GenXR-Bench: Benchmarking Coding Agents for Interactive XR Development
Oct 21, 2026William Huang, David Li, Xiang 'Anthony' Chen +13Code Generation Benchmarks Coding Agents
University of California, Los Angeles · Google XR Labs · Carnegie Mellon University +5
VHDLBench: Benchmarking and Fine-Tuning Large Language Models for VHDL-Based Chip Design
Oct 19, 2026Patrik Drazic, Benaoumeur Senouci, Boualem BenatallahElectronic Design Automation Code Generation
University of Southern Denmark, Sønderborg, Denmark · RINN AI-Research Ireland Centre for AI and Data Science, Dublin City University Dublin, Ireland
SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark
Oct 16, 2026Soham Dan, Jeff Da, Sanyam Bhutani +8Coding Agent Benchmarks Software Engineering Benchmarks
Scale AI · Reflection AI
Between Spoilers and Guesswork: Avoiding Over-Specification in Coding Benchmarks
Oct 16, 2026Yuanzhe Liu, Bastian Lipka, Avraham Shinnar +3Benchmark Design Software Engineering Benchmarks
Rensselaer Polytechnic Institute · IBM
SWE-Nexus: Automated Composition of Software Engineering Tasks from Test Execution
Oct 14, 2026Songwei Yu, Ziqi Miao, Jingxuan Xu +10Software Engineering Benchmarks Code Generation Benchmarks
National University of Singapore, Singapore · Shanghai Artiﬁcial Intelligence Laboratory, Shanghai, China · Independent Researchers +1
DivAlgo: A systematic evaluation of algorithmic diversity for code generation models
Oct 13, 2026Anubhav Jangra, Smaranda MuresanCode Language Models Code Generation Evaluation
Department of Computer Science Columbia University, USA · Barnard College Columbia University, USA
SolBehave: Evaluating the Behavioural Correctness of LLM-Generated Smart Contracts
Oct 13, 2026Giovanni Pio Delvecchio, Stefano Fantazzini, Giacomo Frisoni +4Code Generation Evaluation LLM Code Generation
University of Bologna, Bologna, Italy · University of Copenhagen, Copenhagen, Denmark
EvoWeb: A Benchmark for Coding Agents on Evolving Web Applications
Oct 12, 2026David Nukrai, Bar Alon, Nativ Levy +6Code Generation Benchmarks Web Application Generation
Stealth Startup, Tel Aviv · Tel Aviv University
CodeWalk: Generating Coding Benchmarks by Walking a Problem Graph
Oct 11, 2026Jiaxin Fang, Runyuan He, Arnav Gupta +3Synthetic Benchmark Generation Benchmark Design
University of California, Berkeley
Last Secure Code Benchmark: Is the Code You Vibe Secure?
Oct 10, 2026Tianyi Wu, Chao Ge, Peng Chen +12Secure Code Generation AI Agent Security Benchmarks
National University of Singapore · Qwen Team, Alibaba Token Hub, Alibaba Group · Cloud Safety and Security, Alibaba Cloud, Alibaba Group
TaoD2C-Bench: Benchmarking MLLMs for Industrial UI Code Generation Beyond Visual Fidelity
Oct 7, 2026Chengwei Shi, Yunnong Chen, Tingting Zhou +5Image-to-Code Generation Code Generation
Zhejiang University · Taobao & Tmall Group of Alibaba
From Pixel to Coding: Evaluating the Figure Reproduction Capabilities of MLLMs
Oct 7, 2026Zijian Chen, Zhengyu Chen, Bohan Liang +7Multimodal Large Language Models Multimodal Model Evaluation
Institute of Image Communication and Information Processing, Shanghai Jiao Tong University, Shanghai, 200240, China. · Shanghai Artificial Intelligence Laboratory, Shanghai, 200030, China. · Macao Polytechnic University, Macao, 999078, China.
Large-scale Repository Engineering via Agent-Native Reusable Code Primitives
Oct 6, 2026Haibo Jin, Peng Kuang, Xucheng Yu +3Repository-Level Code Generation Software Engineering
University of Illinois Urbana-Champaign, USA
CheckerBench: Can Long-Horizon Agents Synthesize Static-Analysis Checkers?
Oct 6, 2026Hang He, Li Wang, Hao Chen +9Software Engineering Agents Static Code Analysis
East China Normal University, Humanlaya Data, Shanghai Jiao Tong University, Peking University, Shanghai Innovation Institute
GNN-CB: A Graph Neural Network Competition Benchmark for Human and LLM Evaluation
Oct 4, 2026Murad Hossen, Tasneem Selim, Gurur Gamgam +22LLM Evaluation Graph Neural Networks
University of Houston, Texas,USA · BASIRA Lab, Department of Computing, Imperial College London, United Kingdom · Department of Mathematics and Computer Science, Faculty of Science, Alexandria University, Alexandria, Egypt +23
VHDL-REPOBENCH: A Repository-Level Benchmark for Evaluating Large Language Models on VHDL Design Generation
Oct 4, 2026Prashanth Vijayaraghavan, Akul Malhotra, Ashutosh Jadhav +2LLM Evaluation Code Generation
IBM Research San Jose, CA, USA
Spec2Game: Can LLMs Generate Complete Playable Games from Detailed Specifications?
Oct 3, 2026Yixue Cai, Yuzhe Zhao, Hanxiang Chao +4Code Generation LLM Code Generation
The Chinese University of Hong Kong Hong Kong, China · Nankai University Tianjin, China · Wuhan University Wuhan, China
Are you Synthesizing or Recalling? Evaluating LLMs on Algorithmic Code Retrieval
Oct 1, 2026Nickil Maveli, Antonio Vergari, Shay B. CohenCode Generation Evaluation Code Generation Benchmarks
School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh, EH8 9AB
Self-Spec Verifiable Code Generation
Sep 30, 2026Jiaru Qian, Yihong Dong, Yongmin Li +3LLM Self-Correction Code Generation
School of Computer Science, Peking University · Beijing Key Laboratory of Trustworthy Code Large Language Models · Key Laboratory of High Confidence Software Technologies, Peking University, Ministry of Education +3
E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch
Sep 29, 2026Hantian Ding, Chloe Bi, Jiacheng Zhu +5Software Engineering Agents Repository-Level Code Generation
Meta Superintelligence Labs
Zero2Repo: Can Coding Agents Build Repositories from Scratch?
Sep 29, 2026Pei Yang, Tianyu Shi, Yuhang Yao +23Coding Agents AI Agent Benchmarks
Gradient Data · McGill University · Carnegie Mellon University +6
LoLBench: Evaluating Coding Agents with Long-Horizon Proposals on Large Software Systems
Sep 29, 2026Yun Peng, Zihan Wu, Zeyang Zhuang +6Software Engineering AI Coding Agents
Fudan University, China · City University of Hong Kong, Hong Kong · Chinese University of Hong Kong, Hong Kong +5
GPUPhysBench: Benchmarking Coding Agents for Correct and Efficient GPU Physics Simulation
Sep 28, 2026Yuchen Sun, Jinjin He, Sinan Wang +1GPU Kernel Optimization Benchmark Design
Georgia Institute of Technology
Do Coding Agents Reuse Existing Code or Reinvent the Wheel?
Sep 28, 2026Dongsheng Ma, Sizhe Wang, Xinyi Huang +5Coding Agents Code Generation
Peking University · Fudan University · Shanghai Jiao Tong University +3
MetaBench-Harness: Unlocking End-to-End Optimization of Benchmark Harnesses
Sep 27, 2026Xuanjun Chen, Hua-Hsuan Chen, Wei-Chung Lu +3LLM Evaluation Benchmark Design
National Taiwan University · NTU AI-Core · Queen Mary University of London
Compact Documentation for Coding Agents: A Benchmark, an Optimizer, and Why It Does Not Transfer
Sep 25, 2026Md Shohel Arman, Igor MolybogSoftware Engineering AI Coding Agents
Daffodil International University, Dhaka, Bangladesh · HawAII, University of Hawai‘i at M¯anoa, Honolulu, HI, USA
CraftBench-UE: Deterministic Evaluation for Coding Agents in Unreal Engine
Sep 19, 2026Shutong Wu, Kevin Calderone, Andy TsenCoding Agents AI Agent Evaluation
University of Rochester · RamenVR
ISA-Bench: A Benchmark for Computational Reasoning Across Instruction Set Architectures
Sep 19, 2026Aditya Pola, Arkaprava Majumdar, Vineeth N. BalasubramanianCode Generation Benchmarks LLM-Based Program Synthesis
IIT Hyderabad · Microsoft Research, India
GameLogicBench: Evaluating Coding Agents on Runtime Game Logic with Tick-Level State Assertions
Sep 18, 2026Xinyu Che, Yunfei Ge, Shihao Li +9AI Coding Agents Benchmark Design
Nanjing University

Oct 26, 2026J/K move · Enter open · S save

How Stable Are LLM-Generated Proofs Under Program Refactoring? A Stress Test for Verifiable Code Generation

Aravind Mandiga, Guoming Li

University of Georgia, Athens, USA

Code Generation Evaluation Formal Verification Code Generation Benchmarks Lean Theorem Proving

Open paper AskPDF arXiv

Benchmarks for verifiable code generation usually test a proof against one fixed implementation. We ask whether the same proof remains valid after a program refactoring that preserves behavior. We introduce RefactorProof, which derives 458 Lean-certified variants from the 185 VERINA tasks that compile in our pinned environment while holding the specification and evaluated proof fixed byte-for-byte. VERINA supplies reference proofs for 46 tasks, all from its basic split; the primary model-versus-reference comparisons further match on 21-25 short, non-recursive tasks. Structural refactorings expose substantial brittleness across proof sources: task-normalized structural PSR is 4.3% for the supplied references and 13.3-26.8% for the specialized provers. On the matched tasks, specialized prover proofs nevertheless have higher overall unchanged-proof survival than the supplied reference artifacts: +15.6 points for DeepSeek-Prover-V2-7B, +18.8 for Goedel-Prover-V2-32B, and +20.3 for Goedel-Prover-V2-8B; all three 95% CIs exclude zero. The ordering persists after excluding universally survived T1 variants, under family-balanced aggregation, and under alternative deployable valid-proof selection rules. However, unchanged-proof failure is not the same as high repair cost: predefined one-identifier edits repair 51/54 eligible reference structural failures and 58/85 eligible prover failures. RefactorProof therefore measures unchanged-proof stability under program refactoring, not the cost of maintaining a proof after it breaks.