Code Generation Benchmarks

Overview

Code generation benchmarks test whether AI systems can turn instructions, examples, or visual inputs into working programs, from short algorithm problems to repository-scale changes. They use executable and hidden tests, static analysis, formal verification, browser or rendering checks, and sometimes human judgments to assess correctness, visual fidelity, security, efficiency, and maintainability. Current benchmarks cover competitive programming, software agents, scientific and hardware code, and multimodal tasks such as generating interactive webpages or 3D scenes. A central challenge is that passing tests or compiling does not guarantee that code meets its intended behavior or quality requirements, while benchmark coverage, test reliability, and contamination can also distort measured capability.

Latest papers

CardsList

169 papers10 papers in the last four weeks, down 58% on the four weeks before. 0.1% of all new papers.

  1. PerfArena: Ranking LLMs by Code Performance

    Nov 13, 2026Rui Rua, Hailemariam Mersha, Joao Paulo Fernandes +1Code Generation EvaluationCode Generation Benchmarks

  2. OCTAVE: A Capability-Resolved Benchmark for End-to-End HTML5 Game Generation

    Nov 13, 2026Kehang Li, Xinlan Xu, Yulin Jin +10game generationCode Generation Benchmarks

  3. RealEmbedBench: Benchmarking LLMs on Real-World Application-Layer Embedded Software Tasks

    Nov 8, 2026Yujun Huang, Yuwei Zhang, Yi Zhou +3LLM EvaluationCode Generation Benchmarks

  4. AnimWeb-Bench: Benchmarking Animated Webpage Code Generation from Videos

    Nov 6, 2026Zhongyi Zhang, Jingyu Xiao, Shuqing Li +1Code Generation Benchmarks

  5. A.S.E 2.0: Understanding How Repository Context Impacts the Security of LLM-Generated Code

    Nov 4, 2026Jiongchi Yu, Keke Lian, Junqing Wang +5LLM Code GenerationLLM Security

  6. VisCodeBench: a new benchmark dataset for Text-to-Vis reflecting real-world practice

    Nov 2, 2026Hy Nguyen, Thomas James Elton, Ella Wilson +4LLM EvaluationCode Generation Benchmarks

  7. RiskyPy: Toward Model-Calibrated Prompt Generation for Joint Security and Functionality Evaluation of LLM-Generated Code

    Oct 29, 2026Zacharie Chenail-Larcher, Daniele Lunghi, Melissa Tessa +5Code Generation EvaluationCode Generation Benchmarks

  8. GenXR-Bench: Benchmarking Coding Agents for Interactive XR Development

    Oct 21, 2026William Huang, David Li, Xiang 'Anthony' Chen +13Code Generation BenchmarksCoding Agents

  9. VHDLBench: Benchmarking and Fine-Tuning Large Language Models for VHDL-Based Chip Design

    Oct 19, 2026Patrik Drazic, Benaoumeur Senouci, Boualem BenatallahElectronic Design AutomationCode Generation

  10. SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark

    Oct 16, 2026Soham Dan, Jeff Da, Sanyam Bhutani +8Coding Agent BenchmarksSoftware Engineering Benchmarks

  11. Between Spoilers and Guesswork: Avoiding Over-Specification in Coding Benchmarks

    Oct 16, 2026Yuanzhe Liu, Bastian Lipka, Avraham Shinnar +3Benchmark DesignSoftware Engineering Benchmarks

  12. SWE-Nexus: Automated Composition of Software Engineering Tasks from Test Execution

    Oct 14, 2026Songwei Yu, Ziqi Miao, Jingxuan Xu +10Software Engineering BenchmarksCode Generation Benchmarks

  13. DivAlgo: A systematic evaluation of algorithmic diversity for code generation models

    Oct 13, 2026Anubhav Jangra, Smaranda MuresanCode Language ModelsCode Generation Evaluation

  14. SolBehave: Evaluating the Behavioural Correctness of LLM-Generated Smart Contracts

    Oct 13, 2026Giovanni Pio Delvecchio, Stefano Fantazzini, Giacomo Frisoni +4Code Generation EvaluationLLM Code Generation

  15. EvoWeb: A Benchmark for Coding Agents on Evolving Web Applications

    Oct 12, 2026David Nukrai, Bar Alon, Nativ Levy +6Code Generation BenchmarksWeb Application Generation

  16. CodeWalk: Generating Coding Benchmarks by Walking a Problem Graph

    Oct 11, 2026Jiaxin Fang, Runyuan He, Arnav Gupta +3Synthetic Benchmark GenerationBenchmark Design

  17. Last Secure Code Benchmark: Is the Code You Vibe Secure?

    Oct 10, 2026Tianyi Wu, Chao Ge, Peng Chen +12Secure Code GenerationAI Agent Security Benchmarks

  18. TaoD2C-Bench: Benchmarking MLLMs for Industrial UI Code Generation Beyond Visual Fidelity

    Oct 7, 2026Chengwei Shi, Yunnong Chen, Tingting Zhou +5Image-to-Code GenerationCode Generation

  19. From Pixel to Coding: Evaluating the Figure Reproduction Capabilities of MLLMs

    Oct 7, 2026Zijian Chen, Zhengyu Chen, Bohan Liang +7Multimodal Large Language ModelsMultimodal Model Evaluation

  20. Large-scale Repository Engineering via Agent-Native Reusable Code Primitives

    Oct 6, 2026Haibo Jin, Peng Kuang, Xucheng Yu +3Repository-Level Code GenerationSoftware Engineering

  21. CheckerBench: Can Long-Horizon Agents Synthesize Static-Analysis Checkers?

    Oct 6, 2026Hang He, Li Wang, Hao Chen +9Software Engineering AgentsStatic Code Analysis

  22. GNN-CB: A Graph Neural Network Competition Benchmark for Human and LLM Evaluation

    Oct 4, 2026Murad Hossen, Tasneem Selim, Gurur Gamgam +22LLM EvaluationGraph Neural Networks

  23. VHDL-REPOBENCH: A Repository-Level Benchmark for Evaluating Large Language Models on VHDL Design Generation

    Oct 4, 2026Prashanth Vijayaraghavan, Akul Malhotra, Ashutosh Jadhav +2LLM EvaluationCode Generation

  24. Spec2Game: Can LLMs Generate Complete Playable Games from Detailed Specifications?

    Oct 3, 2026Yixue Cai, Yuzhe Zhao, Hanxiang Chao +4Code GenerationLLM Code Generation

  25. Are you Synthesizing or Recalling? Evaluating LLMs on Algorithmic Code Retrieval

    Oct 1, 2026Nickil Maveli, Antonio Vergari, Shay B. CohenCode Generation EvaluationCode Generation Benchmarks

  26. Self-Spec Verifiable Code Generation

    Sep 30, 2026Jiaru Qian, Yihong Dong, Yongmin Li +3LLM Self-CorrectionCode Generation

  27. Zero2Repo: Can Coding Agents Build Repositories from Scratch?

    Sep 29, 2026Pei Yang, Tianyu Shi, Yuhang Yao +23Coding AgentsAI Agent Benchmarks

  28. LoLBench: Evaluating Coding Agents with Long-Horizon Proposals on Large Software Systems

    Sep 29, 2026Yun Peng, Zihan Wu, Zeyang Zhuang +6Software EngineeringAI Coding Agents