Code Generation Evaluation

Ask about this topic

Overview

Code generation evaluation studies how well AI systems produce programs that work and satisfy the requirements developers intended. Common approaches include executable unit tests and pass@k benchmarks, repository-level issue resolution, and tests of generated applications through browsers or other interactive environments. Researchers also assess qualities beyond passing tests, such as security, maintainability, performance, instruction adherence, and fidelity to domain-specific behavior. A central challenge is that test success can overstate real-world reliability, while stochastic outputs, benchmark contamination, and imperfect automated judges make comparisons difficult.

Latest papers

CardsList

215 papers31 papers in the last four weeks, up 63% on the four weeks before. 0.2% of all new papers.

  1. Beyond Functional Correctness: Design Issues in AI IDE-Generated Large-Scale Projects

    Apr 7, 2026Syed Mohammad Kashif, Ruiyin Li, Peng Liang +4Software EngineeringAI Coding Agents

    School of Computer Science, Wuhan University, China · School of Mathematical and Computational Sciences, Massey University, New Zealand · School of Computer Science, Nanjing University of Science and Technology, China +2

  2. From Paper to Program: Specification-Guided Implementation of Quantum Many-Body Algorithms with Large Language Model

    Apr 5, 2026Yi ZhouScientific Code GenerationCode Generation Evaluation

    Institute of Physics, Chinese Academy of Sciences, Beijing 100190, China

  3. ACES: Who Tests the Tests? Leave-One-Out AUC Consistency for Code Generation

    Apr 5, 2026Hui Sun, Yun-Ji Zhang, Zheng Xie +4Language Model Generation EvaluationCode Generation Evaluation

    National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China

  4. Beyond Isolated Tasks: A Framework for Evaluating Coding Agents on Sequential Software Evolution

    Apr 3, 2026KN Ajay Shastry, Ganesh Senrayan, Shrey Satapara +2Code Generation EvaluationSoftware Engineering

    Fujitsu Research

  5. Benchmarking PhD-Level Coding in 3D Geometric Computer Vision

    Mar 31, 2026Wenyi Li, Renkai Luo, Yue Yu +5Code Generation BenchmarksCode Generation Evaluation

    AIR, Tsinghua University · Qiuzhen College, Tsinghua University · BAAI +3

  6. Factors Influencing the Quality of AI-Generated Code: A Synthesis of Empirical Evidence

    Mar 26, 2026Vehid Geruslu, Zulfiyya Aliyeva, Eray TüzünAI-Assisted Software DevelopmentCode Generation Evaluation

    Queen’s University Belfast, Belfast, United Kingdom · Azerbaijan Technical University, Baku, Azerbaijan · MilliSoft MMC Corporation, Baku, Azerbaijan +1

  7. ReCUBE: Evaluating Repository-Level Context Utilization in Code Generation

    Mar 26, 2026Jiseung Hong, Benjamin G. Ascoli, Jinho D. ChoiRepository-Level Code GenerationCode Generation Evaluation

    Language Technologies Institute, Carnegie Mellon University · Computer Science, Emory University

  8. QuanBench+: A Unified Multi-Framework Benchmark for LLM-Based Quantum Code Generation

    Mar 25, 2026Ali Slim, Haydar Hamieh, Jawad Kotaich +5Code Generation BenchmarksCode Generation Evaluation

    American University of Beirut · King Abdullah University of Science and Technology

  9. Comparing Developer and LLM Biases in Code Evaluation

    Mar 25, 2026Aditya Mittal, Ryan Shar, Zichu Wu +6Bias in LLM-as-a-JudgeLanguage Model Bias Evaluation

    Carnegie Mellon University

  10. Coherence Collapse: Diagnosing Why Code Agents Fail After Reaching the Right Code

    Mar 25, 2026Myeongsoo Kim, Dingmin Wang, Siwei Cui +6Agent Failure AnalysisAI Coding Agents

    AWS AI Labs · Monash University

  11. Evaluating LLM-Based Test Generation Under Software Evolution

    Mar 24, 2026Sabaat Haroon, Mohammad Taha Khan, Muhammad Ali GulzarAutomated Test GenerationAutomated Software Testing

    Virginia Tech, USA · Carnegie Mellon University, USA

  12. Customized User Plane Processing via Code Generating AI Agents for Next Generation Mobile Networks

    Mar 24, 2026Xiaowen Ma, Onur Ayan, Yunpu Ma +1Code GenerationAgentic Code Generation

    Huawei Technologies Duesseldorf GmbH · Ludwig-Maximilians-University Munich

  13. An Agentic Evaluation Framework for AI-Generated Scientific Code in PETSc

    Mar 16, 2026Hong Zhang, Barry Smith, Satish Balay +4High-Performance ComputingCode Generation

    Argonne National Laboratory, Lemont, IL, USA

  14. Breaking Training Bottlenecks: Effective and Stable Reinforcement Learning for Coding Models

    Mar 8, 2026Zongqian Li, Shaohan Huang, Zewen Chi +5RL for Code GenerationLLM Code Generation

    Microsoft Research · University of Cambridge · Princeton University

  15. Vibe Code Bench: Evaluating AI Models on End-to-End Web Application Development

    Mar 4, 2026Hung Tran, Langston Nashold, Rayan Krishnan +2Web Application GenerationCode Generation Evaluation

    Vals AI San Francisco, California, USA · Massachusetts Institute of Technology Cambridge, Massachusetts, USA

  16. SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration

    Mar 4, 2026Jialong Chen, Xander Xu, Hu Wei +2Software EngineeringSoftware Engineering Benchmarks

    Sun Yat-sen University · Skylenage · Alibaba Group

  17. CONCUR: Benchmarking LLMs for Concurrent Code Generation

    Mar 4, 2026Jue Huang, Tarek Mahmud, Corina Pasareanu +1Code Generation EvaluationCode Generation Benchmarks

    The University of Queensland · Texas A&M University - Kingsville · Carnegie Mellon University

  18. NERFIFY: A Multi-Agent Framework for Turning NeRF Papers into Code

    Feb 28, 2026Seemandhar Jain, Keshav Gupta, Kunal Gupta +1Agentic Code GenerationLLM-Based Program Synthesis

    University of California, San Diego

  19. ManiBench: A Benchmark for Testing Visual-Logic Drift and Syntactic Hallucinations in Manim Code Generation

    Feb 24, 2026Nabin OliCitation HallucinationCode Generation

    Sunway College Kathmandu, Birmingham City University

  20. From Prompts to Performance: Evaluating LLMs for Task-based Parallel Code Generation

    Feb 24, 2026Linus Bantel, Moritz Strack, Alexander Strack +1Code GenerationLLM Code Generation

    Institute for Parallel and Distributed Systems, University of Stuttgart, Stuttgart, Germany

  21. ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads?

    Feb 23, 2026Ayush Nangia, Shikhar Mishra, Aman Gokrani +1Coding Agent BenchmarksAI Coding Agents

  22. Operational Robustness of LLMs on Code Generation

    Feb 21, 2026Debalina Ghosh Paul, Hong Zhu, Ian BayleyLanguage Model RobustnessLLM Code Generation

    School of Engineering, Computing and Mathematics, Oxford Brookes University Oxford, OX3 0BP, United Kingdom

  23. Simple Baselines are Competitive with Code Evolution

    Feb 18, 2026Yonatan Gideoni, Sebastian Risi, Yarin GalLLMs for Evolutionary OptimizationCode Generation Evaluation

    University of Oxford · Sakana AI and ITU Copenhagen

  24. A Theoretical Analysis of Test-Driven Code Generation

    Feb 5, 2026Nicolas Menet, Michael Hersche, Andreas Krause +1Execution-Guided Code GenerationCode Generation

  25. Declarative by Design, Assistable Only by Convention: Benchmarking Multi-Agent Frameworks for AI-Assistability

    Feb 3, 2026Shafiuddin Rehan Ahmed, Sourabh DeshpandeAI Agent BenchmarksCode Generation Evaluation

    Center for Advanced AI, Accenture · Mountain View, California, USA

  26. Benchmarking Code LLM-as-a-Judge via Constrained Perturbation and Calibration

    Dec 23, 2025Ruiqi Wang, Xinchen Wang, Cuiyun Gao +3LLM-as-a-JudgeCode Generation Evaluation

    Harbin Institute of Technology, Shenzhen, China · Monash University Malaysia, Malaysia · Zhejiang University, China

  27. Can Coding Agents Migrate to Post-Quantum Cryptography?

    Dec 15, 2025Abdulmalik AlquwayfiliCoding AgentsCode Generation Evaluation

    Saudi Data & AI Authority, Riyadh, Saudi Arabia

  28. From Charts to Code: A Hierarchical Benchmark for Multimodal Models

    Oct 20, 2025Jiahao Tang, Henry Hengyuan Zhao, Lijian Wu +8Data VisualizationChart-to-Code Generation

    CSU-JPG, Central South University · National University of Singapore · Nanyang Technological University

  29. LiveOIBench: Can Large Language Models Outperform Human Contestants in Informatics Olympiads?

    Oct 10, 2025Kaijian Zou, Aaron Xiong, Yunxiang Zhang +6LLM EvaluationCompetitive Programming

    University of Michigan, Ann Arbor.

  30. Investigating The Smells of LLM Generated Code

    Oct 3, 2025Debalina Ghosh Paul, Hong Zhu, Ian BayleySoftware EngineeringCode Generation Evaluation

    School of Engineering, Computing and Mathematics, Oxford Brookes University, Oxford, OX3 0BP, UK