Code Generation Evaluation

Ask about this topic

Overview

Code generation evaluation studies how well AI systems produce programs that work and satisfy the requirements developers intended. Common approaches include executable unit tests and pass@k benchmarks, repository-level issue resolution, and tests of generated applications through browsers or other interactive environments. Researchers also assess qualities beyond passing tests, such as security, maintainability, performance, instruction adherence, and fidelity to domain-specific behavior. A central challenge is that test success can overstate real-world reliability, while stochastic outputs, benchmark contamination, and imperfect automated judges make comparisons difficult.

Latest papers

CardsList

215 papers31 papers in the last four weeks, up 63% on the four weeks before. 0.2% of all new papers.

  1. How Stable Are LLM-Generated Proofs Under Program Refactoring? A Stress Test for Verifiable Code Generation

    Oct 26, 2026Aravind Mandiga, Guoming LiCode Generation EvaluationFormal Verification

    University of Georgia, Athens, USA

  2. Secure and Functional Code Generation with Self-Evaluation

    Oct 24, 2026Md Afif Al Mamun, Gias UddinSecure Code GenerationRetrieval-Augmented Code Generation

    University of Calgary, Calgary, Canada · York University, Toronto, Canada

  3. SWE-Clar: Can Coding Agents Clarify Incomplete Client Requirements and Deliver Greenfield Software?

    Oct 22, 2026Jialei Li, Dingxian Wang, Xiaoyan Zhao +11Coding AgentsSoftware Engineering Benchmarks

    University of Science and Technology of China · National University of Singapore · Upwork Inc.

  4. ProductWebBench: Evaluating Website Continuity Beyond Task Completion

    Oct 22, 2026Jiajun Wu, Jian Yang, Yaxin Du +9Software Engineering BenchmarksCode Generation Evaluation

    Beihang University · IQuest Research · Huawei Technologies Ltd.

  5. Recursive Self-Improvement on a Budget: Scoring Code Edits Before They Run

    Oct 21, 2026Erfan Darzi, Sree Bhargavi Balija, Sheng HeLLM Agent Self-ImprovementLLM Agent Workflow Optimization

    Harvard University · Massachusetts Institute of Technology · UC San Diego

  6. FrontierRefactor: Benchmarking Multi-Scale Performance-Oriented Codebase Refactoring

    Oct 19, 2026Shijian Wang, Fanqing Meng, Jiajun Chen +8Automated Code RefactoringSoftware Engineering Benchmarks

    Evolvent AI · National University of Singapore · University of Southern California +2

  7. When LLM-Generated Refactorings Ripple: An Empirical Study of Repository Adoption Readiness

    Oct 17, 2026Muhammad Waseem, Aakash Ahmad, Pekka AbrahamssonAutomated Code RefactoringLLM Reliability

    Tampere University, Finland. · the School of Computing and Engineering, University of Derby, UK.

  8. Poison Valleys: Diagnosing Path-Dependent Failure in LLM Code Repair

    Oct 15, 2026Veljko SkarichAgent Failure AnalysisLLM Code Generation

  9. SWE-Nexus: Automated Composition of Software Engineering Tasks from Test Execution

    Oct 14, 2026Songwei Yu, Ziqi Miao, Jingxuan Xu +10Software Engineering BenchmarksCode Generation Benchmarks

    National University of Singapore, Singapore · Shanghai Artificial Intelligence Laboratory, Shanghai, China · Independent Researchers +1

  10. DivAlgo: A systematic evaluation of algorithmic diversity for code generation models

    Oct 13, 2026Anubhav Jangra, Smaranda MuresanCode Language ModelsCode Generation Evaluation

    Department of Computer Science Columbia University, USA · Barnard College Columbia University, USA

  11. AesCode: Aesthetic Code Generation with Decoupled Cross-Modal Rewards

    Oct 13, 2026Xin Gao, Yifan Yang, Xin Li +13Code GenerationRL for Code Generation

    Yale University · Microsoft · Nanyang Technological University +3

  12. Beyond the Verdict: Agentic Code Judging with Verifiable Feedback

    Oct 13, 2026Zheye Deng, Sha Li, Changlong Yu +4LLM-as-a-JudgeCode Generation Evaluation

    HKUST · Amazon

  13. SolBehave: Evaluating the Behavioural Correctness of LLM-Generated Smart Contracts

    Oct 13, 2026Giovanni Pio Delvecchio, Stefano Fantazzini, Giacomo Frisoni +4Code Generation EvaluationLLM Code Generation

    University of Bologna, Bologna, Italy · University of Copenhagen, Copenhagen, Denmark

  14. Passing Tests, Missing Semantics: Evidence of Interpretive Lock-In in Plan-Based DataFrame Code Generation

    Oct 11, 2026Jingjing Liu, Dishu Yang, Hongbo Qin +1Code GenerationCode Generation Evaluation

    University of California, Berkeley, Berkeley, CA, USA · Northeastern University, San Jose, CA, USA · Northeastern University, Boston, MA, USA

  15. CodeWalk: Generating Coding Benchmarks by Walking a Problem Graph

    Oct 11, 2026Jiaxin Fang, Runyuan He, Arnav Gupta +3Synthetic Benchmark GenerationBenchmark Design

    University of California, Berkeley

  16. Re-executing Coding-Agent Evaluations: Unchanged Decisions and Unresolved Policy Risk

    Oct 10, 2026Tianyu Ding, Yaocheng Tong, Muzhi Yang +3Benchmark ReproducibilityAutomated Agent Evaluation

    Independent Researchers

  17. Not All Correct Fixes Run Alike: An Empirical Study of Patch Performance in Repository-Level Issue Resolution

    Oct 9, 2026Han Peng, Xing Hu, Feifei Niu +2Code Generation EvaluationSoftware Engineering Benchmarks

    The State Key Laboratory of Blockchain and Data Security, Zhejiang University, China and Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security, China · Graz University of Technology, Austria

  18. Characterizing Overconfident Failure in LLM-Based Code Generation

    Oct 8, 2026Ravishka Rathnasuriya, Wei YangLLM Code GenerationConfidence Estimation in Language Models

    Institute of Systems for Advanced Computing, Fudan University, Shanghai, China

  19. TaoD2C-Bench: Benchmarking MLLMs for Industrial UI Code Generation Beyond Visual Fidelity

    Oct 7, 2026Chengwei Shi, Yunnong Chen, Tingting Zhou +5Image-to-Code GenerationCode Generation

    Zhejiang University · Taobao & Tmall Group of Alibaba

  20. Visible Reasoning Is Not a Universal Optimizer: Persona- and Thinking-Dependent Effects in Analytics Code Generation

    Oct 7, 2026Bhawani Shankar Leelar, Pawan Chorasiya, Davin Hill +2CoT ReasoningLLM Code Generation

    Optum AI

  21. AeroEval: Staged Program and Execution Validation for AI-Generated Drone Missions

    Oct 7, 2026Kautuk Astu, Naina Rabha, Yogesh SimmhanCode Generation EvaluationAerial Robotics

    Department of Computational and Data Sciences, Indian Institute of Science, Bangalore 560012, India

  22. Beyond the Leaderboard: Multi-Dimensional Evaluation of Dense and Mixture-of-Experts Models for Automated Program Repair

    Oct 6, 2026Anvi Kalpesh Shah, Umamaheswara Sharma BSoftware EngineeringAutomated Program Repair

    National Institute of Technology, Calicut

  23. Verification Trap: Understanding Test-Time Selection Failures under False Premises in Code Generation

    Oct 4, 2026Feng He, Hejia Wang, Linghao Meng +2Execution-Guided Code GenerationCode Generation Evaluation

    University of Science and Technology of China, China · Beijing University of Posts and Telecommunications, China · National University of Singapore, Singapore +1

  24. MintEval: Do LLMs Implement the Trading Strategy You Asked For? A Behavioural-Equivalence Benchmark for Natural-Language-to-Strategy Code

    Oct 2, 2026Siyu Wang, Yifan Wang, Yuecheng HeLLM EvaluationCode Generation Evaluation

    Fudan University Shanghai, China

  25. WebUIProof: Benchmarking WebUI Code Generators with UI-Agent Execution Harness

    Oct 2, 2026Yun-Yun Tsai, Yuning Mao, Shiqi Wang +2Code GenerationCode Generation Evaluation

    Columbia University · Meta SuperIntelligence Labs

  26. Improving the Energy-Efficiency of the Code Generated by LLMs through Effective Prompting

    Oct 1, 2026Ritika Rekhi, Bing Zhang, Md Arman Islam +7LLM Code GenerationCode Generation

    University at Buffalo, Buffalo, NY, USA · National Center for Supercomputing Applications, Urbana, IL, USA · IBM Research, Yorktown Heights, NY, USA

  27. Are you Synthesizing or Recalling? Evaluating LLMs on Algorithmic Code Retrieval

    Oct 1, 2026Nickil Maveli, Antonio Vergari, Shay B. CohenCode Generation EvaluationCode Generation Benchmarks

    School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh, EH8 9AB

  28. CONTRA: Discovering and Qualifying Behavior-Changing Questions for Selective Clarification in LLM Code Generation

    Oct 1, 2026Zheng Fang, Yongmin Li, Yichang Zhang +5Clarification Question GenerationAI Coding Agents

    Peking University · Wuhan University

  29. Groundability, Not Scale Alone: When Weak Reviewers Can Audit Strong Coding Agents

    Oct 1, 2026Junyu Guo, Shangding Gu, Ming Jin +1AI Agent AuditingCode Generation Evaluation

    University of California, Berkeley · Virginia Tech

  30. Code That Works, Environments That Don't: Measuring Environment Reproducibility in AI-Generated Software

    Sep 30, 2026Bhanu Prakash Vangala, Tanu MalikCoding AgentsCode Generation

    Department of Electrical Engineering and Computer Science, University of Missouri–Columbia, Missouri, USA