Code Generation Evaluation

Overview

Code generation evaluation studies how well AI systems produce programs that work and satisfy the requirements developers intended. Common approaches include executable unit tests and pass@k benchmarks, repository-level issue resolution, and tests of generated applications through browsers or other interactive environments. Researchers also assess qualities beyond passing tests, such as security, maintainability, performance, instruction adherence, and fidelity to domain-specific behavior. A central challenge is that test success can overstate real-world reliability, while stochastic outputs, benchmark contamination, and imperfect automated judges make comparisons difficult.

Latest papers

CardsList

228 papers13 papers in the last four weeks, down 55% on the four weeks before. 0.1% of all new papers.

  1. Let Execution Do the Teaching

    Nov 15, 2026Abhishek Mishra, Sandeep Srinivasa, Arko Chattopadhyay +2Execution-Guided Code GenerationRL for Code Generation

  2. PerfArena: Ranking LLMs by Code Performance

    Nov 13, 2026Rui Rua, Hailemariam Mersha, Joao Paulo Fernandes +1Code Generation EvaluationCode Generation Benchmarks

  3. RiskyPy: Toward Model-Calibrated Prompt Generation for Joint Security and Functionality Evaluation of LLM-Generated Code

    Oct 29, 2026Zacharie Chenail-Larcher, Daniele Lunghi, Melissa Tessa +5Code Generation EvaluationCode Generation Benchmarks

  4. Secure and Functional Code Generation with Self-Evaluation

    Oct 24, 2026Md Afif Al Mamun, Gias UddinSecure Code GenerationRetrieval-Augmented Code Generation

  5. SWE-Clar: Can Coding Agents Clarify Incomplete Client Requirements and Deliver Greenfield Software?

    Oct 22, 2026Jialei Li, Dingxian Wang, Xiaoyan Zhao +11Coding AgentsSoftware Engineering Benchmarks

  6. ProductWebBench: Evaluating Website Continuity Beyond Task Completion

    Oct 22, 2026Jiajun Wu, Jian Yang, Yaxin Du +9Software Engineering BenchmarksCode Generation Evaluation

  7. Recursive Self-Improvement on a Budget: Scoring Code Edits Before They Run

    Oct 21, 2026Erfan Darzi, Sree Bhargavi Balija, Sheng HeLLM Agent Self-ImprovementLLM Agent Workflow Optimization

  8. FrontierRefactor: Benchmarking Multi-Scale Performance-Oriented Codebase Refactoring

    Oct 19, 2026Shijian Wang, Fanqing Meng, Jiajun Chen +8Automated Code RefactoringSoftware Engineering Benchmarks

  9. When LLM-Generated Refactorings Ripple: An Empirical Study of Repository Adoption Readiness

    Oct 17, 2026Muhammad Waseem, Aakash Ahmad, Pekka AbrahamssonAutomated Code RefactoringLLM Reliability

  10. SWE-Nexus: Automated Composition of Software Engineering Tasks from Test Execution

    Oct 14, 2026Songwei Yu, Ziqi Miao, Jingxuan Xu +10Software Engineering BenchmarksCode Generation Benchmarks

  11. DivAlgo: A systematic evaluation of algorithmic diversity for code generation models

    Oct 13, 2026Anubhav Jangra, Smaranda MuresanCode Language ModelsCode Generation Evaluation

  12. AesCode: Aesthetic Code Generation with Decoupled Cross-Modal Rewards

    Oct 13, 2026Xin Gao, Yifan Yang, Xin Li +13Code GenerationRL for Code Generation

  13. SolBehave: Evaluating the Behavioural Correctness of LLM-Generated Smart Contracts

    Oct 13, 2026Giovanni Pio Delvecchio, Stefano Fantazzini, Giacomo Frisoni +4Code Generation EvaluationLLM Code Generation

  14. Passing Tests, Missing Semantics: Evidence of Interpretive Lock-In in Plan-Based DataFrame Code Generation

    Oct 11, 2026Jingjing Liu, Dishu Yang, Hongbo Qin +1Code GenerationCode Generation Evaluation

  15. CodeWalk: Generating Coding Benchmarks by Walking a Problem Graph

    Oct 11, 2026Jiaxin Fang, Runyuan He, Arnav Gupta +3Synthetic Benchmark GenerationBenchmark Design

  16. Not All Correct Fixes Run Alike: An Empirical Study of Patch Performance in Repository-Level Issue Resolution

    Oct 9, 2026Han Peng, Xing Hu, Feifei Niu +2Code Generation EvaluationSoftware Engineering Benchmarks

  17. Characterizing Overconfident Failure in LLM-Based Code Generation

    Oct 8, 2026Ravishka Rathnasuriya, Wei YangLLM Code GenerationConfidence Estimation in Language Models

  18. TaoD2C-Bench: Benchmarking MLLMs for Industrial UI Code Generation Beyond Visual Fidelity

    Oct 7, 2026Chengwei Shi, Yunnong Chen, Tingting Zhou +5Image-to-Code GenerationCode Generation

  19. AeroEval: Staged Program and Execution Validation for AI-Generated Drone Missions

    Oct 7, 2026Kautuk Astu, Naina Rabha, Yogesh SimmhanCode Generation EvaluationAerial Robotics