Code Generation Evaluation

Ask about this topic

Momentum

17 papers in the last four weeks, up 113% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 147

All topics
CardsList

  1. TaoD2C-Bench: Benchmarking MLLMs for Industrial UI Code Generation Beyond Visual Fidelity

    Oct 7, 2026Chengwei Shi, Yunnong Chen, Tingting Zhou +5Code GenerationCode Generation Evaluation

    Zhejiang University · Taobao & Tmall Group of Alibaba

  2. AeroEval: Staged Program and Execution Validation for AI-Generated Drone Missions

    Oct 7, 2026Kautuk Astu, Naina Rabha, Yogesh SimmhanCode Generation EvaluationAerial Robotics

    Department of Computational and Data Sciences, Indian Institute of Science, Bangalore 560012, India

  3. Beyond the Leaderboard: Multi-Dimensional Evaluation of Dense and Mixture-of-Experts Models for Automated Program Repair

    Oct 6, 2026Anvi Kalpesh Shah, Umamaheswara Sharma BSoftware EngineeringAutomated Program Repair

    National Institute of Technology, Calicut

  4. Verification Trap: Understanding Test-Time Selection Failures under False Premises in Code Generation

    Oct 4, 2026Feng He, Hejia Wang, Linghao Meng +2Execution-Guided Code GenerationCode Generation Evaluation

    University of Science and Technology of China, China · Beijing University of Posts and Telecommunications, China · National University of Singapore, Singapore +1

  5. Groundability, Not Scale Alone: When Weak Reviewers Can Audit Strong Coding Agents

    Oct 1, 2026Junyu Guo, Shangding Gu, Ming Jin +1AI Agent AuditingCode Generation Evaluation

    University of California, Berkeley · Virginia Tech

  6. Code That Works, Environments That Don't: Measuring Environment Reproducibility in AI-Generated Software

    Sep 30, 2026Bhanu Prakash Vangala, Tanu MalikCoding AgentsCode Generation

    Department of Electrical Engineering and Computer Science, University of Missouri–Columbia, Missouri, USA

  7. A2Z GameSpec-Bench: How Faithfully Can Coding Agents Generate Games from Game Design Specifications?

    Sep 30, 2026Seonho Lee, Wonryeol Jeong, Alberto Cereser +4AI Coding AgentsAI Agent Benchmarks

    KRAFTON · KAIST · Korea National University of Arts

  8. Between the Commits: Process, Error, and Claim Reliability in a Wholly AI-Authored Codebase

    Sep 24, 2026Douglas LeithAI Agent ReliabilityData Provenance

    Trinity College Dublin Dublin, Ireland

  9. Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware Screen

    Sep 22, 2026Om Nepal, Sushant Aryal, Oluseyi Olukola +1Automated Program RepairLLMs for Cybersecurity

    School of Computing Sciences and Computer Engineering, The University of Southern Mississippi, Hattiesburg, MS 39406, USA

  10. The Complexity Kink: A Prompt-Side Structural Complexity Index for Code-Generation Reliability

    Sep 17, 2026Michael Hernandez, Tian ZhaoRubric-Based EvaluationCode Generation Evaluation

    University of Wisconsin–Milwaukee

  11. ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks

    Sep 16, 2026Jeonghye Kim, Minseon Kim, Young Jin Kim +5Coding AgentsSoftware Engineering Benchmarks

    KAIST · Microsoft Research Montréal · Microsoft AI

  12. A Study of the Reliability of Agentic AI-Generated Programs

    Sep 16, 2026Ayesha Shafique, Barton P. MIller, Elisa R. HeymannAI Agent ReliabilityAgentic Code Generation

  13. An Empirical Evaluation of Cost-Efficient Large Language Models on Algorithmic Programming Tasks

    Sep 16, 2026Chandimal Adikari, Nandika HerathCode GenerationLLM Reliability

    Faculty of Engineering and Information Sciences University of Wollongong Wollongong, Australia · Faculty of Engineering, Computing and Science Western Sydney University Parramatta, Australia

  14. DepthBenchCAD: When Does Deeper Auditing Yield More Reliable Conclusions?

    Sep 14, 2026Hongye Yang, Zhihao Xie, Shengjun Xiong +1Benchmark AuditingText-to-CAD Generation

    College of Computing Georgia Institute of Technology · Independent Researcher

  15. Beyond Vector Similarity: Hierarchical Context-Aware Graph RAG vs Standard RAG in Enterprise Code Migration

    Sep 14, 2026Nilesh Jaiswal, Aniket Agrawal, Arjit Shukla +4Code TranslationGraph Retrieval-Augmented Generation

    Google Cloud

  16. What is the Difference Between Me and You? Benchmarking the Quality Gap Between Human-Written and AI-Generated Code

    Sep 14, 2026Cristina Improta, Pietro Liguori, Domenico CotroneoLLM SecurityCode Generation Evaluation

    University of Naples Federico II, Naples, Italy · University of North Carolina at Charlotte, Charlotte, NC, USA

  17. The Vibe Shift in Software Engineering: Evaluating AI-Led Conversational Programming for Performance, Cognition, and Responsible Adoption

    Sep 10, 2026Sales G. Aribe Jr., Louie Jay S. LabastidaCode Generation EvaluationHuman-AI Collaboration

    Information Technology Department, Bukidnon State University, Malaybalay City, Philippines

  18. It Is Not My Code Anymore

    Sep 8, 2026Augusto CamargoAlgorithmic AccountabilityCode Generation Evaluation

    Institute of Mathematics, Statistics and Computer Science (IME) University of São Paulo, São Paulo, Brazil

  19. SciFigure2Code: An AI-Reconstructed Benchmark for Scientific Figure-to-Code

    Sep 8, 2026Wentao Li, Yibo Wu, Yizhe Chen +5Scientific VisualizationCode Generation Evaluation

    Tsinghua University

  20. CodeTD: Topology of Attention Detects Hallucinations in Code LLMs

    Sep 7, 2026Daria Voronkova, Ilya Trofimov, Anton Dmitriev +3LLM Hallucination DetectionCode Generation Evaluation

    Applied AI Institute · AXXX · IMJ CNRS

  21. Hints Help But Do They Teach? Evaluating Skills Transfer in Code Generation

    Sep 1, 2026Will BadrLanguage Model Generation EvaluationLLM Prompting

  22. Commit-first LLM judging inherits the judge's own errors

    Aug 31, 2026Idil GozelLLM-as-a-JudgeLLM Auditing

    Evaluator Integrity, London

  23. The reach of a verification tool decides its value: A controlled study of verification surface, artifact quality, and cost in AI coding agents

    Aug 28, 2026Achint MehtaAutomated Software TestingAI Coding Agents

  24. Rubric-to-Code Credit Assignment for Reinforcement Learning

    Aug 28, 2026Rui Jin, Jikai Chen, Yihan Chen +6RL for Code GenerationCode Generation

    Inclusion AI, Ant Group

  25. RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests

    Aug 28, 2026Gyuhyeong Kim, Hyojung Gwon, Jeonghyeon Kim +2LLM PromptingCoding Agents

    Sungkyunkwan University

  26. Metrics That Write Themselves: Evolving an Evaluator from Its Own Blind Spots

    Aug 19, 2026Xing Zhang, Yanwei Cui, Guanghui Wang +2Counterexample-Guided RefinementCode Generation Evaluation

    AWS Generative AI Innovation Center

  27. QuoteBench: How Matched Scores Can Hide Command-Path Failures

    Aug 13, 2026Shangao Li, Yao Zhang, Volker Tresp +1LLM Agent EvaluationAgent Evaluation

    1Stony Brook University · 2LMU Munich · 3Munich Center for Machine Learning

  28. VisEditBench: Can Vision-Language Models Edit Visualization Code from Multimodal Feedback?

    Aug 11, 2026Mizanur Rahman, Arshia Azimlu, Shadikur Rahman +4VLM EvaluationData Visualization

    ‡York University · $Nanyang Technological University · ¶Salesforce AI Research

  29. CAS2UML: A Handwritten Sketch-to-PlantUML Dataset for Class and Activity Diagrams

    Aug 7, 2026Simon Scholz, Mersedeh SadeghiCode Generation EvaluationDocument Image Understanding

    University of Cologne Cologne, Germany

  30. Blind to the Pivotal Vote: Aggregate Independence Metrics Miss Where Verification Actually Helps

    Aug 7, 2026Yang ShuLLM EvaluationLLM-as-a-Judge

    Zhejiang University, Hangzhou, China