Code Generation Evaluation

Ask about this topic

Momentum

17 papers in the last four weeks, up 113% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 145

All topics
CardsList

  1. TaoD2C-Bench: Benchmarking MLLMs for Industrial UI Code Generation Beyond Visual Fidelity

    Oct 7, 2026Chengwei Shi, Yunnong Chen, Tingting Zhou +5Code GenerationCode Generation Evaluation

    Zhejiang University · Taobao & Tmall Group of Alibaba

  2. AeroEval: Staged Program and Execution Validation for AI-Generated Drone Missions

    Oct 7, 2026Kautuk Astu, Naina Rabha, Yogesh SimmhanCode Generation EvaluationAerial Robotics

    Department of Computational and Data Sciences, Indian Institute of Science, Bangalore 560012, India

  3. Beyond the Leaderboard: Multi-Dimensional Evaluation of Dense and Mixture-of-Experts Models for Automated Program Repair

    Oct 6, 2026Anvi Kalpesh Shah, Umamaheswara Sharma BSoftware EngineeringAutomated Program Repair

    National Institute of Technology, Calicut

  4. Verification Trap: Understanding Test-Time Selection Failures under False Premises in Code Generation

    Oct 4, 2026Feng He, Hejia Wang, Linghao Meng +2Execution-Guided Code GenerationCode Generation Evaluation

    University of Science and Technology of China, China · Beijing University of Posts and Telecommunications, China · National University of Singapore, Singapore +1

  5. Groundability, Not Scale Alone: When Weak Reviewers Can Audit Strong Coding Agents

    Oct 1, 2026Junyu Guo, Shangding Gu, Ming Jin +1AI Agent AuditingCode Generation Evaluation

    University of California, Berkeley · Virginia Tech

  6. Code That Works, Environments That Don't: Measuring Environment Reproducibility in AI-Generated Software

    Sep 30, 2026Bhanu Prakash Vangala, Tanu MalikCoding AgentsCode Generation

    Department of Electrical Engineering and Computer Science, University of Missouri–Columbia, Missouri, USA

  7. A2Z GameSpec-Bench: How Faithfully Can Coding Agents Generate Games from Game Design Specifications?

    Sep 30, 2026Seonho Lee, Wonryeol Jeong, Alberto Cereser +4AI Coding AgentsAI Agent Benchmarks

    KRAFTON · KAIST · Korea National University of Arts

  8. Between the Commits: Process, Error, and Claim Reliability in a Wholly AI-Authored Codebase

    Sep 24, 2026Douglas LeithAI Agent ReliabilityData Provenance

    Trinity College Dublin Dublin, Ireland

  9. Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware Screen

    Sep 22, 2026Om Nepal, Sushant Aryal, Oluseyi Olukola +1Automated Program RepairLLMs for Cybersecurity

    School of Computing Sciences and Computer Engineering, The University of Southern Mississippi, Hattiesburg, MS 39406, USA

  10. The Complexity Kink: A Prompt-Side Structural Complexity Index for Code-Generation Reliability

    Sep 17, 2026Michael Hernandez, Tian ZhaoRubric-Based EvaluationCode Generation Evaluation

    University of Wisconsin–Milwaukee

  11. ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks

    Sep 16, 2026Jeonghye Kim, Minseon Kim, Young Jin Kim +5Coding AgentsSoftware Engineering Benchmarks

    KAIST · Microsoft Research Montréal · Microsoft AI

  12. A Study of the Reliability of Agentic AI-Generated Programs

    Sep 16, 2026Ayesha Shafique, Barton P. MIller, Elisa R. HeymannAI Agent ReliabilityAgentic Code Generation

  13. An Empirical Evaluation of Cost-Efficient Large Language Models on Algorithmic Programming Tasks

    Sep 16, 2026Chandimal Adikari, Nandika HerathCode GenerationLLM Reliability

    Faculty of Engineering and Information Sciences University of Wollongong Wollongong, Australia · Faculty of Engineering, Computing and Science Western Sydney University Parramatta, Australia

  14. DepthBenchCAD: When Does Deeper Auditing Yield More Reliable Conclusions?

    Sep 14, 2026Hongye Yang, Zhihao Xie, Shengjun Xiong +1Benchmark AuditingText-to-CAD Generation

    College of Computing Georgia Institute of Technology · Independent Researcher

  15. What is the Difference Between Me and You? Benchmarking the Quality Gap Between Human-Written and AI-Generated Code

    Sep 14, 2026Cristina Improta, Pietro Liguori, Domenico CotroneoLLM SecurityCode Generation Evaluation

    University of Naples Federico II, Naples, Italy · University of North Carolina at Charlotte, Charlotte, NC, USA

  16. Beyond Vector Similarity: Hierarchical Context-Aware Graph RAG vs Standard RAG in Enterprise Code Migration

    Sep 14, 2026Nilesh Jaiswal, Aniket Agrawal, Arjit Shukla +4Code TranslationGraph Retrieval-Augmented Generation

    Google Cloud

  17. The Vibe Shift in Software Engineering: Evaluating AI-Led Conversational Programming for Performance, Cognition, and Responsible Adoption

    Sep 10, 2026Sales G. Aribe Jr., Louie Jay S. LabastidaCode Generation EvaluationHuman-AI Collaboration

    Information Technology Department, Bukidnon State University, Malaybalay City, Philippines

  18. It Is Not My Code Anymore

    Sep 8, 2026Augusto CamargoAlgorithmic AccountabilityCode Generation Evaluation

    Institute of Mathematics, Statistics and Computer Science (IME) University of São Paulo, São Paulo, Brazil

  19. SciFigure2Code: An AI-Reconstructed Benchmark for Scientific Figure-to-Code

    Sep 8, 2026Wentao Li, Yibo Wu, Yizhe Chen +5Scientific VisualizationCode Generation Evaluation

    Tsinghua University

  20. CodeTD: Topology of Attention Detects Hallucinations in Code LLMs

    Sep 7, 2026Daria Voronkova, Ilya Trofimov, Anton Dmitriev +3LLM Hallucination DetectionCode Generation Evaluation

    Applied AI Institute · AXXX · IMJ CNRS

  21. Hints Help But Do They Teach? Evaluating Skills Transfer in Code Generation

    Sep 1, 2026Will BadrLanguage Model Generation EvaluationLLM Prompting

  22. Commit-first LLM judging inherits the judge's own errors

    Aug 31, 2026Idil GozelLLM-as-a-JudgeLLM Auditing

    Evaluator Integrity, London

  23. The reach of a verification tool decides its value: A controlled study of verification surface, artifact quality, and cost in AI coding agents

    Aug 28, 2026Achint MehtaAutomated Software TestingAI Coding Agents

  24. Rubric-to-Code Credit Assignment for Reinforcement Learning

    Aug 28, 2026Rui Jin, Jikai Chen, Yihan Chen +6RL for Code GenerationCode Generation

    Inclusion AI, Ant Group

  25. RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests

    Aug 28, 2026Gyuhyeong Kim, Hyojung Gwon, Jeonghyeon Kim +2LLM PromptingCoding Agents

    Sungkyunkwan University

  26. Metrics That Write Themselves: Evolving an Evaluator from Its Own Blind Spots

    Aug 19, 2026Xing Zhang, Yanwei Cui, Guanghui Wang +2Counterexample-Guided RefinementCode Generation Evaluation

    AWS Generative AI Innovation Center

  27. QuoteBench: How Matched Scores Can Hide Command-Path Failures

    Aug 13, 2026Shangao Li, Yao Zhang, Volker Tresp +1LLM Agent EvaluationAgent Evaluation

    1Stony Brook University · 2LMU Munich · 3Munich Center for Machine Learning

  28. VisEditBench: Can Vision-Language Models Edit Visualization Code from Multimodal Feedback?

    Aug 11, 2026Mizanur Rahman, Arshia Azimlu, Shadikur Rahman +4VLM EvaluationData Visualization

    ‡York University · $Nanyang Technological University · ¶Salesforce AI Research

  29. CAS2UML: A Handwritten Sketch-to-PlantUML Dataset for Class and Activity Diagrams

    Aug 7, 2026Simon Scholz, Mersedeh SadeghiCode Generation EvaluationDocument Image Understanding

    University of Cologne Cologne, Germany

  30. Blind to the Pivotal Vote: Aggregate Independence Metrics Miss Where Verification Actually Helps

    Aug 7, 2026Yang ShuLLM EvaluationLLM-as-a-Judge

    Zhejiang University, Hangzhou, China

  31. DevIntent: How Much Does LLM-Generated Code Violate Developer Intent?

    Aug 7, 2026Susana Haing, Natan Vidra, Spurthi SettyLanguage Model Generation EvaluationCode Generation Evaluation

    Anote AI

  32. Characterizing the Quality Profile of AI-Generated C++ in Production

    Aug 6, 2026Michael Tran, Fred Lewis, Kun Yang +5Code GenerationCode Generation Evaluation

    Google, Mountain View, USA

  33. Coding Agents as Test-Suite Auditors: Finding What Official Suites Miss While Approaching What They Catch

    Aug 3, 2026Shuyang Xie, Shuxiao Xie, Feng Zhu +2Automated Software TestingBenchmark Auditing

    Harbin Institute of Technology · Fudan University · Sun Yat-sen University

  34. Security-First Evaluation of Text-to-Terraform: Benchmarking LLMs and SLMs for Secure IaC Generation

    Aug 2, 2026Francis Luis Santos Vargas, Rodrigo Brandão Mansilha, Diego KreutzLLM EvaluationCode Generation

    AI Horizon Labs, PPGES, Universidade Federal do Pampa (UNIPAMPA), Alegrete, Brazil

  35. TaPR: Test-Aware Policy Refinement for Feedback-Conditioned Code Generation

    Aug 1, 2026Aofan Liu, Jingxiang Meng, Fangxin Liu +1RL for Code GenerationReward Shaping

    ∗These authors contributed equally. · †Corresponding author.

  36. Simulation Code Generation for Fluid Systems using Large Language Models: Benchmarking Models and Prompting Strategies

    Jul 31, 2026Jan Marius Stürmer, Jascha Knack, Tobias Koch +1LLM PromptingCode Generation Evaluation

    German Aerospace Centre (DLR), Institute for the Protection of Terrestrial Infrastructures, Rathausallee 12, 53757, Sankt Augustin, Germany · Algorithms for Computer Vision, Imaging and Data Analysis, Technische Hochschule Würzburg-Schweinfurt, Ignaz-Schön-Straße 11, Schweinfurt, 97421, Germany

  37. SecDrift: Measuring Sector-Conditioned Security Drift in AI-Generated Code

    Jul 28, 2026Narayanaswami Natraj Bharadwaj, Dhivya ChandramouleeswaranCybersecurityLLM Security

    Independent Researcher USA

  38. RIDGE: An Autonomous Framework for Validation and Method Discovery in LLM-Generated Option Pricing

    Jul 28, 2026Liexin Cheng, Xue Cheng, Shuaiqiang Liu +1Quantitative FinanceCode Generation Evaluation

    Mathematical Institute, Utrecht University · School of Mathematical Sciences, Peking University · Delft Institute of Applied Mathematics, Delft University of Technology

  39. Learning from 53.6K Real-World Developer Edits of AI-Generated Code

    Jul 27, 2026Jenny T. Liang, Mihika Bairathi, Wayne Chi +3Code Generation EvaluationCode Language Models

    Carnegie Mellon University

  40. Adversarial Test-Hardening for AI-Written Code: An Instrument Autopsy and a Pre-Registered Causal Estimate of the Critic Loop

    Jul 25, 2026Jeff OttersonAutomated Software TestingLLM Auditing

    W. P. Carey School of Business, Arizona State University

  41. AssumptionMiner: Extracting, Tracing, and Revising Implicit Assumptions in LLM Code Generation

    Jul 24, 2026Jie "JW" WuCode GenerationCode Generation Evaluation

    Michigan Technological University, Houghton, MI, USA

  42. The Best Programming Language for Tokenmaxxing: An Investigation of Coding Agent Behavior Across Programming Languages

    Jul 24, 2026Zixuan Wu, Carolyn Jane Anderson, Arjun GuhaAI Coding AgentsCode Generation

    Northeastern University · Wellesley College

  43. MineValiCoder: Reliable Code Generation with Test Case Quality Mining and Bipartite Graph-Based Mutual Validation

    Jul 24, 2026Zhen Zhao, Qihang Yang, Feifei Dai +2Execution-Guided Code GenerationCode Generation

    Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · University of Electronic Science and Technology of China, Chengdu, China

  44. How Do AI Coding Agents Contribute to Software Development? an Empirical Study of Agentic Pull Requests

    Jul 23, 2026Iren Mazloomzadeh, Mohammad Mehdi Morovati, Foutse KhomhSoftware Engineering AgentsAI Coding Agents

    Polytechnique Montréal, Canada

  45. Benchmarking LLMs for Verilog Design Flows

    Jul 23, 2026Angshuman Chakravertty, Rahul Koshti, Buddhi Prakash Sharma +1Code GenerationCode Generation Evaluation

    School of Technology Management and Engineering, NMIMS, Hyderabad, India · Department of Electronics and Communication Engineering, Indian Institute of Technology, Roorkee, India · Department of Electrical and Electronics Engineering and the Anuradha and Prashanth Palakurthi Centre for Artificial Intelligence Research (APPCAIR), Birla Institute of Technology and Science, Pilani (BITS Pilani), Pilani 333031, India

  46. Code Monitor Red Teaming for Public-Test-Passing Code

    Jul 23, 2026Junchi Liao, Jiawen Deng, Fuji RenLLM AuditingCode Generation Evaluation

    University of Electronic Science and Technology of China Chengdu, China

  47. Cross-Model LLM Code Review: Should you use Claude to review Codex or vice versa?

    Jul 22, 2026Zuodong Xiang, Yike Zhang, YueMing Zhang +1Code Generation EvaluationAutomated Code Review

    University of California, Davis USA · Johns Hopkins University USA · California State University, Long Beach USA

  48. SciCodePile: A 128GB Corpus and Executable Benchmark for Challenging Scientific Code Generation

    Jul 21, 2026Weifeng Sun, Ye Fan, Yuchen Chen +6Scientific Code GenerationCode Generation Evaluation

    Singapore Management University · Nanjing University · SUN YAT-SEN University +1

  49. Spaghetti Architect: A Contamination-Resistant, By-Construction-Labelled, Multi-Language Code Dataset Generator

    Jul 21, 2026Yuxiang JiSynthetic Benchmark GenerationData Contamination in Language Models

    Sunway College

  50. The Librarian Who Refused to Code: Model-Dependent Identity Enactment in LLM Code Generation

    Jul 19, 2026Shayell Aharon Salomon, Noam Israel, Ido Safruti +1Personality Modeling in Language ModelsCode Generation

    Bluebear Security

  51. Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality

    Jul 16, 2026Saima Afrin, Alessandro Midolo, Camilo Escobar-Velásquez +5Multilingual Language Model EvaluationCode Generation

    William & Mary, VA, USA · University of Catania, Italy · Universidad de los Andes, Colombia +2

  52. Quantize with Confidence? An Empirical Study of Quantization for Code Generation

    Jul 15, 2026Saima Afrin, Md. Zahidul Haque, Antonio MastropaoloLLM QuantizationCode Generation Evaluation

    AURA Lab · Department of Computer Science · William & Mary +1

  53. Form, Not Content? A Preregistered, Placebo-Controlled Evaluation of Learned Error-Conditioned Self-Repair Through Prompts and Weights in Frozen Small Code Models

    Jul 14, 2026Mehmet IscanLLM Self-CorrectionAutomated Program Repair

    PythaLab, Yıldız Technical University, Istanbul, Turkey

  54. Line-Anchored Feedback Cuts Token Costs and Improves Correctness in AI Code Editing

    Jul 14, 2026William Franz LambertiCode Generation EvaluationCode Language Models

    Computational and Data Sciences College of Science George Mason University Fairfax, VA, United States

  55. Token Reduction Is Not Cost Reduction

    Jul 13, 2026Sarel Weinberger, Amir HozezAI Coding AgentsCost-Aware Inference

    PointFive

  56. When the Reward Suite Is Leaky: A Preregistered Causal Contrast of Natural Verifier False Positives in RLVR

    Jul 13, 2026Chuyifei ZhangAlgorithmic AuditingReinforcement Learning with Verifiable Rewards

    Beijing Jiaotong University

  57. When Does Restricting a Coding Agent to execute_code Help? A Regime ×\times Agent-Design Ablation

    Jul 12, 2026Hong Yang, Qi Yu, Travis DesellCoding AgentsTool-Using Agents

    Rochester Institute of Technology Rochester, New York, USA

  58. Knowledge-Conditioned, Single-Pass LLM Synthesis of Executable Unity Game Scenes: A Compiler Error Census across 26 Goal Playable Concepts

    Jul 11, 2026Hugh Xuechen Liu, Kıvanç TatarLLM-Based Program SynthesisProcedural Content Generation

    Chalmers University of Technology and University of Gothenburg, SE-412 96 Göteborg, Sweden

  59. The Patchwork Problem in LLM-Generated Code

    Jul 9, 2026Viraaji Mothukuri, Reza M. PariziStatic Code AnalysisLLM Reliability

    Decentralized Science Lab, College of Computing and Software Engineering Kennesaw State University, GA, USA

  60. PHITSBench: an execution-scored benchmark for AI-assisted PHITS radiation-transport input generation using natural language

    Jul 8, 2026Xianglin Ji, Svetlana V. BoriskinaKnowledge Augmentation for Language ModelsCode Generation

    Department of Mechanical Engineering, Massachusetts Institute of Technology, Cambridge, MA 02139, USA