Code Generation Benchmarks

Ask about this topic

Momentum

15 papers in the last four weeks, up 275% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 104

All topics
CardsList

  1. TaoD2C-Bench: Benchmarking MLLMs for Industrial UI Code Generation Beyond Visual Fidelity

    Oct 7, 2026Chengwei Shi, Yunnong Chen, Tingting Zhou +5Code GenerationCode Generation Evaluation

    Zhejiang University · Taobao & Tmall Group of Alibaba

  2. From Pixel to Coding: Evaluating the Figure Reproduction Capabilities of MLLMs

    Oct 7, 2026Zijian Chen, Zhengyu Chen, Bohan Liang +7Multimodal Large Language ModelsMultimodal Model Evaluation

    Institute of Image Communication and Information Processing, Shanghai Jiao Tong University, Shanghai, 200240, China. · Shanghai Artificial Intelligence Laboratory, Shanghai, 200030, China. · Macao Polytechnic University, Macao, 999078, China.

  3. Large-scale Repository Engineering via Agent-Native Reusable Code Primitives

    Oct 6, 2026Haibo Jin, Peng Kuang, Xucheng Yu +3Repository-Level Code GenerationSoftware Engineering

    University of Illinois Urbana-Champaign, USA

  4. GNN-CB: A Graph Neural Network Competition Benchmark for Human and LLM Evaluation

    Oct 4, 2026Murad Hossen, Tasneem Selim, Gurur Gamgam +22LLM EvaluationGraph Neural Networks

    University of Houston, Texas,USA · BASIRA Lab, Department of Computing, Imperial College London, United Kingdom · Department of Mathematics and Computer Science, Faculty of Science, Alexandria University, Alexandria, Egypt +23

  5. VHDL-REPOBENCH: A Repository-Level Benchmark for Evaluating Large Language Models on VHDL Design Generation

    Oct 4, 2026Prashanth Vijayaraghavan, Akul Malhotra, Ashutosh Jadhav +2LLM EvaluationCode Generation

    IBM Research San Jose, CA, USA

  6. Self-Spec Verifiable Code Generation

    Sep 30, 2026Jiaru Qian, Yihong Dong, Yongmin Li +3LLM Self-CorrectionCode Generation

    School of Computer Science, Peking University · Beijing Key Laboratory of Trustworthy Code Large Language Models · Key Laboratory of High Confidence Software Technologies, Peking University, Ministry of Education +3

  7. E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch

    Sep 29, 2026Hantian Ding, Chloe Bi, Jiacheng Zhu +5Software Engineering AgentsRepository-Level Code Generation

    Meta Superintelligence Labs

  8. Zero2Repo: Can Coding Agents Build Repositories from Scratch?

    Sep 29, 2026Pei Yang, Tianyu Shi, Yuhang Yao +23Coding AgentsAI Agent Benchmarks

    Gradient Data · McGill University · Carnegie Mellon University +6

  9. LoLBench: Evaluating Coding Agents with Long-Horizon Proposals on Large Software Systems

    Sep 29, 2026Yun Peng, Zihan Wu, Zeyang Zhuang +6Software EngineeringAI Coding Agents

    Fudan University, China · City University of Hong Kong, Hong Kong · Chinese University of Hong Kong, Hong Kong +5

  10. GPUPhysBench: Benchmarking Coding Agents for Correct and Efficient GPU Physics Simulation

    Sep 28, 2026Yuchen Sun, Jinjin He, Sinan Wang +1GPU Kernel OptimizationBenchmark Design

    Georgia Institute of Technology

  11. Do Coding Agents Reuse Existing Code or Reinvent the Wheel?

    Sep 28, 2026Dongsheng Ma, Sizhe Wang, Xinyi Huang +5Coding AgentsCode Generation

    Peking University · Fudan University · Shanghai Jiao Tong University +3

  12. MetaBench-Harness: Unlocking End-to-End Optimization of Benchmark Harnesses

    Sep 27, 2026Xuanjun Chen, Hua-Hsuan Chen, Wei-Chung Lu +3LLM EvaluationBenchmark Design

    National Taiwan University · NTU AI-Core · Queen Mary University of London

  13. GameLogicBench: Evaluating Coding Agents on Runtime Game Logic with Tick-Level State Assertions

    Sep 18, 2026Xinyu Che, Yunfei Ge, Shihao Li +9AI Coding AgentsBenchmark Design

    Nanjing University

  14. SWE-Proof: Can Language Models Resolve Real-World Issues with Machine-Checked Proofs?

    Sep 18, 2026George Ma, Benjamin Mikek, Haoyu Li +9Formal VerificationAgentic Code Generation

    UC Berkeley · Georgia Tech · UIUC +1

  15. What is the Difference Between Me and You? Benchmarking the Quality Gap Between Human-Written and AI-Generated Code

    Sep 14, 2026Cristina Improta, Pietro Liguori, Domenico CotroneoLLM SecurityCode Generation Evaluation

    University of Naples Federico II, Naples, Italy · University of North Carolina at Charlotte, Charlotte, NC, USA

  16. IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications

    Sep 9, 2026Yiling Ma, Yilun Zhao, Sihong Wu +2Benchmark DesignClarification Question Generation

    Yale University · TCS Research

  17. Retrofitting Code Using LLMs to Support Exceptional Behavior

    Sep 9, 2026Linghan Zhong, Jiyang Zhang, Jayanth Srinivasa +2Code GenerationCode Generation Benchmarks

    The University of Texas at Austin, USA · Cisco Systems, USA

  18. SciFigure2Code: An AI-Reconstructed Benchmark for Scientific Figure-to-Code

    Sep 8, 2026Wentao Li, Yibo Wu, Yizhe Chen +5Scientific VisualizationCode Generation Evaluation

    Tsinghua University

  19. GVS5H: Zero-Shot Self-Orchestration with Ledger-Based Control for Improved LLM Coding Performance

    Aug 27, 2026Victor Gao, Vida Khosrowshahi, Ali Khosrowshahi +4Zero-Shot LearningMulti-Agent LLM Systems

    Sang Won · Persis Capital Inc.

  20. Vero: Can AI Agents Build Formally Verified Software Repositories?

    Aug 13, 2026Zhe Ye, Hantao Lou, Yuechun Sun +8Code GenerationFormal Verification

    1UC Berkeley · University of Chicago · 3California Institute of Technology +3

  21. P3^{3}: Joint Program-and-Proof Planning for Verified Code Generation

    Aug 10, 2026Zenan Li, Ziran Yang, Peiyang Song +2Code GenerationFormal Verification

    1Apodex · 2Princeton University · 3Caltech +1

  22. A Unified Issue Resolution Benchmark for Requirement Clarification, Planning, and Code Generation for Coding Agents

    Aug 10, 2026Xin Zhou, Chun Yong Chong, Kisub Kim +11Requirements EngineeringAI Coding Agents

    Singapore Management University, Singapore · Monash University Malaysia, Malaysia · Daegu Gyeongbuk Institute of Science and Technology (DGIST), Republic of Korea +5

  23. DevIntent: How Much Does LLM-Generated Code Violate Developer Intent?

    Aug 7, 2026Susana Haing, Natan Vidra, Spurthi SettyLanguage Model Generation EvaluationCode Generation Evaluation

    Anote AI

  24. SciCode-Verified: How Benchmark Defects Underestimated the Scientific-Coding Ability of Language Models

    Aug 5, 2026Sihan Hu, Lyuhan Huang, Youjin Deng +1LLM EvaluationScientific Code Generation

    Hefei National Laboratory College of Mechanical and Electrical Engineering University of Science and Technology of China · Hefei National Laboratory Institute of Theoretical Physics University of Science and Technology of China Chinese Academy of Sciences Hefei 230026, China · Chinese Academy of Sciences Beijing 100190, China

  25. MT-Web2Code: Benchmarking Coding Agents on Multi-Turn Regional Reconstruction and Localized Modification

    Aug 4, 2026Qiming Li, Shujie Hu, Haohan Liu +3Code GenerationAI Agent Benchmarks

    1Harbin Institute of Technology · 2Meituan, Longcat Team · 3Peng Cheng Laboratory

  26. FinHardBench: Can LLMs Generate Latency-Aware Hardware for Financial Computing?

    Aug 2, 2026Weimin Fu, Hejia Zhang, Minghao Shao +6Large Language Model-Guided OptimizationElectronic Design Automation

    Lehigh University · University of California, San Diego · NYU Tandon School of Engineering +1

  27. EduPluginBench: Executable Assurance for AI-Generated Educational Plugins

    Aug 1, 2026Nizam KadirSoftware SecurityRuntime Assurance

    Singapore University of Technology and Design, Science, Mathematics and Technology, Singapore, Singapore

  28. Change2Task: From Repository Changes to Executable Coding Agent Tasks and Environments

    Jul 30, 2026Haomin Qi, Xingliang Wang, Xuanqi Gao +9Code Generation BenchmarksAgent Benchmarks

    1Microsoft · 3Zhejiang University · 4Xi’an Jiaotong University +1

  29. Fewer Clarifications, Better Code: Benchmarking Cross-Session Personalized Ambiguity Adaptation in Coding Assistants

    Jul 29, 2026Zijian Xu, Wenshuo Zhang, Zisen Qin +4Knowledge Augmentation for Language ModelsCode Generation

    Southeast University · The Hong Kong University of Science and Technology

  30. AssumptionMiner: Extracting, Tracing, and Revising Implicit Assumptions in LLM Code Generation

    Jul 24, 2026Jie "JW" WuCode GenerationCode Generation Evaluation

    Michigan Technological University, Houghton, MI, USA

  31. Benchmarking LLMs for Verilog Design Flows

    Jul 23, 2026Angshuman Chakravertty, Rahul Koshti, Buddhi Prakash Sharma +1Code GenerationCode Generation Evaluation

    School of Technology Management and Engineering, NMIMS, Hyderabad, India · Department of Electronics and Communication Engineering, Indian Institute of Technology, Roorkee, India · Department of Electrical and Electronics Engineering and the Anuradha and Prashanth Palakurthi Centre for Artificial Intelligence Research (APPCAIR), Birla Institute of Technology and Science, Pilani (BITS Pilani), Pilani 333031, India

  32. ICAE-Bench: Evaluating Coding Agents as Interactive Project Builders

    Jul 23, 2026Zhongyuan Peng, Dan Huang, Chuyu Zhang +8Coding AgentsSoftware Engineering Benchmarks

    Fudan University · Meituan Group · Singapore Management University +1

  33. SciCodePile: A 128GB Corpus and Executable Benchmark for Challenging Scientific Code Generation

    Jul 21, 2026Weifeng Sun, Ye Fan, Yuchen Chen +6Scientific Code GenerationCode Generation Evaluation

    Singapore Management University · Nanjing University · SUN YAT-SEN University +1

  34. Spaghetti Architect: A Contamination-Resistant, By-Construction-Labelled, Multi-Language Code Dataset Generator

    Jul 21, 2026Yuxiang JiSynthetic Benchmark GenerationData Contamination in Language Models

    Sunway College

  35. Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality

    Jul 16, 2026Saima Afrin, Alessandro Midolo, Camilo Escobar-Velásquez +5Multilingual Language Model EvaluationCode Generation

    William & Mary, VA, USA · University of Catania, Italy · Universidad de los Andes, Colombia +2

  36. Alipay-PIBench: A Realistic Payment Integration Benchmark for Coding Agents

    Jul 16, 2026Shiyu Ying, Xuejie Cao, Yingfan Ma +4AI Coding AgentsSoftware Engineering Benchmarks

    Ant Group

  37. BackendForge: Benchmarking Agentic End-to-End Code Generation with Backend Services

    Jul 13, 2026Yuzhe Guo, Mengzhou Wu, Yuan Cao +4Tool-Augmented Language Model AgentsLanguage Model Generation Evaluation

    1Key Lab of HCST (PKU), MOE; SCS, Peking University · 2Beijing Tongming Lake Information Technology Application Innovation Center (TLAIC) · 3Fudan University Institute of Systems for Advanced Computing +1

  38. When Does Restricting a Coding Agent to execute_code Help? A Regime ×\times Agent-Design Ablation

    Jul 12, 2026Hong Yang, Qi Yu, Travis DesellCoding AgentsTool-Using Agents

    Rochester Institute of Technology Rochester, New York, USA

  39. PHITSBench: an execution-scored benchmark for AI-assisted PHITS radiation-transport input generation using natural language

    Jul 8, 2026Xianglin Ji, Svetlana V. BoriskinaKnowledge Augmentation for Language ModelsCode Generation

    Department of Mechanical Engineering, Massachusetts Institute of Technology, Cambridge, MA 02139, USA

  40. Dashboard2Code: Evaluating Multimodal Models on Reconstructing Interactive Dashboards

    Jul 6, 2026Tianhao Niu, Ziyu Han, Qiguang Chen +5Data VisualizationCode Generation Evaluation

    Research Center for Social Computing and Interactive Robotics Harbin Institute of Technology, China

  41. Beyond the Need for Speed: Energy-Aware Code Generation via Simulation-Guided Reinforcement Learning

    Jul 6, 2026Saurabhsingh Rajput, Tushar SharmaRL for Code GenerationReinforcement Learning

    Dalhousie University, Halifax, Canada

  42. Benchmarking API Drift in LLM-Generated Quantum Code Across Successive SDK Versions

    Jul 5, 2026Mohammad Arif Rasyidi, Syahirul FaizCode Generation BenchmarksLLM Code Generation

    Department of Computer Science Khalifa University Abu Dhabi, United Arab Emirates

  43. PairCoder++: Pair Programming as a Universal Paradigm for Verified Code-Driven Multimodal and Structured-Artifact Generation

    Jul 2, 2026Junhao Chen, Xiang Li, Mingjin Chen +9Execution-Guided Code GenerationCode Generation

    1THU · 2PKU · 3PolyU, Hong Kong +7

  44. AxDafny: Agentic Verified Code Generation in Dafny

    Jun 30, 2026Benjamin Breen, Austin Letson, Borja Requena Pozo +1Code GenerationFormal Verification

    Axiomatic AI, Boston, MA, USA.

  45. AlgoBench: Benchmarking Algorithmic Adaptation in Code Generation

    Jun 30, 2026Xinyuan Song, Zekun Cai, Liang ZhaoSynthetic Benchmark GenerationCode Generation Evaluation

    Emory University, Atlanta, GA, USA · The University of Tokyo, Tokyo, Japan · LocationMind, Tokyo, Japan

  46. MirrorCode: AI can rebuild entire programs from behavior alone

    Jun 29, 2026Tom Adamczewski, David Owen, David Rein +4Software Engineering AgentsAI Agent Benchmarks

    1Epoch AI · 2METR · 3Prime Intellect +2

  47. LibEvoBench: Probing Temporal Knowledge Stratification in Code Generation Models

    Jun 24, 2026Daniele Cipollone, Sergey Titov, Maliheh Izadi +2Software EngineeringTemporal Reasoning in Language Models

    JetBrains Research, Amsterdam, Netherlands · Faculty of EEMCS, Delft University of Technology, Delft, Netherlands.

  48. The Verifier is the Curriculum: Execution-Gated Self-Distillation for Cross-Family Game Generation

    Jun 23, 2026Chenyu Zhou, Qiliang Jiang, Shuning Wu +1Execution-Guided Code GenerationLanguage Model Distillation

    School of Engineering, Institute of Science Tokyo, Japan · College of Control Science and Engineering, Zhejiang University, China · Department of Electrical and Computer Engineering, National University of Singapore, Singapore

  49. Text2DSL: LLM-Based Code Generation for Domain-Specific Languages

    Jun 21, 2026Alexander V. Kozachok, Alexander M. Nazimov, Shamil G. MagomedovLLM PromptingCode Generation

    RTU MIREA — Russian Technological University, 78 Vernadsky Avenue, Moscow 119454, Russia · Academy of the Federal Guard Service of the Russian Federation, 35 Priborostroitelnaya Street, Oryol 302020, Russia

  50. NL2Scratch: An Executable Benchmark and Evaluation for Block-Based Programming

    Jun 20, 2026Heejin Do, Alexandre Ballenghien, Yang Wu +1Code GenerationCode Generation Evaluation

    Department of Computer Science, ETH Zurich · ETH AI Center

  51. Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages

    Jun 18, 2026Maria Ivanova, Pavel Zadorozhny, Rodion Levichev +5Code GenerationCode Generation Evaluation

    Yandex School of Data Analysis, Applied AI Institute · GigaCode

  52. JAMER: Project-Level Code Framework Dataset and Benchmark on Professional Game Engines

    Jun 18, 2026Jianwen Sun, Chuanhao Li, Zizhen Li +5Benchmark DesignCode Generation

    1Alaya Studio · 2Nankai University, Tianjin, China · 3Shanghai Innovation Institute, Shanghai, China

  53. StaminaBench: Stress-Testing Coding Agents over 100 Interaction Turns

    Jun 17, 2026Vlad Sobal, Shuo Yang, Yuting Zhang +2AI Coding AgentsBenchmark Design

    AWS Agentic AI

  54. Vibe Coding Ate My Homework: An evaluation of AI approaches to greenfield software engineering and programming

    Jun 15, 2026Callum BarbourSoftware EngineeringCode Generation Evaluation

  55. CODA-BENCH: Can Code Agents Handle Data-Intensive Tasks?

    Jun 13, 2026Yuxin Zhang, Ju Fan, Meihao Fan +2AI Coding AgentsSoftware Engineering Benchmarks

    Renmin University of China

  56. IterCAD: An Iterative Multimodal Agent for Visually-Grounded CAD Generation and Editing

    Jun 11, 2026Tao Hu, Jiaxin Ai, Licheng Wen +12Parametric CAD ModelingText-to-CAD Generation

    Shanghai Artificial Intelligence Laboratory · University of Science and Technology of China · Shanghai Innovation Institute +4

  57. VHDLSuite: Unified Pipeline for LLM VHDL Generation with Data Synthesis and Evaluation

    Jun 11, 2026Yijun Shen, Minghao Shao, Yichen Zhao +4LLM EvaluationCode Generation

    Center for Data Science, NYU Shanghai, China · NYU Tandon School of Engineering, USA · NYU Abu Dhabi, UAE

  58. Beyond Problem Solving: UOJ-Bench for Evaluating Code Generation, Hacking, and Repair in Competitive Programming

    Jun 11, 2026Tingqiang Xu, Hangrui Zhou, Tianle Cai +2Automated Program RepairCompetitive Programming

    Institute for Interdisciplinary Information Sciences, Tsinghua University · Universal Online Judge · ByteDance Seed +1

  59. HybridCodeAuthorship: A Benchmark Dataset for Line-Level Code Authorship Detection

    Jun 10, 2026Luke Patterson, Li Wang, Adam FaulknerAuthorship AttributionCode Generation Benchmarks

    Card Intelligence, Capital One

  60. CodeAlchemy: Synthetic Code Rewriting at Scale

    Jun 8, 2026Ankit Gupta, Aditya Prasad, Rameswar PandaSynthetic Data PretrainingSynthetic Data Generation

    MIT-IBM Watson AI Lab, IBM Research