Code Generation Benchmarks

Ask about this topic

Momentum

15 papers in the last four weeks, up 275% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 104

All topics
CardsList

  1. Context-Augmented Code Generation: How Product Context Improves AI Coding Agent Decision Compliance by 49%

    Apr 27, 2026Drew Dillon, Kasyap VaranasiRetrieval-Augmented GenerationAI Coding Agents

    Brief

  2. BLAST: Benchmarking LLMs with ASP-based Structured Testing

    Apr 24, 2026Manuel Alejandro Borroto Santana, Erica Coppolillo, Francesco Calimeri +3Logic ProgrammingLLM Evaluation

    University of Calabria, Rende, Italy · ICAR-CNR, Rende, Italy · Artificial Intelligence LAB, DeMaCS, University of Calabria, Rende, Italy +1

  3. WebCompass: Towards Multimodal Web Coding Evaluation for Code Language Models

    Apr 20, 2026Xinping Lei, Xinyu Che, Junqi Xiong +16LLM Agent EvaluationCode Language Models

    Nanjing University · Kuaishou Technology

  4. Playing Psychic: Using Thought Trees to Predict Reasoning Models Accuracy on Coding Tasks

    Apr 18, 2026Jiaxin Fang, Runyuan He, Sahil Bhatia +2Structured ReasoningCode Generation Evaluation

    University of California, Berkeley

  5. ACES: Who Tests the Tests? Leave-One-Out AUC Consistency for Code Generation

    Apr 5, 2026Hui Sun, Yun-Ji Zhang, Zheng Xie +4Language Model Generation EvaluationCode Generation Evaluation

    National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China

  6. VectorGym: A Multi-Task Benchmark for SVG Code Generation, Sketching and Editing

    Feb 22, 2026Joan Rodriguez, Haotian Zhang, Abhay Puri +14VLM EvaluationCode Generation

    Mila, Quebec AI Institute · ÉTS Montréal · QuiverAI +11

  7. 1D-Bench: A Benchmark for Iterative UI Code Generation with Visual Feedback in Real-World

    Feb 20, 2026Qiao Xu, Yipeng Yu, Chengxiao Feng +1Benchmark DesignExecution-Guided Code Generation

    Taobao & Tmall Group of Alibaba

  8. HardSecBench: Benchmarking the Security Awareness of LLMs for Hardware Code Generation

    Jan 20, 2026Qirui Chen, Jingxian Shuai, Shuangwu Chen +8LLM SecuritySecure Code Generation

    University of Science and Technology of China, China · Xi’an Jiaotong University, China · Nanjing University, China +1

  9. Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks

    Dec 2, 2025Songwen Zhao, Danqing Wang, Kexun Zhang +3AI Coding AgentsSoftware Security

    Carnegie Mellon University, Language Technologies Institute · Columbia University · Johns Hopkins University +1

  10. LiveOIBench: Can Large Language Models Outperform Human Contestants in Informatics Olympiads?

    Oct 10, 2025Kaijian Zou, Aaron Xiong, Yunxiang Zhang +6LLM EvaluationCompetitive Programming

    University of Michigan, Ann Arbor.

  11. Benchmarking Web API Integration Code Generation

    Sep 24, 2025Daniel Maninger, Leon Chemnitz, Amir Molzam Sharifloo +2Code GenerationCode Generation Evaluation

    Technische Universität Darmstadt, Germany · Hessian Center for Artificial Intelligence (hessian.AI), Germany · Pariton AI, Germany +1

  12. BigO(Bench): Can LLMs Generate Code with Controlled Time and Space Complexity?

    Mar 19, 2025Pierre Chambon, Baptiste Roziere, Benoit Sagot +1Benchmark DesignCode Generation

    FAIR at Meta · Inria · Work done at Meta, now working at Mistral AI

  13. An Agentic Evaluation Framework for AI-Generated Scientific Code in PETSc

    Date pendingHong Zhang, Barry Smith, Satish Balay +4High-Performance ComputingCode Generation

    Argonne National Laboratory, Lemont, IL, USA

  14. PhysCodeBench: Benchmarking Physics-Aware Symbolic Simulation of 3D Scenes via Self-Corrective Multi-Agent Refinement

    Date pendingTianyidan Xie, Peiyu Wang, Hu Jiaxin +9LLM-Based Program SynthesisScientific Code Generation

    Nanjing University · Skywork AI · Jilin University +2