Code Generation Evaluation

Follow this topic Ask about this topic

Momentum

17 papers in the last four weeks, up 113% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 147

All topics

Delulu: A Verified Multi-Lingual Benchmark for Code Hallucination Detection in Fill-in-the-Middle Tasks
May 7, 2026Mahdi Erfanian, Nelson Daniel Troncoso, Aashna Garg +4Language Model Error Detection Language Model Generation Evaluation
University of Illinois Chicago · Microsoft
Correct Code, Vulnerable Dependencies: A Large Scale Measurement Study of LLM-Specified Library Versions
May 7, 2026Chengjie Wang, Jingzheng Wu, Xiang Ling +2Software Engineering LLM Security
Intelligent Software Research Center, Institute of Software, Chinese Academy of Sciences, Beijing, China · Key Laboratory of System Software (Chinese Academy of Sciences), Beijing, China
The Single-File Test: A Longitudinal Public-Interface Evaluation of First-Output LLM Web Generation with Social Reach Tracking
May 6, 2026Diego Cabezas PalaciosCode Generation Code Generation Evaluation
Beyond BLEU: A Semantic Evaluation Method for Code Translation
May 6, 2026Julius Näumann, Sven Keidel, Amir Molzam Sharifloo +1Code Translation Code Generation Evaluation
∗TU Darmstadt · †Hessian Center for Artificial Intelligence, Darmstadt, Germany · ‡National Research Center for Applied Cybersecurity ATHENE
Bridging Generation and Training: A Systematic Review of Quality Issues in LLMs for Code
May 6, 2026Kaifeng He, Xiaojun Zhang, Peiliang Cai +7Training Data Curation Code Generation Evaluation
Sun Yat-sen University, China · Nanyang Technological University, Singapore, Singapore · Tongji University, Shanghai, China +1
Measuring and Mitigating Bias in Code Generated by Large Language Models
May 5, 2026Yuxi Chen, Yutian Tang, Timothy StorerAlgorithmic Bias Language Model Bias Evaluation
School of Computing Science, University of Glasgow, United Kingdom.
MolViBench: Evaluating LLMs on Molecular Vibe Coding
May 4, 2026Jiatong Li, Yuxuan Ren, Weida Wang +4AI-Assisted Scientific Research Scientific Code Generation
The Hong Kong Polytechnic University · National University of Singapore · Fudan University
Social Bias in LLM-Generated Code: Benchmark and Mitigation
May 1, 2026Fazle Rabbi, Lin Ling, Song Wang +1Social Bias in Language Models Agentic Code Generation
Concordia University, Montreal, Canada · Lassonde School of Engineering, York University, Toronto, Canada
Defective Task Descriptions in LLM-Based Code Generation: Detection and Analysis
Apr 27, 2026Amal Akli, Mike Papadakis, Maxime Cordy +1Code Generation Evaluation Language Model Robustness
University of Luxembourg Luxembourg
Characterizing Paraphrase-Induced Failures in Lean 4 Autoformalization
Apr 25, 2026William Feng, Ethan Lou, Aryan SharmaLean Theorem Proving Code Generation Evaluation
Yale University, New Haven, Connecticut, USA.
BLAST: Benchmarking LLMs with ASP-based Structured Testing
Apr 24, 2026Manuel Alejandro Borroto Santana, Erica Coppolillo, Francesco Calimeri +3Logic Programming LLM Evaluation
University of Calabria, Rende, Italy · ICAR-CNR, Rende, Italy · Artificial Intelligence LAB, DeMaCS, University of Calabria, Rende, Italy +1
From If-Statements to ML Pipelines: Revisiting Bias in Code-Generation
Apr 23, 2026Minh Duc Bui, Xenia Heilmann, Mattia Cerrato +2Algorithmic Bias Code Generation Evaluation
Johannes Gutenberg University Mainz, Germany · Universidad Iberoamericana, Ciudad de Mexico · University of Colorado Boulder, USA
Co-Located Tests, Better AI Code: How Test Syntax Structure Affects Foundation Model Code Generation
Apr 20, 2026Éric JacopinCode Generation Code Generation Evaluation
Cosmic AI
Playing Psychic: Using Thought Trees to Predict Reasoning Models Accuracy on Coding Tasks
Apr 18, 2026Jiaxin Fang, Runyuan He, Sahil Bhatia +2Structured Reasoning Code Generation Evaluation
University of California, Berkeley
Bias in the Loop: Auditing LLM-as-a-Judge for Software Engineering
Apr 18, 2026Zixiao Zhao, Amirreza Esmaeili, Fatemeh FardSoftware Engineering LLM-as-a-Judge
University of British Columbia, Canada
QuantCode-Bench: A Benchmark for Evaluating the Ability of Large Language Models to Generate Executable Algorithmic Trading Strategies
Apr 16, 2026Alexey Khoroshilov, Alexey Chernysh, Orkhan Ekhtibarov +2Agentic Code Generation Code Generation Evaluation
Lime
From Paper to Program: Knowledge Externalization and Bottleneck Diagnosis in AI-Assisted Quantum Many-Body Programming
Apr 5, 2026Yi ZhouScientific Code Generation Code Generation Evaluation
Institute of Physics, Chinese Academy of Sciences, Beijing 100190, China
ACES: Who Tests the Tests? Leave-One-Out AUC Consistency for Code Generation
Apr 5, 2026Hui Sun, Yun-Ji Zhang, Zheng Xie +4Language Model Generation Evaluation Code Generation Evaluation
National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China
Declarative by Design, Assistable Only by Convention: Benchmarking Multi-Agent Frameworks for AI-Assistability
Feb 3, 2026Shafiuddin Rehan Ahmed, Sourabh DeshpandeAI Agent Benchmarks Code Generation Evaluation
Center for Advanced AI, Accenture · Mountain View, California, USA
Can Coding Agents Migrate to Post-Quantum Cryptography?
Dec 15, 2025Abdulmalik AlquwayfiliCoding Agents Code Generation Evaluation
Saudi Data & AI Authority, Riyadh, Saudi Arabia
From Charts to Code: A Hierarchical Benchmark for Multimodal Models
Oct 20, 2025Jiahao Tang, Henry Hengyuan Zhao, Lijian Wu +8Data Visualization Code Generation
CSU-JPG, Central South University · National University of Singapore · Nanyang Technological University
LiveOIBench: Can Large Language Models Outperform Human Contestants in Informatics Olympiads?
Oct 10, 2025Kaijian Zou, Aaron Xiong, Yunxiang Zhang +6LLM Evaluation Competitive Programming
University of Michigan, Ann Arbor.
Investigating The Smells of LLM Generated Code
Oct 3, 2025Debalina Ghosh Paul, Hong Zhu, Ian BayleySoftware Engineering Code Generation Evaluation
School of Engineering, Computing and Mathematics, Oxford Brookes University, Oxford, OX3 0BP, UK
Benchmarking Web API Integration Code Generation
Sep 24, 2025Daniel Maninger, Leon Chemnitz, Amir Molzam Sharifloo +2Code Generation Code Generation Evaluation
Technische Universität Darmstadt, Germany · Hessian Center for Artificial Intelligence (hessian.AI), Germany · Pariton AI, Germany +1
Experimental Analysis of Productive Interaction Strategy with ChatGPT: User Study on Function and Project-level Code Generation Tasks
Aug 6, 2025Sangwon Hyun, Hyunjun Kim, Jinhyuk Jang +2LLM Prompting Code Generation
Adelaide University, Adelaide, SA, Australia · Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea
Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility
Jan 18, 2025Jialun Cao, Yuk-Kit Chan, Zixuan Ling +12Benchmark Construction Benchmark Design
The Hong Kong University of Science and Technology · Guangzhou HKUST Fok Ying Tung Research Institute · The Chinese University of Hong Kong +6
An Agentic Evaluation Framework for AI-Generated Scientific Code in PETSc
Date pendingHong Zhang, Barry Smith, Satish Balay +4High-Performance Computing Code Generation
Argonne National Laboratory, Lemont, IL, USA

May 7, 2026, cs.LGJ/K move · Enter open · S save

Delulu: A Verified Multi-Lingual Benchmark for Code Hallucination Detection in Fill-in-the-Middle Tasks

Mahdi Erfanian, Nelson Daniel Troncoso, Aashna Garg, Amabel Gale+3

University of Illinois Chicago · Microsoft

Language Model Error Detection Language Model Generation Evaluation Hallucination in Language Models Code Generation LLM Reliability LLM Hallucination Detection+2

Open paper AskPDF arXiv

Large Language Models for code generation frequently produce hallucinations in Fill-in-the-Middle (FIM) tasks -- plausible but incorrect completions such as invented API methods, invalid parameters, undefined variables, or non-existent imports. These failures pass superficial review yet introduce runtime errors. We introduce Delulu, a verified multi-lingual benchmark of 1,951 FIM samples across 7 languages and 4 hallucination types. Samples are curated through an adversarial pipeline: a frontier LLM generates plausible hallucinations, four diverse judge models evaluate them, embedding-based clustering mines progressively harder examples, self-contained Docker containers verify that golden completions compile while hallucinated variants produce the expected runtime error, and a final human-expert review removes any remaining biased or trivially decidable samples. We evaluate 11 open-weight FIM models from five families spanning 0.5B-32B parameters: a six-point Qwen2.5-Coder scaling slate, plus a cross-family slate (CodeLlama, DeepSeek-Coder-V2, StarCoder2). The strongest model reaches only 84.5% pass@1, no family exceeds 0.77 Edit Similarity, and every family produces hallucination-aligned completions on a non-trivial share of samples, confirming that the difficulty exposed by Delulu is task-intrinsic rather than family-specific. We release the benchmark, containers, and evaluation framework at https://github.com/microsoft/delulu.

Momentum

Related topics

Latest papers 147