LLM Code Generation

LLM: Large Language Model

This field studies how large language models generate, edit, translate, and test programs from natural-language instructions, examples, or existing code. Common approaches include prompting and retrieval, fine-tuning and reinforcement learning, constrained decoding, and agentic loops that use execution, tests, or static analysis to refine outputs. Evaluations range from HumanEval, MBPP, and LiveCodeBench to repository-level and domain-specific tasks, increasingly measuring security, efficiency, and adherence to requirements alongside whether code runs. Models have improved on standard benchmarks, but reliability on unfamiliar APIs, complex projects, specialized languages, and implicit requirements remains a challenge, as do trade-offs among correctness, safety, cost, and maintainability.

Latest papers

314 papers23 papers in the last four weeks, down 12% on the four weeks before. 0.1% of all new papers.

Beyond Passing Tests: A Systematic Review and Controlled Re-evaluation of Techniques for Improving the Quality of LLM-Generated Code
Oct 28, 2026Prashank KadamLLM Code Generation Code Generation Evaluation
Cerby Inc., USA
Reasoning Patterns for Efficient AI Coding: An Empirical Study
Oct 26, 2026Zhe Yin, Yuling Shi, Xiaodong Gu +1Efficient Language Model Reasoning Large Reasoning Models
Shanghai Jiao Tong University, China
Software Engineering LLMs: Why We Need Them and How Software Engineering Helps Build Them
Oct 24, 2026Amir M. Ebrahimi, Mohammed Mehedi Hasan, Seyed Shayan Daneshvar +3Code Language Models LLM Training
Queen’s University, Canada · University of Manitoba, Canada · Center for Software Excellence, Huawei Technologies, Canada
Self-Decomposing Diffusion for Code Generation
Oct 20, 2026Lize Shao, Zichen Xie, Zhizhen Chen +2Code Generation LLM Code Generation
University of Virginia
Many Repairs, One Direction: Steering Code Models Toward Safer Cryptographic API Use
Oct 18, 2026Tianhe Lu, Sakuna Harinda Jayasundara, Eric Spero +3Secure Code Generation LLM Code Generation
University of Auckland Auckland, New Zealand
Do LMs Know What Variables Do? Interpreting Code Representations via Variable Roles
Oct 16, 2026Célian Vasson, Daniil Ognev, Kentaro Inui +1LLM Interpretability Code Language Models
Sorbonne Université · MBZUAI · RIKEN AIP +1
Poison Valleys: Diagnosing Path-Dependent Failure in LLM Code Repair
Oct 15, 2026Veljko SkarichAgent Failure Analysis LLM Code Generation
LLMs program using Reusable and Composable Abstract Representations
Oct 13, 2026Casper L. Christensen, Kyle J. Ray, Dmitry Manning-Coe +2LLM Code Generation Compositional Representation Learning
Simplex · Astera Institute · MATS +1
HERO: Harnessing Evidence for Test-time Refinement and Prompt Optimization in Program Generation
Oct 13, 2026Mingyu Lu, Timothy Ossowski, Tristan Naumann +2Automatic Prompt Optimization LLM Code Generation
University of Washington · Microsoft Research · Recursion
SolBehave: Evaluating the Behavioural Correctness of LLM-Generated Smart Contracts
Oct 13, 2026Giovanni Pio Delvecchio, Stefano Fantazzini, Giacomo Frisoni +4Code Generation Evaluation LLM Code Generation
University of Bologna, Bologna, Italy · University of Copenhagen, Copenhagen, Denmark
SpecCloak: Probing Harmful Code Generation through Engineering Specifications
Oct 13, 2026Wenbin Chen, Jiexin Wang, Guanting Wu +2LLM Security LLM Code Generation
South China University of Technology
PhySimCode: A Benchmark and Evaluation Method for Physics Video to Code Generation
Oct 11, 2026Souraja Kundu, Aditya Gupta, Joel Julin +3Physical Reasoning in Video LLM Code Generation
Carnegie Mellon University · Amazon AGI
Characterizing Overconfident Failure in LLM-Based Code Generation
Oct 8, 2026Ravishka Rathnasuriya, Wei YangLLM Code Generation Confidence Estimation in Language Models
Institute of Systems for Advanced Computing, Fudan University, Shanghai, China
LLM-Assisted Generation of Transparent, Open-Source Multiphysics Models of Electrochemical Devices
Oct 7, 2026Sebastian Castro, Maya F. Schuchert, Spencer A. McCluskey +2Multiphysics Simulation LLM Code Generation
Department of Chemical and Biomolecular Engineering, New York University, Brooklyn, NY, USA · Department of Chemical and Biological Engineering, University of British Columbia, Vancouver, BC, Canada
Visible Reasoning Is Not a Universal Optimizer: Persona- and Thinking-Dependent Effects in Analytics Code Generation
Oct 7, 2026Bhawani Shankar Leelar, Pawan Chorasiya, Davin Hill +2CoT Reasoning LLM Code Generation
Optum AI
WorldBench: Evaluating LLMs on Three.js Voxel World Generation
Oct 7, 2026Krish BakshiLLM Code Generation 3D Scene Generation
VHDL-REPOBENCH: A Repository-Level Benchmark for Evaluating Large Language Models on VHDL Design Generation
Oct 4, 2026Prashanth Vijayaraghavan, Akul Malhotra, Ashutosh Jadhav +2LLM Evaluation Code Generation
IBM Research San Jose, CA, USA
Grammar-Guided Code Watermarking with Green Temperature
Oct 4, 2026Hyundong Jin, Hyeseon An, Soohan Lim +1Text Watermarking LLM Watermarking
Department of Computer Science Yonsei University, Seoul, Republic of Korea
Hidden in the Comments: A Context-Injection Attack Surface in Code LLMs
Oct 4, 2026Noor Munir, Francesco Quinzan, Stephen RobertsLLM Security Indirect Prompt Injection
Department of Engineering Science, University of Oxford, Oxford OX1 3PJ, UK
Spec2Game: Can LLMs Generate Complete Playable Games from Detailed Specifications?
Oct 3, 2026Yixue Cai, Yuzhe Zhao, Hanxiang Chao +4Code Generation LLM Code Generation
The Chinese University of Hong Kong Hong Kong, China · Nankai University Tianjin, China · Wuhan University Wuhan, China
Improving the Energy-Efficiency of the Code Generated by LLMs through Effective Prompting
Oct 1, 2026Ritika Rekhi, Bing Zhang, Md Arman Islam +7LLM Code Generation Code Generation
University at Buffalo, Buffalo, NY, USA · National Center for Supercomputing Applications, Urbana, IL, USA · IBM Research, Yorktown Heights, NY, USA
Where LLMs Fail with Visualization DSLs
Oct 1, 2026Chang Han, Andrew McNutt, Katherine IsaacsData Visualization LLM Evaluation
All authors are with University of Utah.
QuantCode Model: Specializing Language Models for Executable Algorithmic Trading Code
Sep 30, 2026Alexey Chernysh, Orkhan Ekhtibarov, Dmitry ZmitrovichLLM Fine-Tuning Domain-Adaptive Pretraining
ReMCTS: Reflection-Enhanced Monte Carlo Tree Search for Code Generation
Sep 28, 2026Huifei Wang, Xinying Huang, Yiheng Sun +1Execution-Guided Code Generation LLM-Based Program Synthesis
College of Computer Science and Software Engineering, Shenzhen University · School of Artificial Intelligence, Shenzhen University
Improving Large Language Models for Code through Runtime Program-State Reasoning
Sep 28, 2026Hongwei Li, Spandan Garg, Yufan HuangAutomated Program Repair Automated Test Generation
University of California, Santa Barbara · Microsoft
How code helps different tasks? A decompositional lens on LLM post-training
Sep 27, 2026Zheng Yu, Yiwei Li, Yishen Chen +4LLM Fine-Tuning Instruction-Tuning Data Selection
Sun Yat-sen University · Shenzhen Loop Area Institute · The Chinese University of Hong Kong, Shenzhen +1
Noisy Test-Time Reinforcement Learning for Code LLMs
Sep 26, 2026Xikai Yang, Hieu Trung Nguyen, Dunyuan Xu +4Language Model Robustness RL for Code Generation
The Chinese University of Hong Kong · Huazhong University of Science and Technology
Improving Medical Calculation of LLMs with Embedded Coding
Sep 25, 2026Tianshi Ming, Yingying Zhang, Xian WuMedical LLMs Numerical Reasoning in Language Models
Carnegie Mellon University · Tencent YouTu Lab
A Framework for Identifying, Categorizing, and Explaining Bias in AI-Generated Code
Sep 25, 2026Manaal Basha, Aimee M. Ribeiro, Gema Rodriguez-PerezLanguage Model Bias Evaluation LLM Code Generation
University of British Columbia, Kelowna, Canada · Federal University of Pará, Belém, Brazil
Beyond Rule-Based Mutation Testing: Test-Aware Mutant Generation Using Large Language Models
Sep 24, 2026Nils Kiele, Zainab Saad, Zirui Wang +2Automated Software Testing Mutation Testing
Department of Electrical and Software Engineering, University of Calgary, AB, Canada

Oct 28, 2026J/K move · Enter open · S save

Beyond Passing Tests: A Systematic Review and Controlled Re-evaluation of Techniques for Improving the Quality of LLM-Generated Code

Prashank Kadam

Cerby Inc., USA

LLM Code Generation Code Generation Evaluation LLM Reliability Best-of-N Sampling

Open paper AskPDF arXiv

Large language models pass most functional-correctness benchmarks, but passing tests says little about whether their code is readable, efficient or secure. Many techniques claim to improve these qualities, yet their evidence is scattered, usually covers one quality dimension and is rarely compared at an equal compute budget. We contribute a systematic review and a controlled re-evaluation. A systematic literature review of 628 primary studies (July 2021 to October 2026), with disclosed, author-reviewed LLM assistance, organizes the techniques by stage, signal source and quality target. Most studies measure only the dimension they target, gains often come with unchanged or unreported correctness, single automated quality signals are gamed, and compute, variance and contamination are rarely reported. A controlled re-evaluation runs ten techniques on four models at matched budgets on correctness, maintainability, efficiency and security benchmarks, extends it to thirteen current models and repository-level tasks, and compares 66 highlighted techniques head-to-head on Gemma 4 31B. At one call, quality prompting raises analyzer issue counts and leaves complexity and judge scores almost unchanged. At matched budgets, best-of-n sampling with an analyzer-weighted ranker matches or beats prompting, planning, self-refinement, a reviewer agent and retrieval on measures the ranker does not optimize, while an analyzer feedback loop and fine-tuning beat it only on analyzer issue counts. Large correctness losses on HumanEval+, a likely contaminated benchmark near its ceiling, do not appear on contests held after the models' training cutoffs. In the head-to-head, 1 of 66 techniques beats best-of-n on its primary target beyond the 95% interval, a rate consistent with chance. Gains should be reported against a budget-matched sampling baseline and on measures independent of the technique's own signal.