LLM Code Generation

LLM: Large Language Model

Overview

This field studies how large language models generate, edit, translate, and test programs from natural-language instructions, examples, or existing code. Common approaches include prompting and retrieval, fine-tuning and reinforcement learning, constrained decoding, and agentic loops that use execution, tests, or static analysis to refine outputs. Evaluations range from HumanEval, MBPP, and LiveCodeBench to repository-level and domain-specific tasks, increasingly measuring security, efficiency, and adherence to requirements alongside whether code runs. Models have improved on standard benchmarks, but reliability on unfamiliar APIs, complex projects, specialized languages, and implicit requirements remains a challenge, as do trade-offs among correctness, safety, cost, and maintainability.

Latest papers

336 papers16 papers in the last four weeks, down 45% on the four weeks before. 0.1% of all new papers.

Jan 19, 2025

Evaluating LLM-Based Regression Test Generation

Large Language Models (LLMs) have shown tremendous promise in automated software engineering. In this paper, we investigate LLMs for just-in-time regression test generation for programs, like parsers, interpreters, or compilers, that take highly structured, human-readable inputs. When a bug fix or code change is committed, the repository (as part of CI/CD) runs an LLM for a few minutes to generate regression tests that exercise the changed code and potentially trigger bugs. We frame LLM-based regression test generation as a machine translation task that takes the developer-provided commit message, the code change, and the input format name (e.g., XML), and produces regression tests for the described change in that format. Testing 72 commits to Mujs, Libxml2, Poppler, JerryScript, Z3, PHP, JQ, and MicroPython, our feedback-directed, zero-shot prototype Cleverest performed well, even without the code change. In under 2 minutes on average, Cleverest found as many bugs as the state-of-the-art directed greybox fuzzer WAFLGo did in 24 hours, even though WAFLGo started with a commit-reaching seed corpus in most cases. Using the Cleverest-generated tests as a seed corpus in coverage-guided greybox fuzzing doubles the number of bugs found; we call this integration ClevFuzz. We also find that some commit messages are more expressive than others, thus we ask how it impacts Cleverest's effectiveness. Cleverest picks up on the change intention: e.g., given a commit message that a patch changes how floating point variables are treated in the Mujs JavaScript interpreter, Cleverest generates JavaScript programs that contain floating point variables. Minimally changing commit messages to reduce or increase their information substantially impacts effectiveness: adding 17 words on average (max. 43) to make ineffective commit messages more expressive significantly increased the number of bugs found.
Jan 14, 2025

Leveraging Metamemory Agent for Enhanced Data-Free Code Generation in Large Language Models

Large language models (LLMs) have shown strong performance in automated code generation, with few-shot prompting widely used for its simplicity and effectiveness. However, few-shot methods depend on curated or manually crafted reference examples, limiting their applicability in data-free coding scenarios such as real-world data-free coding scenarios and benchmarks without training sets. Existing methods that generate reference examples via recitation or analogy cannot guarantee their authenticity or accuracy. Inspired by human metamemory, we propose a novel metamemory agent to enhance one-time code generation in data-free coding scenarios. The agent guides LLMs to recall relevant prior knowledge, evaluate confidence in recalled information, and selectively exploit reliable content for problem solving. This agent removes the need for external reference examples, improves the authenticity and accuracy of recalled knowledge, and adaptively tailors the recall&evaluation process to each task. Extensive experiments demonstrate that the proposed metamemory agent significantly improves one-time code generation quality across data-free coding scenarios. The AI contribution is the metamemory agent, which makes self-recalled examples reliable through confidence evaluation and selection; the engineering application is data-free automated code generation, validated on eight public benchmarks.
Oct 29, 2024

Can Language Models Replace Programmers for Coding? REPOCOD Says 'Not Yet'

Recently, a number of repository-level code generation benchmarks-such as CoderEval, DevEval, RepoEval, RepoBench, and LongCodeArena-have emerged to evaluate the capabilities of large language models (LLMs) beyond standalone benchmarks like HumanEval and MBPP. Thus, a natural question is, would LLMs have similar performance in real world coding tasks as their performance in these benchmarks? Unfortunately, one cannot answer this question, since these benchmarks consist of short completions, synthetic examples, or focus on limited scale repositories, failing to represent real-world coding tasks. To address these challenges, we create REPOCOD, a Python code-generation benchmark containing complex tasks with realistic dependencies in real-world large projects and appropriate metrics for evaluating source code. It includes 980 whole-function generation tasks from 11 popular projects, 50.8% of which require repository-level context. REPOCOD includes 314 developer-written test cases per instance for better evaluation. We evaluate ten LLMs on REPOCOD and find that none achieves more than 30% pass@1 on REPOCOD, indicating the necessity of building stronger LLMs that can help developers in real-world software development. In addition, we found that retrieval-augmented generation achieves better results than using target function dependencies as context.
Jun 22, 2024

BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions

Task automation has been greatly empowered by the recent advances in Large Language Models (LLMs) via Python code, where the tasks ranging from software engineering development to general-purpose reasoning. While current benchmarks have shown that LLMs can solve tasks using programs like human developers, the majority of their evaluations are limited to short and self-contained algorithmic tasks or standalone function calls. Solving challenging and practical tasks requires the capability of utilizing diverse function calls as tools to efficiently implement functionalities like data analysis and web development. In addition, using multiple tools to solve a task needs compositional reasoning by accurately understanding complex instructions. Fulfilling both of these characteristics can pose a great challenge for LLMs.To assess how well LLMs can solve challenging and practical tasks via programs, we introduce BigCodeBench, a benchmark that challenges LLMs to invoke multiple function calls as tools from 139 libraries and 7 domains for 1,140 fine-grained tasks. To evaluate LLMs rigorously, each task encompasses 5.6 test cases with an average branch coverage of 99%. In addition, we propose a natural-language-oriented variant of BigCodeBench, BigCodeBench-Instruct, that automatically transforms the original docstrings into short instructions only with essential information. Our extensive evaluation of 60 LLMs shows that LLMs are not yet capable of following complex instructions to use function calls precisely, with scores up to 60%, significantly lower than the human performance of 97%. The results underscore the need for further advancements in this area.
May 31, 2024

Grammar-Aligned Decoding

Large Language Models (LLMs) struggle with reliably generating highly structured outputs, such as program code, mathematical formulas, or well-formed markup. Constrained decoding approaches mitigate this problem by greedily restricting what tokens an LLM can output at each step to guarantee that the output matches a given constraint. Specifically, in grammar-constrained decoding (GCD), the LLM's output must follow a given grammar. In this paper, we demonstrate that GCD techniques (and in general constrained decoding techniques) can distort the LLM's distribution, leading to outputs that are grammatical but appear with likelihoods that are not proportional to the ones given by the LLM, and so ultimately are low-quality. We call the problem of aligning sampling with a grammar constraint, grammar-aligned decoding (GAD), and propose adaptive sampling with approximate expected futures (ASAp), a decoding algorithm that guarantees the output to be grammatical while provably producing outputs that match the conditional probability of the LLM's distribution conditioned on the given grammar constraint. Our algorithm uses prior sample outputs to soundly overapproximate the future grammaticality of different output prefixes. Our evaluation on code generation and structured NLP tasks shows how ASAp often produces outputs with higher likelihood (according to the LLM's distribution) than existing GCD techniques, while still enforcing the desired grammatical constraints.
Nov 6, 2023

Leveraging High-Level Synthesis and Large Language Models to Generate, Simulate, and Deploy a Uniform Random Number Generator Hardware Design

We present a new high-level synthesis methodology for using large language model tools to generate hardware designs. The methodology uses exclusively open-source tools excluding the large language model. As a case study, we use our methodology to generate a permuted congruential random number generator design with a wishbone interface. We verify the functionality and quality of the random number generator design using large language model-generated simulations and the Dieharder randomness test suite. We document all the large language model chat logs, Python scripts, Verilog scripts, and simulation results used in the case study. We believe that our method of hardware design generation coupled with the open source silicon 130 nm design tools will revolutionize application-specific integrated circuit design. Our methodology significantly lowers the bar to entry when building domain-specific computing accelerators for the Internet of Things and proof of concept prototypes for later fabrication in more modern process nodes.