Towards Functional Correctness of Code Models with Selective Generation

Jaewoo Jeong; Taesoo Kim; Sangdon Park

cs.SEMay 19, 2025

Towards Functional Correctness of Code Models with Selective Generation

Authors: Jaewoo Jeong, Taesoo Kim, Sangdon Park

Organizations: CSE, POSTECH · SCS&SCP GaTech · Microsoft · GSAI, POSTECH

Abstract

The hallucination of code generation models hinders their applicability to systems requiring higher safety standards. One critical bottleneck in addressing code hallucination is the difficulty of identifying the functional correctness of generated code, due to its unnatural form. We address this core bottleneck by automatically generating unit tests using dynamic code analysis tools, leveraging the executable nature of code. Accordingly, we propose a selective code generator that abstains from uncertain generations -- based on the functional correctness evaluated by generated unit tests -- to theoretically control the correctness among non-abstained answers, \ie the false discovery rate. Finally, we propose to use generated unit tests in evaluation as well as in learning for precise code evaluation, calling this paradigm FuzzEval. We demonstrate the efficacy of our method along with the controllability of code hallucination and reasonable selection efficiency.

Figures & tables

Figure 1: Overview of our proposed selective code generation. We leverage an abstaining option to selectively generate code to control the rate of hallucination in an FDR. Our selective generator learns a selection function by leveraging dynamic code analysis tools to automatically generate unit tests and use them as a calibration set for the selection function and also as a test set for evaluation.

Panel of Figure 2 — Figure 2: The box plots of the FDR-CE and selection efficiency for various LLMs ( $\delta_{S}=0.1$ , $\varepsilon_{S}=0.3$ , $\varepsilon_{E}=0.05$ , $\alpha=0.35$ ). See Figure for results with CodeLlama 13B Instruct.

Panel of Figure 3 — Figure 3: The FDR-CE results for GPT-4o with varying parameters and scoring functions. We use $\varepsilon_{S}=0.3,\delta_{S}=0.1,\alpha=0.35$ , and $\varepsilon_{E}=0.05$ for Figure 3(a) and 3(b) and $\varepsilon_{S}=0.2,\delta_{S}=0.1,\alpha=0.15$ , and $\varepsilon_{E}=0.05$ for Figure 3(c)

Code Generation

Code generation studies methods for turning natural-language, visual, or formal specifications into executable programs and code edits.

See all 322 papers

Functional Entropy: Predicting Functional Correctness in LLM-Generated Code with Uncertainty Quantification
May 27, 2026Dylan Bouchard, Mohit Singh Chauhan, Zeya Ahmad +1Language Model Calibration LLM Uncertainty Estimation
CVS Health®, Wellesley, MA, USA
Route-Align-Verify for Functional Correctness in Code Generation
Aug 4, 2026Erxue Zhou, Jingxiang Meng, Aofan LiuExecution-Guided Code Generation Code Generation
Software Engineering Institute, East China Normal University, Shanghai 200062, China · University of Chicago · Peking University
Verification Trap: Understanding Test-Time Selection Failures under False Premises in Code Generation
Oct 4, 2026Feng He, Hejia Wang, Linghao Meng +2Execution-Guided Code Generation Code Generation Evaluation
University of Science and Technology of China, China · Beijing University of Posts and Telecommunications, China · National University of Singapore, Singapore +1

May 27, 2026J/K move · Enter open · S save

Functional Entropy: Predicting Functional Correctness in LLM-Generated Code with Uncertainty Quantification

Dylan Bouchard, Mohit Singh Chauhan, Zeya Ahmad, Ho-Kyeong Ra

CVS Health®, Wellesley, MA, USA

Language Model Calibration LLM Uncertainty Estimation Automated Evaluation LLM Code Generation

Open paper AskPDF arXiv

Large language models have shown impressive capabilities in code generation, yet they often produce functionally incorrect code. Uncertainty quantification (UQ) methods have emerged as a promising approach for detecting hallucinations in natural language generation, but their effectiveness for code generation tasks remains underexplored. We systematically evaluate how UQ techniques transfer to code generation across three programming languages, five LLMs, and over 1,700 problems. We find that some token-probability-based methods generalize effectively without modification, while sampling-based methods relying on natural language inference (NLI) fail because NLI models cannot distinguish functionally different code, causing most responses to collapse into a single semantic cluster. To address this, we introduce functional equivalence methods, a family of code-specific methods that replace NLI-based semantic equivalence with an LLM-based functional equivalence assessment, including functional entropy, a code-specific analog of semantic entropy. Functional equivalence methods achieve top AUROC in 11 out of 15 model-benchmark combinations and the best calibration across most settings, consistently outperforming both NLI-based counterparts and all other methods evaluated.

Abstract

Code Generation

Explore similar work

Functional Entropy: Predicting Functional Correctness in LLM-Generated Code with Uncertainty Quantification

Route-Align-Verify for Functional Correctness in Code Generation

Verification Trap: Understanding Test-Time Selection Failures under False Premises in Code Generation