An Empirical Analysis of Static Analysis Methods for Detection and Mitigation of Code Library Hallucinations

Clarissa Miranda-Pena; Andrew Reeson; Cécile Paris; Josiah Poon; Jonathan K. Kummerfeld

cs.CLApr 9, 2026

An Empirical Analysis of Static Analysis Methods for Detection and Mitigation of Code Library Hallucinations

Authors: Clarissa Miranda-Pena, Andrew Reeson, Cécile Paris, Josiah Poon, Jonathan K. Kummerfeld

Organizations: The University of Sydney · CSIRO’s Data61

Abstract

Despite extensive research, Large Language Models continue to hallucinate when generating code, particularly when using libraries. On NL-to-code benchmarks that require library use, we find that LLMs generate code that uses non-existent library features in 8.1-40% of responses. One intuitive approach for detection and mitigation of hallucinations is static analysis. In this paper, we analyse the potential of static analysis tools, both in terms of what they can solve and what they cannot. We find that static analysis tools can detect 16-70% of all errors, and 14-85% of library hallucinations, with performance varying by LLM and dataset. Through manual analysis, we identify cases a static method could not plausibly catch, which gives an upper bound on their potential from 48.5% to 77%. Overall, we show that static analysis methods are cheap method for addressing some forms of hallucination, and we quantify how far short of solving the problem they will always be.

Figures & tables

All Errors claude-3 gpt-3.5 gpt-4 Granite OP OR OP OR OP OR OP OR DS-1000 Mypy 0.22 0.27 0.26 0.34 0.16 0.23 0.63 0.26 Pyright 0.61 0.70 0.57 0.60 0.50 0.62 0.95 0.55 o3-mini 0.63 0.43 0.61 0.45 0.55 0.43 0.79 0.51 Grammar 0.36 0.14 0.45 0.11 0.53 0.15 0.57 0.10 Odex Mypy 0.85 0.15 0.39 0.66 0.43 0.81 0.81 0.18 Pyright 0.76 0.22 0.39 0.68 0.43 0.82 0.72 0.34 o3-mini 0.61 0.09 0.61 0.22 0.79 0.32 0.74 0.34 Grammar 0.35 0.08 0.43 0.08 0.52 0.11 0.59 0.15 BigCode Mypy 0.16 0.05 0.22 0.07 0.24 0.09 0.47 0.09 Pyright 0.32 0.20 0.28 0.16 0.34 0.20 0.62 0.30 o3-mini 0.64 0.12 0.56 0.16 0.57 0.11 0.77 0.48 Grammar 0.19 0.04 0.24 0.12 0.23 0.06 0.45 0.09 Hallucinations claude-3 gpt-3.5 gpt-4 Granite IF R IF R IF R IF R 0.22 0.23 0.26 0.21 0.85 0.75 0.84 0.52 0.29 0.29 0.26 0.68 0.16 0.15 0.20 0.17 0.17 0.60 0.70 0.17 0.26 0.64 0.71 0.36 0.08 0.11 0.08 0.34 0.11 0.10 0.12 0.17 0.03 0.03 0.08 0.10 0.24 0.14 0.24 0.36 0.15 0.14 0.14 0.50 0.06 0.11 0.06 0.06

Table 1: On the left: Detection results for all query models, detection methods, and datasets. We defined OP as the overall precision and OR as the overall recall when detecting all types of bugs. On the right: IF R represents recall only on imaginary features. We do not show IF precision as it was 1 in all cases, except for o3-mini in IBM-Granite on DS-1000, which was 0.98.

	Caught	TP	Overlap	Capable
DS-1000
Static	56.7	37.6	17.9	77.0
Grammar	11.2	7.9	85.7	18.0
Odex
Static	19.0	4.0	12.5	70.5
Grammar	6.5	2.5	20.0	10.5

Table 2: Manual analysis of a sample of hallucinations across benchmarks. Where Static is the union of a hallucination been flagged by either Mypy or Pyright. See 5.2 for metric definitions.

Hallucination cause	DS-1000	Odex	BigCode
During generation
Flow	57.1	42.4	34.2
Library	14.7	6.6	6.0
Data-Logic	8.5	6.1	7.5
Ambiguous input	0.0	6.6	7.5
During test

Table 3: Categories that triggered a hallucination and their percentages on the sample of each benchmark.

claude-3 gpt-3.5 gpt-4 Granite ER P@1 ER P@1 ER P@1 ER P@1 DS-1000 o3+Self 82.3 44.3 82.8 37.9 85.3 49.3 56.3 12.0 o3+Mypy 80.0 43.7 80.3 37.7 82.5 49.1 45.5 10.1 o3+Pyright 80.3 44.3 79.9 37.4 82.1 48.7 44.5 10.1 No repair 73.1 42.7 73.2 36.6 76.8 47.7 33.6 8.9 Odex o3+Self 70.3 43.6 67.3 40.4 67.5 43.2 50.3 19.6 o3+Mypy 71.1 44.2 67.3 40.2 67.5 43.0 48.5 19.0 o3+Pyright 71.3 44.0 67.7 40.6 68.1 43.2 50.3 19.4 No repair 66.9 42.2 59.6 35.8 57.0 35.2 46.3 18.0 BigCode o3+Self 82.6 47.3 82.1 41.3 82.6 47.2 77.7 26.7 o3+Mypy 79.0 45.5 77.9 39.4 80.0 46.1 62.2 21.1 o3+Pyright 80.0 45.9 78.3 39.4 80.4 46.1 66.7 23.2 No repair 79.0 45.5 77.6 39.3 79.6 46.0 61.0 20.6 claude-3 gpt-3.5 gpt-4 Granite RIF RIF RIF RIF 2.7 2.6 1.9 5.2 2.9 3.3 1.7 8.1 3.2 3.5 2.4 8.6 9.1 9.7 8.1 11.4 16.6 15.8 16.0 23.0 16.2 16.8 15.6 24.2 16.2 16.4 16.0 23.4 20.4 15.8 13.7 24.0 29.8 34.3 35.4 39.0 30.5 35.7 36.4 39.0 28.9 36.4 34.5 38.2 30.1 35.7 36.6 40.0

Table 4: Code performance with various repair methods. On the left: The percentage of responses that are executable (ER) and the ones that are correct (P@1). On the right: The percentage of imaginary features that remain after repair.

	ER	P@1	RIF
DS-1000
Unconstrained	33.6	8.9	11.4
Constrained	36.3	6.5	8.0
Odex
Unconstrained	46.3	18.0	24.0
Constrained	45.9	18.6	23.0

Table 5: Benchmark results with and without constrained generation using our grammar.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Imaginary Features	Description
TypeError	A common example is a call to a function that does not
	match the function’s definition, such as parameters
	with incorrect datatype or name.
Generated code	result = pd.DataFrame(df, columns=df.columns, suffixes=(’_d’, ’_z’) )
AttributeError	This occurs when trying to access an attribute that does not belong
	to the class, module or submodule.

Appendix

Table 6: Examples and descriptions of type of bugs considered as imaginary features. The tokens considered as imaginary are highlighted in red.

Label	N (valid pairs)	% Agreement	Cohen’s Kappa	Interpretation
Open-close (Yes / No / NA)
Caught Static	98	94.9	0.8849	Almost Perfect
Capable Static	163	89.57	0.7181	Substantial
Caught Grammar	17	94.12	0.8496	Almost Perfect
Capable Grammar	163	92.02	0.767	Substantial
Categorical (capabilities / NA)

Appendix

Table 7: Cohen’s kappa analysis of inter-annotator agreement between 25% of annotated labels.

Static Analysers

Can do the same as in the grammar ( Syntax + Library )

Can keep data and control flow ( Flow )

cont = CountVectorizer()

is of type CountVectorizer, and should have N attributes.

Should keep track of ( Lambda )

Can know global and local scopes ( Scope )

Appendix

Table 8: Examples and descriptions of annotation categories for the capability of tools to detect errors and their source.

Figure 4: Distribution of time taken on each step to build the grammar.

Figure 5: Comparison of time taken on bug detection analysis tools.

Figure 6: Distribution of the number of rules in the grammar for each benchmark.

Figure 7: Distribution on resampling in each benchmark.

Figure 8: Comparison of time taken between unconstrained and constrained.

Figure 9: Comparison of time taken for approaches on grammar-constrained parsing.

LLM Hallucination Mitigation

Large language model hallucination mitigation develops ways to detect and reduce unsupported, fabricated, or inconsistent outputs.

See all 178 papers

Evaluating Inference-Time Defenses Against Package Hallucination in LLM-Generated Code
Aug 23, 2026Alberick Euraste Djire, Iyiola E. Olatunji, Melissa Tessa +3LLM Hallucination Mitigation LLM Code Generation
University of Luxembourg Luxembourg, Luxembourg AI4D (CITADEL) · University College London London, United Kingdom
Hallucinations of Large Language Models in Cross-Language Code Translation: Phenomena and Mitigation
Oct 20, 2026Yali Du, Yunji Zhang, Ming LiCode Translation Hallucination in Language Models
National Key Laboratory for Novel Software Technology, and the School of Artificial Intelligence, Nanjing University, China
CodeTD: Topology of Attention Detects Hallucinations in Code LLMs
Sep 7, 2026Daria Voronkova, Ilya Trofimov, Anton Dmitriev +3LLM Hallucination Detection Code Generation Evaluation
Applied AI Institute · AXXX · IMJ CNRS

Aug 23, 2026J/K move · Enter open · S save

Evaluating Inference-Time Defenses Against Package Hallucination in LLM-Generated Code

Alberick Euraste Djire, Iyiola E. Olatunji, Melissa Tessa, Earl T. Barr+2

University of Luxembourg Luxembourg, Luxembourg AI4D (CITADEL) · University College London London, United Kingdom

LLM Hallucination Mitigation LLM Code Generation Retrieval-Augmented Generation

Open paper AskPDF arXiv

LLMs are increasingly used for code generation, yet they frequently hallucinate non-existent software packages, creating exploitable entry points into the software supply chain. We make four contributions to this problem. First, we show that prior evaluation methodologies systematically inflate hallucination rates by misclassifying standard-library modules as hallucinations in some languages. For Python, the overestimation reaches 9.4 percentage points. Second, we evaluate seven inference-time defenses for mitigating package hallucinations, including five guided decoding strategies (Greedy, Contrastive, DoLa, Nudging, and Active Layer-Contrastive Decoding), an iterative self-refinement approach (Self-Refine), and a Retrieval-Augmented Generation (RAG)-based defense.. Across eight models spanning five families and four programming languages (Python, JavaScript, Ruby, Rust), RAG reduces the package hallucination rate (PHR) in 18 of 32 model--language configurations. Third, we introduce Package Utility (PU) to assess whether defenses preserve valid and task-relevant recommendations. Among strategies evaluated, Greedy decoding provides the strongest average mitigation--utility trade-off. Fourth, we stress-test all strategies under adversarial prompts seeded with fabricated package names and find that PHR surges by up to 45 percentage points relative to standard prompts, with Ruby consistently the most vulnerable language (80.9--95.2%). Under adversarial conditions, RAG and Self-Refine outperform all decoding-only strategies, indicating that robust defense requires either external grounding or iterative self-verification when prompts are actively hostile. Our results recast package hallucination as both a measurement problem and a decoding-time control problem, and they demonstrate that the choice of defense must be matched to the threat model and recommendation utility.

Abstract

LLM Hallucination Mitigation

Explore similar work

Evaluating Inference-Time Defenses Against Package Hallucination in LLM-Generated Code

Hallucinations of Large Language Models in Cross-Language Code Translation: Phenomena and Mitigation

CodeTD: Topology of Attention Detects Hallucinations in Code LLMs