LLMs are increasingly used for code generation, yet they frequently hallucinate non-existent software packages, creating exploitable entry points into the software supply chain. We make four contributions to this problem. First, we show that prior evaluation methodologies systematically inflate hallucination rates by misclassifying standard-library modules as hallucinations in some languages. For Python, the overestimation reaches 9.4 percentage points. Second, we evaluate seven inference-time defenses for mitigating package hallucinations, including five guided decoding strategies (Greedy, Contrastive, DoLa, Nudging, and Active Layer-Contrastive Decoding), an iterative self-refinement approach (Self-Refine), and a Retrieval-Augmented Generation (RAG)-based defense.. Across eight models spanning five families and four programming languages (Python, JavaScript, Ruby, Rust), RAG reduces the package hallucination rate (PHR) in 18 of 32 model--language configurations. Third, we introduce Package Utility (PU) to assess whether defenses preserve valid and task-relevant recommendations. Among strategies evaluated, Greedy decoding provides the strongest average mitigation--utility trade-off. Fourth, we stress-test all strategies under adversarial prompts seeded with fabricated package names and find that PHR surges by up to 45 percentage points relative to standard prompts, with Ruby consistently the most vulnerable language (80.9--95.2%). Under adversarial conditions, RAG and Self-Refine outperform all decoding-only strategies, indicating that robust defense requires either external grounding or iterative self-verification when prompts are actively hostile. Our results recast package hallucination as both a measurement problem and a decoding-time control problem, and they demonstrate that the choice of defense must be matched to the threat model and recommendation utility.
Figures & tables
Strategy
Hyperparameters
Baseline (Vanilla)
T=1.0 , top- k=50 , top- p=1.0
Greedy
T=0
Contrastive Decoding (CD)
α=0.5 ; amateur/expert layer
pairing (model-specific)
DoLa
mature_layer = final layer of
target model;
Table 1. Hyperparameter configuration per strategy.
Lang.
Strategy
Ngen
PHR w/ stdlib (%)
PHR w/o stdlib (%)
Δ (pp)
Python
Vanilla
79,965
24.7 ±0.67
32.3 ±0.53
+7.6
Greedy
47,796
19.3 ±0.00
27.4 ±0.00
+8.1
S-Ref
39,465
19.0 ±1.14
28.4 ±1.26
+9.4
CD
24,489
26.7 ±0.55
32.6 ±0.50
+5.9
DoLa
51,540
32.3 ±0.60
38.9 ±0.58
+6.6
Nudge
17,628
21.3 ±1.40
30.4 ±0.91
+9.1
Table 2. Micro PHR with and without standard library (stdlib) correction, averaged over all models. Δ>0 means overestimation without the correction.
Model
Lang.
Ngen
Nhall
micro PHR (%)
ngen
nhall
macro PHR (%)
deepseek-1.3b
Python
3,024
1,077
35.6 ±1.16
2.02
0.72
20.7 ±1.64
JavaScript
1,950
585
30.0 ±5.49
1.30
0.39
13.9 ±1.77
Ruby
1,470
600
40.8 ±3.35
0.98
0.40
17.6 ±1.20
Rust
2,658
1,305
49.1 ±0.55
1.78
0.87
35.4 ±0.23
Overall
9,102
3,567
39.2 ±0.55
4.55
1.78
21.9 ±0.23
deepseek-6.7b
Python
5,013
630
12.6 ±1.77
3.34
0.42
10.6 ±0.77
Table 3. Vanilla decoding PHR per model and language. nˉ = mean per prompt. Overall is column aggregates of all languages and for micro and macro PHR, it is the mean. Higher PHR per model is in bold .
Model
Lang.
Vanilla
Greedy
Self-Refine
CD
DoLa
Nudge
RAG
ALCD
deepseek-1.3b
Python
35.6
44.3
36.2
44.1
87.3
42.7
18.3
66.3
JavaScript
30.0
38.8
20.0
38.5
83.5
42.8
41.5
70.0
Ruby
40.8
52.9
50.9
56.8
91.2
69.4
22.3
73.7
Rust
49.1
70.7
40.4
58.3
88.9
81.3
24.6
77.4
deepseek-6.7b
Python
12.6
12.6
26.4
46.6
79.2
33.1
9.4
41.4
JavaScript
10.2
10.2
15.6
40.6
88.8
32.0
36.0
46.6
Table 4. Micro PHR (%) per model, language, and strategy on the package recommendation task. Bold = lowest (best) per row.
Figure 1. Valid and hallucinated package references extracted from full code generation outputs, aggregated by language (left) and by model (right) across all seven strategies. Blue segments denote registry-validated packages; red segments denote hallucinated references. Bar plots showing the distribution of valid and hallucinated package references for code generation outputs, separated by language and model. Blue bars represent valid packages, red bars represent hallucinated packages.
Figure 2. Structural quality of generated code per model, strategy, and language. Left: syntax error rate (%); green cells denote error-free output, warmer colors indicate increasing parse failure rates. Right: mean Semgrep-detected code smell count per sample.
Figure 3. Package Utility ( PU ) across five reference-set definitions for all seven strategies and eight models on the package recommendation task. Values are multiplied by 100 for readability. Rows: strategies; columns: models.
Model
Lang.
Vanilla
Greedy
Self-Refine
DoLa
RAG
Nudge
CD
ALCD
deepseek-1.3b
JavaScript
84.4 ( 320/379 )
87.4 ( 340/389 )
75.2 ( 194/258 )
84.9 ( 253/298 )
71.8 ( 492/685 )
76.9 ( 230/299 )
87.7 ( 193/220 )
76.1 ( 242/318 )
Python
80.3 ( 392/488 )
90.4 ( 339/375 )
81.2 ( 234/288 )
68.5 ( 204/298 )
48.6 ( 470/968 )
80.0 ( 419/524 )
93.9 ( 845/900 )
73.6 ( 245/333 )
Ruby
92.5 ( 368/398 )
93.2 ( 449/482 )
91.8 ( 180/196 )
93.1 ( 244/262 )
87.2 ( 429/492 )
91.2 ( 466/511 )
96.8 ( 987/1,020 )
87.3 ( 207/237 )
Rust
72.4 ( 422/583 )
38.6 ( 248/643 )
91.2 ( 312/342 )
94.1 ( 636/676 )
79.3 ( 466/588 )
26.6 ( 238/895 )
91.9 ( 570/620 )
86.0 ( 301/350 )
deepseek-6.7b
JavaScript
73.1 ( 425/581 )
81.3 ( 231/284 )
86.0 ( 505/587 )
91.6 ( 164/179 )
61.8 ( 777/1,257 )
96.7 ( 205/212 )
76.4 ( 292/382 )
86.1 ( 167/194 )
Python
73.2 ( 423/578 )
83.6 ( 316/378 )
59.5 ( 601/1,010 )
81.0 ( 111/137 )
41.0 ( 1,156/2,819 )
88.1 ( 244/277 )
60.1 ( 196/326 )
75.0 ( 165/220 )
Table 5. Micro PHR (%) and raw counts ( Nhall / Ngen ) in parentheses per model, language, and strategy under adversarial prompts. Lowest PHR per row in bold .
Spracklen et al. (USENIX Security '25) showed that code-generating large language models hallucinate package names that do not exist on PyPI or npm at rates ranging from 5.2% on commercial models to 21.7% on open-source models, creating an attack surface for slopsquatting -- the registration of malicious packages under hallucinated names. We replicate their methodology on five frontier code-capable LLMs released between October 2025 and March 2026: Claude Sonnet 4.6, Claude Haiku 4.5, GPT-5.4-mini, Gemini 2.5 Pro, and DeepSeek V3.2. Across 199,845 paired Python and JavaScript prompts validated against PyPI and npm master lists, we measure overall hallucination rates between 4.62% (Claude Haiku 4.5) and 6.10% (GPT-5.4-mini) -- an order-of-magnitude compression of the inter-model spread observed by Spracklen, but not a retirement of the threat. Beyond replication, we identify a set of 127 package names (109 on PyPI, 18 on npm) that all five evaluated models invent identically; following coordinated disclosure with PyPI Security and Socket.dev, 53 of these (41 on PyPI, 12 on npm) remain registrable by an attacker after each registry's existing defenses, constituting a model-agnostic supply-chain attack surface that no single-model study can reveal. We further document a Python-over-JavaScript hallucination asymmetry that inverts Spracklen's 2024 finding, identify a Haiku-below-Sonnet inversion within the Anthropic family, and observe a Jaccard-similarity peak between DeepSeek V3.2 and GPT-5.4-mini (J = 0.343) suggestive of shared training-data origins.
Hallucinations remain an unsolved problem for LLMs, and package hallucinations are a particularly dangerous instance of this phenomenon. Package hallucinations occur during code generation when a model fabricates non-existent software packages, recommending imports and installation commands for fictional libraries. This creates a critical supply-chain vulnerability; an attacker can proactively register such packages on public registries with malicious payloads that are subsequently installed and executed by developers or autonomous agents. These hallucinations enable a class of package confusion attack known as slopsquatting. To address this issue, we present Adaptive Unlearning (AU), a post-deployment framework that surgically suppresses package hallucinations while preserving general model utility. AU introduces a hybrid token-level objective that simultaneously reinforces valid outputs and suppresses hallucinated ones. Combined with an adaptive discovery loop that continuously surfaces new hallucination-inducing contexts without human supervision, AU enables generalization to unseen prompts and hallucinations. We demonstrate that AU reduces package hallucination rates by 88%, while maintaining performance on standard coding benchmarks. Our analysis shows that distributional changes are concentrated on package-related generations, leaving general coding behavior largely unaffected and confirming that AU's effect is isolated to the targeted distribution. AU relies entirely on model-generated data and requires no human annotation, representing a post-deployment hallucination mitigation framework.
Joseph Spracklen, Pedram Aghazadeh, Farinaz Koushanfar +1
University of Texas San Antonio San Antonio, TX, USA · University of California San Diego La Jolla, CA, USA
LLM-powered coding agents increasingly participate in software development workflows by generating code, selecting dependencies, and producing package installation commands. This creates a new software supply chain risk: when an agent hallucinates a non-existent package, an attacker may register the hallucinated name and later compromise users who install it. Existing package hallucination attacks and defenses primarily focus on naturally occurring hallucinations, targeted dependency steering, or post-hoc package validation. In this paper, we introduce Neutral Prompting Attack (NPA), a highly stealthy attack paradigm in which semantically benign instructions, such as encouraging imagination and exhaustiveness, increase package hallucination propensity without containing explicit malicious intent. Unlike targeted dependency steering, NPA does not specify an attacker-chosen package. Instead, it shifts the model's dependency generation behavior toward more speculative package names. We evaluate NPA across multiple coding-oriented LLMs and package hallucination benchmarks. Our results show that NPA increases both Hallucination ASR and Pip Install ASR, changes the distribution of hallucinated package names, and evades existing static-analysis, LLM-based, and agent-based Skill defenses. These findings reveal that harmless-looking prompts can covertly manipulate hallucination behavior and create downstream software supply chain risks.
Chia-Yi Hsu, Chia-Mu Yu, Chun-Ying Huang +1
Department of Computer Science National Yang Ming Chiao Tung University · Department of Electronics and Electrical Engineering National Yang Ming Chiao Tung University · School of Computing Institute of Science Tokyo