Organizations: School of Computing and Artificial Intelligence (SCAI), Nazarbayev University, Astana, Kazakhstan · University of Manchester, Manchester, United Kingdom
Open large language models (LLMs) small enough to run on a laptop are an obvious candidate for automatic bug severity prediction. We measure what they add over cheaper classifiers on one task: assigning one of four Bugzilla severity levels (Critical, Major, Minor, Trivial) to Eclipse bug reports. From a public dump of 361,006 unique Eclipse reports we draw a class-balanced sample of 20,000 rows and compare eleven configurations of nine model types: six TF-IDF classifiers, a fine-tuned CodeBERT encoder, and Llama 3 8B and Qwen 2.5 7B, each run zero-shot and after QLoRA fine-tuning. All eleven are scored on the same 2,000 held-out reports, with bootstrap intervals and McNemar tests. The zero-shot runs reach macro-F1 0.305 and 0.321, below every TF-IDF model (0.430-0.489). The fine-tuned runs reach 0.526 and 0.485, but the zero-shot and fine-tuned setups also differ in checkpoint, serving runtime and decoding, so this gap cannot be credited to fine-tuning alone. Fine-tuned Llama 3 shows no detectable difference from CodeBERT (0.518; difference +0.007, 95% CI -0.017 to +0.032), and fine-tuned Qwen 2.5 is below it on macro-F1. The models differ more by class than in aggregate: fine-tuned Llama 3 finds 36% of Critical reports, Qwen 2.5 finds 69% but only 21% of Minor ones. The dump repeats 60,000 rows, and 8% of the test reports duplicate a training report; removing them lowers TF-IDF scores by up to 0.031 and leaves the LLMs almost unchanged. We release split identifiers, per-report predictions and the analysis script at https://github.com/tbayan/bug_finetuning_llm_artifacts, together with CyberTriage, a tool that serves the fine-tuned classifier locally.
Figures & tables
Study
Data
Labels
Models
Menzies and Marcus [23]
NASA PITS
five NASA levels
Ripper rule learner
Lamkanfi et al. [20]
three open-source projects
severe vs. non-severe
Naive Bayes
Lamkanfi et al. [21]
Eclipse, GNOME
severe vs. non-severe
NB, multinomial NB, k -NN, SVM
Tian et al. [29]
three open-source projects
blocker to trivial
BM25 nearest neighbour
Dao and Yang [10]
Eclipse, Mozilla
—
CNN with multi-aspect features
Silva de Oliveira [27]
Eclipse, Mozilla
severe vs. non-severe
Llama 2 (prompted, fine-tuned)
Table 1: Selected severity-prediction studies from Section 2 . Label schemes and data differ, so reported scores cannot be compared across rows. A dash marks information we could not confirm from the published abstract or record.
Label in the dump
Rows
Unique reports
Mapped class
Pool rows
Sampled (train / test)
blocker
6,972
6,223
Critical
22,127
4,000 / 1,000
critical
15,255
12,845
major
40,743
35,367
Major
40,530
4,000 / 1,000
minor
15,539
13,438
Minor
15,461
4,000 / 1,000
trivial
5,411
4,815
Trivial
5,370
4,000 / 1,000
normal
282,829
242,819
excluded
—
—
Table 2: Labels in the Hugging Face dump and how the experimental sample was drawn. “Rows” counts every row in the file; “Unique reports” drops the 60,000 repeated rows. The pool is what the sampler drew from; it still contains repeated rows.
Figure 1: Study design. Grey boxes are data, with the counts found in the audit; coloured boxes are the eleven model configurations, in the family colours of Figure 3 . The TF-IDF classifiers were fit on 12,800 of the 16,000 training reports. The shared subset is the 2,000-report sample on which the LLMs were run; every other model was re-scored on it from its stored files.
Figure 2: Architecture of the two trained neural classifiers. (a) CodeBERT, a 12-layer RoBERTa encoder fine-tuned end-to-end with a classification head. (b) The decoder-only LLMs, whose pre-trained weights stay frozen while LoRA adapters are trained on all seven linear projections of every block; the label is read from the generated text. During training the block projections are stored in NF4 with double quantisation, while the embeddings, normalisation layers and LM head stay in 16-bit. (c) One adapted projection (Eq. 1 ). Dimensions are taken from the CodeBERT configuration file and the headers of the deployed GGUF files, and trainable-parameter counts from the training logs.
CodeBERT
Llama 3 8B (QLoRA)
Qwen 2.5 7B (QLoRA)
Initial checkpoint
microsoft/codebert-base
unsloth/llama-3-8b-bnb-4bit
unsloth/qwen2.5-7b-bnb-4bit
Trainable / total parameters
124,648,708 / 124,648,708
83,886,080 / 8,114,147,328
80,740,352 / 7,696,356,864
Adaptation
full fine-tuning, classification head
LoRA r=32 , α=32 , dropout 0.05, on q, k, v, o, gate, up, down
Precision in training
fp32 weights, fp16 mixed precision
block projections frozen in NF4 with double quantisation (bitsandbytes); embeddings, norms and LM head in bf16; bf16 compute
Objective
cross-entropy on 4 labels
next-token cross-entropy on the whole sequence (prompt not masked)
Optimiser, learning rate
AdamW, 2×10−5 , linear decay, no warm-up
paged_adamw_8bit, 1×10−4 , cosine, 10% warm-up
Table 3: Training configuration of the learned models, as recorded in the training scripts and logs. LLM losses are averaged over each epoch and cover all tokens of each record; CodeBERT’s log keeps no per-epoch average. The NF4 settings come from the quantisation configuration of the base checkpoints.
Family
Model
Acc.
Macro-P
Macro-R
Macro-F1 [95% CI]
F1, dedup.
TF-IDF
Naive Bayes
0.435
0.456
0.437
0.430 [0.409, 0.452]
0.425
Logistic regression
0.491
0.488
0.490
0.489 [0.467, 0.510]
0.474
Linear SVM
0.490
0.484
0.489
0.484 [0.462, 0.505]
0.471
Random forest
0.491
0.486
0.491
0.483 [0.461, 0.504]
0.455
XGBoost
0.479
0.475
0.479
0.476 [0.454, 0.498]
0.445
Soft-voting ensemble
0.486
0.482
0.486
0.479 [0.457, 0.501]
0.458
Table 4: Results on the 2,000 shared test reports (495 Critical, 505 Major, 491 Minor, 509 Trivial). Acc. is accuracy; brackets give 95% bootstrap intervals. The last column (dedup.) repeats macro-F1 after removing the 159 shared reports whose text also appears in the training set ( n=1,841 ). Best macro-F1 in bold.
Figure 3: Macro-F1 of all eleven configurations on the shared 2,000 reports, with 95% bootstrap intervals.
Comparison (A vs. B)
Δ macro-F1 [95% CI]
pHolm , bootstrap
b / c
pHolm , McNemar
Llama 3 FT vs. CodeBERT
+0.007 [ −0.017 , +0.032 ]
1.000
336 / 318
1.000
Qwen 2.5 FT vs. CodeBERT
−0.033 [ −0.058 , −0.008 ]
0.031
333 / 332
1.000
Llama 3 FT vs. Qwen 2.5 FT
+0.041 [ +0.015 , +0.066 ]
0.010
380 / 363
1.000
Llama 3 FT vs. Llama 3 ZS
+0.221 [ +0.195 , +0.247 ]
≤ 0.001
581 / 262
< 0.001
Qwen 2.5 FT vs. Qwen 2.5 ZS
+0.164 [ +0.133 , +0.194 ]
≤ 0.001
748 / 459
< 0.001
CodeBERT vs. TF-IDF ensemble
+0.039 [ +0.014 , +0.065 ]
0.010
373 / 309
0.079
Table 5: Selected pairwise comparisons on the shared 2,000 reports. FT is fine-tuned, ZS zero-shot, and the TF-IDF ensemble is the soft-voting ensemble. Δ is A minus B. b and c count reports that only A or only B classifies correctly. Bootstrap p -values cannot fall below 1/10,000 , hence the floor after Holm correction.
Critical
Major
Minor
Trivial
Critical recall
Model
P
R
P
R
P
R
P
R
95% CI
Soft-voting ensemble
0.518
0.513
0.447
0.311
0.408
0.458
0.553
0.660
[0.471, 0.557]
CodeBERT
0.557
0.471
0.423
0.440
0.454
0.511
0.655
0.648
[0.427, 0.515]
Llama 3 8B (zero-shot)
0.741
0.162
0.388
0.450
0.319
0.829
0.677
0.041
[0.129, 0.195]
Qwen 2.5 7B (zero-shot)
0.700
0.127
0.396
0.366
0.322
0.886
0.722
0.128
[0.099, 0.157]
Llama 3 8B (fine-tuned)
0.632
0.358
0.412
0.707
0.517
0.493
0.720
0.546
[0.316, 0.400]
Table 6: Per-class precision (P) and recall (R) on the shared 2,000 reports, with a 95% bootstrap interval for Critical recall.
Figure 4: Confusion matrices on the shared 2,000 reports for six configurations. Cells show counts; shading shows the share of each true class (row). Rows are true labels, columns predicted labels.
Model
Reports
Total (s)
Per report
Timed loop covers
TF-IDF classifiers (six)
4,000
0.39–1.09
0.10–0.27 ms
load model, vectorise, predict
CodeBERT
4,000
196.3
49 ms
per-report tokenise and forward pass
Llama 3 8B (zero-shot)
2,000
2,742
1.37 s
HTTP call to local Ollama, 150-token cap, 50 ms pause
Qwen 2.5 7B (zero-shot)
2,000
3,076
1.54 s
HTTP call to local Ollama, 150-token cap, 50 ms pause
Llama 3 8B (fine-tuned)
2,000
2,619
1.31 s
llama-cpp-python call, 32-token cap
Qwen 2.5 7B (fine-tuned)
2,000
2,693
1.35 s
llama-cpp-python call, 32-token cap
Table 7: Inference wall-clock time on the Apple M1 Pro, from the most recent logged evaluation runs. The LLM loops also include prompt building and parsing; in the CodeBERT and LLM loops the first call also loads the model.
Figure 5: Two-stage analysis in CyberTriage. Any severity the Stage 2 model writes is discarded by the back end; the stored label is always Stage 1’s.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Model
Accuracy
Macro-P
Macro-R
Macro-F1
Logged F1
Logged time (s)
Naive Bayes
0.450
0.472
0.450
0.444
0.4439
0.52
Logistic regression
0.493
0.489
0.493
0.491
0.4908
0.39
Linear SVM
0.495
0.488
0.495
0.489
0.4887
0.60
Random forest
0.501
0.495
0.501
0.494
0.4939
1.09
XGBoost
0.495
0.490
0.495
0.492
0.4917
0.58
Soft-voting ensemble
0.498
0.492
0.498
0.490
0.4904
1.02
Appendix
Table 8: TF-IDF classifiers and CodeBERT on the full 4,000-report test split.
The rapid adoption of LLM-powered code generation has dramatically accelerated software development, yet effective verification methods remain severely underdeveloped. Existing bug localization techniques are either prohibitively expensive, requiring minutes of agentic reasoning and thousands of generated tokens per file, and/or operate at coarse function-level granularity unsuitable for precise debugging. While works that focus on line-level granularity and are more light-weight are often limited in their performance or context size. We introduce a novel line-level bug localization approach that addresses these limitations through three key contributions: (1) a token alignment algorithm that overcomes fundamental tokenization challenges in previous work, (2) a lightweight multi-task LLM for bug localization (MLC) enabling efficient line-level bug classification, and (3) an optimized training recipe for multi-line prediction. Our method achieves state-of-the-art performance among similar setups on line-level bug localization with full-file context. At the same time we reach comparable performance to agentic approaches on Defects4J and PypiBugs benchmarks while reducing inference latency by orders of magnitudes, requiring only a single generated token per file. We further demonstrate strong generalization by introducing and evaluating on a small out-of-domain evaluation datasets in Python. We will open source our code, models, and datasets upon acceptance.
Nikolai Rozanov
Department of Computing, Imperial College London, London, UK
Evaluating the ability of large language models (LLMs) to discover software bugs is increasingly important. Existing benchmarks typically evaluate this capability by asking the model to generate a proof-of-concept input that triggers a predefined target vulnerability. However, this setup may overlook valid crashes discovered by the model when they do not match the predefined target. As a result, the evaluation may not reflect the model's real capability. We present FuzzingBrain-Bench, a benchmark for assessing AI models' ability to discover bugs in open-source software. Models are given an open-source project and a sanitizer-instrumented harness in a self-contained Docker image. Their goal is to generate inputs that trigger as many distinct crashes as possible through the harness. A model's performance on each challenge is scored based on the number of distinct crash signatures it produces, capped at a predefined maximum and weighted by a difficulty coefficient. FuzzingBrain-Bench V1 consists of 77 challenges drawn from 43 open-source projects, with 36 C, 32 C++, and 9 Java/JVM challenges. We evaluate Claude Haiku 4.5, Claude Sonnet 4.6, and Claude Opus 4.8 on the full benchmark. Claude Opus 4.8 performs best, triggering crashes in 60 of 77 challenges and achieving a score of 196 out of 579. None of the three models triggers a crash in 13 challenges. The FuzzingBrain-Bench corpus and harnesses are publicly available at https://github.com/fuzzingbrain/FuzzingBrain-Bench.
Large language models often generate code with bugs. Existing methods rely on feedback signals such as test failures and self-critiques to iteratively refine the generated code. Such signals are either too coarse-grained or too high-level, which is not sufficient to inform the model where to fix the bug. In this work, we present Flare, an iterative framework with a lightweight diagnostic model that predicts line-level suspiciousness signals for bug localization and code refinement. Given the inherent uncertainty of diagnostic predictions, Flare searches over the top-k suspicious regions and selects the best candidate according to execution outcomes. Experiments on LiveCodeBench and BigCodeBench with five base LLMs show that, even without candidate search (k=1), Flare outperforms the strongest baseline with an absolute improvement from 1.72% to 7.42%. Furthermore, searching over 10 candidates yields an average improvement of 8.50% compared with no candidate search. When evaluated in isolation, our lightweight diagnostic model achieves the best performance compared with recent fault localization methods, demonstrating that it can provide reliable fine-grained guidance for code refinement.