cs.AIOct 16, 2026

FuzzInsight: Navigating the Efficiency-Effectiveness Trade-off in Coverage-Guided DNN Fuzzing

Authors: Vivek Vekariya, Fathiyeh Faghih, Alexander Pretschner

Organizations: Technical University of Munich, Germany · Birmingham City University, United Kingdom · Technical University of Munich & fortiss GmbH, Germany

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Collecting and labeling diverse test datasets for evaluating Deep Neural Networks (DNNs) is costly and time-consuming. This has motivated coverage-guided fuzzing techniques that generate defect-revealing corner cases from limited test data. However, their effectiveness depends heavily on configuration choices, particularly coverage criteria and seed-selection strategies. We present a configuration-oriented analysis and decision guidelines for scalable DNN fuzzing targeting two bottlenecks: inefficient seed utilization and the cost of coverage guidance. Our analysis shows that existing seed-selection strategies repeatedly prioritize a small subset of inputs, leaving many collected seeds unexplored. To address this issue, we propose Equalized Sampling, a constraint-based strategy that increases mean seed queue utilization to 77%, improving on prior approaches by 13 percentage points. We further evaluate 84 fuzzing configurations on four benchmark datasets: MNIST, SVHN, GTSRB, and CIFAR-100. Results show that the non-structural criteria LSCD, LSC, and MDSC achieve coverage outcomes comparable to structural and alternative non-structural criteria while requiring substantially less computation. These criteria also promote broad behavioral exploration of the DNN under test, reducing the need to monitor multiple coverage metrics. Among them, LSCD provides a modest but consistent runtime advantage. Finally, we show that retraining with generated corner cases improves generalization robustness by up to 77 percentage points on average without degrading performance on the original test distribution. Together, these findings provide a practical blueprint for selecting fuzzing configurations that balance efficiency, behavioral exploration, and model hardening.

Figures & tables

Figure 1. Overview of coverage-guided fuzzing for DNNs.
DatasetNumber of ImagesOutputAcc.
 TrainValTestClasses(in %)
MNIST48,00012,00010,0001098.98
SVHN62,26810,98926,0321092.83
GTSRB35,6103,87012,6314399.69
CIFAR-10040,00010,00010,00010074.00
Table 1. Overview of Datasets and DNN Performance
Figure 2. Architectural overview of the evaluated DNNs, ranging from classic convolutional stacks (LeNet-5) to spatial transformer networks and modular VGG-style networks. The final FC layer size corresponds to the number of output classes for each dataset: 10 for MNIST/SVHN, 43 for GTSRB, and 100 for CIFAR-100. Final FC layer is used to compute non-structural coverage criteria.
Figure 3. Overview of re-training pipeline using generated corner cases.
Figure 4. Seed queue utilization across datasets using various seed selection strategies.
Figure 5. Impact of the ( max_timesmax\_times ) on equalized sampling strategy.
Figure 6. Comparison of computational time required by various coverage criteria during fuzzing.
DatasetCorner CasesMean Accuracy (in %)Gain (in p.p.)
  BaselineRetrained 
MNIST163,3272.6689.44+86.78
SVHN261,5097.1268.63+61.51
GTSRB190,9841.1377.94+76.81
CIFAR-100923,2890.3985.44+85.05
Average1,539,1092.8280.36+77.53
Table 3. RQ 4 : Impact of Retraining using Corner Cases on DNN’s Robustness Note: Baseline and retrained values report mean accuracy evaluated on corner cases (in %), and gains report absolute accuracy differences in percentage points. Results are averaged across all evaluated coverage criteria because per-criterion differences are small.
Figure 7. Corner-case examples generated across the four studied benchmarks.
Figure 8. Impact of retraining using corner case data generated using CGF on DNN performance.
Figure 9. Impact of seed selection strategies on robustness and stability of retrained DNNs.
More in this topic

Automated Test Generation

Automated test generation creates inputs, scenarios, and checks to expose software and system defects with less manual effort.

See all 133 papers

Explore similar work

CardsList
Jun 3, 2026

Latent Anchor-Driven Test Generation for Deep Neural Networks

Deep Neural Networks (DNNs) are increasingly being deployed in security-critical and safety-sensitive applications, which makes rigorous testing essential to identify and mitigate model weaknesses. Existing DNN testing approaches explore either the input space or a learned latent space. While latent-space generation can better maintain plausibility than direct input-space mutation, current methods still face a trade-off among exploration controllability, failure diversity, and seed-relative semantic drift. To overcome these limitations, we propose Latte, a black-box testing framework that generates semantically proximate, diverse, and fault-revealing test cases by leveraging the latent space. Specifically, Latte encodes each input seed with a pre-trained VQ-VAE and performs a seed-centered, one-step latent mutation along directions defined by anchors sampled from alternative classes, followed by quantization and decoding back to the input space. This explores local neighborhoods around each seed within the learned latent manifold, resulting in a larger number and broader diversity of oracle-triggering prediction discrepancies under the same budget. We evaluated Latte on 5 datasets and 10 DNN models in single-model and multi-model testing scenarios. Across the evaluated datasets and models, Latte improves fault exposure and behavioral diversity under matched testing budgets. Under the single-model setting, it also maintains low seed-relative semantic drift with respect to the source seeds.

Bin Duan, Matthew B. Dwyer, Guowei Yang

School of Electrical Engineering and Computer Science, The University of Queensland, Australia · Department of Computer Science, University of Virginia, United States

DNN TestingAutomated Test Generation
AskPDF
May 4, 2026

FunFuzz: An LLM-Powered Evolutionary Fuzzing Framework

Modern fuzzers increasingly use Large Language Models (LLMs) to generate structured inputs, but LLM-driven fuzzing is sensitive to prompt initialization and sampling variance, which can reduce exploration efficiency and lead to redundant inputs. We present FunFuzz, a multi-island evolutionary fuzzing framework that runs several isolated searches in parallel and periodically migrates high-value candidates to maintain diversity. FunFuzz derives initial generation prompts from documentation and initializes islands with topic-specific instructions, then continuously adapts prompts using feedback-guided selection. During fuzzing, candidates are prioritized by incremental compiler coverage, while compiler-internal failure signals are used to identify crash-inducing inputs. We evaluate FunFuzz on compiler fuzzing, where inputs are source programs and success is measured by compiler coverage and unique compiler-internal failures. Across repeated 24-hour campaigns on GCC and Clang, FunFuzz achieves higher compiler coverage than previous LLM-driven baselines and discovers more unique failure-triggering inputs.

Mario Rodríguez Béjar, B. Romera-Paredes, Jose L. Hernández-Ramos

Universidad de Murcia Murcia, Spain · Hiverge London, England

Evolutionary OptimizationSoftware Vulnerability DetectionLLMs for CybersecurityAutomated Test Generation
AskPDF
Jul 6, 2026

AdaStop: Cost-Aware Early Stopping for DNN Test Selection

Existing methods for testing deep neural networks (DNNs) primarily prioritize test inputs likely to reveal model faults under a fixed labeling budget. In practice, choosing that budget is difficult: too little testing misses failures, while too much incurs unnecessary labeling costs. This work studies the stopping problem in DNN testing. We formulate testing as a cost--benefit decision process in which labeling an input incurs cost cc and discovering a fault yields value vv. Based on this formulation, we introduce AdaStop, a framework that estimates the marginal fault discovery rate during testing and stops labeling when the estimated rate falls below the threshold τ=c/vτ= c/v. Experiments across multiple datasets, architectures, and selection strategies show that 6565--84%84\% of faults can be discovered using only 99--31%31\% of the labeling budget.

Bonan Shen, Wei-Jung Huang, Xin Liu +2

Independent Researcher

Early StoppingDNN Testing
AskPDF