TensorMutation: Learning Weight-Space Mutations of Deep Learning Models on Neural Graphs

Mutants support testing and debugging in two roles: (i) as test goals and (ii) as substitutes for real faults. Hard-to-kill mutants provide better guidance for test improvement, while realism is essential when mutants are used to simulate real bugs. Building on these roles, selective mutation for deep learning (DL) aims to reduce the cost of mutant generation and execution by choosing operator configurations that yield resistant and realistic mutants. However, the DL literature lacks a unified measure that captures both aspects. This study presents a probabilistic framework to quantify mutant quality along two complementary axes: resistance and realism. Resistance adapts the classical notion of hard-to-kill mutants to the DL setting using statistical killing probabilities, while realism is measured via the generalized Jaccard similarity between mutant and real-fault detectability patterns. The framework enables ranking and filtering of low-quality mutation-operator configurations without assuming a specific use case. We empirically evaluate the approach on four datasets of real DL faults. Three datasets (CleanML, DeepFD, and DeepLocalize) are used to estimate and select high-quality operator configurations, and the held-out defect4ML dataset is used for validation. Results show that quality-driven selection reduces the number of generated mutants by up to 55.6% while preserving typical levels of resistance and realism under baseline-aligned selection thresholds. These findings confirm that dual-objective selection can lower cost without compromising the usefulness of mutants for either role.

Data flow underlies deep learning (DL) framework execution, including automatic differentiation and operator-fusion optimization. Bugs involving data flow can cause incorrect results or unexpected behavior. Existing testing methods represent models primarily by operator topology, omitting runtime tensor values and properties and operator input requirements. They therefore cannot distinguish data flows in models with similar topologies, even when these differences affect framework behavior. We introduce the Tensor Flow Graph (TFG) to capture runtime tensor states and operator input requirements. Its nodes represent operators and their input requirements, while its edges record tensor values and properties. Building on the TFG, we propose FlowMuT, a mutation-based testing method grounded in data flow. Mutations are represented as constrained subgraph replacements, with mutation operators derived from prior work and an analysis of mutable data flow objects. In each round, FlowMuT builds a TFG for the seed model, pairs mutation operators with TFG sites, and filters illegal candidates using data flow constraints. It uses LinUCB to select a candidate based on coverage and impact, executes the mutant, updates selection feedback, and uses each accepted mutant as the next seed. We evaluate FlowMuT on PyTorch and MindSpore using representative models from diverse application scenarios. FlowMuT submitted 55 bug reports for cases missed by all baseline methods; developers confirmed 41 reports and fixed the bugs described in 7 reports. FlowMuT achieves 43.4% higher data flow diversity than the strongest baseline and a legal mutant rate of 96.7%, compared with 86.4% without constraints. These results show that testing from a data flow perspective is practical and effective.

Deep learning systems often fail due to subtle implementation faults that alter training behavior. Recent work has studied how to detect and diagnose such failures from changes observed across training epochs. However, the software engineering community still lacks a public dataset of per-epoch training runs with documented fault history, feature extraction details, and clear reuse support for fault detection and diagnosis tasks. We present Deep4ge, a controlled benchmark of 14,227 training runs generated from 59 adapted TensorFlow/Keras deep neural network (DNN) programs collected from Stack Overflow. We generated faulty variants using 27 source-code transformations that introduce known faults across seven categories. The dataset contains 9,845 faulty runs and 4,382 correct baseline runs. For each run, we record 4 evaluation metrics and 26 features that measure training behavior at every epoch. These features capture weights, gradients, activations, accuracy and loss trends, learning rate, and hardware use. Deep4ge supports binary fault detection, multi-class fault diagnosis, and early fault prediction from partial training runs. We release the dataset and fault-injection framework at https://doi.org/10.5281/zenodo.20337241.