Data flow underlies deep learning (DL) framework execution, including automatic differentiation and operator-fusion optimization. Bugs involving data flow can cause incorrect results or unexpected behavior. Existing testing methods represent models primarily by operator topology, omitting runtime tensor values and properties and operator input requirements. They therefore cannot distinguish data flows in models with similar topologies, even when these differences affect framework behavior. We introduce the Tensor Flow Graph (TFG) to capture runtime tensor states and operator input requirements. Its nodes represent operators and their input requirements, while its edges record tensor values and properties. Building on the TFG, we propose FlowMuT, a mutation-based testing method grounded in data flow. Mutations are represented as constrained subgraph replacements, with mutation operators derived from prior work and an analysis of mutable data flow objects. In each round, FlowMuT builds a TFG for the seed model, pairs mutation operators with TFG sites, and filters illegal candidates using data flow constraints. It uses LinUCB to select a candidate based on coverage and impact, executes the mutant, updates selection feedback, and uses each accepted mutant as the next seed. We evaluate FlowMuT on PyTorch and MindSpore using representative models from diverse application scenarios. FlowMuT submitted 55 bug reports for cases missed by all baseline methods; developers confirmed 41 reports and fixed the bugs described in 7 reports. FlowMuT achieves 43.4% higher data flow diversity than the strongest baseline and a legal mutant rate of 96.7%, compared with 86.4% without constraints. These results show that testing from a data flow perspective is practical and effective.
Figures & tables
Figure 1. Three data flow challenges illustrated by scaled dot-product attention (SDPA). The top panel shows a scaled dot-product attention computation. Panel (a) compares a correct execution with separate contiguous copies of K and V against a buggy execution with expanded, aliased storage under the same topology. Panel (b) shows that slice-and-expand head sharing preserves values for equal K and V heads but changes unequal heads, making that candidate illegal. Panel (c) shows two legal mutations of equal heads, cloning and sharing, whose behavioral novelty is unknown.
Figure 2. The FlowMuT workflow. Four stages are arranged from left to right and continue along a lower feedback loop. A seed model is profiled into a TFG, a candidate is selected and applied to obtain a mutant, and the execution outcome guides the next iteration.
Figure 3. Construction of a TFG from a PyTorch model, combining operator topology G , runtime tensor values and properties F , and operator input requirements R . Three panels show the PyTorch source model, its computation graph, and the resulting TFG. The TFG preserves operator connections, annotates tensor edges with runtime values and properties, and attaches input requirements to consumer operators.
Figure 4. Construction of the FlowMuT mutation operator pool from CG-derived and TFG-derived operators. The two sources produce 120 unique CG-derived operators and 48 TFG-derived operators, of which 21 are shared. Their union is implemented and validated into a pool of 147 operators.
Mutation category
CG-derived construction
TFG-derived construction
Final pool
Object
Primitive
Extracted
Removed
Retained
Derived
Overlap
TFG-only
Total
Tensor
Update
19
14
5
8
5
3
8
Tensor
Insertion
1
0
1
4
1
3
4
Tensor subtotal
20
14
6
12
6
6
12
Operator
Update
46
9
37
6
5
1
38
Operator
Insertion
6
3
3
3
2
1
4
Table 1. Construction of the 147-operator pool by mutable TFG object, mutation primitive, and source.
Scene Task
Model
Input
Parameters
Image Classification
ConvNeXt V2-Tiny
224 × 224
28,635,496
Swin V2-Tiny
256 × 256
28,347,154
EfficientNetV2-S
384 × 384
21,458,488
Object Detection
YOLOv10-S
640 × 640
8,128,272
RT-DETRv2-S
640 × 640
20,226,500
Semantic Segmentation
SegFormer-B1
512 × 512
13,715,798
Table 2. Seed models by application task, input size, and parameter count.
Developer status
Symptom category
Status
PyTorch
MindSpore
Category
PyTorch
MindSpore
Reported
14 (100.0%)
41 (100.0%)
Crash
0 (0.0%)
31 (75.6%)
Confirmed
8 (57.1%)
33 (80.5%)
Numerical Error
12 (85.7%)
6 (14.6%)
Fixed
7 (50.0%)
0 (0.0%)
State Error
2 (14.3%)
4 (9.8%)
Table 3. Cumulative developer status and symptom categories of 55 new bug reports.
Figure 5. (a) Bug overlap from discovery campaigns; (b) C++ line overlap from one run per method. Two four-set overlap diagrams compare LEMON, DevMuT, FlowMuT, and COMET. Panel (a) shows detected-bug overlap, including 55 cases found only by FlowMuT and submitted as bug reports. Panel (b) shows C++ source-line coverage from one run per method: 32.2 percent for LEMON, 43.9 percent for DevMuT, 44.4 percent for FlowMuT, and 35.8 percent for COMET.
Test Case Diversity at Round 250
Framework Coverage (%)
Execution Time (s/test case)
Method
LIC ( ×10−2 )
LPC ( ×10−2 )
LSC ( ×10−2 )
DFSD ( ×10−5 )
Python
API
C++
Generation
Detection
LEMON
2.89
1.37
0.44
28.08
27.8
28.0
34.8
0.85
3.19
COMET
3.37
3.09
0.65
32.11
28.0
28.1
35.0
7.60
3.76
DevMuT
3.17
1.36
0.58
30.80
30.8
32.0
41.5
1.20
3.98
FlowMuT
5.95
4.48
1.04
46.06
37.8
42.2
48.1
6.04
4.58
Table 4. Test case diversity, framework coverage, and execution cost of FlowMuT and the three baseline methods at mutation round 250.
Figure 6. Test case diversity over 250 mutation rounds. Columns (a)–(d) show LIC, LPC, LSC, and DFSD; the rows compare methods (top) and operator pools (bottom). Lines are means with 95.0% intervals. Eight line charts form a two-by-four grid. The columns show LIC, LPC, LSC, and DFSD. The top row compares LEMON, COMET, DevMuT, and FlowMuT, while the bottom row compares the full, CG-derived, and TFG-derived operator pools.
Configuration
Ops
Test Case Diversity
Framework Coverage (%)
Configuration
LIC ( ×10−2 )
LPC ( ×10−2 )
LSC ( ×10−2 )
DFSD ( ×10−5 )
Python
API
C++
Operator-source ablation
Full operator pool
147
5.95
4.48
1.04
46.06
37.8
42.2
48.1
CG-derived pool
120
3.17
3.42
0.71
30.51
29.9
32.4
37.7
TFG-derived pool
48
2.04
0.68
0.18
20.01
20.1
20.1
21.8
TFG object ablation (within the TFG-derived pool)
Table 5. Test case diversity and framework coverage results for operator-pool and TFG ablations at mutation round 250.
Mutation legality
Filter outcomes
Filter selectivity (%)
Configuration
Ret. (%)
LMR (%)
TN
FN
FP
TP
Prec.
Rec.
FRR
Leak.
Full constraints
88.0
96.7
851
29
13
107
89.2
78.7
1.5
21.3
w/o input
92.7
92.3
856
71
8
65
89.0
47.8
0.9
52.2
w/o internal
90.7
94.2
854
53
10
83
89.2
61.0
1.2
39.0
w/o output
91.6
93.3
855
61
9
75
89.3
55.1
1.0
44.9
w/o all
100.0
86.4
864
136
0
0
N/A
0.0
0.0
100.0
Table 6. Mutation legality and filter selectivity under constraint ablations. TN/FN count legal/illegal retained candidates; FP/TP count legal/illegal rejected candidates.
Test Case Diversity
Framework Coverage (%)
Distinct signatures
Configuration
LIC ( ×10−2 )
LPC ( ×10−2 )
LSC ( ×10−2 )
DFSD ( ×10−5 )
C++
API
Selection-policy comparison
One-shot
3.68
1.32
0.41
43.63
35.5
27.3
460.9
Random search
4.56
1.50
0.62
45.55
33.3
22.7
481.2
Fixed priority
1.99
0.56
0.17
10.97
28.4
17.2
115.9
UCB1
4.38
1.44
0.52
41.65
32.2
20.3
440.0
Table 7. Test case diversity and framework coverage under selection-policy and LinUCB context-feature ablations at mutation round 250.
Modern deep learning compilers compile models lazily and cache artifacts across invocations. Retained compiler state can make a later invocation incorrect even when it is correct in isolation. An analysis of 26 publicly reported issues in PyTorch, JAX, and TensorFlow shows that detecting such bugs requires sequences that first establish compiler state and then expose a failure. We present Rondo, a stateful testing method that represents each test case as a model paired with an invocation sequence. Rondo mutates invocation sequences and uses a Markov chain Monte Carlo sampler guided by compilation behavior to favor rarely observed behaviors. Its three-way differential oracle compares execution without compilation, compiled execution with compiler state reset, and compiled execution with state retained. It confirms stateful bugs by replaying the same invocation with state retained and reset while preserving inputs and program state, then shortens sequences while preserving the original bug. We evaluate Rondo on four deep learning frameworks using 24 common model architectures. Rondo detects 117 bugs, including 35 stateful bugs. We submit 39 bug reports, including 18 for stateful bugs; developers confirm 35 reports, including 16 for stateful bugs, and fix 18 bugs, including 11 stateful bugs. Rondo achieves the highest behavior coverage and compiler Python source-line coverage among the compared methods, with 99.26% alarm precision. It shortens 44 of 117 invocation sequences, reducing the lengths of these 44 sequences by 83.34% on average.
Jiatao Ge, Ruixiang Qian, Yibo Hu +5
State Key Laboratory for Novel Software Technology, Nanjing University, China · UNIST, Republic of Korea
Tensor programs, as used in deep learning models, are a prime target for optimization, as small performance improvements can have a large impact across training or inference workloads. However, such optimizations are complicated and can produce subtle bugs. Traditionally, correctness is assumed when differential testing against a reference on random inputs fails to reveal bugs. However, the inputs to these programs are massive tensors, and finding bugs can require generating extremely low likelihood inputs with precise relationships among their values. We propose a novel way to find bugs more consistently by flipping the quantifiers. Rather than generating a single input and checking all output tensor locations for equivalence, what if you could check a single output tensor location's equivalence for all inputs? We implement this idea in a system, \dirigo, by using a novel symbolic execution strategy. We demonstrate that \dirigo can find bugs effectively in a public dataset of 6,988 AI-written CUDA kernels that are all marked correct by differential testing. Of these, \dirigo finds 600 kernels that are actually buggy, and finds 97.3% of those bugs within two minutes.
Test-input generation for tensor kernels is folkloric. Most projects pick a representative shape and dtype, run a fixed-shape allclose-style check, and ship. We make the choices explicit and measure them. Using the gpuemu op-schema-aware seeded fuzzer (arXiv:2606.20128), we evaluate seven test-generation strategies across a 26-op corpus (16 correct controls and 10 LLM-style buggy variants seeded with documented transcription patterns) on an RTX 3060 GPU instance. Strategies vary the shape candidate set, the dtype mix, and the input value distribution. We report each strategy on two axes: bug recall and control false-positive (FP) rate. Boundary-only shape sampling is the operationally safe winner: 78% recall on the 10 buggy kernels with 0% FP on the 16 controls. Adversarial value sampling reaches higher recall (99%) but inflates control FP to 94% because the strategy injects NaN and Inf inputs and the validator's NaN check fires on every kernel that propagates them, not only on buggy kernels. On the two softmax tail-mask bugs the "regular" strategy (no boundary shapes) catches 0%, while boundary raises recall to 100% and 62% respectively. That gap is the clearest single signal in the data. The corpus result is about which seeded bug patterns each strategy catches, not about the bug rate of any specific deployed LLM.