Grounding Deep Learning Framework Testing in Data Flow

Jiatao Ge; Yanzhou Mu; Yinglong Zou; Yibo Hu; Zhenyu Chen; Chunrong Fang

cs.SEOct 11, 2026

Grounding Deep Learning Framework Testing in Data Flow

Authors: Jiatao Ge, Yanzhou Mu, Yinglong Zou, Yibo Hu, Zhenyu Chen, Chunrong Fang

Organizations: State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · UNIST, Ulsan, Republic of Korea

Abstract

Data flow underlies deep learning (DL) framework execution, including automatic differentiation and operator-fusion optimization. Bugs involving data flow can cause incorrect results or unexpected behavior. Existing testing methods represent models primarily by operator topology, omitting runtime tensor values and properties and operator input requirements. They therefore cannot distinguish data flows in models with similar topologies, even when these differences affect framework behavior. We introduce the Tensor Flow Graph (TFG) to capture runtime tensor states and operator input requirements. Its nodes represent operators and their input requirements, while its edges record tensor values and properties. Building on the TFG, we propose FlowMuT, a mutation-based testing method grounded in data flow. Mutations are represented as constrained subgraph replacements, with mutation operators derived from prior work and an analysis of mutable data flow objects. In each round, FlowMuT builds a TFG for the seed model, pairs mutation operators with TFG sites, and filters illegal candidates using data flow constraints. It uses LinUCB to select a candidate based on coverage and impact, executes the mutant, updates selection feedback, and uses each accepted mutant as the next seed. We evaluate FlowMuT on PyTorch and MindSpore using representative models from diverse application scenarios. FlowMuT submitted 55 bug reports for cases missed by all baseline methods; developers confirmed 41 reports and fixed the bugs described in 7 reports. FlowMuT achieves 43.4% higher data flow diversity than the strongest baseline and a legal mutant rate of 96.7%, compared with 86.4% without constraints. These results show that testing from a data flow perspective is practical and effective.

Figures & tables

Figure 1. Three data flow challenges illustrated by scaled dot-product attention (SDPA). The top panel shows a scaled dot-product attention computation. Panel (a) compares a correct execution with separate contiguous copies of K and V against a buggy execution with expanded, aliased storage under the same topology. Panel (b) shows that slice-and-expand head sharing preserves values for equal K and V heads but changes unequal heads, making that candidate illegal. Panel (c) shows two legal mutations of equal heads, cloning and sharing, whose behavioral novelty is unknown.

Figure 2. The FlowMuT workflow. Four stages are arranged from left to right and continue along a lower feedback loop. A seed model is profiled into a TFG, a candidate is selected and applied to obtain a mutant, and the execution outcome guides the next iteration.

Figure 3. Construction of a TFG from a PyTorch model, combining operator topology

G

, runtime tensor values and properties

F

, and operator input requirements

R

. Three panels show the PyTorch source model, its computation graph, and the resulting TFG. The TFG preserves operator connections, annotates tensor edges with runtime values and properties, and attaches input requirements to consumer operators.

Figure 4. Construction of the FlowMuT mutation operator pool from CG-derived and TFG-derived operators. The two sources produce 120 unique CG-derived operators and 48 TFG-derived operators, of which 21 are shared. Their union is implemented and validated into a pool of 147 operators.

Mutation category		CG-derived construction			TFG-derived construction			Final pool
Object	Primitive	Extracted	Removed	Retained	Derived	Overlap	TFG-only	Total
Tensor	Update	19	14	5	8	5	3	8
Tensor	Insertion	1	0	1	4	1	3	4
Tensor subtotal		20	14	6	12	6	6	12
Operator	Update	46	9	37	6	5	1	38
Operator	Insertion	6	3	3	3	2	1	4

Table 1. Construction of the 147-operator pool by mutable TFG object, mutation primitive, and source.

Scene Task	Model	Input	Parameters
Image Classification	ConvNeXt V2-Tiny	224 $\times$ 224	28,635,496
	Swin V2-Tiny	256 $\times$ 256	28,347,154
	EfficientNetV2-S	384 $\times$ 384	21,458,488
Object Detection	YOLOv10-S	640 $\times$ 640	8,128,272
Object Detection	RT-DETRv2-S	640 $\times$ 640	20,226,500
Semantic Segmentation	SegFormer-B1	512 $\times$ 512	13,715,798

Table 2. Seed models by application task, input size, and parameter count.

Developer status			Symptom category
Status	PyTorch	MindSpore	Category	PyTorch	MindSpore
Reported	14 (100.0%)	41 (100.0%)	Crash	0 (0.0%)	31 (75.6%)
Confirmed	8 (57.1%)	33 (80.5%)	Numerical Error	12 (85.7%)	6 (14.6%)
Fixed	7 (50.0%)	0 (0.0%)	State Error	2 (14.3%)	4 (9.8%)

Table 3. Cumulative developer status and symptom categories of 55 new bug reports.

Figure 5. (a) Bug overlap from discovery campaigns; (b) C++ line overlap from one run per method. Two four-set overlap diagrams compare LEMON, DevMuT, FlowMuT, and COMET. Panel (a) shows detected-bug overlap, including 55 cases found only by FlowMuT and submitted as bug reports. Panel (b) shows C++ source-line coverage from one run per method: 32.2 percent for LEMON, 43.9 percent for DevMuT, 44.4 percent for FlowMuT, and 35.8 percent for COMET.

	Test Case Diversity at Round 250				Framework Coverage (%)			Execution Time (s/test case)
Method	LIC ( $\times 10^{-2}$ )	LPC ( $\times 10^{-2}$ )	LSC ( $\times 10^{-2}$ )	DFSD ( $\times 10^{-5}$ )	Python	API	C++	Generation	Detection
LEMON	2.89	1.37	0.44	28.08	27.8	28.0	34.8	0.85	3.19
COMET	3.37	3.09	0.65	32.11	28.0	28.1	35.0	7.60	3.76
DevMuT	3.17	1.36	0.58	30.80	30.8	32.0	41.5	1.20	3.98
FlowMuT	5.95	4.48	1.04	46.06	37.8	42.2	48.1	6.04	4.58

Table 4. Test case diversity, framework coverage, and execution cost of FlowMuT and the three baseline methods at mutation round 250.

Figure 6. Test case diversity over 250 mutation rounds. Columns (a)–(d) show LIC, LPC, LSC, and DFSD; the rows compare methods (top) and operator pools (bottom). Lines are means with 95.0% intervals. Eight line charts form a two-by-four grid. The columns show LIC, LPC, LSC, and DFSD. The top row compares LEMON, COMET, DevMuT, and FlowMuT, while the bottom row compares the full, CG-derived, and TFG-derived operator pools.

Configuration	Ops	Test Case Diversity				Framework Coverage (%)
Configuration		LIC ( $\times 10^{-2}$ )	LPC ( $\times 10^{-2}$ )	LSC ( $\times 10^{-2}$ )	DFSD ( $\times 10^{-5}$ )	Python	API	C++
Operator-source ablation
Full operator pool	147	5.95	4.48	1.04	46.06	37.8	42.2	48.1
CG-derived pool	120	3.17	3.42	0.71	30.51	29.9	32.4	37.7
TFG-derived pool	48	2.04	0.68	0.18	20.01	20.1	20.1	21.8
TFG object ablation (within the TFG-derived pool)

Table 5. Test case diversity and framework coverage results for operator-pool and TFG ablations at mutation round 250.

	Mutation legality		Filter outcomes				Filter selectivity (%)
Configuration	Ret. (%)	LMR (%)	TN	FN	FP	TP	Prec.	Rec.	FRR	Leak.
Full constraints	88.0	96.7	851	29	13	107	89.2	78.7	1.5	21.3
w/o input	92.7	92.3	856	71	8	65	89.0	47.8	0.9	52.2
w/o internal	90.7	94.2	854	53	10	83	89.2	61.0	1.2	39.0
w/o output	91.6	93.3	855	61	9	75	89.3	55.1	1.0	44.9
w/o all	100.0	86.4	864	136	0	0	N/A	0.0	0.0	100.0

Table 6. Mutation legality and filter selectivity under constraint ablations. TN/FN count legal/illegal retained candidates; FP/TP count legal/illegal rejected candidates.

	Test Case Diversity				Framework Coverage (%)		Distinct signatures
Configuration	LIC ( $\times 10^{-2}$ )	LPC ( $\times 10^{-2}$ )	LSC ( $\times 10^{-2}$ )	DFSD ( $\times 10^{-5}$ )	C++	API
Selection-policy comparison
One-shot	3.68	1.32	0.41	43.63	35.5	27.3	460.9
Random search	4.56	1.50	0.62	45.55	33.3	22.7	481.2
Fixed priority	1.99	0.56	0.17	10.97	28.4	17.2	115.9
UCB1	4.38	1.44	0.52	41.65	32.2	20.3	440.0

Table 7. Test case diversity and framework coverage under selection-policy and LinUCB context-feature ablations at mutation round 250.

Automated Software Testing

Automated software testing uses tools and algorithms to generate, run, and evaluate tests that find defects and prevent regressions.

See all 135 papers

Stateful Testing for Deep Learning Compilers
Oct 15, 2026Jiatao Ge, Ruixiang Qian, Yibo Hu +5Automated Software Testing Differential Testing
State Key Laboratory for Novel Software Technology, Nanjing University, China · UNIST, Republic of Korea
The Output-Space Hypothesis: Enumerative Equivalence Checking for Tensor Programs
Sep 17, 2026Paul Biberstein, Joseph Devietti, Mayur NaikDifferential Testing GPU Kernel Correctness Verification
University of Pennsylvania Philadelphia, PA, USA
Test-Input Generation for Tensor Programs: What Actually Finds Kernel Bugs
Jun 23, 2026Dipankar SarkarAutomated Software Testing Benchmark Design
Arizona State University, USA

Oct 15, 2026J/K move · Enter open · S save

Stateful Testing for Deep Learning Compilers

Jiatao Ge, Ruixiang Qian, Yibo Hu, Yanzhou Mu+4

State Key Laboratory for Novel Software Technology, Nanjing University, China · UNIST, Republic of Korea

Automated Software Testing Differential Testing

Open paper AskPDF arXiv

Modern deep learning compilers compile models lazily and cache artifacts across invocations. Retained compiler state can make a later invocation incorrect even when it is correct in isolation. An analysis of 26 publicly reported issues in PyTorch, JAX, and TensorFlow shows that detecting such bugs requires sequences that first establish compiler state and then expose a failure. We present Rondo, a stateful testing method that represents each test case as a model paired with an invocation sequence. Rondo mutates invocation sequences and uses a Markov chain Monte Carlo sampler guided by compilation behavior to favor rarely observed behaviors. Its three-way differential oracle compares execution without compilation, compiled execution with compiler state reset, and compiled execution with state retained. It confirms stateful bugs by replaying the same invocation with state retained and reset while preserving inputs and program state, then shortens sequences while preserving the original bug. We evaluate Rondo on four deep learning frameworks using 24 common model architectures. Rondo detects 117 bugs, including 35 stateful bugs. We submit 39 bug reports, including 18 for stateful bugs; developers confirm 35 reports, including 16 for stateful bugs, and fix 18 bugs, including 11 stateful bugs. Rondo achieves the highest behavior coverage and compiler Python source-line coverage among the compared methods, with 99.26% alarm precision. It shortens 44 of 117 invocation sequences, reducing the lengths of these 44 sequences by 83.34% on average.

Abstract

Automated Software Testing

Explore similar work

Stateful Testing for Deep Learning Compilers

The Output-Space Hypothesis: Enumerative Equivalence Checking for Tensor Programs

Test-Input Generation for Tensor Programs: What Actually Finds Kernel Bugs