Stateful Testing for Deep Learning Compilers

Jiatao Ge; Ruixiang Qian; Yibo Hu; Yanzhou Mu; Yinglong Zou; Jiawei Liu; Zhenyu Chen; Chunrong Fang

cs.SEOct 15, 2026

Stateful Testing for Deep Learning Compilers

Authors: Jiatao Ge, Ruixiang Qian, Yibo Hu, Yanzhou Mu, Yinglong Zou, Jiawei Liu, Zhenyu Chen, Chunrong Fang

Organizations: State Key Laboratory for Novel Software Technology, Nanjing University, China · UNIST, Republic of Korea

Abstract

Modern deep learning compilers compile models lazily and cache artifacts across invocations. Retained compiler state can make a later invocation incorrect even when it is correct in isolation. An analysis of 26 publicly reported issues in PyTorch, JAX, and TensorFlow shows that detecting such bugs requires sequences that first establish compiler state and then expose a failure. We present Rondo, a stateful testing method that represents each test case as a model paired with an invocation sequence. Rondo mutates invocation sequences and uses a Markov chain Monte Carlo sampler guided by compilation behavior to favor rarely observed behaviors. Its three-way differential oracle compares execution without compilation, compiled execution with compiler state reset, and compiled execution with state retained. It confirms stateful bugs by replaying the same invocation with state retained and reset while preserving inputs and program state, then shortens sequences while preserving the original bug. We evaluate Rondo on four deep learning frameworks using 24 common model architectures. Rondo detects 117 bugs, including 35 stateful bugs. We submit 39 bug reports, including 18 for stateful bugs; developers confirm 35 reports, including 16 for stateful bugs, and fix 18 bugs, including 11 stateful bugs. Rondo achieves the highest behavior coverage and compiler Python source-line coverage among the compared methods, with 99.26% alarm precision. It shortens 44 of 117 invocation sequences, reducing the lengths of these 44 sequences by 83.34% on average.

Figures & tables

Subject	Collected	Filtered				Studied
		Irrelevant	Enhancement	Duplicate	Not an Issue
PyTorch	77	38	1	12	5	21
JAX	27	20	0	0	3	4
TensorFlow	16	14	0	0	1	1
MindSpore	15	15	0	0	0	0
Total	135	87	1	12	9	26

Table 1. Bug report filtering results.

Figure 1. A stateful compiler bug across two invocations. An earlier addition call with inputs six and three returns nine and leaves compiled addition code. A later multiplication call with the same inputs should return eighteen. With compiler state retained, it reuses the addition code and returns nine. Resetting compiler state while keeping the multiplication function and its inputs unchanged makes the same later invocation return the correct result eighteen.

Figure 2. Overview of Rondo . Three panels show the complete testing pipeline. The first generates legal test cases through mutation or SMT-based synthesis, runs them, and feeds behavior keys to an MCMC sampler that guides subsequent test case generation. Selected test cases enter the second panel, where Without compilation, Compiled without state, and Compiled with state produce structured observations. Result comparison and independent semantic checks use shared numeric rules to identify candidate failures. In the third panel, the same invocation is repeated with compiler state retained and reset from aligned program state to determine the witness class. Sequence deletions and invocation simplifications are accepted only when the preservation condition holds, producing a witness and reproducer.

Operator	Operation	Example change
Shape	Change tensor shape	$(1,4)\rightarrow(2,4)$
Dtype	Change tensor dtype	float32 $\rightarrow$ float64
Layout	Change memory layout	contiguous $\rightarrow$ strided
Device	Change tensor device	CPU $\rightarrow$ GPU
Resample	Resample input values	$x=3\rightarrow x=5$
Rescale	Scale input magnitudes	$x=3\rightarrow x=3000$

Table 2. Mutation operators used by Rondo and illustrative examples.

Figure 3. One iteration of test-space exploration. Four stages show the current sequence, candidate generation, compilation-behavior feedback, and the acceptance decision. A return mutation appends invocation B to the sequence A, C, U. B repeats A after U updates the model parameter. Previously unobserved reuse after U raises the score from zero point two to one point two. With the illustrative proposal probabilities, the acceptance probability is approximately zero point six eight. A draw of zero point five accepts the candidate as the current sequence for the next iteration.

Figure 4. Test Oracle. Two panels show invocation modes and semantic checking. Without compilation, Compiled without state, and Compiled with state run the same test case from the same initial program state, with separate program-state trajectories. Compiled without state resets compiler state before each invocation, while Compiled with state retains it. Their observations contain returned values and declared observable state. These observations enter two parallel paths: result comparison, which traverses paired trees and checks tensor attributes and numeric leaves, and independent semantic checks, which enforce declared invariants when their preconditions hold. Both paths share numeric comparison rules and can produce candidates for refinement.

Framework	Bugs	Submitted	Confirmed	Fixed
PyTorch	30 (22)	15 (14)	13 (12)	8 (8)
TensorFlow	30 (8)	9 (1)	7 (1)	6 (1)
JAX	17 (2)	3 (2)	3 (2)	3 (2)
MindSpore	40 (3)	12 (1)	12 (1)	1 (0)
Total	117 (35)	39 (18)	35 (16)	18 (11)

Table 3. Bug detection and reporting results. Numbers in parentheses denote counts for bugs with at least one stateful witness.

	Behavior coverage				Compiler coverage
Method	Input	Path	Guard	HBC	Code (%)	Covered lines
NNSmith	0.107	0.091	0.061	0.000	14.12	31,751
OATest	0.069	0.056	0.035	0.000	14.15	31,820
AlignGuard	0.061	0.059	0.082	0.023	14.07	31,627
RandomHistory	0.109	0.106	0.154	0.117	14.17	31,864
Rondo	0.356	0.370	0.384	0.242	14.94	33,591

Table 4. Behavior coverage and compiler Python source-line coverage.

Figure 6. Coverage growth. Columns (a)–(d) show Input, Path, Guard, and HBC. The rows compare methods (top) and exploration-component configurations (bottom). Eight line plots show cumulative behavior coverage over test cases. The top row compares NNSmith, OATest, AlignGuard, RandomHistory, and Rondo. The bottom row compares full Rondo with variants that remove feedback, SMT synthesis, the return operator, or state operations. The four columns show Input, Path, Guard, and History-Behavior Coverage.

	Behavior coverage				Compiler coverage
Configuration	Input	Path	Guard	HBC	Code (%)	Covered lines
Full Rondo	0.356	0.370	0.384	0.242	14.94	33,591
w/o Feedback	0.198	0.193	0.199	0.166	14.79	33,239
w/o SMT	0.337	0.352	0.385	0.232	14.94	33,594
w/o Return	0.269	0.295	0.229	0.160	14.90	33,501
w/o State Operations	0.240	0.231	0.209	0.124	14.75	33,171

Table 5. Behavior coverage and compiler Python source-line coverage under exploration-component ablations.

	Detection results			Alarm reliability
Configuration	TP	New TP	Alarms	FP	FDR (%)
Crash-only	31	31	31	0	0.00
Two-mode numerical	354	323	357	3	0.84
Three-mode numerical	382	28	385	3	0.78
Three-mode structured	487	105	490	3	0.61
Full Oracle	539	52	543	4	0.74

Table 6. Oracle detections and false discovery rates across five levels of checking. TP counts are cumulative.

	All test cases (117)			Shortened test cases (44)
Metric	Mean	SD	Median	Mean	SD	Median
Original invocations	12.26	32.89	3	22.18	42.22	13
Refined invocations	4.56	18.96	1	1.73	2.56	1
Oracle checks	7.32	9.71	5	9.70	6.87	7
Compression ratio (%)	31.34	41.49	0.00	83.34	15.60	89.44
Refinement time (s)	184.55	295.42	42.54	357.78	355.53	189.26

Table 7. Sequence refinement statistics for all 117 test cases and the 44 effectively shortened test cases.

Automated Software Testing

Automated software testing uses tools and algorithms to generate, run, and evaluate tests that find defects and prevent regressions.

See all 135 papers

Grounding Deep Learning Framework Testing in Data Flow
Oct 11, 2026Jiatao Ge, Yanzhou Mu, Yinglong Zou +3Automated Software Testing DNN Testing
State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · UNIST, Ulsan, Republic of Korea
Demystifying the Silence of Correctness Bugs in PyTorch Compiler
Apr 9, 2026Meiziniu Li, Dongze Li, Jianmeng Liu +1
The Hong Kong University of Science and Technology, China · Guangzhou HKUST Fok Ying Tung Research Institute, China · Carnegie Mellon University, USA
The Output-Space Hypothesis: Enumerative Equivalence Checking for Tensor Programs
Sep 17, 2026Paul Biberstein, Joseph Devietti, Mayur NaikDifferential Testing GPU Kernel Correctness Verification
University of Pennsylvania Philadelphia, PA, USA

Oct 11, 2026J/K move · Enter open · S save

Grounding Deep Learning Framework Testing in Data Flow

Jiatao Ge, Yanzhou Mu, Yinglong Zou, Yibo Hu+2

State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · UNIST, Ulsan, Republic of Korea

Automated Software Testing DNN Testing Automated Test Generation Mutation Testing

Open paper AskPDF arXiv

Data flow underlies deep learning (DL) framework execution, including automatic differentiation and operator-fusion optimization. Bugs involving data flow can cause incorrect results or unexpected behavior. Existing testing methods represent models primarily by operator topology, omitting runtime tensor values and properties and operator input requirements. They therefore cannot distinguish data flows in models with similar topologies, even when these differences affect framework behavior. We introduce the Tensor Flow Graph (TFG) to capture runtime tensor states and operator input requirements. Its nodes represent operators and their input requirements, while its edges record tensor values and properties. Building on the TFG, we propose FlowMuT, a mutation-based testing method grounded in data flow. Mutations are represented as constrained subgraph replacements, with mutation operators derived from prior work and an analysis of mutable data flow objects. In each round, FlowMuT builds a TFG for the seed model, pairs mutation operators with TFG sites, and filters illegal candidates using data flow constraints. It uses LinUCB to select a candidate based on coverage and impact, executes the mutant, updates selection feedback, and uses each accepted mutant as the next seed. We evaluate FlowMuT on PyTorch and MindSpore using representative models from diverse application scenarios. FlowMuT submitted 55 bug reports for cases missed by all baseline methods; developers confirmed 41 reports and fixed the bugs described in 7 reports. FlowMuT achieves 43.4% higher data flow diversity than the strongest baseline and a legal mutant rate of 96.7%, compared with 86.4% without constraints. These results show that testing from a data flow perspective is practical and effective.

Abstract

Automated Software Testing

Explore similar work

Grounding Deep Learning Framework Testing in Data Flow

Demystifying the Silence of Correctness Bugs in PyTorch Compiler

The Output-Space Hypothesis: Enumerative Equivalence Checking for Tensor Programs