Organizations: The Hong Kong University of Science and Technology, China · Guangzhou HKUST Fok Ying Tung Research Institute, China · Carnegie Mellon University, USA
Performance optimization of AI infrastructure is key to the fast adoption of large language models (LLMs). The PyTorch compiler (torch.compile), a core optimization tool for deep learning (DL) models (including LLMs), has received due attention. However, torch.compile is prone to correctness bugs, which cause incorrect outputs of compiled DL models without triggering exceptions, crashes, or warnings. These bugs pose a serious threat to the reliability of downstream LLM applications. Data from the PyTorch community shows that 19.2% of high-priority issues are incorrect outputs of compiled DL models induced by torch.compile bugs, the second-most-common bug category (only behind program crashes at 19.57%). However, no systematic study has been conducted to specifically characterize and thereby detect these bugs. In this paper, we present the first empirical study of the correctness bugs in torch.compile, examine their characteristics, and assess the effectiveness of existing fuzzers in detecting them. Based on our findings, we propose a proof-of-concept testing technique named AlignGuard, tailored specifically for detecting correctness bugs in torch.compile. AlignGuard incorporates bug characteristics distilled from our empirical study, applying LLM-based test mutation to existing test cases for correctness bug detection. At the time of writing, AlignGuard has successfully detected 23 new correctness bugs in recent torch.compile. All these bugs have been confirmed or fixed by the PyTorch development team, and over half (14/23) of them are even marked as high-priority bugs, underscoring the usefulness of our technique.
Figures & tables
Figure 1. An industrial LLM fails to converge during training due to a correctness bug in torch.compile . A correctness bug in \subject{} causes an industrial LLM to fail to saturate during training.
Figure 2. Architecture of PyTorch compiler. A diagram showing the typical deep learning compiler stack.
Figure 3. Distribution of faulty components (bottom and top left) and symptoms of high-priority issues in torch.compile (top right). Two bar charts showing the distribution of faulty components and symptoms of high-priority issues in PyTorch compiler.
Step
Filtering Criterion / Category
# Excluded
# Remaining
—
Initial issues gathered with keyword ‘torch.compile’
—
2,542
1
Excluded Issues: Feature requests and general help requests
581
1,961
2
Excluded Issues: Issues not discussing a bug within torch.compile
379
1,582
3
Excluded Issues: Issues not related to correctness bugs
1,329
253
4
Excluded Issues: Issues not confirmed by developers
93
160
5
Excluded Issues: Duplicated issues (confirmed by developers)
34
126
Table 1. Step-by-step filtering process for identifying correctness bug issues in torch.compile
Figure 4. Distribution of Bug Types A diagram showing the distribution of bug types in \subject{}.
Technique
Year
Generation Strategy
Opera
2025
Test Migration
DeepConstr
2024
LLM-based generation
WhiteFox
2024
LLM-based generation
NeuRI
2023
Grammar-based generation
NNSmith
2022
Grammar-based generation
Table 3. Characteristics of DL compiler testing techniques used in this study.
Step
Filtering Criterion / Category
# Excluded
# Remaining
—
Studied Dataset (Confirmed Correctness Bugs)
—
116
1
Excluded: no reproducible code
6
110
2
Excluded: not torch.compile bug
1
109
3
Excluded: patch not identified
17
92
4
Excluded: not reproducible on release
7
85
5
Excluded: require non-CPU/GPU hardware
8
77
Table 4. Step-by-step filtering process for correctness bug benchmark construction
Figure 6. Distribution of detected and missed bugs. The name of each bug category is abbreviated using its initials (e.g., ‘OTB’ for Operator-Transformation Bugs , ‘LCG’ for Low-Level Code Generation Bugs ). Distribution of Detected and Missed Bugs by All Fuzzers
Figure 7. Workflow of AlignGuard Workflow of \toolname{}
Bug Type
Reported
Confirmed
Fixed
High-Priority
Memory-related Bugs
12
12
7
8
Operator-related Bugs
7
7
1
3
Graph-related Bugs
3
3
1
2
Precision Bugs
1
1
1
1
Total
23
23
10
14
Table 5. New correctness bugs detected by AlignGuard
Modern deep learning compilers compile models lazily and cache artifacts across invocations. Retained compiler state can make a later invocation incorrect even when it is correct in isolation. An analysis of 26 publicly reported issues in PyTorch, JAX, and TensorFlow shows that detecting such bugs requires sequences that first establish compiler state and then expose a failure. We present Rondo, a stateful testing method that represents each test case as a model paired with an invocation sequence. Rondo mutates invocation sequences and uses a Markov chain Monte Carlo sampler guided by compilation behavior to favor rarely observed behaviors. Its three-way differential oracle compares execution without compilation, compiled execution with compiler state reset, and compiled execution with state retained. It confirms stateful bugs by replaying the same invocation with state retained and reset while preserving inputs and program state, then shortens sequences while preserving the original bug. We evaluate Rondo on four deep learning frameworks using 24 common model architectures. Rondo detects 117 bugs, including 35 stateful bugs. We submit 39 bug reports, including 18 for stateful bugs; developers confirm 35 reports, including 16 for stateful bugs, and fix 18 bugs, including 11 stateful bugs. Rondo achieves the highest behavior coverage and compiler Python source-line coverage among the compared methods, with 99.26% alarm precision. It shortens 44 of 117 invocation sequences, reducing the lengths of these 44 sequences by 83.34% on average.
Jiatao Ge, Ruixiang Qian, Yibo Hu +5
State Key Laboratory for Novel Software Technology, Nanjing University, China · UNIST, Republic of Korea
Compilers are critical to modern computing, yet fixing compiler bugs is difficult. While recent large language model (LLM) advancements enable automated bug repair, compiler bugs pose unique challenges due to their complexity, deep cross-domain expertise requirements, and sparse, non-descriptive bug reports, necessitating compiler-specific harnesses. To bridge the gap, we introduce llvm-harness, the first harness designed to assist LLM agents in understanding and fixing compiler bugs. Our current focus is on the middle end of LLVM, one of the most widely used compiler infrastructures. Central to llvm-harness are agent-friendly LLVM tools, a benchmark llvm-bench of 334 reproducible LLVM middle-end bugs, and a tailored mini agent llvm-autofix-mini for fixing LLVM middle-end bugs automatically. We evaluate five frontier models and find that they exhibit a performance decline when tackling compiler bugs with the state-of-the-art agent. With llvm-harness' enhancement, their performance improves by 62%. Our specialized mini agent llvm-autofix-mini further outperforms the llvm-harness-enhanced state-of-the-art by 22%. This emphasizes the necessity for specialized harnesses like ours to assist LLMs in compiler engineering tasks. Despite promising results, our expert review also reveals several open challenges that remain when applying LLMs for compiler engineering tasks. GitHub: https://github.com/dtcxzyw/llvm-harness
Yingwei Zheng, Cong Li, Shaohua Li +2
Southern University of Science and Technology · ETH Zurich · The Chinese University of Hong Kong
Benchmarks for LLM-generated GPU kernels (KernelBench, TritonBench, GEAK) score correctness through fixed-shape, small-sample allclose-style checks. The number of inputs varies between benchmarks. The shape, dtype, and tolerance are fixed for each kernel. We test that oracle empirically. We construct a controlled corpus of 24 Triton and CPU stand-in kernels (15 correct controls and 9 LLM-style buggy variants seeded with documented transcription errors) and re-evaluate it under op-schema-aware seeded fuzzing with a high-precision (fp64) CPU reference and per-(op, dtype) absolute tolerances. The seeded oracle flags 9 of 9 buggy kernels and passes 15 of 15 correct controls, at zero precision cost on controls. We extend the corpus to 26 ops (adding a flash-attention pair) and re-run the same protocol on five GPU classes (RTX 3060, A10, L40S, A100 SXM4, H100 NVL). The verdicts are identical across all five GPUs: 10 of 10 illusions caught and 16 of 16 controls clean. The corpus result is about LLM-style transcription bugs that the allclose-on-one-shape oracle certifies as correct, not about the bug rate of any specific deployed LLM. Every flagged failure replays byte-for-byte from a stored seed.