Message Passing Interface (MPI) is a foundational technology in high-performance computing (HPC), widely used for large-scale simulations and distributed training (e.g., in machine learning frameworks such as PyTorch and TensorFlow). However, maintaining MPI programs remains challenging due to their complex interplay among processes and the intricacies of message passing and synchronization. With the advancement of large language models like ChatGPT, it is tempting to adopt such technology for automated error detection and repair. Yet, our studies reveal that directly applying large language models (LLMs) yields suboptimal results, largely because these models lack essential knowledge about correct and incorrect usage, particularly the bugs found in MPI programs. In this paper, we design a bug detection and repair technique alongside Few-Shot Learning (FSL), Chain-of-Thought (CoT) reasoning, and Retrieval Augmented Generation (RAG) techniques in LLMs to enhance the large language model's ability to detect and repair errors. Surprisingly, such enhancements lead to a significant improvement, from 44% to 77%, in error detection accuracy compared to baseline methods that use ChatGPT directly. Additionally, our experiments demonstrate our bug referencing technique generalizes well to other large language models.
Figures & tables
Figure 1: Comparison of Zero-Shot, Few-Shot, Few-Shot+Chain-of-Thought (CoT), and Few-Shot+CoT+RAG prompting techniques. The inclusion of Few-Shot and CoT reasoning significantly enhances performance across all metrics.
Tool
TP
TN
FP
FN
Acc(%)
Prec(%)
Rec(%)
F1(%)
CIVL [ 23 ]
61
18
61
101
32.78
37.65
50.00
42.96
ISP [ 35 ]
116
17
94
14
55.19
89.23
55.24
68.24
MPISV [ 4 ]
68
15
64
94
34.44
41.98
51.52
46.26
PARCOACH [ 26 ]
140
10
79
12
62.24
92.11
63.93
75.47
Table 4: Performance metrics of Non-LLM Approaches.
Figure 2: Detailed Performance Metrics Across Experimental ChatGPT Trials: Comparison of True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN) across Zero-Shot, Few-Shot, CoT, and RAG experimental setups, providing an in-depth insight into the strengths and weaknesses of each method in identifying MPI program defects.
Figure 3: Comparison among different RAG. As the blue bar is the highest, RAG_100% is the best.
Tool
FP
FN
FP Ratio
FN Ratio
(Fall-out %)
(Miss-rate %)
CIVL [ 23 ]
61
101
60.40
62.35
ISP [ 35 ]
94
14
70.15
10.77
MPISV [ 4 ]
64
94
61.54
58.02
PARCOACH [ 26 ]
79
12
66.39
7.89
Few-Shot+CoT (ours)
50
4
55.56
2.30
Table 7: False Positive and False Negative Ratios. The best results are in bold .
Figure 4: Distribution of Repair Successes and Failures by Evaluation Metric including a summary of ChatGPT repair success rates across different evaluation criteria: Successful Compilation, Resource Leak Removal, and Deadlock Removal. Depicting the specific areas of strength and opportunities for future enhancement in MPI program repair.
Figure 5: Comparison of Zero-Shot, Few-Shot, Few-Shot+CoT and Few-Shot+CoT+RAG True Positive(TP), True Negative(TN), False Positive(FP), and False Negative(FP) results. The inclusion of our bug referencing Few-Shot+CoT reasoning exhibits the largest improvement in true positives and reduction in false negatives across all three LLMs.
Figure 6: Comparison of Zero-Shot, Few-Shot, Few-Shot+CoT and Few-Shot+CoT+RAG prompting strategies based on accuracy, precision, recall, and F1-score. Our bug referencing technique provides the best classification and detection metrics across all three LLMs
Figure 7: Repair success and failure rates by defect type across all three LLMs.
Figure 8: Accuracy, Precision, Recall, and F1 Score by Retrieval Corpus Composition using Llama2, Code Llama, and Qwen2.5-coder, showing repeated trends for each retrieval corpus by LLM.
Large language models (LLMs) are increasingly integrated into high-performance computing (HPC) workflows, accelerating scientific discovery through diverse perspectives such as code generation and domain-specific decision-making. Yet, how soft errors propagate and affect LLM inference remains largely unexplored. To bridge this gap, we present a comprehensive study on error propagation in LLM inference, enabled by our proposed LLMFI, a configurable and deterministic fault-injection framework. Using LLMFI, we systematically inject faults across three open-weighted LLMs and thirteen representative tasks, covering reasoning, multilingual, mathematical, and coding domains. In addition, we conduct fine-grained case studies that reveal critical vulnerability patterns. Overall, our study yields 17 takeaways that advance the understanding of error propagation in LLM inference and introduces four low-overhead directions to improve reliability through software-only modification, offering practical guidance for future error detection and mitigation.
Yafan Huang, Sheng Di, Guanpeng Li
University of Iowa · Argonne National Laboratory · University of Florida
Repository-level benchmarks for evaluating Large Language Model (LLM) code repair on Secure Multi-Party Computation (MPC) software do not yet exist, and directly transplanting general-purpose benchmarks such as SWE-bench fails on three structural fronts: (i) MPC repositories are dominated by generic Python infrastructure rather than cryptographic logic; (ii) high-value MPC fixes lack the standardized tests rigid extraction pipelines require; and (iii) standard fail-to-pass evaluation is insufficient for code that must also be cryptographically safe. MPC is increasingly deployed for privacy-preserving machine learning, biomedical collaboration, and secure analytics. Existing MPC-specific code-synthesis efforts cover only operator-level or single-framework tasks; evaluating LLM agents on real repository-level MPC repair instead demands MPC-aware data curation and a verifier matched to the security and numerical-fidelity guarantees MPC programs must obey neither of which existing benchmarks provide. We introduce MPC-Patch-Bench, a repository-level benchmark organised around two frameworks. (1)The Data Curation Framework combines a domain-specific curation agent that filters raw pull requests through three cryptographic layers with a human-AI completion engine that synthesizes missing problem statements and Fail-to-Pass/Pass-to-Pass tests, yielding 205 fully verified instances. (2)The MPC Verifier provides dedicated security and numerical-fidelity checks via dynamic differential testing against plaintext oracles and MPC-specific static analysis rules that flag unsafe reveals, insecure arithmetic, and illegal public/private casts. The strongest evaluated LLM functionally resolves only 22.9% of MPC-Patch-Bench tasks; the MPC Verifier further reduces verified resolution to 17.1%, with up to 40% of functionally-passing patches rejected for cryptographic or numerical-fidelity violations.
State-of-the-art large language model (LLM) training takes tens of thousands of graphics processing units (GPUs) for months and encounters failures across the software and hardware stack. Existing fault-tolerance mechanisms either impose non-trivial overhead during failure-free execution or suffer from prolonged recovery latency, particularly under scenarios where a small subset of compute nodes experience permanent failures. %The tradeoff between failure-free overhead and recovery latency forms a space forms a Pareto frontier We present PHOENIX to simultaneously address both optimization objectives. PHOENIX incorporates a fault-tolerance mechanism that restores LLM training via hot-swapping, namely by replacing failed nodes with spare nodes without terminating the complete job. The hot-swapping of PHOENIX is enabled by two ideas: First, it exploits an off-critical-path in-memory checkpointing mechanism for spatial redundancy. Second, it introduces a communicator reconstruction protocol that replaces failed nodes with spare nodes at runtime. PHOENIX efficiently overlaps the in-memory checkpointing with computation, thus introducing zero overhead during error-free execution. Upon permanent node failures, PHOENIX can rebuild memory states with minimal recomputation by leveraging in-memory checkpoints. We evaluate PHOENIX across scales (up to 512 NVIDIA A100 GPUs) and LLMs (up to 65B parameters), and observe zero checkpoint overhead with hot-swapping recovery completing in under 40 seconds. These results show that PHOENIX simultaneously achieves both zero-overhead error-free execution and extremely low recovery cost.