Improving MPI Error Detection and Repair with Large Language Models and Bug References

Scott Piersall; Yang Gao; Shenyang Liu; Liqiang Wang

cs.SEApr 2, 2026

Improving MPI Error Detection and Repair with Large Language Models and Bug References

Authors: Scott Piersall, Yang Gao, Shenyang Liu, Liqiang Wang

Organizations: Dept. of Computer Science, University of Central Florida, Orlando, 32816, FL, US

Abstract

Message Passing Interface (MPI) is a foundational technology in high-performance computing (HPC), widely used for large-scale simulations and distributed training (e.g., in machine learning frameworks such as PyTorch and TensorFlow). However, maintaining MPI programs remains challenging due to their complex interplay among processes and the intricacies of message passing and synchronization. With the advancement of large language models like ChatGPT, it is tempting to adopt such technology for automated error detection and repair. Yet, our studies reveal that directly applying large language models (LLMs) yields suboptimal results, largely because these models lack essential knowledge about correct and incorrect usage, particularly the bugs found in MPI programs. In this paper, we design a bug detection and repair technique alongside Few-Shot Learning (FSL), Chain-of-Thought (CoT) reasoning, and Retrieval Augmented Generation (RAG) techniques in LLMs to enhance the large language model's ability to detect and repair errors. Surprisingly, such enhancements lead to a significant improvement, from 44% to 77%, in error detection accuracy compared to baseline methods that use ChatGPT directly. Additionally, our experiments demonstrate our bug referencing technique generalizes well to other large language models.

Figures & tables

Figure 1: Comparison of Zero-Shot, Few-Shot, Few-Shot+Chain-of-Thought (CoT), and Few-Shot+CoT+RAG prompting techniques. The inclusion of Few-Shot and CoT reasoning significantly enhances performance across all metrics.

Tool	TP	TN	FP	FN	Acc(%)	Prec(%)	Rec(%)	F1(%)
CIVL [ 23 ]	61	18	61	101	32.78	37.65	50.00	42.96
ISP [ 35 ]	116	17	94	14	55.19	89.23	55.24	68.24
MPISV [ 4 ]	68	15	64	94	34.44	41.98	51.52	46.26
PARCOACH [ 26 ]	140	10	79	12	62.24	92.11	63.93	75.47

Table 4: Performance metrics of Non-LLM Approaches.

Figure 2: Detailed Performance Metrics Across Experimental ChatGPT Trials: Comparison of True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN) across Zero-Shot, Few-Shot, CoT, and RAG experimental setups, providing an in-depth insight into the strengths and weaknesses of each method in identifying MPI program defects.

Figure 3: Comparison among different RAG. As the blue bar is the highest, RAG_100% is the best.

Tool	FP	FN	FP Ratio	FN Ratio
Tool	FP	FN	(Fall-out %)	(Miss-rate %)
CIVL [ 23 ]	61	101	60.40	62.35
ISP [ 35 ]	94	14	70.15	10.77
MPISV [ 4 ]	64	94	61.54	58.02
PARCOACH [ 26 ]	79	12	66.39	7.89
Few-Shot+CoT (ours)	50	4	55.56	2.30

Table 7: False Positive and False Negative Ratios. The best results are in bold .

Figure 4: Distribution of Repair Successes and Failures by Evaluation Metric including a summary of ChatGPT repair success rates across different evaluation criteria: Successful Compilation, Resource Leak Removal, and Deadlock Removal. Depicting the specific areas of strength and opportunities for future enhancement in MPI program repair.

Panel of Figure 5 — Figure 5: Comparison of Zero-Shot, Few-Shot, Few-Shot+CoT and Few-Shot+CoT+RAG True Positive(TP), True Negative(TN), False Positive(FP), and False Negative(FP) results. The inclusion of our bug referencing Few-Shot+CoT reasoning exhibits the largest improvement in true positives and reduction in false negatives across all three LLMs.

Panel of Figure 6 — Figure 6: Comparison of Zero-Shot, Few-Shot, Few-Shot+CoT and Few-Shot+CoT+RAG prompting strategies based on accuracy, precision, recall, and F1-score. Our bug referencing technique provides the best classification and detection metrics across all three LLMs

Panel of Figure 7 — Figure 7: Repair success and failure rates by defect type across all three LLMs.

Panel of Figure 8 — Figure 8: Accuracy, Precision, Recall, and F1 Score by Retrieval Corpus Composition using Llama2, Code Llama, and Qwen2.5-coder, showing repeated trends for each retrieval corpus by LLM.

Language Model Error Detection

Language model error detection identifies unreliable answers, reasoning steps, and actions so systems can flag, verify, or repair them.

See all 90 papers

Not All Errors Are Equal: A Systematic Study of Error Propagation in Large Language Model Inference
Jun 1, 2026Yafan Huang, Sheng Di, Guanpeng LiLLM Inference Language Model Robustness
University of Iowa · Argonne National Laboratory · University of Florida
MPC-Patch-Bench: Security-Aware LLM Code Patch for Multi-Party Computation
Jun 9, 2026Yukuan Zhang, Mengxin Zheng, Qian LouSecure Multi-Party Computation LLM Evaluation
University of Central Florida
PHOENIX: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpoint
Jul 2, 2026Haotian Xie, Junlin Chen, Mingkai Zheng +2Fault-Tolerant Distributed Training Distributed Training
Rutgers University · George Mason University

Jun 1, 2026J/K move · Enter open · S save

Not All Errors Are Equal: A Systematic Study of Error Propagation in Large Language Model Inference

Yafan Huang, Sheng Di, Guanpeng Li

University of Iowa · Argonne National Laboratory · University of Florida

LLM Inference Language Model Robustness

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly integrated into high-performance computing (HPC) workflows, accelerating scientific discovery through diverse perspectives such as code generation and domain-specific decision-making. Yet, how soft errors propagate and affect LLM inference remains largely unexplored. To bridge this gap, we present a comprehensive study on error propagation in LLM inference, enabled by our proposed LLMFI, a configurable and deterministic fault-injection framework. Using LLMFI, we systematically inject faults across three open-weighted LLMs and thirteen representative tasks, covering reasoning, multilingual, mathematical, and coding domains. In addition, we conduct fine-grained case studies that reveal critical vulnerability patterns. Overall, our study yields 17 takeaways that advance the understanding of error propagation in LLM inference and introduces four low-overhead directions to improve reliability through software-only modification, offering practical guidance for future error detection and mitigation.

Abstract

Language Model Error Detection

Explore similar work

Not All Errors Are Equal: A Systematic Study of Error Propagation in Large Language Model Inference

MPC-Patch-Bench: Security-Aware LLM Code Patch for Multi-Party Computation

PHOENIX: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpoint