From Prompts to Performance: Evaluating LLMs for Task-based Parallel Code Generation

Linus Bantel; Moritz Strack; Alexander Strack; Dirk Pflüger

cs.PLFeb 24, 2026

From Prompts to Performance: Evaluating LLMs for Task-based Parallel Code Generation

Authors: Linus Bantel, Moritz Strack, Alexander Strack, Dirk Pflüger

Organizations: Institute for Parallel and Distributed Systems, University of Stuttgart, Stuttgart, Germany

Abstract

Large Language Models (LLM) show strong abilities in code generation, but their skill in creating efficient parallel programs is less studied. This paper explores how LLMs generate task-based parallel code from three kinds of input prompts: natural language problem descriptions, sequential reference implementations, and parallel pseudo code. We focus on three programming frameworks: OpenMP Tasking, C++ standard parallelism, and the asynchronous many-task runtime HPX. Each framework offers different levels of abstraction and control for task execution. We evaluate LLM-generated solutions for correctness and scalability. Our results reveal both strengths and weaknesses of LLMs with regard to problem complexity and framework. Finally, we discuss what these findings mean for future LLM-assisted development in high-performance and scientific computing.

Figures & tables

Panel of Figure 1 — Figure 1: Pass@1 metric for the respective llm without any correction.

Panel of Figure 2 — Figure 2: Pass@1 metric for the level of fixes, such that the program is correct and runs

Category	#Lines	#Code	#Comments	Complexity
LLM
ChatGPT	67.25	51.27	4.66	8.65
Gemini	75.97	51.07	12.80	9.06
Qwen	71.89	50.78	8.63	9.66
Framework
OpenMP	70.11	51.14	8.16	9.06

Table 1: Comparison of generated code with regards to length and complexity.

Panel of Figure 3 — Figure 3: Scaling of the generated code grouped by benchmark problem.

Panel of Figure 4 — Figure 4: Scaling of the generated code grouped by framework.

	ChatGPT	Gemini	Qwen
PCGQS-Score	0.7425	0.7008	0.5702

Table 2: PCGQS-Score for the evaluated llm .

Code Generation

Code generation studies methods for turning natural-language, visual, or formal specifications into executable programs and code edits.

See all 313 papers

CodegenBench: Can LLMs Write Efficient Code Across Architectures?
Jun 1, 2026Jie Li, Wenzhao Wu, Junqi Hu +5High-Performance Computing Code Generation
1Sun Yat-sen University · 2National Supercomputing Center in Wuxi · 3National Supercomputing Center in Shenzhen +2
PerfCodeBench: Benchmarking LLMs for System-Level High-Performance Code Optimization
May 13, 2026Huihao Jing, Wenbin Hu, Shaojin Chen +5Benchmark Design High-Performance Computing
HKUST · NYU · SWUPL
Operational Robustness of LLMs on Code Generation
Feb 21, 2026Debalina Ghosh Paul, Hong Zhu, Ian BayleyLanguage Model Robustness LLM Code Generation
School of Engineering, Computing and Mathematics, Oxford Brookes University Oxford, OX3 0BP, United Kingdom

Jun 1, 2026J/K move · Enter open · S save

CodegenBench: Can LLMs Write Efficient Code Across Architectures?

Jie Li, Wenzhao Wu, Junqi Hu, Qinrui Zheng+4

1Sun Yat-sen University · 2National Supercomputing Center in Wuxi · 3National Supercomputing Center in Shenzhen+2

High-Performance Computing Code Generation Heterogeneous Computing Code Generation Benchmarks LLM Code Generation

Open paper AskPDF arXiv

While large language models (LLMs) have been extensively evaluated on code generation tasks for general-purpose programming and GPU-accelerated environments (e.g., PyTorch, CUDA), their capabilities in CPU-oriented high-performance computing (HPC) across diverse architectures remain underexplored. To bridge this gap, we introduce CodegenBench, a comprehensive benchmark suite designed to evaluate the generation of efficient parallel code across three distinct hardware platforms: x86_64, Sunway, and Kunpeng. Our benchmark comprises 106 standard Basic Linear Algebra Subprograms (BLAS) routines establishing a fundamental baseline, alongside 20 specialized computational kernels adapted for each of the unique supercomputing architectures (LeetSunway and LeetKunpeng). Our extensive evaluation reveals that while state-of-the-art LLMs can generate optimized code for ubiquitous architectures like x86_64, they exhibit significant performance degradation on domain-specific architectures with limited public documentation and training data, highlighting critical limitations in cross-platform generalization. Furthermore, our analysis of factors influencing code quality such as implementation length and task complexity indicates that current LLMs are most effective for moderately difficult problems requiring concise code snippets. We open-source our dataset and automated evaluation infrastructure to facilitate future research in LLM-driven high-performance code generation. The resources are available at https://anonymous.4open.science/r/CodegenBench-EDE1/ and https://anonymous.4open.science/r/CodegenBenchDataset-2551.

Abstract

Code Generation

Explore similar work

CodegenBench: Can LLMs Write Efficient Code Across Architectures?

PerfCodeBench: Benchmarking LLMs for System-Level High-Performance Code Optimization

Operational Robustness of LLMs on Code Generation