Leveraging Large Language Models (LLMs) for code generation has increasingly emerged as a common practice in the domain of software engineering. Relevant benchmarks have been established to evaluate the code generation capabilities of LLMs. However, existing benchmarks focus primarily on sequential code, lacking the ability to effectively evaluate LLMs on concurrent code generation. Compared to sequential code, concurrent code exhibits greater complexity and possesses unique types of bugs, such as deadlocks and race conditions, that do not occur in sequential code. Therefore, a benchmark for evaluating sequential code generation cannot be useful for evaluating concurrent code generation with LLMs. To address this gap, we designed a benchmark CONCUR specifically aimed at evaluating the capability of LLMs to generate concurrent code. CONCUR consists of a base set of 43 concurrency problems derived from a standard concurrency textbook, together with 72 validated mutant variants, resulting in 115 total problems. The base problems serve as the semantic core of the benchmark, while the mutants expand linguistic and structural diversity. We conducted an evaluation of a range of LLMs on CONCUR, highlighting limitations of current models. Overall, our work provides a novel direction for evaluating the capability of LLMs to generate code with focus on concurrency.
Figures & tables
Figure 2: Code Evaluation Framework of CONCUR.
Error Type
Description
Deadlock (DL)
A deadlock occurs when multiple threads are running and each thread waits indefinitely for others to release their locks.
Race Condition (RC)
A race condition occurs when concurrent read and write operations access the same shared variable without proper synchronization.
Starvation (SV)
Starvation occurs when one or more threads are perpetually denied access to shared resources because higher-priority threads monopolize execution.
Uncaught Exception (UE)
Exceptions that arise during concurrent program execution, including both concurrency-related errors and all exceptions defined in the Java 8 standard libraries.
No Entry Method (NEM)
The generated code lacks a public class with a main method, resulting in no valid entry point for execution.
Single Thread (ST)
The generated code executes with only a single thread, which violates the prompt requirement for multi-threaded concurrent code.
Table 1: Error Types Detected by JPF
Model
Passing Rate
CodeBLEU
k=1
k=3
k=1
k=3
gpt-5
89/115 (77.39%)
105/115 (91.30%)
0.576122
0.576229
claude-opus-4-1
78/115 (67.83%)
91/115 (79.13%)
0.579037
0.580484
gemini-3-pro
76/115 (66.09%)
85/115 (73.91%)
0.574317
0.575100
gpt-4o
69/115 (60.00%)
95/115 (82.61%)
0.547317
0.541937
llama3.3:70b
57/115 (49.57%)
71/115 (61.74%)
0.514043
0.514464
Table 2: Comparison of 23 Large Language Models on Concurrent Code Generation for 115 Problems (Pass@1 vs. Pass@3).
Model
Compilation Errors
JPF Errors
DL
RC
SV
UE
NEM
ST
TE
claude-opus-4-1
1
5
7
0
9
0
0
15
codellama:34b
88
2
2
0
8
3
2
9
codeqwen:7b
64
1
3
0
1
6
3
4
codestral:22b
42
6
3
0
5
2
1
10
deepseek-r1:32b
43
1
3
0
4
0
1
17
Table 3: Summary of Errors in Generated Code (Pass@1)
LiveCodeBench (LCB) has recently become a widely adopted benchmark for evaluating large language models (LLMs) on code-generation tasks. By curating competitive programming problems, constantly adding fresh problems to the set, and filtering them by release dates, LCB provides contamination-aware evaluation and offers a holistic view of coding capability. However, LCB remains restricted to Python, leaving open the question of whether LLMs can generalize across the diverse programming languages required in real-world software engineering. We introduce Multi-LCB, a benchmark for evaluating LLMs across twelve programming languages, including Python. Multi-LCB transforms Python tasks from the LCB dataset into equivalent tasks in other languages while preserving LCB's contamination controls and evaluation protocol. Because it is fully compatible with the original LCB format, Multi-LCB will automatically track future LCB updates, enabling systematic assessment of cross-language code generation competence and requiring models to sustain performance well beyond Python. We evaluated 24 LLMs for instruction and reasoning on Multi-LCB, uncovering evidence of Python overfitting, language-specific contamination, and substantial disparities in multilingual performance. Our results establish Multi-LCB as a rigorous new benchmark for multi-programming-language code evaluation, directly addressing LCB's primary limitation and exposing critical gaps in current LLM capabilities.
Maria Ivanova, Pavel Zadorozhny, Rodion Levichev +5
Yandex School of Data Analysis, Applied AI Institute · GigaCode
This paper introduces Code Bench, a benchmark capable of evaluating Large Language Models (LLMs) concise code generation abilities in 60 programming languages. Based on code golf, a recreational programming competition focused on minimal character or byte solutions, the benchmark provides a distinctive measure of LLMs ability to produce efficient, concise code. Unlike existing benchmarks limited by fixed problem sets and language coverage, CodeGolf Bench leverages the code.golf platform to provide new problems and live human performance baselines. Evaluation of nine LLMs on Python and C++ tasks demonstrates that reasoning models significantly outperform non-reasoning models, achieving best average percentile of 70.97%. This performance gap is particularly pronounced in C++, highlighting reasoning's importance for languages with strict syntax requirements. Non-reasoning models struggle more with efficiency optimization across both languages, with best percentiles significantly lower than reasoning counterparts. CodeGolf Bench offers a dynamic framework for evaluating LLM code generation capabilities against evolving human performance on code golf.
Large language models (LLMs) can often generate functionally correct code, but their ability to produce efficient implementations for performance-critical systems tasks remains limited. Existing code benchmarks mainly emphasize correctness or algorithmic problem solving, while realistic systems-level optimization is still underexplored. To address this gap, we introduce PerfCodeBench, an executable benchmark for evaluating LLMs on high-performance code optimization. The tasks require system-level implementation choices, hardware-aware optimization, and careful handling of performance bottlenecks. Each task includes executable correctness checks, a baseline implementation, and a reference optimized solution. This allows us to evaluate both correctness and runtime-oriented efficiency. Our evaluation on a broad set of state-of-the-art LLMs shows a clear gap between model-generated code and expert-optimized implementations. The gap is especially large on tasks involving parallelism and GPU operations. Current models also show weaknesses in cross-language robustness and in consistently reaching expert-level efficiency. These results suggest that performance-aware evaluation are still needed. LLMs should move beyond generating merely correct code toward producing efficient systems software. We submit the benchmark data, evaluation infrastructure, and complete logs of all LLMs-generated code at https://anonymous.4open.science/r/perfcodebench-7CDE.