EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code

Yuhao Qing; Boyu Zhu; Mingzhe Du; Zhijiang Guo; Terry Yue Zhuo; Qianru Zhang; Jie M. Zhang; Heming Cui

cs.CLMay 19, 2025

EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code

Authors: Yuhao Qing, Boyu Zhu, Mingzhe Du, Zhijiang Guo, Terry Yue Zhuo, Qianru Zhang, Jie M. Zhang, Heming Cui,

Organizations: HKU · UCL · NTU · NUS ·

Abstract

Existing code generation benchmarks primarily evaluate functional correctness, with limited focus on code efficiency and often restricted to a single language like Python. To address this gap, we introduce EffiBench-X, the first multi-language benchmark designed to measure the efficiency of LLM-generated code. EffiBench-X supports Python, C++, Java, JavaScript, Ruby, and Golang. It comprises competitive programming tasks with human-expert solutions as efficiency baselines. Evaluating state-of-the-art LLMs on EffiBench-X reveals that while models generate functionally correct code, they consistently underperform human experts in efficiency. Even the most efficient LLM-generated solutions (Qwen3-32B) achieve only around 62% of human efficiency on average, with significant language-specific variations. LLMs show better efficiency in Python, Ruby, and JavaScript than in Java, C++, and Golang. For instance, DeepSeek-R1's Python code is significantly more efficient than its Java code. These results highlight the critical need for research into LLM optimization techniques to improve code efficiency across diverse languages. The dataset and evaluation infrastructure are submitted and available at https://github.com/EffiBench/EffiBench-X.git and https://huggingface.co/datasets/EffiBench/effibench-x.

Figures & tables

Dataset	Tasks	Test Cases	Solutions	Metrics	Language	Source From
EvalPerf	121	1/284.78	7.6	$DPS/DPS_{norm}$	Python	EvalPlus/HumanEval/MBPP
ENAMEL	142	20	1	Eff $@$ k	Python	HumanEval
COFFE (Function)	398	10.71	1	Efficiency $@$ k	Python	HumanEval/MBPP/APPS/CodeContests
COFFE (File)	358	48.63	66.93	Efficiency $@$ k	Python	HumanEval/MBPP/APPS/CodeContests
PIE	41	104	23.8	%Opt/Speedup/%Correct	C++	CodeNet
ECCO	48	20.0	16.5	Time/Memory	Python	CodeNet

Table 1: Comparison of EffiBench-X to other code efficiency benchmarks. EffiBench-X covers six programming languages, with one human-written solution for each language per task.

Figure 1: Overview of the construction pipeline. The process begins with collecting efficiency-critical problems from competitive programming platforms, followed by constructing canonical solutions from expert programmers, and then generating test suites with test case generators and solution evaluators.

Model Name	Execution Time	Memory Peak	Memory Integral	Pass@1
DeepSeek-V3-0324	40.46%	51.52%	39.38%	53.29%
DeepSeek-R1	61.33%	69.41%	60.06%	72.79%
Llama-4-Scout-17B-16E-Instruct	23.16%	28.09%	22.61%	28.44%
Llama-4-Maverick-17B-128E-Instruct	16.28%	36.47%	15.52%	37.32%
Qwen3-8B	45.44%	51.64%	45.11%	53.50%
Qwen3-14B	59.75%	60.79%	58.58%	63.30%

Table 2: Results averaged across six programming languages on EffiBench-X .

Model Name	ET	MP	MI	Pass@1	ET	MP	MI	Pass@1
	C++				Python
DeepSeek-V3-0324	39.02%	52.54%	32.23%	54.41%	48.02%	53.22%	48.59%	55.38%
DeepSeek-R1	60.89%	71.57%	51.89%	75.12%	67.30%	69.66%	66.27%	74.64%
Qwen3-14B	61.22%	63.84%	57.03%	66.77%	65.28%	64.36%	63.64%	68.38%
Qwen3-32B	63.89%	70.30%	56.48%	74.80%	69.28%	70.94%	68.34%	75.44%
Qwen2.5-Coder-14B-Instruct	32.02%	35.09%	28.17%	35.63%	31.89%	34.26%	31.89%	34.99%

Table 3: Results on EffiBench-X -C++ and EffiBench-X -Python.

Figure 2: Distribution of inefficiency causes across representative proprietary models.

Inefficiency Category	Gemini-2.5-Pro	GPT-4o	Claude-3.7-Sonnet
Algorithmic Complexity	29.9%	37.1%	28.0%
Suboptimal Implementation	28.2%	16.3%	29.0%
Redundant Computation	22.9%	22.9%	29.5%
I/O Overhead	11.4%	17.5%	5.3%
Inefficient Data Structure	7.6%	6.2%	8.2%

Table 4: Inefficiency Analysis by Category (higher = more frequent cause).

Language	ET	MP	MI	Pass@1	ET	MP	MI	Pass@1
	DeepSeek-R1				Claude-3.7-Sonnet
JavaScript	63.34%	69.19%	63.03%	71.43%	50.15%	57.33%	49.48%	58.59%
Ruby	64.01%	66.11%	63.65%	69.02%	49.44%	52.48%	49.59%	53.61%
Python	67.30%	69.66%	66.27%	74.64%	51.90%	55.88%	52.03%	57.78%
Java	52.23%	69.36%	54.98%	73.19%	46.33%	55.11%	46.41%	56.66%
C++	60.89%	71.57%	51.89%	75.12%	45.45%	55.26%	40.68%	56.98%

Table 5: DeepSeek-R1 and Claude-3.7-Sonnet on EffiBench-X across different languages.

Appendix figures & tables34 assets

Supplementary material from the paper’s appendix.

Appendix

Open Source Models
Model Name	URL
DeepSeek-V3-0324	https://huggingface.co/deepseek-ai/DeepSeek-V3-0324
DeepSeek-R1	https://huggingface.co/deepseek-ai/DeepSeek-R1
Llama-4-Scout-17B-16E-Instruct	https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct
Llama-4-Maverick-17B-128E-Instruct	https://huggingface.co/meta-llama/Llama-4-Maverick-17B-128E-Instruct
Qwen3-8B	https://huggingface.co/Qwen/Qwen3-8B

Appendix

Table 6: Model list and URLs.

All
Model Name	ET	MP	MI	Pass@1
DeepSeek-R1	62.45% (61.33%, 63.55%)	69.42% (69.41%, 69.44%)	61.15% (60.06%, 62.19%)	72.79%
Claude-3.7-Sonnet	48.63% (47.79%, 49.31%)	54.63% (54.60%, 54.65%)	47.78% (46.98%, 48.40%)	56.21%
C++
DeepSeek-R1	61.79% (60.89%, 62.77%)	71.54% (71.44%, 71.61%)	52.99% (51.89%, 54.10%)	75.12%
Claude-3.7-Sonnet	46.36% (45.45%, 47.18%)	55.12% (55.03%, 55.26%)	41.58% (40.68%, 42.05%)	56.82%

Appendix

Table 7: Robustness evaluation of code efficiency metrics through triple execution of DeepSeek-R1 and Claude-3.7-Sonnet generated solutions. Results are reported in Mean (min, max) format to demonstrate consistency across multiple runs.

Attempts	# of Generators	Ratio (%)	Cumulative Ratio (%)
1	510	81.86	81.86
2	71	11.40	93.26
3	20	3.21	96.47
4	4	0.64	97.11
5	3	0.48	97.59
$>$ 5	15	2.41	100.00

Appendix

Table 8: Success rate of test-case generator.

Metric	Results (%)
Line Coverage	100.00
Branch Coverage	100.00

Appendix

Table 9: Coverage analysis of generated tests.

Model	Median Unclipped ET Ratio	% of Tasks with ET $>$ 1.0
GPT-4o-mini	0.77	1.61
GPT-4o	0.82	7.54
Claude-3.5-Haiku	0.91	9.15
Gemini-2.5-Flash	0.87	13.80
Gemini-2.5-Pro	0.91	19.26
Claude-3.7-Sonnet	0.97	20.71

Appendix

Table 10: Unclipped median ET ratio and proportion of tasks with faster-than-expert performance (

\text{ET}>1.0

Optimization Category	Gemini-2.5-Pro	Claude-3.7-Sonnet	GPT-4o
Implementation-Level Optimization	55.83	62.02	13.79
Algorithmic Complexity Optimization	26.67	20.93	37.93
Problem Reformulation	7.50	6.98	41.38
Advanced Data Structure Usage	5.00	3.88	0.00
Pruning and Heuristic Optimization	5.00	6.20	6.90

Appendix

Table 11: Categorization of optimization strategies observed in faster-than-expert code.

Field Name	Definition
id	Problem index in the corresponding source.
title	Task name.
title_slug	URL-friendly slug of the task name, used for referencing.
description	Task description.
description_md	Task description in markdown format.
source	The source platform from which the problem was collected.

Appendix

Table 12: Dataset Card of EffiBench-X

Website	Aizu	CodeChef	Codeforces	AtCoder	LeetCode	Total
Tasks	33	93	10	149	338	623

Appendix

Table 13: Task distribution by source websites in EffiBench-X

Model Name ET	MP	MI	Pass@1
DeepSeek-V3-0324	34.46%	52.11%	34.35%	53.93%
DeepSeek-R1	52.23%	69.36%	54.98%	73.19%
Llama-4-Scout-17B-16E-Instruct	20.92%	26.16%	20.46%	26.48%
Llama-4-Maverick-17B-128E-Instruct	11.24%	38.80%	8.45%	39.81%
Qwen3-8B	32.63%	53.83%	37.54%	56.18%
Qwen3-14B	62.32%	65.04%	62.02%	68.70%

Appendix

Table 14: Evaluation of code generated by various LLMs on EffiBench-X -Java.

Model Name ET	MP	MI	Pass@1
DeepSeek-V3-0324	40.18%	51.22%	39.75%	52.17%
DeepSeek-R1	63.34%	69.19%	63.03%	71.43%
Llama-4-Scout-17B-16E-Instruct	24.70%	31.20%	24.61%	31.46%
Llama-4-Maverick-17B-128E-Instruct	14.10%	44.75%	14.45%	45.10%
Qwen3-8B	52.38%	54.11%	52.37%	55.22%
Qwen3-14B	60.65%	61.65%	60.05%	63.08%

Appendix

Table 15: Evaluation of code generated by various LLMs on EffiBench-X -JavaScript.

Open-source LLMs
Model Name ET	MP	MI	Pass@1
DeepSeek-V3-0324	45.61%	49.28%	45.65%	50.88%
DeepSeek-R1	64.01%	66.11%	63.65%	69.02%
Llama-4-Scout-17B-16E-Instruct	21.30%	23.19%	21.51%	23.60%
Llama-4-Maverick-17B-128E-Instruct	28.05%	32.59%	28.93%	33.23%
Qwen3-8B	44.44%	45.99%	44.35%	47.35%

Appendix

Table 16: Evaluation of code generated by various LLMs on EffiBench-X -Ruby.

Open-source LLMs
Model Name ET	MP	MI	Pass@1
DeepSeek-V3-0324	35.49%	50.75%	35.71%	52.97%
DeepSeek-R1	60.24%	70.57%	60.57%	73.35%
Llama-4-Scout-17B-16E-Instruct	20.84%	24.89%	20.89%	25.52%
Llama-4-Maverick-17B-128E-Instruct	7.12%	30.20%	7.12%	32.10%
Qwen3-8B	42.88%	46.34%	43.33%	47.83%

Appendix

Table 17: Evaluation of code generated by various LLMs on EffiBench-X -Go.

Open-source LLMs
Model Name	ET	MP	MI	Pass@1
DeepSeek-V3-0324	45.95%	58.46%	45.01%	59.52%
DeepSeek-R1	74.53%	83.26%	73.26%	85.60%
Llama-4-Scout-17B-16E-Instruct	29.47%	32.48%	28.80%	32.79%
Llama-4-Maverick-17B-128E-Instruct	20.71%	42.62%	19.91%	43.29%
Qwen3-8B	55.88%	63.69%	55.83%	65.09%

Appendix

Table 18: Evaluation of code generated by various LLMs on the EffiBench-X -functional subset.

Model Name	ET	MP	MI	Pass@1
DeepSeek-V3-0324	33.95%	43.29%	32.70%	45.91%
DeepSeek-R1	45.68%	52.99%	44.42%	57.60%
Llama-4-Scout-17B-16E-Instruct	15.66%	22.89%	15.28%	23.27%
Llama-4-Maverick-17B-128E-Instruct	11.03%	29.18%	10.31%	30.23%
Qwen3-8B	33.06%	37.35%	32.40%	39.77%
Qwen3-14B	44.12%	44.19%	42.22%	47.60%

Appendix

Table 19: Evaluation of code generated by various LLMs on the EffiBench-X -standard I/O subset.

Algorithm Tag	MP (%)	ET (%)	MI (%)
string-matching	90.91	86.85	86.48
simulation	89.29	80.37	77.99
counting	82.74	76.85	74.72
hash-table	76.75	71.61	70.16
two-pointers	76.14	79.85	77.02
enumeration	72.19	71.52	70.16

Appendix

Table 20: Algorithm Tag Analysis: per-tag efficiency by metric (%).

Language	Docker Image	Compilation/Execution Flags
C++	gcc:14.2.0-bookworm	-O2, -fsanitize=address
Java	openjdk:21-jdk-bookworm	-
JavaScript	node:22.14.0-bookworm	–harmony
Ruby	ruby:3.2.7-bookworm	-
Go	golang:1.23.7-bookworm	-
Python3	python:3.11.11-bookworm	-

Appendix

Table 21: Language Runtime Specifications and Flags

Model Name	all	C++	Java	JavaScript	Ruby	Go	Python
DeepSeek-V3-0324	32/3738 (0.86%)	5/623 (0.80%)	4/623 (0.64%)	5/623 (0.80%)	8/623 (1.28%)	4/623 (0.64%)	6/623 (0.96%)
DeepSeek-R1	6/3738 (0.16%)	2/623 (0.32%)	1/623 (0.16%)	1/623 (0.16%)	1/623 (0.16%)	1/623 (0.16%)	0/623 (0.00%)
Gemini-2.0-Flash	53/3738 (1.42%)	2/623 (0.32%)	2/623 (0.32%)	5/623 (0.80%)	10/623 (1.61%)	13/623 (2.09%)	21/623 (3.37%)
Gemini-2.0-Flash-Thinking	32/3738 (0.86%)	2/623 (0.32%)	3/623 (0.48%)	7/623 (1.12%)	7/623 (1.12%)	8/623 (1.28%)	5/623 (0.80%)
Gemini-2.5-Flash	78/3738 (2.09%)	12/623 (1.93%)	17/623 (2.73%)	13/623 (2.09%)	11/623 (1.77%)	15/623 (2.41%)	10/623 (1.61%)
Gemini-2.5-Pro	2/3738 (0.05%)	0/623 (0.00%)	0/623 (0.00%)	0/623 (0.00%)	0/623 (0.00%)	2/623 (0.32%)	0/623 (0.00%)

Appendix

Table 22: Rate of LLM-generated responses that do not extract code, presented per model.

Model	Runtime Score (%)	Memory Score (%)	Integral Score (%)	Pass Rate (%)
DeepSeek-V2	41.80	44.49	43.04	46.07
Qwen-32B	27.64	29.13	28.61	29.86
GPT-4o	26.15	28.09	26.89	28.57
Claude 3.5 Sonnet	31.88	34.83	32.97	35.47

Appendix

Table 23: Most efficient solution scores under stochastic decoding ( pass@5 ).

Model	Average Runtime Score (%)	Average Memory Score (%)	Average Integral Score (%)
DeepSeek-V2	34.88	35.76	34.28
Qwen-32B	22.03	22.80	21.93
GPT-4o	46.89	47.74	46.88
Claude 3.5 Sonnet	28.28	29.09	28.16

Appendix

Table 24: Average efficiency scores of correct solutions under stochastic decoding ( pass@5 ).

Time Limit	ET (%)
10s (Original)	53.63
7s	53.63
5s	53.63
3s	53.50

Appendix

Table 25: Performance of Claude 3.7 Sonnet at various execution time limits.

Model	ET @ 10s Cutoff (%)	ET @ 5s Cutoff (%)	Change (%)
Gemini-2.5-Pro	64.92	64.92	0.00
Gemini-2.5-Flash	58.95	58.88	-0.07
Claude-3.7-Sonnet	53.63	53.63	0.00
GPT-4o	40.12	40.12	0.00
Claude-3.5-Haiku	35.97	35.95	-0.02
GPT-4o-mini	28.63	28.62	-0.01

Appendix

Table 26: Ranking of models under different execution time cutoffs.

Model Name	Execution Time (ET) (%)	Memory Peak (MP) (%)	Memory Integral (MI) (%)	Pass@1 (%)
DeepSeek-V3-0324	25.81%	31.38%	24.61%	34.34%
DeepSeek-R1	36.09%	42.02%	34.96%	45.96%
Llama-4-Scout-17B-16E-Instruct	6.83%	10.73%	7.01%	11.11%
Llama-4-Maverick-17B-128E-Instruct	5.05%	12.68%	4.66%	14.14%
Qwen3-8B	18.69%	21.02%	18.07%	22.22%
Qwen3-14B	24.87%	24.70%	23.30%	27.27%

Appendix

Table 27: Evaluation of code generated by various LLMs on EffiBench-X -Aizu.

Model Name	Execution Time (ET) (%)	Memory Peak (MP) (%)	Memory Integral (MI) (%)	Pass@1 (%)
DeepSeek-V3-0324	29.97%	37.61%	28.06%	41.16%
DeepSeek-R1	37.36%	42.25%	35.33%	48.21%
Llama-4-Scout-17B-16E-Instruct	10.78%	19.41%	10.41%	19.80%
Llama-4-Maverick-17B-128E-Instruct	9.69%	26.13%	9.00%	27.29%
Qwen3-8B	25.09%	28.46%	23.77%	31.32%
Qwen3-14B	35.39%	35.32%	32.87%	39.60%

Appendix

Table 28: Evaluation of code generated by various LLMs on EffiBench-X -AtCoder.

Model Name	Execution Time (ET) (%)	Memory Peak (MP) (%)	Memory Integral (MI) (%)	Pass@1 (%)
DeepSeek-V3-0324	42.74%	55.97%	42.48%	57.17%
DeepSeek-R1	62.31%	73.85%	62.22%	76.88%
Llama-4-Scout-17B-16E-Instruct	26.57%	33.12%	25.91%	33.51%
Llama-4-Maverick-17B-128E-Instruct	15.46%	40.29%	14.56%	41.04%
Qwen3-8B	50.15%	56.55%	50.55%	58.78%
Qwen3-14B	64.57%	65.03%	63.54%	67.56%

Appendix

Table 29: Evaluation of code generated by various LLMs on EffiBench-X -CodeChef.

Model Name	Execution Time (ET) (%)	Memory Peak (MP) (%)	Memory Integral (MI) (%)	Pass@1 (%)
DeepSeek-V3-0324	38.47%	49.32%	37.71%	50.00%
DeepSeek-R1	46.62%	55.19%	45.52%	56.67%
Llama-4-Scout-17B-16E-Instruct	16.13%	19.72%	16.23%	20.00%
Llama-4-Maverick-17B-128E-Instruct	9.50%	25.88%	9.02%	26.67%
Qwen3-8B	40.19%	45.15%	39.54%	46.67%
Qwen3-14B	47.63%	46.83%	45.74%	48.33%

Appendix

Table 30: Evaluation of code generated by various LLMs on EffiBench-X -Codeforces.

Open-source LLMs
Model Name	Execution Time (ET) (%)	Memory Peak (MP) (%)	Memory Integral (MI) (%)	Pass@1 (%)
DeepSeek-V3-0324	45.95%	58.46%	45.01%	59.52%
DeepSeek-R1	74.53%	83.26%	73.26%	85.60%
Llama-4-Scout-17B-16E-Instruct	29.47%	32.48%	28.80%	32.79%
Llama-4-Maverick-17B-128E-Instruct	20.71%	42.62%	19.91%	43.29%
Qwen3-8B	55.88%	63.69%	55.83%	65.09%

Appendix

Table 31: Evaluation of code generated by various LLMs on EffiBench-X -LeetCode.

Method	Parameters	Contaminated Tasks	Rate (%)
N-gram Overlap	$n=20$	11 / 623	1.77
N-gram Overlap	$n=30$	6 / 623	0.96
Embedding Search	Cosine similarity $>0.90$	3 / 623	0.48

Appendix

Table 32: Contamination analysis on EffiBench-X (623 tasks).

Hardware	Model	Runtime	Memory	Integral
AMD EPYC	Gemini-2.5-Pro	56.83%	57.45%	54.13%
	DeepSeek-R1	48.92%	49.44%	47.03%
	GPT-4o	39.04%	39.00%	38.59%
Intel Xeon	Gemini-2.5-Pro	57.00%	58.54%	54.86%
	DeepSeek-R1	52.08%	53.06%	49.93%
	GPT-4o	39.46%	40.80%	38.93%

Appendix

Table 33: Efficiency comparison across different hardware platforms.

Prompt Strategy	Runtime Pass Rate (%)	Memory Pass Rate (%)	Integral Score (%)
Baseline Prompt	27.12	36.21	28.26
aggressive_opt	31.36	31.34	31.24
ecco	31.99	31.49	31.73
hardware_hint	30.64	30.52	30.63
eco_aware	32.40	32.20	32.42

Appendix

Table 34: Performance of single-shot generation under different prompt strategies.

Refinement Strategy	Runtime Pass Rate (%)	Memory Pass Rate (%)	Integral Score (%)
Unsupervised Self-Refine	34.39	36.63	34.17
Result Aware	37.16	37.93	37.24
Line Profiler	35.83	36.64	35.02
Memory Profiler	35.89	40.59	38.33
EffiLearner (Full Feedback)	39.19	40.59	38.83

Appendix

Table 35: Performance of iterative refinement under different feedback strategies.

Panel of Figure 3 — Figure 3: This figure shows a Python problem where DeepSeek-R1 generated code with efficiency nearly matching the human expert solution (ratio = 0.985). Both calculate adjacent character ASCII differences using equivalent linear-time algorithms with constant space complexity, demonstrating that well-trained LLMs can identify optimal approaches matching expert solutions.

Panel of Figure 4 — Figure 4: This figure shows a C++ problem where Claude-3.7-Sonnet’s solution achieves exceptional efficiency by choosing an optimal data structure.

Panel of Figure 5 — Figure 5: This figure shows Claude-3.7-Sonnet’s inefficient solution using 4D dynamic programming to count strings containing “leet.”

Panel of Figure 6 — Figure 6: This figure presents the human expert’s elegant solution using the Inclusion-Exclusion Principle, which directly calculates the answer by subtracting invalid configurations from the total possible strings.

Code Generation Benchmarks

Code generation benchmarks measure how well AI models produce code across programming tasks, languages, and real-world settings.

See all 154 papers

CodeGolf Bench: A Multi-Language Benchmark for Evaluating Concise Code Generation Capabilities of Large Language Models
May 28, 2026Vedant PadwalLanguage Model Generation Evaluation Code Generation
Independent
PerfCodeBench: Benchmarking LLMs for System-Level High-Performance Code Optimization
May 13, 2026Huihao Jing, Wenbin Hu, Shaojin Chen +5Benchmark Design High-Performance Computing
HKUST · NYU · SWUPL
Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages
Jun 18, 2026Maria Ivanova, Pavel Zadorozhny, Rodion Levichev +5Code Generation Code Generation Evaluation
Yandex School of Data Analysis, Applied AI Institute · GigaCode

May 28, 2026J/K move · Enter open · S save

CodeGolf Bench: A Multi-Language Benchmark for Evaluating Concise Code Generation Capabilities of Large Language Models

Vedant Padwal

Independent

Language Model Generation Evaluation Code Generation Code Generation Evaluation Code Generation Benchmarks LLM Code Generation Multilingual Code Generation

Open paper AskPDF arXiv

This paper introduces Code Bench, a benchmark capable of evaluating Large Language Models (LLMs) concise code generation abilities in 60 programming languages. Based on code golf, a recreational programming competition focused on minimal character or byte solutions, the benchmark provides a distinctive measure of LLMs ability to produce efficient, concise code. Unlike existing benchmarks limited by fixed problem sets and language coverage, CodeGolf Bench leverages the code.golf platform to provide new problems and live human performance baselines. Evaluation of nine LLMs on Python and C++ tasks demonstrates that reasoning models significantly outperform non-reasoning models, achieving best average percentile of 70.97%. This performance gap is particularly pronounced in C++, highlighting reasoning's importance for languages with strict syntax requirements. Non-reasoning models struggle more with efficiency optimization across both languages, with best percentiles significantly lower than reasoning counterparts. CodeGolf Bench offers a dynamic framework for evaluating LLM code generation capabilities against evolving human performance on code golf.

Abstract

Code Generation Benchmarks

Explore similar work

CodeGolf Bench: A Multi-Language Benchmark for Evaluating Concise Code Generation Capabilities of Large Language Models

PerfCodeBench: Benchmarking LLMs for System-Level High-Performance Code Optimization

Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages