BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
Authors: Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian Wang, Binyuan Hui, Niklas Muennighoff, David Lo, Daniel Fried, Xiaoning Du, Harm de Vries, Leandro Von Werra
Organizations: Monash University · CSIRO’s Data61 · Singapore Management University · Detomo Inc., Japan · Queen Mary University of London · Tencent AI Lab · TU Darmstadt · Independent · University of Virginia · Inria · Intel · Cornell University · University College London · Uber · University of Notre Dame · UNC Chapel Hill · UC Berkeley · MIT · Shanghai Jiaotong University · UIUC · Sea AI Lab · AWS AI Labs∗ · Contextual AI · Carnegie Mellon University · ServiceNow Research · Hugging Face
Task automation has been greatly empowered by the recent advances in Large Language Models (LLMs) via Python code, where the tasks ranging from software engineering development to general-purpose reasoning. While current benchmarks have shown that LLMs can solve tasks using programs like human developers, the majority of their evaluations are limited to short and self-contained algorithmic tasks or standalone function calls. Solving challenging and practical tasks requires the capability of utilizing diverse function calls as tools to efficiently implement functionalities like data analysis and web development. In addition, using multiple tools to solve a task needs compositional reasoning by accurately understanding complex instructions. Fulfilling both of these characteristics can pose a great challenge for LLMs.To assess how well LLMs can solve challenging and practical tasks via programs, we introduce BigCodeBench, a benchmark that challenges LLMs to invoke multiple function calls as tools from 139 libraries and 7 domains for 1,140 fine-grained tasks. To evaluate LLMs rigorously, each task encompasses 5.6 test cases with an average branch coverage of 99%. In addition, we propose a natural-language-oriented variant of BigCodeBench, BigCodeBench-Instruct, that automatically transforms the original docstrings into short instructions only with essential information. Our extensive evaluation of 60 LLMs shows that LLMs are not yet capable of following complex instructions to use function calls precisely, with scores up to 60%, significantly lower than the human performance of 97%. The results underscore the need for further advancements in this area.
Figures & tables
Figure 1: Programming tasks in BigCodeBench are structured with complex instructions in the docstrings, annotated by experts. The behavior of the solution is evaluated against a class of rigorous test cases with the proper environment setup.
Figure 2: Each programming task in BigCodeBench is created through a three-stage construction process. The task quality is controlled by the human-LLM collaboration.
Figure 3: Each task prompt in BigCodeBench - Instruct are transformed from BigCodeBench - Complete via the pre-defined rules. We omit non-essential details during the transformation.
Figure 4: Examples of tools in BigCodeBench are illustrated. Each function call belongs to a domain-specific library. The distribution of each domain is computed based on the frequency of domain-specific libraries appearing per task. For example, “63%” in “Computation” means that there are 63% tasks in BigCodeBench using at least one computation library.
Overall Statistics
Benchmark
# Task
Test (Avg.)
Prompt (Avg.)
Solution (Avg.)
#
Cov.
Char. (Code)
Line
Char.
Line
C.C.
HumanEval
164
7.8
98%
450.6 (450.6)
13.7
180.9
6.8
3.6
DS-1000
1,000
1.6
98%
871.8 (193.9)
29.1
138.1
5.1
1.6
DS-1000 (Orig.)
452
1.5
98%
831.4 (201.2)
26.2
115.5
4.2
1.4
ODEX
945
1.8
96%
87.5 (87.5)
1.0
50.4
1.9
1.4
Table 1: Summarized statistics of representative function-level Python programming benchmarks , partially inspired by ( Zan et al., 2023 ) . BigCodeBench are more complex regarding the depth of task complexity and breadth of tool-use knowledge. Cov. : Branch Coverage. Char. : Characters. C.C. : Cyclomatic Complexity (starting from 0, higher values indicate more complex code). Lib. : Library. Dom. : Domain. Std. : Python Standard Library . Ext. : Python External Library .
Figure 5: Complexity - tool comparisons with various benchmarks.
Figure 6: Pass@1 results of instruction-tuned LLMs on BigCodeBench - Complete (Top) and BigCodeBench - Instruct (Bottom). We only highlight the calibrated results having at least a difference of 1% from the original Pass@1.
Figure 7: Top 5 instruction-tuned LLMs on BigCodeBench - Complete .
Library (%)
Function Call (%)
Sol.⊆GT.
Sol.⊈GT.
Sol.⊆GT.
Sol.⊈GT.
Pass
79.84
20.16
40.46
59.54
Fail
71.78
28.22
22.47
77.53
Table 2: Tool-use comparisons between all generated solutions ( Sol. ) and ground truths ( GT. ) on BigCodeBench - Complete .
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
N-Gram
Overlap Percentage (%)
ODEX
Stack Overflow
StarCoderData
13
0.00
0.18
—
10
0.09
1.49
2.49
Appendix
Table 3: Comparison of N-Gram data across different datasets.
Figure 8: BigCodeBench - Hard construct workflow.
Figure 9: StackOverflow user query.
Figure 10: BigCodeBench task prompt.
Figure 11: BigCodeBench rankings v.s. BigCodeBench - Hard rankings
Figure 12: Library density comparisons. We sort the libraries by frequency count, showcasing the long-tail distribution to highlight the broad diversity within BigCodeBench .
Figure 13: Function call density comparisons. We sort function calls by frequency count, showcasing the long-tail distribution to highlight the broad diversity within BigCodeBench .
Benchmark
Complexity
Diversity
APPS ( Hendrycks et al., 2021 )
352.9
137
DS-1000 ( Lai et al., 2023 )
138.1
328
ODEX ( Wang et al., 2023c )
50.4
230
APIBench ( Patil et al., 2023 )
77.5
171
MBPP ( Austin et al., 2021 )
181.1
51
NumpyEval ( Zan et al., 2022b )
30.1
52
Appendix
Table 5: Depth (Complexity — Solution Characters) and breath (Diversity – Function Calls) comparisons to existing programming benchmarks in Python.
Model
Size / Checkpoint
Greedy Decoding
Random Sampling
Original
Calibrated
Pass@1
Pass@5
Instruction-tuned LLMs
GPT-4o ( OpenAI, 2024b )
2024-05-13
0.602
0.611
0.557
0.711
GPT-4-Turbo ( OpenAI, 2024a )
2024-04-09
0.582
0.582
0.563
0.699
GPT-4 ( Achiam et al., 2023 )
0613
0.484
0.572
0.407
0.682
GPT-3.5-Turbo ( OpenAI, 2023 )
0125
0.506
0.506
0.504
0.657
Appendix
Table 6: Evaluating LLMs on BigCodeBench - Complete . Due to the training flaws in StarCoder2 and Granite-Code, we additionally strip the trailing newlines for model inference.
Model
Size / Checkpoint
Greedy Decoding
Δ(NL2C−C2C)
Original
Calibrated
Original
Calibrated
GPT-4o
2024-05-13
0.499
0.511
-0.103
-0.100
GPT-4-Turbo
2024-04-09
0.480
0.482
-0.102
-0.100
GPT-4
0613
0.459
0.460
-0.025
-0.112
GPT-3.5-Turbo
0125
0.382
0.391
-0.124
-0.115
Claude-3
Opus
0.452
0.455
-0.116
-0.119
Appendix
Table 7: Evaluating instruction-tuned LLMs on BigCodeBench - Instruct . We report the results of greedy decoding, paired with calibrated results. To understand the performance difference between BigCodeBench - Complete and BigCodeBench - Instruct , we compute Δ(NL2C−C2C) for each model. We do not include the results of Granite-Code-Instruct 8B & 3B as they constantly have empty outputs.
Figure 18: Top 5 base LLMs
Split-Subset
GPT-4o-2024-05-13 (%)
Gemini-1.5-Pro-API-0514 (%)
Full-Complete
61.1 → 59.4
57.5 → 55.9
Full-Instruct
51.1 → 49.5
43.8 → 44.2 ( ↑ )
Hard-Complete
29.1 → 34.5 ( ↑ )
31.1 → 27.7
Hard-Instruct
25.0 → 23.0
19.6 → 18.2
Appendix
Table 8: Calibrated Pass@1 comparison of plain prompting and zero-shot CoT prompting on both BigCodeBench and BigCodeBench - Hard .
BigCodeBench - Complete
BigCodeBench - Instruct
r
p
r
p
MBPP+
0.963
0.937
0.926
0.881
NaturalCodeBench
0.757
0.803
0.913
0.857
HumanEval+
0.849
0.861
0.864
0.894
LiveCodeBench
0.853
0.898
0.778
0.815
Appendix
Table 9: Correlation coefficients measured by Pearson’s r and Spearman’s p against existing benchmarks.
Original Model
Original
Rephrased
Δ↓
Qwen2.5-Coder-32B-Instruct
27.7
8.8
-18.9
Llama-3.1-70B-Instruct
23.6
11.5
-12.1
GPT-4o-mini-2024-07-18
23.6
10.1
-13.5
GPT-4o-2024-05-13
25.0
12.8
-12.2
Appendix
Table 10: Performance comparison across models for original BigCodeBench - Instruct ( Original ), rephrased, and the differences.
This paper introduces Code Bench, a benchmark capable of evaluating Large Language Models (LLMs) concise code generation abilities in 60 programming languages. Based on code golf, a recreational programming competition focused on minimal character or byte solutions, the benchmark provides a distinctive measure of LLMs ability to produce efficient, concise code. Unlike existing benchmarks limited by fixed problem sets and language coverage, CodeGolf Bench leverages the code.golf platform to provide new problems and live human performance baselines. Evaluation of nine LLMs on Python and C++ tasks demonstrates that reasoning models significantly outperform non-reasoning models, achieving best average percentile of 70.97%. This performance gap is particularly pronounced in C++, highlighting reasoning's importance for languages with strict syntax requirements. Non-reasoning models struggle more with efficiency optimization across both languages, with best percentiles significantly lower than reasoning counterparts. CodeGolf Bench offers a dynamic framework for evaluating LLM code generation capabilities against evolving human performance on code golf.
Large language models (LLMs) can often generate functionally correct code, but their ability to produce efficient implementations for performance-critical systems tasks remains limited. Existing code benchmarks mainly emphasize correctness or algorithmic problem solving, while realistic systems-level optimization is still underexplored. To address this gap, we introduce PerfCodeBench, an executable benchmark for evaluating LLMs on high-performance code optimization. The tasks require system-level implementation choices, hardware-aware optimization, and careful handling of performance bottlenecks. Each task includes executable correctness checks, a baseline implementation, and a reference optimized solution. This allows us to evaluate both correctness and runtime-oriented efficiency. Our evaluation on a broad set of state-of-the-art LLMs shows a clear gap between model-generated code and expert-optimized implementations. The gap is especially large on tasks involving parallelism and GPU operations. Current models also show weaknesses in cross-language robustness and in consistently reaching expert-level efficiency. These results suggest that performance-aware evaluation are still needed. LLMs should move beyond generating merely correct code toward producing efficient systems software. We submit the benchmark data, evaluation infrastructure, and complete logs of all LLMs-generated code at https://anonymous.4open.science/r/perfcodebench-7CDE.
Large language models (LLMs) increasingly rely on explicit reasoning to solve coding tasks, yet evaluating the quality of this reasoning remains challenging. Existing reasoning evaluators are not designed for coding, and current benchmarks focus primarily on code generation, leaving other coding tasks largely unexplored. We introduce CodeRQ-Bench, the first benchmark for evaluating LLM reasoning quality across three coding task categories: generation, summarization, and classification. Using this benchmark, we analyze 1,069 mismatch cases from existing evaluators, identify five recurring limitations, and derive four design insights for reasoning evaluation in coding tasks. Guided by these insights, we propose VERA, a two-stage evaluator that combines evidence-grounded verification with ambiguity-aware score correction. Experiments on CodeRQ-Bench show that VERA consistently outperforms strong baselines across four datasets, improving AUCROC by up to 0.26 and AUPRC by up to 0.21. We release CodeRQ-Bench at https://github.com/MrLYG/CodeRQ-Bench, supporting future investigations.