BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
Authors: Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian Wang, Binyuan Hui, Niklas Muennighoff, David Lo, Daniel Fried, Xiaoning Du, Harm de Vries, Leandro Von Werra
Organizations: Monash University · CSIRO’s Data61 · Singapore Management University · Detomo Inc., Japan · Queen Mary University of London · Tencent AI Lab · TU Darmstadt · Independent · University of Virginia · Inria · Intel · Cornell University · University College London · Uber · University of Notre Dame · UNC Chapel Hill · UC Berkeley · MIT · Shanghai Jiaotong University · UIUC · Sea AI Lab · AWS AI Labs∗ · Contextual AI · Carnegie Mellon University · ServiceNow Research · Hugging Face
Task automation has been greatly empowered by the recent advances in Large Language Models (LLMs) via Python code, where the tasks ranging from software engineering development to general-purpose reasoning. While current benchmarks have shown that LLMs can solve tasks using programs like human developers, the majority of their evaluations are limited to short and self-contained algorithmic tasks or standalone function calls. Solving challenging and practical tasks requires the capability of utilizing diverse function calls as tools to efficiently implement functionalities like data analysis and web development. In addition, using multiple tools to solve a task needs compositional reasoning by accurately understanding complex instructions. Fulfilling both of these characteristics can pose a great challenge for LLMs.To assess how well LLMs can solve challenging and practical tasks via programs, we introduce BigCodeBench, a benchmark that challenges LLMs to invoke multiple function calls as tools from 139 libraries and 7 domains for 1,140 fine-grained tasks. To evaluate LLMs rigorously, each task encompasses 5.6 test cases with an average branch coverage of 99%. In addition, we propose a natural-language-oriented variant of BigCodeBench, BigCodeBench-Instruct, that automatically transforms the original docstrings into short instructions only with essential information. Our extensive evaluation of 60 LLMs shows that LLMs are not yet capable of following complex instructions to use function calls precisely, with scores up to 60%, significantly lower than the human performance of 97%. The results underscore the need for further advancements in this area.
Figures & tables
Figure 1: Programming tasks in BigCodeBench are structured with complex instructions in the docstrings, annotated by experts. The behavior of the solution is evaluated against a class of rigorous test cases with the proper environment setup.
Figure 2: Each programming task in BigCodeBench is created through a three-stage construction process. The task quality is controlled by the human-LLM collaboration.
Figure 3: Each task prompt in BigCodeBench - Instruct are transformed from BigCodeBench - Complete via the pre-defined rules. We omit non-essential details during the transformation.
Figure 4: Examples of tools in BigCodeBench are illustrated. Each function call belongs to a domain-specific library. The distribution of each domain is computed based on the frequency of domain-specific libraries appearing per task. For example, “63%” in “Computation” means that there are 63% tasks in BigCodeBench using at least one computation library.
Overall Statistics
Benchmark
# Task
Test (Avg.)
Prompt (Avg.)
Solution (Avg.)
#
Cov.
Char. (Code)
Line
Char.
Line
C.C.
HumanEval
164
7.8
98%
450.6 (450.6)
13.7
180.9
6.8
3.6
DS-1000
1,000
1.6
98%
871.8 (193.9)
29.1
138.1
5.1
1.6
DS-1000 (Orig.)
452
1.5
98%
831.4 (201.2)
26.2
115.5
4.2
1.4
ODEX
945
1.8
96%
87.5 (87.5)
1.0
50.4
1.9
1.4
Table 1: Summarized statistics of representative function-level Python programming benchmarks , partially inspired by ( Zan et al., 2023 ) . BigCodeBench are more complex regarding the depth of task complexity and breadth of tool-use knowledge. Cov. : Branch Coverage. Char. : Characters. C.C. : Cyclomatic Complexity (starting from 0, higher values indicate more complex code). Lib. : Library. Dom. : Domain. Std. : Python Standard Library . Ext. : Python External Library .
Figure 5: Complexity - tool comparisons with various benchmarks.
Figure 6: Pass@1 results of instruction-tuned LLMs on BigCodeBench - Complete (Top) and BigCodeBench - Instruct (Bottom). We only highlight the calibrated results having at least a difference of 1% from the original Pass@1.
Figure 7: Top 5 instruction-tuned LLMs on BigCodeBench - Complete .
Library (%)
Function Call (%)
Sol.⊆GT.
Sol.⊈GT.
Sol.⊆GT.
Sol.⊈GT.
Pass
79.84
20.16
40.46
59.54
Fail
71.78
28.22
22.47
77.53
Table 2: Tool-use comparisons between all generated solutions ( Sol. ) and ground truths ( GT. ) on BigCodeBench - Complete .
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
N-Gram
Overlap Percentage (%)
ODEX
Stack Overflow
StarCoderData
13
0.00
0.18
—
10
0.09
1.49
2.49
Appendix
Table 3: Comparison of N-Gram data across different datasets.
Figure 8: BigCodeBench - Hard construct workflow.
Figure 9: StackOverflow user query.
Figure 10: BigCodeBench task prompt.
Figure 11: BigCodeBench rankings v.s. BigCodeBench - Hard rankings
Figure 12: Library density comparisons. We sort the libraries by frequency count, showcasing the long-tail distribution to highlight the broad diversity within BigCodeBench .
Figure 13: Function call density comparisons. We sort function calls by frequency count, showcasing the long-tail distribution to highlight the broad diversity within BigCodeBench .
Benchmark
Complexity
Diversity
APPS ( Hendrycks et al., 2021 )
352.9
137
DS-1000 ( Lai et al., 2023 )
138.1
328
ODEX ( Wang et al., 2023c )
50.4
230
APIBench ( Patil et al., 2023 )
77.5
171
MBPP ( Austin et al., 2021 )
181.1
51
NumpyEval ( Zan et al., 2022b )
30.1
52
Appendix
Table 5: Depth (Complexity — Solution Characters) and breath (Diversity – Function Calls) comparisons to existing programming benchmarks in Python.
Model
Size / Checkpoint
Greedy Decoding
Random Sampling
Original
Calibrated
Pass@1
Pass@5
Instruction-tuned LLMs
GPT-4o ( OpenAI, 2024b )
2024-05-13
0.602
0.611
0.557
0.711
GPT-4-Turbo ( OpenAI, 2024a )
2024-04-09
0.582
0.582
0.563
0.699
GPT-4 ( Achiam et al., 2023 )
0613
0.484
0.572
0.407
0.682
GPT-3.5-Turbo ( OpenAI, 2023 )
0125
0.506
0.506
0.504
0.657
Appendix
Table 6: Evaluating LLMs on BigCodeBench - Complete . Due to the training flaws in StarCoder2 and Granite-Code, we additionally strip the trailing newlines for model inference.
Model
Size / Checkpoint
Greedy Decoding
Δ(NL2C−C2C)
Original
Calibrated
Original
Calibrated
GPT-4o
2024-05-13
0.499
0.511
-0.103
-0.100
GPT-4-Turbo
2024-04-09
0.480
0.482
-0.102
-0.100
GPT-4
0613
0.459
0.460
-0.025
-0.112
GPT-3.5-Turbo
0125
0.382
0.391
-0.124
-0.115
Claude-3
Opus
0.452
0.455
-0.116
-0.119
Appendix
Table 7: Evaluating instruction-tuned LLMs on BigCodeBench - Instruct . We report the results of greedy decoding, paired with calibrated results. To understand the performance difference between BigCodeBench - Complete and BigCodeBench - Instruct , we compute Δ(NL2C−C2C) for each model. We do not include the results of Granite-Code-Instruct 8B & 3B as they constantly have empty outputs.
Figure 18: Top 5 base LLMs
Split-Subset
GPT-4o-2024-05-13 (%)
Gemini-1.5-Pro-API-0514 (%)
Full-Complete
61.1 → 59.4
57.5 → 55.9
Full-Instruct
51.1 → 49.5
43.8 → 44.2 ( ↑ )
Hard-Complete
29.1 → 34.5 ( ↑ )
31.1 → 27.7
Hard-Instruct
25.0 → 23.0
19.6 → 18.2
Appendix
Table 8: Calibrated Pass@1 comparison of plain prompting and zero-shot CoT prompting on both BigCodeBench and BigCodeBench - Hard .
BigCodeBench - Complete
BigCodeBench - Instruct
r
p
r
p
MBPP+
0.963
0.937
0.926
0.881
NaturalCodeBench
0.757
0.803
0.913
0.857
HumanEval+
0.849
0.861
0.864
0.894
LiveCodeBench
0.853
0.898
0.778
0.815
Appendix
Table 9: Correlation coefficients measured by Pearson’s r and Spearman’s p against existing benchmarks.
Original Model
Original
Rephrased
Δ↓
Qwen2.5-Coder-32B-Instruct
27.7
8.8
-18.9
Llama-3.1-70B-Instruct
23.6
11.5
-12.1
GPT-4o-mini-2024-07-18
23.6
10.1
-13.5
GPT-4o-2024-05-13
25.0
12.8
-12.2
Appendix
Table 10: Performance comparison across models for original BigCodeBench - Instruct ( Original ), rephrased, and the differences.