BigCodeArena: Unveiling More Reliable Human Preferences in Code Generation via Execution
Authors: Terry Yue Zhuo, Xiaolong Jin, Hange Liu, Juyong Jiang, Tianyang Liu, Chen Gong, Bhupesh Bishnoi, Vaisakhi Mishra, Marek Suppa, Noah Ziems, Saiteja Utpala, Ming Xu, Guangyu Song, Kaixin Li, Yuhan Cao, Bo Liu, Zheng Liu, Sabina Abdurakhmanova, Wenhao Yu, Mengzhao Jia, Jihan Yao, Kenneth Hamilton, Kumar Shridhar, Minh Chien Vu, Dingmin Wang, Jiawei Liu, Zijian Wang, Qian Liu, Binyuan Hui, Meg Risdal, Ahsen Khaliq, Atin Sood, Zhenchang Xing, Wasi Uddin Ahmad, John Grundy, David Lo, Banghua Zhu, Xiaoning Du, Torsten Scholak, Leandro von Werra
Organizations: Monash University · CSIRO’s Data61 · Purdue University · Independent · HKUST (Guangzhou) · UCSD · UVA · CNRS, France · IBM · Cisco · Comenius University in Bratislava · University of Notre Dame · Uber · Tano Labs · NUS · Institute of Automation, CAS · Tencent AI Lab · University of Washington · Nevsky Collective · ETH Zurich · Detomo Inc · University of Oxford · UIUC · Google · NVIDIA · Singapore Management University · ServiceNow Research · Hugging Face
Crowdsourced model evaluation platforms, such as Chatbot Arena, enable real-time evaluation from human perspectives to assess the quality of model responses. In the coding domain, manually examining the quality of LLM-generated content is extremely challenging, as it requires understanding long chunks of raw code and deliberately simulating code execution. To this end, we introduce BigCodeArena, an open human evaluation platform for code generation backed by a comprehensive and on-the-fly execution environment. Built on top of Chatbot Arena, BigCodeArena enables the execution of LLM-generated code and allows humans to interact with the execution process and outcomes. We collected over 14,000 raw code-centric conversation sessions across 10 widely used LLMs, spanning 10 languages and 8 types of execution environments. Among these conversations, we identified more than 4,700 multi-turn samples with pairwise human preferences. Further analysis uncovers underexplored preferences of LLMs in fine-grained domains characterized by tasks, languages, and frameworks. To systematically examine code understanding and generation capabilities of frontier LLMs, we curated two benchmarks based on the collected data, namely BigCodeReward and AutoCodeArena. For BigCodeReward, we post-processed the 4,700 conversations and evaluated the consistency between reward models and human preferences. The evaluation shows that most LLMs have superior performance in judging coding preferences when the execution results are available. Inspired by these findings, we propose AutoCodeArena, an automatic Elo rating benchmark designed to assess the coding quality of LLMs without human involvement. We find that proprietary LLMs like GPT-5, Claude-Sonnet-4, and Claude-Opus-4 still lead in code generation performance among recent emerging models.
Figures & tables
Figure 1: BigCodeArena enables user evaluation based on execution outcomes beyond raw code.
Domain
Evaluation Platform
Real-time
Multi-turn
Verified
Codebase
Data
# Instances
Vision
3D Arena ( Ebert, 2025 )
✗
✗
✗
✓
✗
0
Vision
Genai Arena ( Jiang et al., 2024 )
✓
✗
✗
✓
✗
0
Vision
K-sort Arena ( Li et al., 2025b )
✓
✗
✗
✓
✗
0
Vision
Vision Arena ( Chou et al., 2025 )
✓
✗
✗
✓
✓
919
Speech
S2S-Arena ( Jiang et al., 2025 )
✗
✗
✗
✗
✗
0
Text
Chatbot Arena ( Chiang et al., 2024 )
✓
✓
✗
✓
✓
106K
Table 1: BigCodeArena is the first crowdsourced evaluation platform on code generation via execution, publicly releasing crowdsourced preference data. We collect 14K raw conversation sessions (analyzed in Appendix E ) and a high-quality subset of 4.7K multi-turn conversations with human preference (discussed in Section 4 ). Real-time : whether the evaluation requires real-time interactions; Multi-turn : whether the evaluation supports multi-turn conversations; Verified : whether the output of LLMs is verified by human experts and executable environments; Codebase : whether the codebase is publicly available; Data : whether the data is fully open. We note that WebDev Arena only focuses on the Next.js framework for web design.
Figure 2: Examples of programming topics, prompts, and execution results in BigCodeArena .
Figure 3: Elo ratings of models under three settings: All Data ( left ), (Execution) Environment Matched ( middle ), and Language Matched ( right ). These settings progressively control runtime and language factors by using all data, matching environments, and restricting to the same language.
Figure 4: Overall win rate heatmaps (percentage of all pairwise comparisons won) of each model in the sessions across languages ( left ) and execution environments ( right ). For each category, we only keep models that appear in at least 3 conversation sessions.
Models
Web
Game
Creative
Diagram
Scientific
Problem
Overall
–
+
–
+
–
+
–
+
–
+
–
+
–
+
Proprietary Models
Claude-Sonnet-4 ( Anthropic, 2025 )
59.1
62.4
58.1
66.2
64.5
67.4
55.0
71.8
52.7
59.9
52.0
57.9
56.7
62.3
Claude-3.7-Sonnet ( Anthropic, 2025 )
57.3
63.1
55.5
61.8
65.5
72.4
52.3
71.1
50.7
59.9
45.3
57.8
53.9
62.2
Claude-3.5-Sonnet ( Anthropic, 2025 )
61.2
63.7
58.5
63.7
69.5
69.7
54.4
63.1
56.6
62.7
57.3
64.2
59.7
64.1
GPT-4.1 ( OpenAI, 2025 )
57.4
60.3
59.2
65.0
64.7
64.2
55.0
67.8
52.5
58.4
45.2
54.5
54.7
60.0
Table 2: Accuracy results (%) for reward models across different task categories with and without execution outputs. “–” denotes evaluation without execution outputs, “+” denotes evaluation with execution outputs. While most models benefit from execution outputs, a minority of models exhibit drops, suggesting instability and insufficient robustness when incorporating multimodal signals. Best results in each category are highlighted in bold.
Figure 5: Overall performance of more recent LLMs on AutoCodeArena . We use GPT-4.1 as the baseline system and Claude-3.7-Sonnet as the judge. To avoid potential judgment bias toward self-generated responses, we exclude Claude-3.7-Sonnet from the rankings. GPT-4.1 is shown only to indicate the 50% win-rate baseline and is not compared against itself during evaluation.
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: User interface of BigCodeArena . The left and right panes display outputs from two different models (A and B) in a code execution sandbox, while the bottom section allows users to view the prompt, inspect code, and cast comparative judgments on which model performed better. The example prompt is “Write HTML with P5.js that simulates 25 particles in a vacuum space of a cylindrical container, bouncing within its boundaries. Use different colors for each ball and ensure they leave a trail showing their movement. Add a slow rotation of the container to give better view of what’s going on in the scene. Make sure to create proper collision detection and physic rules to ensure particles remain in the container. Add an external spherical container. Add a slow zoom in and zoom out effect to the whole scene.”
Figure 7: Overview of BigCodeArena Pipeline.
Figure 8: User interaction trajectories on initial functional testing and subsequent UI enhancement evaluation.
Figure 9: Distribution of conversation turns.
Figure 10: The distribution of character counts for conversations.
Figure 11: Distributions of languages ( left ) and programming frameworks ( right ) among the collected conversations.
Function
Description
click(x, y)
Perform a mouse click at screen coordinates (x,y)
keyboard(’enter’)
Sends an Enter/Return key input
keyboard(’x’)
Sends a character key input (e.g., typing “x”)
scroll(x)
Scroll up within x units on the interface
resize(x, y)
Adjusts the window size to width x and height y
Appendix
Table 3: Examples of the mouse and keyboard actions in BigCodeArena .
Environment
# Sessions
# Keyboard
# Click
Duration (s)
React
3,107
12.0
6.2
32.3
Core Web
1,699
18.8
6.3
59.6
PyGame
964
21.4
5.4
32.0
Vue
201
21.0
6.0
29.9
Streamlit
45
8.2
6.0
45.8
Gradio
14
13.7
4.2
26.1
Appendix
Table 4: User interaction statistics across different sandbox environments in BigCodeArena . Session counts cover all 6,030 recorded sessions; the duration breakdown in the text is computed on the 5,557 sessions retained after filtering.
Figure 12: Distribution of programming topics ( left ) and model win rates across topics ( right ).
Figure 13: Spearman correlations and ranking shifts between BigCodeArena and other benchmarks, including Copilot Arena ( Chi et al., 2025 ) , BigCodeBench ( Zhuo et al., 2024 ) , Chatbot Arena ( Chiang et al., 2024 ) , Chatbot Arena (Coding), and WebDev Arena ( lma, 2025b ) .
Table 6: Macro F1 scores (%) for reward models across different task categories with and without execution outputs. ”–” denotes evaluation without execution outputs, ”+” denotes evaluation with execution outputs. Best results in each category are highlighted in bold.
Figure 20: Web design performance of open and proprietary models on AutoCodeArena . We use GPT-4.1 as the baseline system and Claude-3.7-Sonnet as the judge. To avoid the potential judgement bias towards self-generated responses, we exclude Claude-3.7-Sonnet from the rankings.
Figure 21: Game development performance of open and proprietary models on AutoCodeArena . We use GPT-4.1 as the baseline system and Claude-3.7-Sonnet as the judge. To avoid the potential judgement bias towards self-generated responses, we exclude Claude-3.7-Sonnet from the rankings.
Figure 22: Creative coding performance of open and proprietary models on AutoCodeArena . We use GPT-4.1 as the baseline system and Claude-3.7-Sonnet as the judge. To avoid the potential judgement bias towards self-generated responses, we exclude Claude-3.7-Sonnet from the rankings.
Figure 23: Diagram creation performance of open and proprietary models on AutoCodeArena . We use GPT-4.1 as the baseline system and Claude-3.7-Sonnet as the judge. To avoid the potential judgement bias towards self-generated responses, we exclude Claude-3.7-Sonnet from the rankings.
Figure 24: Scientific computing performance of open and proprietary models on AutoCodeArena . We use GPT-4.1 as the baseline system and Claude-3.7-Sonnet as the judge. To avoid the potential judgement bias towards self-generated responses, we exclude Claude-3.7-Sonnet from the rankings.
Figure 25: Problem solving performance of open and proprietary models on AutoCodeArena . We use GPT-4.1 as the baseline system and Claude-3.7-Sonnet as the judge. To avoid the potential judgement bias towards self-generated responses, we exclude Claude-3.7-Sonnet from the rankings.
Instructions
Model Outputs
Write an HTML page with embedded JavaScript that creates an interactive human anatomy diagram. Each body part (e.g., heart, lungs, brain) should: Display its name, function, and related diseases on hover. Allow filtering by system (e.g., circulatory, respiratory, nervous). Include a search bar to find body parts by name or function.
Could you use React to create a Kanban board with Tailwind CSS and swimlanes (rows) using ’react-dnd’, where users can add and drag tasks within and across swimlanes?
Build a mobile app using Gradio for real-time weather forecasting. The app should allow users to view forecasts, receive alerts for severe weather, and track conditions in multiple locations. Provide an intuitive interface for navigation and personalization. Optimize the app for performance on mobile devices.
Using streamlit build an user friendly income tax calculator and explainer that determines an individual’s estimated annual income tax based on their gross annual salary and US state location. Allow users to enter their gross annual salary and select their US state location. Then do the tax calculation and output the estimated annual income tax based on the user’s input, taking into account federal and state income tax rates.
This paper introduces Code Bench, a benchmark capable of evaluating Large Language Models (LLMs) concise code generation abilities in 60 programming languages. Based on code golf, a recreational programming competition focused on minimal character or byte solutions, the benchmark provides a distinctive measure of LLMs ability to produce efficient, concise code. Unlike existing benchmarks limited by fixed problem sets and language coverage, CodeGolf Bench leverages the code.golf platform to provide new problems and live human performance baselines. Evaluation of nine LLMs on Python and C++ tasks demonstrates that reasoning models significantly outperform non-reasoning models, achieving best average percentile of 70.97%. This performance gap is particularly pronounced in C++, highlighting reasoning's importance for languages with strict syntax requirements. Non-reasoning models struggle more with efficiency optimization across both languages, with best percentiles significantly lower than reasoning counterparts. CodeGolf Bench offers a dynamic framework for evaluating LLM code generation capabilities against evolving human performance on code golf.
Industry now relies on LLMs to write a large share of production code. Although that code is mostly, but not always, correct, correctness alone does not determine whether it is fit for repeated and extensive use, which depends on what the code costs in time, memory and energy. Most code-generation benchmarks nonetheless evaluate correctness in isolation, and most performance-oriented benchmarks cover few languages and rarely measure energy consumption. To address this gap, we present PERFARENA, an open-source, reproducible arena that ranks LLMs by the performance of their generated code across multiple languages and competitions. PERFARENA executes the solutions of each LLM on a Unix-like host and reports correctness together with latency, peak memory, and energy consumption. Correctness oracles, performance oracles, profilers, and code-generation workflows are pluggable in PERFARENA, enabling the integration of new judging backends and measurement tools without changing the pipeline. We evaluate PERFARENA on two competitions: CLBGComp, 10 compute-intensive benchmarks from the Computer Language Benchmarks Game, and LeetCodeComp, 99 LeetCode problems whose accepted solutions differ measurably in latency. Currently, PERFARENA holds 9,455 correct and measured solutions from 13 state-of-the-art LLMs across 7 programming languages. Across both competitions, got-6-astra is the only LLM that no other LLM beats significantly in latency, energy consumption, or peak memory, solving 99.7% of LeetCodeComp and 100% of CLBGComp. All PERFARENA leaderboards and measured solutions are publicly available, providing a continuously updated reference that helps developers choose an LLM per language and metric, instead of relying only on correctness rates or pricing.
Rui Rua, Hailemariam Mersha, Joao Paulo Fernandes +1
New York University Abu Dhabi, Abu Dhabi, UAE. · Universidade do Algarve, Faro, Portugal.
Code generation models are typically compared using compact execution benchmarks and aggregate pass rates, but such summaries obscure how performance varies across programming languages, problem families, and failure modes. We present a large-scale, execution-grounded evaluation of 9 openly accessible LLMs specialized for coding on 2,707 free LeetCode problems across 12 programming languages. Our corpus contains 325,343 problem-model-language jobs, each linked to prompt metadata, extracted code, LeetCode execution outcomes, and static-analysis signals. The results show that current open models remain far from the human acceptance reference: the best model, Yi-Coder-9B-Chat, reaches 23.64% mean correctness, compared with a 57.2% human acceptance baseline. Rankings are also slice-dependent: Qwen2.5-Coder-14B-Instruct is strongest on hard problems and distinct-problem coverage, while Gemma-2-27B-IT achieves the highest all-language lint pass rate. Failure analysis shows that compile errors account for 63.25% of non-accepted best submissions, indicating that many failures occur before semantic correctness can be tested. Static quality further diverges from functional correctness. Together, these findings show that multilingual, artifact-preserving evaluation reveals tradeoffs hidden by single-language or single-metric leaderboards.
Sayed Erfan Arefin
Department of Computer Science University of Dayton Dayton, OH 45469, USA