BigCodeArena: Unveiling More Reliable Human Preferences in Code Generation via Execution
Authors: Terry Yue Zhuo, Xiaolong Jin, Hange Liu, Juyong Jiang, Tianyang Liu, Chen Gong, Bhupesh Bishnoi, Vaisakhi Mishra, Marek Suppa, Noah Ziems, Saiteja Utpala, Ming Xu, Guangyu Song, Kaixin Li, Yuhan Cao, Bo Liu, Zheng Liu, Sabina Abdurakhmanova, Wenhao Yu, Mengzhao Jia, Jihan Yao, Kenneth Hamilton, Kumar Shridhar, Minh Chien Vu, Dingmin Wang, Jiawei Liu, Zijian Wang, Qian Liu, Binyuan Hui, Meg Risdal, Ahsen Khaliq, Atin Sood, Zhenchang Xing, Wasi Uddin Ahmad, John Grundy, David Lo, Banghua Zhu, Xiaoning Du, Torsten Scholak, Leandro von Werra
Organizations: Monash University · CSIRO’s Data61 · Purdue University · Independent · HKUST (Guangzhou) · UCSD · UVA · CNRS, France · IBM · Cisco · Comenius University in Bratislava · University of Notre Dame · Uber · Tano Labs · NUS · Institute of Automation, CAS · Tencent AI Lab · University of Washington · Nevsky Collective · ETH Zurich · Detomo Inc · University of Oxford · UIUC · Google · NVIDIA · Singapore Management University · ServiceNow Research · Hugging Face
Crowdsourced model evaluation platforms, such as Chatbot Arena, enable real-time evaluation from human perspectives to assess the quality of model responses. In the coding domain, manually examining the quality of LLM-generated content is extremely challenging, as it requires understanding long chunks of raw code and deliberately simulating code execution. To this end, we introduce BigCodeArena, an open human evaluation platform for code generation backed by a comprehensive and on-the-fly execution environment. Built on top of Chatbot Arena, BigCodeArena enables the execution of LLM-generated code and allows humans to interact with the execution process and outcomes. We collected over 14,000 raw code-centric conversation sessions across 10 widely used LLMs, spanning 10 languages and 8 types of execution environments. Among these conversations, we identified more than 4,700 multi-turn samples with pairwise human preferences. Further analysis uncovers underexplored preferences of LLMs in fine-grained domains characterized by tasks, languages, and frameworks. To systematically examine code understanding and generation capabilities of frontier LLMs, we curated two benchmarks based on the collected data, namely BigCodeReward and AutoCodeArena. For BigCodeReward, we post-processed the 4,700 conversations and evaluated the consistency between reward models and human preferences. The evaluation shows that most LLMs have superior performance in judging coding preferences when the execution results are available. Inspired by these findings, we propose AutoCodeArena, an automatic Elo rating benchmark designed to assess the coding quality of LLMs without human involvement. We find that proprietary LLMs like GPT-5, Claude-Sonnet-4, and Claude-Opus-4 still lead in code generation performance among recent emerging models.
Figures & tables
Figure 1: BigCodeArena enables user evaluation based on execution outcomes beyond raw code.
Domain
Evaluation Platform
Real-time
Multi-turn
Verified
Codebase
Data
# Instances
Vision
3D Arena ( Ebert, 2025 )
✗
✗
✗
✓
✗
0
Vision
Genai Arena ( Jiang et al., 2024 )
✓
✗
✗
✓
✗
0
Vision
K-sort Arena ( Li et al., 2025b )
✓
✗
✗
✓
✗
0
Vision
Vision Arena ( Chou et al., 2025 )
✓
✗
✗
✓
✓
919
Speech
S2S-Arena ( Jiang et al., 2025 )
✗
✗
✗
✗
✗
0
Text
Chatbot Arena ( Chiang et al., 2024 )
✓
✓
✗
✓
✓
106K
Table 1: BigCodeArena is the first crowdsourced evaluation platform on code generation via execution, publicly releasing crowdsourced preference data. We collect 14K raw conversation sessions (analyzed in Appendix E ) and a high-quality subset of 4.7K multi-turn conversations with human preference (discussed in Section 4 ). Real-time : whether the evaluation requires real-time interactions; Multi-turn : whether the evaluation supports multi-turn conversations; Verified : whether the output of LLMs is verified by human experts and executable environments; Codebase : whether the codebase is publicly available; Data : whether the data is fully open. We note that WebDev Arena only focuses on the Next.js framework for web design.
Figure 2: Examples of programming topics, prompts, and execution results in BigCodeArena .
Figure 3: Elo ratings of models under three settings: All Data ( left ), (Execution) Environment Matched ( middle ), and Language Matched ( right ). These settings progressively control runtime and language factors by using all data, matching environments, and restricting to the same language.
Figure 4: Overall win rate heatmaps (percentage of all pairwise comparisons won) of each model in the sessions across languages ( left ) and execution environments ( right ). For each category, we only keep models that appear in at least 3 conversation sessions.
Models
Web
Game
Creative
Diagram
Scientific
Problem
Overall
–
+
–
+
–
+
–
+
–
+
–
+
–
+
Proprietary Models
Claude-Sonnet-4 ( Anthropic, 2025 )
59.1
62.4
58.1
66.2
64.5
67.4
55.0
71.8
52.7
59.9
52.0
57.9
56.7
62.3
Claude-3.7-Sonnet ( Anthropic, 2025 )
57.3
63.1
55.5
61.8
65.5
72.4
52.3
71.1
50.7
59.9
45.3
57.8
53.9
62.2
Claude-3.5-Sonnet ( Anthropic, 2025 )
61.2
63.7
58.5
63.7
69.5
69.7
54.4
63.1
56.6
62.7
57.3
64.2
59.7
64.1
GPT-4.1 ( OpenAI, 2025 )
57.4
60.3
59.2
65.0
64.7
64.2
55.0
67.8
52.5
58.4
45.2
54.5
54.7
60.0
Table 2: Accuracy results (%) for reward models across different task categories with and without execution outputs. “–” denotes evaluation without execution outputs, “+” denotes evaluation with execution outputs. While most models benefit from execution outputs, a minority of models exhibit drops, suggesting instability and insufficient robustness when incorporating multimodal signals. Best results in each category are highlighted in bold.
Figure 5: Overall performance of more recent LLMs on AutoCodeArena . We use GPT-4.1 as the baseline system and Claude-3.7-Sonnet as the judge. To avoid potential judgment bias toward self-generated responses, we exclude Claude-3.7-Sonnet from the rankings. GPT-4.1 is shown only to indicate the 50% win-rate baseline and is not compared against itself during evaluation.
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: User interface of BigCodeArena . The left and right panes display outputs from two different models (A and B) in a code execution sandbox, while the bottom section allows users to view the prompt, inspect code, and cast comparative judgments on which model performed better. The example prompt is “Write HTML with P5.js that simulates 25 particles in a vacuum space of a cylindrical container, bouncing within its boundaries. Use different colors for each ball and ensure they leave a trail showing their movement. Add a slow rotation of the container to give better view of what’s going on in the scene. Make sure to create proper collision detection and physic rules to ensure particles remain in the container. Add an external spherical container. Add a slow zoom in and zoom out effect to the whole scene.”
Figure 7: Overview of BigCodeArena Pipeline.
Figure 8: User interaction trajectories on initial functional testing and subsequent UI enhancement evaluation.
Figure 9: Distribution of conversation turns.
Figure 10: The distribution of character counts for conversations.
Figure 11: Distributions of languages ( left ) and programming frameworks ( right ) among the collected conversations.
Function
Description
click(x, y)
Perform a mouse click at screen coordinates (x,y)
keyboard(’enter’)
Sends an Enter/Return key input
keyboard(’x’)
Sends a character key input (e.g., typing “x”)
scroll(x)
Scroll up within x units on the interface
resize(x, y)
Adjusts the window size to width x and height y
Appendix
Table 3: Examples of the mouse and keyboard actions in BigCodeArena .
Environment
# Sessions
# Keyboard
# Click
Duration (s)
React
3,107
12.0
6.2
32.3
Core Web
1,699
18.8
6.3
59.6
PyGame
964
21.4
5.4
32.0
Vue
201
21.0
6.0
29.9
Streamlit
45
8.2
6.0
45.8
Gradio
14
13.7
4.2
26.1
Appendix
Table 4: User interaction statistics across different sandbox environments in BigCodeArena . Session counts cover all 6,030 recorded sessions; the duration breakdown in the text is computed on the 5,557 sessions retained after filtering.
Figure 12: Distribution of programming topics ( left ) and model win rates across topics ( right ).
Figure 13: Spearman correlations and ranking shifts between BigCodeArena and other benchmarks, including Copilot Arena ( Chi et al., 2025 ) , BigCodeBench ( Zhuo et al., 2024 ) , Chatbot Arena ( Chiang et al., 2024 ) , Chatbot Arena (Coding), and WebDev Arena ( lma, 2025b ) .
Table 6: Macro F1 scores (%) for reward models across different task categories with and without execution outputs. ”–” denotes evaluation without execution outputs, ”+” denotes evaluation with execution outputs. Best results in each category are highlighted in bold.
Figure 20: Web design performance of open and proprietary models on AutoCodeArena . We use GPT-4.1 as the baseline system and Claude-3.7-Sonnet as the judge. To avoid the potential judgement bias towards self-generated responses, we exclude Claude-3.7-Sonnet from the rankings.
Figure 21: Game development performance of open and proprietary models on AutoCodeArena . We use GPT-4.1 as the baseline system and Claude-3.7-Sonnet as the judge. To avoid the potential judgement bias towards self-generated responses, we exclude Claude-3.7-Sonnet from the rankings.
Figure 22: Creative coding performance of open and proprietary models on AutoCodeArena . We use GPT-4.1 as the baseline system and Claude-3.7-Sonnet as the judge. To avoid the potential judgement bias towards self-generated responses, we exclude Claude-3.7-Sonnet from the rankings.
Figure 23: Diagram creation performance of open and proprietary models on AutoCodeArena . We use GPT-4.1 as the baseline system and Claude-3.7-Sonnet as the judge. To avoid the potential judgement bias towards self-generated responses, we exclude Claude-3.7-Sonnet from the rankings.
Figure 24: Scientific computing performance of open and proprietary models on AutoCodeArena . We use GPT-4.1 as the baseline system and Claude-3.7-Sonnet as the judge. To avoid the potential judgement bias towards self-generated responses, we exclude Claude-3.7-Sonnet from the rankings.
Figure 25: Problem solving performance of open and proprietary models on AutoCodeArena . We use GPT-4.1 as the baseline system and Claude-3.7-Sonnet as the judge. To avoid the potential judgement bias towards self-generated responses, we exclude Claude-3.7-Sonnet from the rankings.
Instructions
Model Outputs
Write an HTML page with embedded JavaScript that creates an interactive human anatomy diagram. Each body part (e.g., heart, lungs, brain) should: Display its name, function, and related diseases on hover. Allow filtering by system (e.g., circulatory, respiratory, nervous). Include a search bar to find body parts by name or function.
Could you use React to create a Kanban board with Tailwind CSS and swimlanes (rows) using ’react-dnd’, where users can add and drag tasks within and across swimlanes?
Build a mobile app using Gradio for real-time weather forecasting. The app should allow users to view forecasts, receive alerts for severe weather, and track conditions in multiple locations. Provide an intuitive interface for navigation and personalization. Optimize the app for performance on mobile devices.
Using streamlit build an user friendly income tax calculator and explainer that determines an individual’s estimated annual income tax based on their gross annual salary and US state location. Allow users to enter their gross annual salary and select their US state location. Then do the tax calculation and output the estimated annual income tax based on the user’s input, taking into account federal and state income tax rates.