PerfArena: Ranking LLMs by Code Performance
Organizations: New York University Abu Dhabi, Abu Dhabi, UAE. · Universidade do Algarve, Faro, Portugal.
Abstract
Industry now relies on LLMs to write a large share of production code. Although that code is mostly, but not always, correct, correctness alone does not determine whether it is fit for repeated and extensive use, which depends on what the code costs in time, memory and energy. Most code-generation benchmarks nonetheless evaluate correctness in isolation, and most performance-oriented benchmarks cover few languages and rarely measure energy consumption. To address this gap, we present PERFARENA, an open-source, reproducible arena that ranks LLMs by the performance of their generated code across multiple languages and competitions. PERFARENA executes the solutions of each LLM on a Unix-like host and reports correctness together with latency, peak memory, and energy consumption. Correctness oracles, performance oracles, profilers, and code-generation workflows are pluggable in PERFARENA, enabling the integration of new judging backends and measurement tools without changing the pipeline. We evaluate PERFARENA on two competitions: CLBGComp, 10 compute-intensive benchmarks from the Computer Language Benchmarks Game, and LeetCodeComp, 99 LeetCode problems whose accepted solutions differ measurably in latency. Currently, PERFARENA holds 9,455 correct and measured solutions from 13 state-of-the-art LLMs across 7 programming languages. Across both competitions, got-6-astra is the only LLM that no other LLM beats significantly in latency, energy consumption, or peak memory, solving 99.7% of LeetCodeComp and 100% of CLBGComp. All PERFARENA leaderboards and measured solutions are publicly available, providing a continuously updated reference that helps developers choose an LLM per language and metric, instead of relying only on correctness rates or pricing.
Figures & tables
| LLM | Provider | $ per 1M (in/out) | Temp. 0 |
|---|---|---|---|
| gpt-6-astra | OpenAI | 5 / 25 | ✗ |
| gpt-5.6-sol | OpenAI | 4 / 20 | ✗ |
| gpt-5.6-luna | OpenAI | 0.2 / 1.2 | ✗ |
| gpt-5.5 | OpenAI | 5 / 30 | ✗ |
| gpt-5.4-mini | OpenAI | 0.37 / 2.25 | ✗ |
| claude-opus-5-5 | Anthropic | 4 / 20 | ✗ |
| Latency | Energy | Peak memory | Latency ratio | |||||
|---|---|---|---|---|---|---|---|---|
| LLM | LeetCodeComp | CLBGComp | LeetCodeComp | CLBGComp | LeetCodeComp | CLBGComp | LeetCodeComp | CLBGComp |
| gpt-6-astra | 62 | 72 | 61 | 72 | 50 | 65 | 1.73 | 1.52 |
| claude-opus-5-5 | 58 | 63 | 58 | 61 | 51 | 43 | 1.97 | 1.76 |
| gpt-5.6-sol | 56 | 59 | 50 | 58 | 50 | 49 | 1.96 | 1.87 |
| claude-sonnet-5-5 | 54 | 44 | 54 | 52 | 50 | 47 | 2.10 | 2.29 |
| claude-fable-5-1 | 53 | 65 | 52 | 63 | 51 | 48 | 2.00 | 1.76 |
| (%) | Response tokens | Cost (USD) | Time (h) | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Per sol. | Sweep | Per sol. | Sweep | Gen. | Meas. | |||||||||
| LLM | LC | CLBG | LC | CLBG | LC | CLBG | LC | CLBG | LC | CLBG | LC | CLBG | LC | CLBG |
| gpt-5.6-luna | 93.1 | 74.3 | 626 | 3,837 | 404K | 200K | 0.0009 | 0.0047 | 0.56 | 0.25 | 1.6 | 0.7 | 21.3 | 5.8 |
| gpt-5.4-mini | 88.5 | 34.3 | 307 | 1,942 | 188K | 47K | 0.0009 | 0.0049 | 0.57 | 0.12 | 0.5 | 0.1 | 18.4 | 2.5 |
| deepseek-v4.1-flash | 97.4 | 64.3 | 2,323 | 42,035 | 1,568K | 1,892K | 0.0033 | 0.0559 | 2.25 | 2.51 | 3.1 | 2.2 | 19.0 | 9.2 |
| gemini-3.8-flash | 95.8 | 92.9 | 1,190 | 13,235 | 790K | 860K | 0.0049 | 0.0501 | 3.28 | 3.25 | 1.9 | 1.5 | 18.6 | 2.3 |
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| TS | Py | JS | Java | C# | PHP | C++ | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| LLM | LC | CLBG | LC | CLBG | LC | CLBG | LC | CLBG | LC | CLBG | LC | CLBG | LC | CLBG |
| gpt-6-astra | 62 | 77 | 62 | 61 | 63 | 66 | 62 | 69 | 61 | 69 | 55 | 71 | 67 | 85 |
| claude-opus-5-5 | 58 | 60 | 63 | 68 | 57 | 51 | 53 | 69 | 60 | 65 | 68 | 77 | 51 | 49 |
| gpt-5.6-sol | 55 | 64 | 62 | 55 | 50 | 51 | 57 | 56 | 59 | 60 | 56 | 51 | 51 | 72 |
| claude-sonnet-5-5 | 56 | 35 | 61 | 50 | 55 | 44 | 49 | 42 | 53 | 39 | 59 | 48 | 46 | 51 |
| claude-fable-5-1 | 59 | 59 | 47 | 80 | 57 | 53 | 52 | 77 | 46 | 77 | 56 | 56 | 52 | 50 |
| TS | Py | JS | Java | C# | PHP | C++ | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| LLM | LC | CLBG | LC | CLBG | LC | CLBG | LC | CLBG | LC | CLBG | LC | CLBG | LC | CLBG |
| gpt-6-astra | 61 | 76 | 64 | 61 | 62 | 66 | 62 | 70 | 64 | 70 | 50 | 73 | 67 | 83 |
| claude-opus-5-5 | 58 | 56 | 63 | 67 | 57 | 50 | 52 | 69 | 53 | 52 | 71 | 77 | 52 | 49 |
| gpt-5.6-sol | 53 | 66 | 49 | 48 | 50 | 51 | 57 | 57 | 45 | 58 | 46 | 52 | 51 | 72 |
| claude-sonnet-5-5 | 56 | 50 | 62 | 54 | 55 | 45 | 48 | 42 | 48 | 66 | 63 | 51 | 46 | 50 |
| claude-fable-5-1 | 57 | 57 | 49 | 79 | 56 | 54 | 51 | 77 | 48 | 57 | 49 | 56 | 51 | 52 |
| TS | Py | JS | Java | C# | PHP | C++ | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| LLM | LC | CLBG | LC | CLBG | LC | CLBG | LC | CLBG | LC | CLBG | LC | CLBG | LC | CLBG |
| gpt-6-astra | 52 | 77 | 50 | 58 | 50 | 71 | 51 | 60 | 50 | 61 | 50 | 68 | 48 | 56 |
| claude-opus-5-5 | 52 | 45 | 52 | 24 | 49 | 49 | 50 | 46 | 51 | 48 | 55 | 46 | 51 | 37 |
| gpt-5.6-sol | 48 | 50 | 50 | 54 | 48 | 57 | 51 | 46 | 50 | 55 | 50 | 39 | 52 | 42 |
| claude-sonnet-5-5 | 50 | 57 | 52 | 25 | 50 | 51 | 49 | 55 | 50 | 43 | 54 | 43 | 50 | 50 |
| claude-fable-5-1 | 53 | 53 | 50 | 40 | 50 | 46 | 50 | 50 | 50 | 47 | 54 | 44 | 48 | 54 |
| Category | Problems | Example problem |
|---|---|---|
| Array | 59 | frog-jump |
| String | 32 | slowest-key |
| Dynamic Programming | 24 | 2-keys-keyboard |
| Hash Table | 22 | word-ladder |
| Math | 18 | clumsy-factorial |
| Sorting | 12 | longest-string-chain |
| Benchmark | Workload | Meaning |
|---|---|---|
| binary-trees | 18 | tree depth |
| fannkuch-redux | 11 | permutation length |
| fasta | 12,000,000 | output size |
| mandelbrot | 3,504 | image side in pixels |
| n-body | 15,000,000 | simulation steps |
| pidigits | 10,000 | digits |
| Language | Metric | Duels | Same | Reversed | |
|---|---|---|---|---|---|
| Python | Latency | 780 | 86.4% | 1.4% | 0.78 (0.21) |
| Python | Energy | 780 | 85.5% | 0.4% | 0.82 (0.31) |
| C++ | Latency | 780 | 84.1% | 1.7% | 0.79 (0.38) |
| C++ | Energy | 780 | 80.8% | 0.9% | 0.71 (0.33) |
| Both | Latency | 1,560 | 85.3% | 1.5% | 0.79 (0.21) |
| Both | Energy | 1,560 | 83.1% | 0.6% | 0.79 (0.31) |
| LLM | Easy | Med. | Hard |
|---|---|---|---|
| gpt-6-astra | 100 | 100 | 99 |
| claude-opus-5-5 | 100 | 100 | 97 |
| gpt-5.6-sol | 98 | 100 | 99 |
| gpt-5.5 | 99 | 100 | 97 |
| grok-4.6 | 100 | 99 | 97 |
| claude-sonnet-5-5 | 99 | 100 | 96 |
| Verdict across languages | Pairs | Share |
|---|---|---|
| Accepted in every language | 1,148 | 89.2% |
| Mixed (accepted in some, rejected in others) | 133 | 10.3% |
| Rejected in every language | 6 | 0.5% |
| Total | 1,287 | 100% |
| Sig. pairs | Language | |||||||
| Metric | Duels | Cells | Problems | Leaders | Chance | |||
| LeetCodeComp | ||||||||
| Latency | 661 | 53/78 | 42/78 | 2 | ||||
| Energy | 661 | 52/78 | 39/78 | 3 | ||||
| Peak memory | 661 | 0/78 | 0/78 | 13 | ||||
| CLBGComp | ||||||||
Code Generation Evaluation
Code generation evaluation measures whether AI-produced programs are correct, reliable, secure, efficient, and faithful to their specifications.
See all 223 papers