InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information
Authors: Jiaze Li, Aocheng Shen, Bing Liu, Boyu Zhang, Xiaoxuan Fan, Qiankun Zhang, Xianjun Deng
Organizations: School of Cyber Science and Engineering, Huazhong University of Science and Technology, Wuhan, China · Key Laboratory of Cyberspace Security, Ministry of Education, Zhengzhou, China · Hubei Key Laboratory of Distributed System Security, Wuhan, China.
Competitive programming is increasingly being used to evaluate the algorithmic reasoning capabilities of large language models (LLMs). However, existing benchmarks primarily focus on full-information tasks where all problem inputs are provided upfront. This overlooks a critical dimension of algorithmic reasoning: the ability of generated programs to operate when key information is not revealed upfront. Interactive problems, a distinctive component of competitive programming, embody this challenge. These problems require programs to engage in multi-round interaction with an interactor (a judge program) under strict protocol constraints and limited query budgets, with new information revealed only in response to queries. To address this gap, we introduce InteractBench, a benchmark comprising 322 high-quality interactive problems curated from Codeforces, AtCoder, IOI, and ICPC. Each problem is packaged with executable local interactors, enabling fully offline evaluation. Unlike existing benchmarks, InteractBench assesses whether model-generated code can acquire information and track state dynamically. Our evaluation reveals a significant interaction gap: even the most advanced reasoning models achieve limited success on interactive problems. Beyond success rates, we propose a fine-grained failure taxonomy to diagnose the root causes of these deficiencies. Although algorithmic logic errors remain dominant, protocol violations and query-budget overruns are frequent. Code is available at https://github.com/kmsgk0/InteractBench.
Figures & tables
Figure 1: An illustration of the difference between batch-style and interactive problems. We use “Find the K-th Zero” as an example. In the batch-style setting (left), the program receives the complete input array at the start of execution and outputs the answer in a single shot. In the interactive setting (right), the array is hidden and the program must interact with an interactor (judge program) under a strict protocol and a limited query budget to gather information. The program can make several queries that ask the sum of all elements in a specific interval, and receive the answer from the interactor. Each query consumes part of the budget, and the interactor’s response depends on the query and the hidden input. The program reasons over multiple rounds before submitting the final answer.
Benchmark
# Problems
Interactive
Interface
Self-contained
Tags
Difficulty
HumanEval ( Chen et al., 2021 )
164
✗
–
✓
✗
✗
MBPP ( Austin et al., 2021 )
974
✗
–
✓
✗
✗
APPS ( Hendrycks et al., 2021 )
10,000
✗
–
✓
✗
✓
CodeContests ( Li et al., 2022 )
13,328
✗
–
✓
✓
✓
LiveCodeBench ( Jain et al., 2024 )
511
✗
–
✓
✗
✓
USACO ( Shi et al., 2024 )
307
✗
–
✓
✗
✓
Table 1: Comparison of representative code and competitive programming benchmarks. Columns report the number of problems; whether the benchmark has interactive support (with counts when available); interface type ( stdio or grader-linked ( CMS Development Team, 2017 ) ); and whether it provides self-contained evaluation, as well as tags and difficulty. Interactive counts may vary for continuously updated benchmarks.
Figure 2: An overview of the InteractBench construction pipeline. We select interactive tasks with multi-round protocols and explicit query budgets and construct offline evaluator artifacts (generator and local interactor) via a propose–validate–adjudicate loop validated on a per-task verification pool. Finally, where permitted, we post-hoc audit agreement by submitting a small sample of solutions to the official evaluation interface and comparing Accepted/Rejected outcomes with our offline harness.
Model
Easy
Medium
Hard
Overall
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
Reasoning Configurations
Gemini-3-Pro-Preview
0.803
0.961
0.512
0.716
0.367
0.597
0.554
0.752
GPT-5.2
0.742
0.921
0.530
0.690
0.340
0.492
0.542
0.705
DeepSeek-V3.2-Thinking
0.645
0.816
0.276
0.484
0.104
0.239
0.332
0.513
Gemini-2.5-Pro
0.587
0.763
0.259
0.452
0.092
0.224
0.305
0.480
Table 2: pass @1 and pass @5 (higher is better) stratified by difficulty on InteractBench. We sample n=10 solutions per task. Configurations are grouped into reasoning and non-reasoning.
Model
Graph
Search
Greedy
Bit
Data Structures
Math
Game
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
Reasoning Configurations
Gemini-3-Pro-Preview
0.475
0.705
0.561
0.770
0.470
0.725
0.613
0.711
0.487
0.750
0.589
0.744
0.480
0.743
GPT-5.2
0.450
0.591
0.547
0.708
0.530
0.725
0.569
0.711
0.500
0.711
0.563
0.712
0.566
0.800
DeepSeek-V3.2-Thinking
0.295
0.511
0.340
0.504
0.345
0.580
0.338
0.467
0.295
0.513
0.314
0.472
0.377
0.657
Gemini-2.5-Pro
0.255
0.443
0.311
0.469
0.293
0.478
0.342
0.444
0.247
0.447
0.307
0.472
0.309
0.600
Table 3: pass @1 and pass @5 by category. Configurations are grouped into reasoning and non-reasoning.
Model
IDLE
PE
CE
TLE
MLE
RE
WA
QLE
Reasoning Configurations
Gemini-3-Pro-Preview
0.018
0.120
0.027
0.078
0.003
0.011
0.624
0.119
GPT-5.2
0.019
0.158
0.031
0.064
0.000
0.015
0.612
0.101
DeepSeek-V3.2-Thinking
0.024
0.132
0.063
0.045
0.002
0.012
0.647
0.074
Gemini-2.5-Pro
0.046
0.157
0.024
0.035
0.002
0.011
0.658
0.068
GPT-OSS-120B
0.401
0.105
0.013
0.013
0.002
0.010
0.424
0.031
Table 4: Failure composition among unsuccessful executions (higher means more common). We condition on failed runs and report the fraction of failures attributable to each primary failure type for each model (i.e., normalized by that model’s failed runs; Section 4.2 ). Configurations are grouped into reasoning and non-reasoning.
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
Field
Description
problem_id
Unique identifier of the task.
desc
Problem statement, including the stdin/stdout interaction protocol, termination rules, and constraints (e.g., query limits).
interactor_mode
Interactor mode metadata: non-adaptive, adaptive, or both.
cpu_time_limit_ms
CPU time limit (in milliseconds).
memory_limit_mb
Memory limit (in megabytes).
test_cases
Hidden test cases used for offline evaluation and consumed by the interactor.
Appendix
Table 5: Task card fields in InteractBench.
Source
Easy
Medium
Hard
Total
Codeforces
52
115
53
220
ICPC
16
33
12
61
AtCoder
8
7
2
17
IOI
9
7
8
24
Total
85
162
75
322
Appendix
Table 6: Task counts by difficulty, broken down by source.
Category
Easy
Medium
Hard
Total
Graph
23
45
20
88
Search
31
58
24
113
Greedy
16
42
11
69
Bit
12
22
11
45
Data Structures
14
51
11
76
Math
31
60
34
125
Appendix
Table 7: Task counts by category stratified by difficulty. A task may be assigned multiple categories.
Model
Mode
Size
Hyperlink
OpenAI (Closed-source)
GPT-5.2
Thinking
–
https://chatgpt.com
GPT-4.1
NonThinking
–
https://chatgpt.com
OpenAI (Open-source)
GPT-OSS-120B
Thinking
120B
https://huggingface.co/openai/gpt-oss-120b
GPT-OSS-20B
Thinking
20B
https://huggingface.co/openai/gpt-oss-20b
Appendix
Table 8: Model card for models used in InteractBench. Models are grouped by provider and access mode (closed-source or open-source). We list parameter counts when publicly specified and provide official hyperlinks.
Model
Setting
pass @1
WA
PE
QLE
IDLE
CE
RE
TLE
MLE
GPT-5.2
Batch
0.767
0.800
0.086
0.000
0.029
0.029
0.000
0.057
0.000
Interactive
0.511
0.629
0.164
0.076
0.005
0.038
0.008
0.079
0.000
DeepSeek-V3.2-Thinking
Batch
0.560
0.788
0.106
0.015
0.015
0.030
0.015
0.030
0.000
Interactive
0.313
0.692
0.115
0.059
0.014
0.060
0.010
0.047
0.004
DeepSeek-V3.2
Batch
0.367
0.695
0.084
0.032
0.021
0.137
0.011
0.021
0.000
Interactive
0.147
0.653
0.178
0.033
0.028
0.041
0.017
0.046
0.003
Appendix
Table 9: Batch vs. interactive evaluation. pass @1 and failure composition on 150 matched problems. Failure-type fractions are computed over unsuccessful executions.
Model
pass @5
pass @5 (few-shot)
refine @5
GPT-5.2
0.705
0.714
0.677
DeepSeek-V3.2-Thinking
0.513
0.506
0.491
DeepSeek-V3.2
0.262
0.279
0.227
Qwen3-14B
0.121
0.143
0.071
Qwen3-14B-NonThinking
0.037
0.056
0.019
Appendix
Table 10: Comparison of mitigation strategies. Independent resampling ( pass @5), few-shot prompting, and iterative refinement ( refine @5).
Model
Round
CE
PE
IDLE
WA
QLE
TLE
MLE
RE
refine @k
GPT-5.2
r1
0.040
0.139
0.020
0.609
0.113
0.060
0.007
0.013
0.531
r3
0.016
0.107
0.008
0.590
0.115
0.049
0.000
0.115
0.621
r5
0.010
0.038
0.010
0.625
0.135
0.048
0.000
0.135
0.677
DeepSeek-V3.2-Thinking
r1
0.087
0.097
0.015
0.655
0.092
0.039
0.000
0.015
0.360
r3
0.034
0.069
0.006
0.632
0.098
0.034
0.006
0.121
0.460
r5
0.024
0.030
0.006
0.646
0.098
0.030
0.006
0.159
0.491
Appendix
Table 11: Per-round failure composition under iterative refinement. Error-type fractions among still-unsolved executions at rounds 1 , 3 , and 5 , together with the cumulative solve rate refine @k, across five representative model configurations.
Figure 3: Temporal contamination diagnostic via time splits. For each cut year Y , we compute the mean pass @1 over problems released in years ≥Y (the post set).
Model
Setting
C++
Python
Java
Go
GPT-5.2
no template
0.542
0.550
0.564
0.550
I/O template
0.557
0.544
0.544
0.540
DeepSeek-V3.2-Thinking
no template
0.332
0.309
0.312
0.242
I/O template
0.319
0.289
0.285
0.285
DeepSeek-V3.2
no template
0.148
0.060
0.134
0.104
I/O template
0.144
0.131
0.107
0.124
Appendix
Table 12: Multi-language and I/O template evaluation. pass @1 on interactive tasks executable across C++/Python/Java/Go.
Category
Easy
Medium
Hard
Overall
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
Graph
0.713
0.870
0.467
0.711
0.220
0.500
0.475
0.705
Search
0.794
0.968
0.531
0.724
0.333
0.625
0.561
0.770
Greedy
0.738
0.938
0.390
0.667
0.382
0.636
0.470
0.725
Bit
0.883
1.000
0.645
0.727
0.255
0.364
0.613
0.711
Data Structures
0.843
1.000
0.416
0.686
0.364
0.727
0.487
0.750
Appendix
Table 15: Performance by category and difficulty: Gemini-3-Pro-Preview.
Category
Easy
Medium
Hard
Overall
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
Graph
0.757
0.870
0.409
0.556
0.190
0.350
0.450
0.591
Search
0.735
0.968
0.545
0.690
0.308
0.417
0.547
0.708
Greedy
0.613
0.812
0.529
0.738
0.418
0.545
0.530
0.725
Bit
0.850
1.000
0.545
0.727
0.309
0.364
0.569
0.711
Data Structures
0.743
0.929
0.475
0.706
0.309
0.455
0.500
0.711
Appendix
Table 16: Performance by category and difficulty: GPT-5.2.
Category
Easy
Medium
Hard
Overall
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
Graph
0.626
0.783
0.236
0.511
0.050
0.200
0.295
0.511
Search
0.639
0.806
0.286
0.466
0.083
0.208
0.340
0.504
Greedy
0.650
0.875
0.295
0.548
0.091
0.273
0.345
0.580
Bit
0.767
0.833
0.173
0.364
0.200
0.273
0.338
0.467
Data Structures
0.600
0.786
0.271
0.529
0.018
0.091
0.295
0.513
Appendix
Table 17: Performance by category and difficulty: DeepSeek-V3.2-Thinking.
Category
Easy
Medium
Hard
Overall
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
Graph
0.530
0.739
0.218
0.444
0.020
0.100
0.255
0.443
Search
0.587
0.742
0.266
0.448
0.067
0.167
0.312
0.469
Greedy
0.550
0.688
0.252
0.476
0.073
0.182
0.293
0.478
Bit
0.650
0.750
0.273
0.364
0.145
0.273
0.342
0.444
Data Structures
0.529
0.714
0.208
0.451
0.073
0.091
0.247
0.447
Appendix
Table 18: Performance by category and difficulty: Gemini-2.5-Pro.
Category
Easy
Medium
Hard
Overall
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
Graph
0.522
0.783
0.151
0.333
0.010
0.050
0.216
0.386
Search
0.535
0.677
0.197
0.345
0.100
0.208
0.269
0.407
Greedy
0.525
0.750
0.171
0.310
0.000
0.000
0.226
0.362
Bit
0.583
0.917
0.227
0.409
0.055
0.273
0.280
0.511
Data Structures
0.414
0.643
0.157
0.275
0.018
0.091
0.184
0.316
Appendix
Table 19: Performance by category and difficulty: GPT-OSS-120B.
Category
Easy
Medium
Hard
Overall
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
Graph
0.296
0.565
0.120
0.222
0.000
0.000
0.139
0.261
Search
0.271
0.484
0.148
0.259
0.008
0.042
0.152
0.274
Greedy
0.375
0.688
0.095
0.238
0.018
0.091
0.148
0.319
Bit
0.317
0.667
0.045
0.182
0.055
0.091
0.120
0.289
Data Structures
0.343
0.714
0.098
0.255
0.000
0.000
0.129
0.303
Appendix
Table 20: Performance by category and difficulty: Claude-Opus-4.5.
Category
Easy
Medium
Hard
Overall
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
Graph
0.287
0.478
0.076
0.156
0.000
0.000
0.114
0.205
Search
0.310
0.484
0.162
0.328
0.008
0.042
0.170
0.310
Greedy
0.312
0.688
0.157
0.357
0.018
0.091
0.171
0.391
Bit
0.400
0.667
0.082
0.182
0.018
0.091
0.151
0.289
Data Structures
0.186
0.429
0.141
0.333
0.000
0.000
0.129
0.303
Appendix
Table 21: Performance by category and difficulty: Gemini-2.5-Flash.
Category
Easy
Medium
Hard
Overall
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
Graph
0.330
0.609
0.067
0.178
0.000
0.000
0.120
0.250
Search
0.310
0.516
0.090
0.207
0.025
0.083
0.136
0.265
Greedy
0.425
0.750
0.076
0.238
0.000
0.000
0.145
0.319
Bit
0.233
0.500
0.055
0.136
0.036
0.182
0.098
0.244
Data Structures
0.200
0.571
0.047
0.157
0.000
0.000
0.068
0.211
Appendix
Table 22: Performance by category and difficulty: GPT-OSS-20B.
Category
Easy
Medium
Hard
Overall
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
Graph
0.174
0.348
0.107
0.178
0.010
0.050
0.102
0.193
Search
0.232
0.452
0.097
0.241
0.008
0.042
0.115
0.257
Greedy
0.200
0.625
0.167
0.286
0.036
0.182
0.154
0.348
Bit
0.217
0.500
0.055
0.091
0.000
0.000
0.084
0.178
Data Structures
0.171
0.429
0.094
0.196
0.000
0.000
0.095
0.211
Appendix
Table 23: Performance by category and difficulty: Qwen3-30B-A3B-Thinking.
Category
Easy
Medium
Hard
Overall
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
Graph
0.270
0.435
0.076
0.111
0.010
0.050
0.111
0.182
Search
0.245
0.419
0.062
0.138
0.008
0.042
0.101
0.195
Greedy
0.300
0.625
0.067
0.119
0.018
0.091
0.113
0.232
Bit
0.267
0.417
0.055
0.091
0.000
0.000
0.098
0.156
Data Structures
0.271
0.500
0.035
0.078
0.000
0.000
0.074
0.145
Appendix
Table 24: Performance by category and difficulty: Qwen3-32B.
Category
Easy
Medium
Hard
Overall
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
Graph
0.139
0.261
0.049
0.133
0.000
0.000
0.061
0.136
Search
0.187
0.290
0.041
0.086
0.000
0.000
0.073
0.124
Greedy
0.212
0.438
0.029
0.095
0.000
0.000
0.067
0.159
Bit
0.133
0.167
0.027
0.045
0.000
0.000
0.049
0.067
Data Structures
0.214
0.429
0.020
0.078
0.000
0.000
0.053
0.132
Appendix
Table 25: Performance by category and difficulty: Qwen3-14B.
Category
Easy
Medium
Hard
Overall
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
Graph
0.070
0.217
0.040
0.111
0.000
0.000
0.039
0.114
Search
0.181
0.323
0.045
0.086
0.000
0.000
0.073
0.133
Greedy
0.162
0.438
0.024
0.071
0.000
0.000
0.052
0.145
Bit
0.017
0.083
0.027
0.045
0.000
0.000
0.018
0.044
Data Structures
0.157
0.286
0.012
0.059
0.000
0.000
0.037
0.092
Appendix
Table 26: Performance by category and difficulty: DeepSeek-R1-Distill-Llama-70B.
Category
Easy
Medium
Hard
Overall
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
Graph
0.304
0.478
0.107
0.244
0.020
0.050
0.139
0.261
Search
0.323
0.548
0.138
0.224
0.025
0.083
0.165
0.283
Greedy
0.362
0.562
0.124
0.262
0.000
0.000
0.159
0.290
Bit
0.350
0.500
0.082
0.182
0.018
0.091
0.138
0.244
Data Structures
0.329
0.643
0.051
0.176
0.000
0.000
0.095
0.237
Appendix
Table 27: Performance by category and difficulty: DeepSeek-V3.2.
Category
Easy
Medium
Hard
Overall
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
Graph
0.130
0.391
0.044
0.111
0.000
0.000
0.057
0.159
Search
0.187
0.290
0.024
0.069
0.000
0.000
0.064
0.115
Greedy
0.212
0.438
0.048
0.119
0.000
0.000
0.078
0.174
Bit
0.133
0.250
0.000
0.000
0.000
0.000
0.036
0.067
Data Structures
0.129
0.286
0.043
0.118
0.000
0.000
0.053
0.132
Appendix
Table 28: Performance by category and difficulty: GPT-4.1.
Category
Easy
Medium
Hard
Overall
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
Graph
0.026
0.087
0.013
0.022
0.000
0.000
0.014
0.034
Search
0.039
0.129
0.003
0.017
0.000
0.000
0.012
0.044
Greedy
0.050
0.188
0.005
0.024
0.000
0.000
0.014
0.058
Bit
0.000
0.000
0.045
0.045
0.000
0.000
0.022
0.022
Data Structures
0.043
0.143
0.000
0.000
0.000
0.000
0.008
0.026
Appendix
Table 29: Performance by category and difficulty: Qwen3-32B-NonThinking.
Category
Easy
Medium
Hard
Overall
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
pass @1
pass @5
Graph
0.026
0.087
0.027
0.044
0.000
0.000
0.020
0.045
Search
0.032
0.129
0.007
0.034
0.000
0.000
0.012
0.053
Greedy
0.050
0.125
0.005
0.024
0.000
0.000
0.014
0.043
Bit
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
Data Structures
0.043
0.143
0.004
0.020
0.000
0.000
0.011
0.039
Appendix
Table 30: Performance by category and difficulty: Qwen3-14B-NonThinking.
Competitive programming problems are increasingly used to evaluate the coding capabilities of large language models (LLMs) due to their complexity and ease of verification. Yet, current coding benchmarks face limitations such as a lack of exceptionally challenging problems, insufficient test case coverage, and reliance on online platform APIs that limit accessibility. To address these issues, we introduce LiveOIBench, a large-scale competitive programming benchmark featuring 403 expert-curated problems, averaging 60 official test cases each, drawn from 72 contests across 14 Informatics Olympiads held between 2023 and 2025. LiveOIBench has four key features: (1) expert-designed tasks with detailed subtask rubrics and extensive test cases; (2) direct comparison to elite human contestants; (3) continuous updates to reduce contamination risk; and (4) a fully offline, reproducible evaluation system. Benchmarking 34 popular general-purpose and reasoning LLMs, we find that GPT-5 achieves an 81.76th percentile, still falling short of top human contestants, while among the open-weight models, GPT-OSS-120B reaches only the 60th percentile. Reasoning-trace analyses indicate that robust reasoning models prioritize precise problem analysis over excessive exploration. Finally, analyses across release dates, task familiarity, and code similarity find minimal evidence of data contamination in our benchmark. Our leaderboard, code, and data are available at: https://liveoibench.github.io/.
High pass rates on established programming benchmarks such as HumanEval and LiveCodeBench do not always show whether a model can reason about algorithms. Many fixed benchmarks eventually become part of the public training ecosystem through released problem statements, editorials, and generated solutions, allowing later models to improve partly by exposure rather than by stronger algorithmic ability. We introduce ALGOBENCH, a framework that automatically builds novel algorithmic problems from known competitive-programming problems through structured constraint-shifting transformations. Each accepted ALGOBENCH variant is traceable to a source problem, but must make the original reference algorithm fail. Beyond pass@k, we introduce complexity-aware metrics -- including OPTT, OPTS, TRAPRATE, GAPT, and CONSENS -- to test whether a solution is not only functionally correct but also asymptotically suitable for the generated problem. Experiments across multiple LLMs and prompting strategies show that performance drops sharply on ALGOBENCH variants, retrieval can increase reuse of the old algorithm, and many correct-looking solutions fail to meet the required complexity. Error analysis shows that failures are mainly algorithmic rather than implementation-level, suggesting that ALGOBENCH evaluates adaptation beyond functional correctness.
Xinyuan Song, Zekun Cai, Liang Zhao
Emory University, Atlanta, GA, USA · The University of Tokyo, Tokyo, Japan · LocationMind, Tokyo, Japan
Despite strong performance in competitive programming, the role of Large Language Models (LLMs) in supporting human learning in the same setting remains largely unexplored. In this work, we introduce UOJ-Bench, a benchmark designed to evaluate not only the problem-solving ability of LLMs, but also their ability to identify errors in human-written code -- a crucial educational activity traditionally supported by running test cases over online judge systems. UOJ-Bench consists of three distinct tasks: code generation, code hacking, and code repair, all constructed from real-world code submissions on the Universal Online Judge (UOJ) and evaluated through UOJ's native judging infrastructure. Our results show that under one-shot evaluation, even the strongest models fail to identify errors in more than 50% of a set of submissions that have been found to be incorrect by UOJ users. While test-time scaling improves success rates to above 90%, the substantial computational costs incurred from model inference limit its practicality for large-scale deployment. Despite these limitations, we find that the best-performing models under test-time scaling can uncover errors in over 5% of full-score submissions across roughly 30 problems, suggesting that frontier LLMs can already provide complementary signals beyond standard judging systems.
Tingqiang Xu, Hangrui Zhou, Tianle Cai +2
Institute for Interdisciplinary Information Sciences, Tsinghua University · Universal Online Judge · ByteDance Seed +1