InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information

Jiaze Li; Aocheng Shen; Bing Liu; Boyu Zhang; Xiaoxuan Fan; Qiankun Zhang; Xianjun Deng

cs.SEAug 30, 2026

InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information

Authors: Jiaze Li, Aocheng Shen, Bing Liu, Boyu Zhang, Xiaoxuan Fan, Qiankun Zhang, Xianjun Deng

Organizations: School of Cyber Science and Engineering, Huazhong University of Science and Technology, Wuhan, China · Key Laboratory of Cyberspace Security, Ministry of Education, Zhengzhou, China · Hubei Key Laboratory of Distributed System Security, Wuhan, China.

Abstract

Competitive programming is increasingly being used to evaluate the algorithmic reasoning capabilities of large language models (LLMs). However, existing benchmarks primarily focus on full-information tasks where all problem inputs are provided upfront. This overlooks a critical dimension of algorithmic reasoning: the ability of generated programs to operate when key information is not revealed upfront. Interactive problems, a distinctive component of competitive programming, embody this challenge. These problems require programs to engage in multi-round interaction with an interactor (a judge program) under strict protocol constraints and limited query budgets, with new information revealed only in response to queries. To address this gap, we introduce InteractBench, a benchmark comprising 322 high-quality interactive problems curated from Codeforces, AtCoder, IOI, and ICPC. Each problem is packaged with executable local interactors, enabling fully offline evaluation. Unlike existing benchmarks, InteractBench assesses whether model-generated code can acquire information and track state dynamically. Our evaluation reveals a significant interaction gap: even the most advanced reasoning models achieve limited success on interactive problems. Beyond success rates, we propose a fine-grained failure taxonomy to diagnose the root causes of these deficiencies. Although algorithmic logic errors remain dominant, protocol violations and query-budget overruns are frequent. Code is available at https://github.com/kmsgk0/InteractBench.

Figures & tables

Figure 1: An illustration of the difference between batch-style and interactive problems. We use “Find the K-th Zero” as an example. In the batch-style setting (left), the program receives the complete input array at the start of execution and outputs the answer in a single shot. In the interactive setting (right), the array is hidden and the program must interact with an interactor (judge program) under a strict protocol and a limited query budget to gather information. The program can make several queries that ask the sum of all elements in a specific interval, and receive the answer from the interactor. Each query consumes part of the budget, and the interactor’s response depends on the query and the hidden input. The program reasons over multiple rounds before submitting the final answer.

Benchmark	# Problems	Interactive	Interface	Self-contained	Tags	Difficulty
HumanEval ( Chen et al., 2021 )	164	✗	–	✓	✗	✗
MBPP ( Austin et al., 2021 )	974	✗	–	✓	✗	✗
APPS ( Hendrycks et al., 2021 )	10,000	✗	–	✓	✗	✓
CodeContests ( Li et al., 2022 )	13,328	✗	–	✓	✓	✓
LiveCodeBench ( Jain et al., 2024 )	511	✗	–	✓	✗	✓
USACO ( Shi et al., 2024 )	307	✗	–	✓	✗	✓

Table 1: Comparison of representative code and competitive programming benchmarks. Columns report the number of problems; whether the benchmark has interactive support (with counts when available); interface type ( stdio or grader-linked ( CMS Development Team, 2017 ) ); and whether it provides self-contained evaluation, as well as tags and difficulty. Interactive counts may vary for continuously updated benchmarks.

Figure 2: An overview of the InteractBench construction pipeline. We select interactive tasks with multi-round protocols and explicit query budgets and construct offline evaluator artifacts (generator and local interactor) via a propose–validate–adjudicate loop validated on a per-task verification pool. Finally, where permitted, we post-hoc audit agreement by submitting a small sample of solutions to the official evaluation interface and comparing Accepted/Rejected outcomes with our offline harness.

Reasoning Configurations
Model	Easy		Medium		Hard		Overall
	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5
Gemini-3-Pro-Preview	0.803	0.961	0.512	0.716	0.367	0.597	0.554	0.752
GPT-5.2	0.742	0.921	0.530	0.690	0.340	0.492	0.542	0.705
DeepSeek-V3.2-Thinking	0.645	0.816	0.276	0.484	0.104	0.239	0.332	0.513
Gemini-2.5-Pro	0.587	0.763	0.259	0.452	0.092	0.224	0.305	0.480

Table 2: pass @1 and pass @5 (higher is better) stratified by difficulty on InteractBench. We sample

n=10

solutions per task. Configurations are grouped into reasoning and non-reasoning.

Reasoning Configurations
Model	Graph		Search		Greedy		Bit		Data Structures		Math		Game
	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5
Gemini-3-Pro-Preview	0.475	0.705	0.561	0.770	0.470	0.725	0.613	0.711	0.487	0.750	0.589	0.744	0.480	0.743
GPT-5.2	0.450	0.591	0.547	0.708	0.530	0.725	0.569	0.711	0.500	0.711	0.563	0.712	0.566	0.800
DeepSeek-V3.2-Thinking	0.295	0.511	0.340	0.504	0.345	0.580	0.338	0.467	0.295	0.513	0.314	0.472	0.377	0.657
Gemini-2.5-Pro	0.255	0.443	0.311	0.469	0.293	0.478	0.342	0.444	0.247	0.447	0.307	0.472	0.309	0.600

Table 3: pass @1 and pass @5 by category. Configurations are grouped into reasoning and non-reasoning.

Reasoning Configurations
Model	IDLE	PE	CE	TLE	MLE	RE	WA	QLE
Gemini-3-Pro-Preview	0.018	0.120	0.027	0.078	0.003	0.011	0.624	0.119
GPT-5.2	0.019	0.158	0.031	0.064	0.000	0.015	0.612	0.101
DeepSeek-V3.2-Thinking	0.024	0.132	0.063	0.045	0.002	0.012	0.647	0.074
Gemini-2.5-Pro	0.046	0.157	0.024	0.035	0.002	0.011	0.658	0.068
GPT-OSS-120B	0.401	0.105	0.013	0.013	0.002	0.010	0.424	0.031

Table 4: Failure composition among unsuccessful executions (higher means more common). We condition on failed runs and report the fraction of failures attributable to each primary failure type for each model (i.e., normalized by that model’s failed runs; Section 4.2 ). Configurations are grouped into reasoning and non-reasoning.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Field	Description
problem_id	Unique identifier of the task.
desc	Problem statement, including the stdin/stdout interaction protocol, termination rules, and constraints (e.g., query limits).
interactor_mode	Interactor mode metadata: non-adaptive, adaptive, or both.
cpu_time_limit_ms	CPU time limit (in milliseconds).
memory_limit_mb	Memory limit (in megabytes).
test_cases	Hidden test cases used for offline evaluation and consumed by the interactor.

Appendix

Table 5: Task card fields in InteractBench.

Source	Easy	Medium	Hard	Total
Codeforces	52	115	53	220
ICPC	16	33	12	61
AtCoder	8	7	2	17
IOI	9	7	8	24
Total	85	162	75	322

Appendix

Table 6: Task counts by difficulty, broken down by source.

Category	Easy	Medium	Hard	Total
Graph	23	45	20	88
Search	31	58	24	113
Greedy	16	42	11	69
Bit	12	22	11	45
Data Structures	14	51	11	76
Math	31	60	34	125

Appendix

Table 7: Task counts by category stratified by difficulty. A task may be assigned multiple categories.

Model	Mode	Size	Hyperlink
OpenAI (Closed-source)
GPT-5.2	Thinking	–	https://chatgpt.com
GPT-4.1	NonThinking	–	https://chatgpt.com
OpenAI (Open-source)
GPT-OSS-120B	Thinking	120B	https://huggingface.co/openai/gpt-oss-120b
GPT-OSS-20B	Thinking	20B	https://huggingface.co/openai/gpt-oss-20b

Appendix

Table 8: Model card for models used in InteractBench. Models are grouped by provider and access mode (closed-source or open-source). We list parameter counts when publicly specified and provide official hyperlinks.

Model	Setting	pass @1	WA	PE	QLE	IDLE	CE	RE	TLE	MLE
GPT-5.2	Batch	0.767	0.800	0.086	0.000	0.029	0.029	0.000	0.057	0.000
GPT-5.2	Interactive	0.511	0.629	0.164	0.076	0.005	0.038	0.008	0.079	0.000
DeepSeek-V3.2-Thinking	Batch	0.560	0.788	0.106	0.015	0.015	0.030	0.015	0.030	0.000
DeepSeek-V3.2-Thinking	Interactive	0.313	0.692	0.115	0.059	0.014	0.060	0.010	0.047	0.004
DeepSeek-V3.2	Batch	0.367	0.695	0.084	0.032	0.021	0.137	0.011	0.021	0.000
DeepSeek-V3.2	Interactive	0.147	0.653	0.178	0.033	0.028	0.041	0.017	0.046	0.003

Appendix

Table 9: Batch vs. interactive evaluation. pass @1 and failure composition on

150

matched problems. Failure-type fractions are computed over unsuccessful executions.

Model	pass @5	pass @5 (few-shot)	refine @5
GPT-5.2	0.705	0.714	0.677
DeepSeek-V3.2-Thinking	0.513	0.506	0.491
DeepSeek-V3.2	0.262	0.279	0.227
Qwen3-14B	0.121	0.143	0.071
Qwen3-14B-NonThinking	0.037	0.056	0.019

Appendix

Table 10: Comparison of mitigation strategies. Independent resampling ( pass @5), few-shot prompting, and iterative refinement ( refine @5).

Model	Round	CE	PE	IDLE	WA	QLE	TLE	MLE	RE	refine @k
GPT-5.2	r1	0.040	0.139	0.020	0.609	0.113	0.060	0.007	0.013	0.531
	r3	0.016	0.107	0.008	0.590	0.115	0.049	0.000	0.115	0.621
	r5	0.010	0.038	0.010	0.625	0.135	0.048	0.000	0.135	0.677
DeepSeek-V3.2-Thinking	r1	0.087	0.097	0.015	0.655	0.092	0.039	0.000	0.015	0.360
	r3	0.034	0.069	0.006	0.632	0.098	0.034	0.006	0.121	0.460
	r5	0.024	0.030	0.006	0.646	0.098	0.030	0.006	0.159	0.491

Appendix

Table 11: Per-round failure composition under iterative refinement. Error-type fractions among still-unsolved executions at rounds

1

3

, and

5

, together with the cumulative solve rate refine @k, across five representative model configurations.

Figure 3: Temporal contamination diagnostic via time splits. For each cut year

Y

, we compute the mean pass @1 over problems released in years

\geq Y

(the post set).

Model	Setting	C++	Python	Java	Go
GPT-5.2	no template	0.542	0.550	0.564	0.550
GPT-5.2	I/O template	0.557	0.544	0.544	0.540
DeepSeek-V3.2-Thinking	no template	0.332	0.309	0.312	0.242
DeepSeek-V3.2-Thinking	I/O template	0.319	0.289	0.285	0.285
DeepSeek-V3.2	no template	0.148	0.060	0.134	0.104
DeepSeek-V3.2	I/O template	0.144	0.131	0.107	0.124

Appendix

Table 12: Multi-language and I/O template evaluation. pass @1 on interactive tasks executable across C++/Python/Java/Go.

Category	Easy		Medium		Hard		Overall
	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5
Graph	0.713	0.870	0.467	0.711	0.220	0.500	0.475	0.705
Search	0.794	0.968	0.531	0.724	0.333	0.625	0.561	0.770
Greedy	0.738	0.938	0.390	0.667	0.382	0.636	0.470	0.725
Bit	0.883	1.000	0.645	0.727	0.255	0.364	0.613	0.711
Data Structures	0.843	1.000	0.416	0.686	0.364	0.727	0.487	0.750

Appendix

Table 15: Performance by category and difficulty: Gemini-3-Pro-Preview.

Category	Easy		Medium		Hard		Overall
	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5
Graph	0.757	0.870	0.409	0.556	0.190	0.350	0.450	0.591
Search	0.735	0.968	0.545	0.690	0.308	0.417	0.547	0.708
Greedy	0.613	0.812	0.529	0.738	0.418	0.545	0.530	0.725
Bit	0.850	1.000	0.545	0.727	0.309	0.364	0.569	0.711
Data Structures	0.743	0.929	0.475	0.706	0.309	0.455	0.500	0.711

Appendix

Table 16: Performance by category and difficulty: GPT-5.2.

Category	Easy		Medium		Hard		Overall
	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5
Graph	0.626	0.783	0.236	0.511	0.050	0.200	0.295	0.511
Search	0.639	0.806	0.286	0.466	0.083	0.208	0.340	0.504
Greedy	0.650	0.875	0.295	0.548	0.091	0.273	0.345	0.580
Bit	0.767	0.833	0.173	0.364	0.200	0.273	0.338	0.467
Data Structures	0.600	0.786	0.271	0.529	0.018	0.091	0.295	0.513

Appendix

Table 17: Performance by category and difficulty: DeepSeek-V3.2-Thinking.

Category	Easy		Medium		Hard		Overall
	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5
Graph	0.530	0.739	0.218	0.444	0.020	0.100	0.255	0.443
Search	0.587	0.742	0.266	0.448	0.067	0.167	0.312	0.469
Greedy	0.550	0.688	0.252	0.476	0.073	0.182	0.293	0.478
Bit	0.650	0.750	0.273	0.364	0.145	0.273	0.342	0.444
Data Structures	0.529	0.714	0.208	0.451	0.073	0.091	0.247	0.447

Appendix

Table 18: Performance by category and difficulty: Gemini-2.5-Pro.

Category	Easy		Medium		Hard		Overall
	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5
Graph	0.522	0.783	0.151	0.333	0.010	0.050	0.216	0.386
Search	0.535	0.677	0.197	0.345	0.100	0.208	0.269	0.407
Greedy	0.525	0.750	0.171	0.310	0.000	0.000	0.226	0.362
Bit	0.583	0.917	0.227	0.409	0.055	0.273	0.280	0.511
Data Structures	0.414	0.643	0.157	0.275	0.018	0.091	0.184	0.316

Appendix

Table 19: Performance by category and difficulty: GPT-OSS-120B.

Category	Easy		Medium		Hard		Overall
	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5
Graph	0.296	0.565	0.120	0.222	0.000	0.000	0.139	0.261
Search	0.271	0.484	0.148	0.259	0.008	0.042	0.152	0.274
Greedy	0.375	0.688	0.095	0.238	0.018	0.091	0.148	0.319
Bit	0.317	0.667	0.045	0.182	0.055	0.091	0.120	0.289
Data Structures	0.343	0.714	0.098	0.255	0.000	0.000	0.129	0.303

Appendix

Table 20: Performance by category and difficulty: Claude-Opus-4.5.

Category	Easy		Medium		Hard		Overall
	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5
Graph	0.287	0.478	0.076	0.156	0.000	0.000	0.114	0.205
Search	0.310	0.484	0.162	0.328	0.008	0.042	0.170	0.310
Greedy	0.312	0.688	0.157	0.357	0.018	0.091	0.171	0.391
Bit	0.400	0.667	0.082	0.182	0.018	0.091	0.151	0.289
Data Structures	0.186	0.429	0.141	0.333	0.000	0.000	0.129	0.303

Appendix

Table 21: Performance by category and difficulty: Gemini-2.5-Flash.

Category	Easy		Medium		Hard		Overall
	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5
Graph	0.330	0.609	0.067	0.178	0.000	0.000	0.120	0.250
Search	0.310	0.516	0.090	0.207	0.025	0.083	0.136	0.265
Greedy	0.425	0.750	0.076	0.238	0.000	0.000	0.145	0.319
Bit	0.233	0.500	0.055	0.136	0.036	0.182	0.098	0.244
Data Structures	0.200	0.571	0.047	0.157	0.000	0.000	0.068	0.211

Appendix

Table 22: Performance by category and difficulty: GPT-OSS-20B.

Category	Easy		Medium		Hard		Overall
	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5
Graph	0.174	0.348	0.107	0.178	0.010	0.050	0.102	0.193
Search	0.232	0.452	0.097	0.241	0.008	0.042	0.115	0.257
Greedy	0.200	0.625	0.167	0.286	0.036	0.182	0.154	0.348
Bit	0.217	0.500	0.055	0.091	0.000	0.000	0.084	0.178
Data Structures	0.171	0.429	0.094	0.196	0.000	0.000	0.095	0.211

Appendix

Table 23: Performance by category and difficulty: Qwen3-30B-A3B-Thinking.

Category	Easy		Medium		Hard		Overall
	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5
Graph	0.270	0.435	0.076	0.111	0.010	0.050	0.111	0.182
Search	0.245	0.419	0.062	0.138	0.008	0.042	0.101	0.195
Greedy	0.300	0.625	0.067	0.119	0.018	0.091	0.113	0.232
Bit	0.267	0.417	0.055	0.091	0.000	0.000	0.098	0.156
Data Structures	0.271	0.500	0.035	0.078	0.000	0.000	0.074	0.145

Appendix

Table 24: Performance by category and difficulty: Qwen3-32B.

Category	Easy		Medium		Hard		Overall
	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5
Graph	0.139	0.261	0.049	0.133	0.000	0.000	0.061	0.136
Search	0.187	0.290	0.041	0.086	0.000	0.000	0.073	0.124
Greedy	0.212	0.438	0.029	0.095	0.000	0.000	0.067	0.159
Bit	0.133	0.167	0.027	0.045	0.000	0.000	0.049	0.067
Data Structures	0.214	0.429	0.020	0.078	0.000	0.000	0.053	0.132

Appendix

Table 25: Performance by category and difficulty: Qwen3-14B.

Category	Easy		Medium		Hard		Overall
	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5
Graph	0.070	0.217	0.040	0.111	0.000	0.000	0.039	0.114
Search	0.181	0.323	0.045	0.086	0.000	0.000	0.073	0.133
Greedy	0.162	0.438	0.024	0.071	0.000	0.000	0.052	0.145
Bit	0.017	0.083	0.027	0.045	0.000	0.000	0.018	0.044
Data Structures	0.157	0.286	0.012	0.059	0.000	0.000	0.037	0.092

Appendix

Table 26: Performance by category and difficulty: DeepSeek-R1-Distill-Llama-70B.

Category	Easy		Medium		Hard		Overall
	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5
Graph	0.304	0.478	0.107	0.244	0.020	0.050	0.139	0.261
Search	0.323	0.548	0.138	0.224	0.025	0.083	0.165	0.283
Greedy	0.362	0.562	0.124	0.262	0.000	0.000	0.159	0.290
Bit	0.350	0.500	0.082	0.182	0.018	0.091	0.138	0.244
Data Structures	0.329	0.643	0.051	0.176	0.000	0.000	0.095	0.237

Appendix

Table 27: Performance by category and difficulty: DeepSeek-V3.2.

Category	Easy		Medium		Hard		Overall
	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5
Graph	0.130	0.391	0.044	0.111	0.000	0.000	0.057	0.159
Search	0.187	0.290	0.024	0.069	0.000	0.000	0.064	0.115
Greedy	0.212	0.438	0.048	0.119	0.000	0.000	0.078	0.174
Bit	0.133	0.250	0.000	0.000	0.000	0.000	0.036	0.067
Data Structures	0.129	0.286	0.043	0.118	0.000	0.000	0.053	0.132

Appendix

Table 28: Performance by category and difficulty: GPT-4.1.

Category	Easy		Medium		Hard		Overall
	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5
Graph	0.026	0.087	0.013	0.022	0.000	0.000	0.014	0.034
Search	0.039	0.129	0.003	0.017	0.000	0.000	0.012	0.044
Greedy	0.050	0.188	0.005	0.024	0.000	0.000	0.014	0.058
Bit	0.000	0.000	0.045	0.045	0.000	0.000	0.022	0.022
Data Structures	0.043	0.143	0.000	0.000	0.000	0.000	0.008	0.026

Appendix

Table 29: Performance by category and difficulty: Qwen3-32B-NonThinking.

Category	Easy		Medium		Hard		Overall
	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5	pass @1	pass @5
Graph	0.026	0.087	0.027	0.044	0.000	0.000	0.020	0.045
Search	0.032	0.129	0.007	0.034	0.000	0.000	0.012	0.053
Greedy	0.050	0.125	0.005	0.024	0.000	0.000	0.014	0.043
Bit	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000
Data Structures	0.043	0.143	0.004	0.020	0.000	0.000	0.011	0.039

Appendix

Table 30: Performance by category and difficulty: Qwen3-14B-NonThinking.

Code Generation Benchmarks

Code generation benchmarks measure how well AI models produce code across programming tasks, languages, and real-world settings.

See all 154 papers

LiveOIBench: Can Large Language Models Outperform Human Contestants in Informatics Olympiads?
Oct 10, 2025Kaijian Zou, Aaron Xiong, Yunxiang Zhang +6LLM Evaluation Competitive Programming
University of Michigan, Ann Arbor.
AlgoBench: Benchmarking Algorithmic Adaptation in Code Generation
Jun 30, 2026Xinyuan Song, Zekun Cai, Liang ZhaoSynthetic Benchmark Generation Code Generation Evaluation
Emory University, Atlanta, GA, USA · The University of Tokyo, Tokyo, Japan · LocationMind, Tokyo, Japan
Beyond Problem Solving: UOJ-Bench for Evaluating Code Generation, Hacking, and Repair in Competitive Programming
Jun 11, 2026Tingqiang Xu, Hangrui Zhou, Tianle Cai +2Automated Program Repair Competitive Programming
Institute for Interdisciplinary Information Sciences, Tsinghua University · Universal Online Judge · ByteDance Seed +1

Oct 10, 2025J/K move · Enter open · S save

LiveOIBench: Can Large Language Models Outperform Human Contestants in Informatics Olympiads?

Kaijian Zou, Aaron Xiong, Yunxiang Zhang, Frederick Zhang+5

University of Michigan, Ann Arbor.

LLM Evaluation Competitive Programming Code Generation Evaluation Code Generation Benchmarks

Open paper AskPDF arXiv

Competitive programming problems are increasingly used to evaluate the coding capabilities of large language models (LLMs) due to their complexity and ease of verification. Yet, current coding benchmarks face limitations such as a lack of exceptionally challenging problems, insufficient test case coverage, and reliance on online platform APIs that limit accessibility. To address these issues, we introduce LiveOIBench, a large-scale competitive programming benchmark featuring 403 expert-curated problems, averaging 60 official test cases each, drawn from 72 contests across 14 Informatics Olympiads held between 2023 and 2025. LiveOIBench has four key features: (1) expert-designed tasks with detailed subtask rubrics and extensive test cases; (2) direct comparison to elite human contestants; (3) continuous updates to reduce contamination risk; and (4) a fully offline, reproducible evaluation system. Benchmarking 34 popular general-purpose and reasoning LLMs, we find that GPT-5 achieves an 81.76th percentile, still falling short of top human contestants, while among the open-weight models, GPT-OSS-120B reaches only the 60th percentile. Reasoning-trace analyses indicate that robust reasoning models prioritize precise problem analysis over excessive exploration. Finally, analyses across release dates, task familiarity, and code similarity find minimal evidence of data contamination in our benchmark. Our leaderboard, code, and data are available at: https://liveoibench.github.io/.

Abstract

Code Generation Benchmarks

Explore similar work

LiveOIBench: Can Large Language Models Outperform Human Contestants in Informatics Olympiads?

AlgoBench: Benchmarking Algorithmic Adaptation in Code Generation

Beyond Problem Solving: UOJ-Bench for Evaluating Code Generation, Hacking, and Repair in Competitive Programming