Credit-Budgeted ICPC-Style Coding: When Agents Must Pay for Every Decision

Lingfeng Zhou; Junhao Shi; Jin Gao; Dequan Wang

cs.AIApr 11, 2026

Credit-Budgeted ICPC-Style Coding: When Agents Must Pay for Every Decision

Authors: Lingfeng Zhou, Junhao Shi, Jin Gao, Dequan Wang

Organizations: Shanghai Jiao Tong University · Shanghai Innovation Institute

Abstract

Current evaluations of autonomous coding agents assume an unrealistic, infinite-resource environment. However, real-world software engineering is a resource-bound competition. As we scale toward large agent swarms, ignoring compute and time costs risks catastrophic budget exhaustion. To shift the focus from isolated accuracy to cost-aware problem-solving, we introduce USACOArena, an interactive ACM-ICPC-style arena driven by a strict "credit" economy. Every generated token, local test, and elapsed second depletes a fixed budget, forcing agents to make strategic trade-offs. Our comprehensive profiling reveals that frontier single agents and swarms currently fail to optimally balance accuracy with these constraints, exhibiting divergent, path-dependent behaviors. Ultimately, USACOArena provides an essential dynamic training ground for developing highly efficient, resource-aware agent architectures.

Figures & tables

Figure 1: From Temporal Vacuum to Cost-Aware Autonomy. Without economic feedback, traditional swarms waste time and compute, inevitably crashing into real-world Budget Limits (top). USACOArena solves this by translating absolute time, tokens, and testing overhead into a Unified Credit Feedback signal (bottom-left funnel). By forcing swarms to "pay" for every decision, it induces efficient, cost-aware coordination for successful deployment (bottom right).

Figure 2: The Unified Credit Economy of USACOArena. Our environment evaluates agents on cost-aware decision-making rather than static correctness. The agents face distinct strategic trade-offs: aggressively parallelizing to minimize absolute delivery time at the expense of high inference costs (top) , conserving compute while burning valuable time credit (middle) , or spending resources to acquire strategic hints to overcome bottlenecks (bottom). Every action—along with the continuous passage of wall-clock time—drains a shared Budget Limit. To maximize the Final Score , the swarm must optimally balance speed, compute, and reliability without going bankrupt.

Figure 3: Average agent scores and consumed credit across the four contests of the 2024–2025 USACO season. Each subplot shows the results for a single contest, with agents sorted by rank. Blue bars represent the average score (left axis), while the orange line indicates the average consumed credit (right axis, log scale). Error bars and the shaded area denote the standard deviation over five independent runs; for clarity, only agents that achieved a non-zero average score are shown. The results reveal a stable and significant performance hierarchy: across all four contests of varying difficulty, Gemini-2.5-pro and GPT-5-Codex consistently rank first and second, respectively.

Figure 4

Figure 5: Emergent behavioral diversity and strategic divergence in self-play. (a) Final scores and credit consumed across nine competitions between identical gemini-2.5-pro agents, revealing a wide spectrum of outcomes with no trivial correlation between cost and performance. (b) A trajectory analysis of a single match provides a granular explanation, showing how different strategic paths lead to a decisive win-loss result. This demonstrated diversity, resulting from complex path-dependent decisions, validates USACOArena’s suitability as a rich training environment.

Figure 6: Performance and resource profiling of Codex agent swarms. (a) Absolute Time vs. Cost: Illustrates the trade-off between delivery time, token expenditure, and final score. The Speedy Spendthrift minimizes time but suffers a catastrophic performance drop (Score: 3), whereas the Frugal Perfectionist achieves a high score (Score: 8) at the cost of impractical delays (300 mins). (b) Token Breakdown: Reveals that blindly maximizing parallel workers (Speedy) triggers a massive explosion in Communication Overhead (orange hatched area). This rapidly depletes the budget without improving coding accuracy, empirically highlighting the critical need for dynamic, cost-aware resource management in future multi-agent systems.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Main Result	Low Credit	High Credit	Free Test	High Test	Free Hint	High Hint	Flat Score	Exp. Score
	(Table 10 config)	(10M)	(40M)	($0)	($1k)	($0)	(100 $\times$ )
Gemini-2.5-pro	13.2	8.3	13.0	10.7	9.7	12.0	13.0	5.0	19.7
GPT-5-Codex	12.0	4.3	4.3	5.7	4.3	3.7	3.7	3.3	16.3
GLM-4.5	3.2	1.0	0.7	1.0	1.0	1.0	1.7	1.3	1.0
Qwen3-235B	2.8	1.7	1.7	3.3	1.7	1.3	2.7	1.3	0.0
Claude-4-Sonnet	1.0	2.0	3.3	2.7	2.0	3.0	2.0	2.0	5.3

Appendix

Table 2: The Mega-Ablation Matrix on USACO 2025 February Contest. The “Main Result” column reflects the standard configuration used in Section 4.2 . “Exp. Score” uses exponential scoring weights {1, 10, 100, 1000} instead of the default {1, 2, 5, 10}.

Problem ID	Level	Result	Test Cases	Cons. Credit	LLM Calls	Sub.
1526	platinum	WA	1/20	10167024	48	37
1525	platinum	WA	1/15	10306989	45	36
1524	platinum	TLE	1/16	10018138	63	49
1523	gold	TLE	1/20	10151386	57	47
1522	gold	AC	15/15	1446214	6	1
1521	gold	AC	25/25	2818862	18	8

Appendix

Table 3: Difficulty baselining results for the 48 problems of the USACO 2024-2025 season, using Gemini-2.5-pro. This data provides a grounded reference for expected problem difficulty in our main experiments.

Model	Result	Cons. Credit	LLM Calls	Submissions	Qualified
Gemini-2.5-Pro	AC	373,173	3	2	Yes
GPT-5-Codex	AC	26,986	3	1	Yes
Claude-4-Sonnet	AC	57,405	3	2	Yes
DeepSeek-V3.1	AC	69,514	19	7	Yes
DeepSeek-V3	AC	536,880	130	47	Yes
Kimi-K2-0905	AC	449,065	35	17	Yes

Appendix

Table 4: Representative qualification results for the USACO 2024 December Contest. Agents are required to solve the easiest Bronze problem 1445. Cons. Credit means Consumed Credit.

Model	Result	Cons. Credit	LLM Calls	Submissions	Qualified
Gemini-2.5-Pro	AC	472,262	4	2	Yes
GPT-5-Codex	AC	177,127	9	1	Yes
Claude-4-Sonnet	WA	10,035,455	168	90	No
	TLE	10,037,109	508	232
	TLE	10,013,656	227	127
DeepSeek-V3.1	AC	497,429	121	46	Yes

Appendix

Table 5: Representative qualification results for the USACO 2025 January Contest. To qualify, agents were required to solve at least one of the three available Bronze problems (1467, 1468, 1469).

Model	Result	Cons. Credit	LLM Calls	Submissions	Qualified
Gemini-2.5-Pro	AC	104,275	2	1	Yes
GPT-5-Codex	AC	19,739	3	1	Yes
Claude-4-Sonnet	AC	57,486	3	2	Yes
DeepSeek-V3.1	AC	14,048	5	1	Yes
DeepSeek-V3	AC	6,918	4	1	Yes
Kimi-K2-0905	AC	213,684	18	9	Yes

Appendix

Table 6: Representative qualification results for the USACO 2025 February Contest. Agents are required to solve the easiest Bronze problem 1491. Cons. Credit means Consumed Credit.

Model	Result	Cons. Credit	LLM Calls	Submissions	Qualified
Gemini-2.5-Pro	AC	137433	3	1	Yes
GPT-5-Codex	AC	37583	4	1	Yes
Claude-4-sonnet	AC	813746	20	9	Yes
DeepSeek-V3.1	AC	1153921	257	47	Yes
DeepSeek-V3	AC	239910	55	17	Yes
Kimi-K2	AC	1366068	95	23	Yes

Appendix

Table 7: Representative qualification results for the USACO 2025 US Open Contest. Agents are required to solve the easiest Bronze problem 1515. Cons. Credit means Consumed Credit.

Model	Avg. Rank	Avg. Score	Avg. Consumed Credit	Inference Credit	Hint Credit	Penalty Credit
Gemini-2.5-pro	1.30 $\pm$ 0.47	14.00 $\pm$ 3.88	13,762,787 $\pm$ 4.3M	13.76M $\pm$ 4.3M	2.5K $\pm$ 2.5K	4.1K $\pm$ 1.9K
GPT-5-Codex	1.70 $\pm$ 0.47	9.39 $\pm$ 7.59	4,707,464 $\pm$ 3.1M	4.71M $\pm$ 3.1M	1.1K $\pm$ 2.0K	0.3K $\pm$ 0.3K
Qwen3-235b	4.00 $\pm$ 1.59	1.61 $\pm$ 0.92	11,732,391 $\pm$ 6.9M	11.17M $\pm$ 6.5M	560.2K $\pm$ 375.7K	3.6K $\pm$ 2.6K
GLM-4.5	4.35 $\pm$ 1.57	2.33 $\pm$ 2.28	7,249,215 $\pm$ 4.0M	7.06M $\pm$ 3.9M	161.7K $\pm$ 93.2K	22.7K $\pm$ 16.8K
DeepSeek-V3	5.70 $\pm$ 1.13	0.11 $\pm$ 0.32	194,050 $\pm$ 0.2M	0.17M $\pm$ 0.2M	19.8K $\pm$ 24.9K	1.2K $\pm$ 1.3K
DeepSeek-V3.1	6.00 $\pm$ 1.30	0.06 $\pm$ 0.24	253,013 $\pm$ 0.4M	0.23M $\pm$ 0.4M	21.9K $\pm$ 32.7K	1.4K $\pm$ 2.7K

Appendix

Table 8: Aggregated results from the main experiment, averaged over 5 runs across four contests. The data shows each agent’s final rank, score, and credit consumption, reflecting their strategic priorities. Values are presented as mean

\pm

standard deviation.

Agent	Win Rate	Avg. Score	Avg. Credit	Attempted	Submission
			Consumed	Problems	Precision (%)
Experiment 1: Generalist vs. Specialist
GPT-5 (Base)	100%	20.3	14M	26	12.9%
GPT-5-Codex	0%	8.0	8M	16	68.2%
Experiment 2: Specialist vs. Agentic Framework
GPT-5-Codex	0%	5	4M	16	44.4%

Appendix

Table 9: Performance and Strategy Comparison within the GPT-5 Agent Family. The value in parenthesis indicates the head-to-head win rate for each agent within its matchup.

Provider	Model	Input Cost	Output Cost
OpenAI	GPT-5-2025-08-07	$1.25	$10.00
	GPT-5-Codex	$1.25	$10.00
Google	Gemini 2.5 Pro	$1.25	$10.00
Anthropic	Claude-Sonnet-4-20250514	$3.00	$15.00
DeepSeek	DeepSeek-v3	$0.27	$1.10
	DeepSeek-v3.1	$0.27	$1.10

Appendix

Table 10: Specifications of Large Language Models used in our evaluation. Costs are denoted in USD per million tokens.

Parameter	Description	Default Value
Basic Setup
max_credits_per_participant	Maximum credits per participant	20,000,000
Scoring System
bronze_score	Points for Bronze problems	1
silver_score	Points for Silver problems	2
gold_score	Points for Gold problems	5

Appendix

Table 11: USACOArena Competition Configuration Parameters

Software Engineering Agents

Software Engineering Agents are AI systems that use code, tools, and feedback to perform software development tasks.

See all 344 papers

AgentSLABench: Evaluating and Benchmarking Agentic Systems Under Resource Constraints
Aug 1, 2026Meher Bhaskar Madiraju, Meher Sai Preetam MadirajuAI Agent Evaluation Agent Evaluation
CodeRescue: Budget-Calibrated Recovery Routing for Coding Agents
Jul 21, 2026Qijia He, Jiayi Cheng, Chenqian Le +8AI Coding Agents Cost-Aware Inference
University of Washington · New York University · ByteDance +1
Analyzing and Mitigating Cost-Inefficient Behaviors in Coding Agents
Sep 25, 2026Yiran Hu, Nan Jiang, Shanchao Liang +3Coding Agents Cost-Aware Inference
Purdue University

Aug 1, 2026J/K move · Enter open · S save

AgentSLABench: Evaluating and Benchmarking Agentic Systems Under Resource Constraints

Meher Bhaskar Madiraju, Meher Sai Preetam Madiraju

AI Agent Evaluation Agent Evaluation AI Agent Benchmarks

Open paper AskPDF arXiv

We present AgentSLABench, a resource-aware evaluation framework for autonomous AI agents that measures correctness alongside latency, cost, compute, memory, and network usage under declared resource budgets. Unlike standard benchmarks that report only accuracy, AgentSLABench produces a multi-dimensional profile per agent per task - the same way systems profilers (perf, pprof, cProfile) measure resource consumption of code, but extended with task correctness as a first-class dimension. AgentSLABench provides 16 task environments across 6 categories (5 core: multi-hop QA, retail substitution, code generation, web shopping, travel planning; 11 extended) with isolated Docker containers, declared CPU/memory/time/network budgets, sealed test sets with SHA256 hashes, and a standardized profiling protocol. We profile 5 general-purpose baseline agents (ReAct, PlanAndSolve, Reflexion, CoT, Random) plus 4 task-specialized agents, finding that specialized agents achieve 100% success on 3/5 core tasks (fact_qa, web_shopping, travel_planning) and 66.7-83.3% on retail and code_gen, while general baselines fail entirely on 4/5 domain tasks. Crucially, we report the Efficiency-Adjusted Success Rate (EASR) - success weighted by resource consumption relative to declared budgets - revealing that high accuracy at unbounded cost is not production-viable. We release the full infrastructure, sealed test sets, and profiling results to enable reproducible, resource-aware agent evaluation.

Abstract

Software Engineering Agents

Explore similar work

AgentSLABench: Evaluating and Benchmarking Agentic Systems Under Resource Constraints

CodeRescue: Budget-Calibrated Recovery Routing for Coding Agents

Analyzing and Mitigating Cost-Inefficient Behaviors in Coding Agents