cs.LGFeb 23, 2026

ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads?

Authors: Ayush Nangia, Shikhar Mishra, Aman Gokrani, Paras Chopra

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We introduce ISO-Bench, a benchmark for coding agents to test their capabilities on real-world inference optimization tasks. These tasks were taken from vLLM and SGLang, two of the most popular LLM serving frameworks. Each task provides an agent with a codebase and bottleneck description, whereby the agent must produce an optimization patch evaluated against expert human solutions. We curated 54 tasks from merged pull requests with measurable performance improvements. While existing benchmarks heavily use runtime-based metrics, such approaches can be gamed to pass tests without capturing the actual intent of the code changes. Therefore, we combine both hard (execution-based) and soft (LLM-based) metrics to show that both are necessary for complete evaluation. While evaluating both closed and open-source coding agents, we find no single agent dominates across codebases. Surprisingly, agents often identify correct bottlenecks but fail to execute working solutions. We also show that agents with identical underlying models differ substantially, suggesting scaffolding is as important as the model.

Figures & tables

Figure 1: ISO-Bench evaluation pipeline. Given a codebase and task description, a coding agent produces an optimization patch. We compare this patch against the human commit using hard metrics (TTFT, throughput) and soft metrics (bottleneck targeting, implementation approach). Hard metrics measure performance improvement; soft metrics assess whether the agent targeted the correct code.
CategoryCriteria
BeatsAgent improves on human by >5%>5\%
SimilarAgent within ±5%\pm 5\% of human
WorseAgent degrades by >5%>5\%
FailedPatch causes benchmarking error
Table 1: Classification of hard metrics based on performance delta.
Bottleneck Targeting
Same targetIdentical code locations as human
Related targetSame module or subsystem
Different targetUnrelated code areas
No optimizationNo performance-relevant changes
Implementation Approach
Similar approachSame technique as human
Table 2: Soft metric categories.
Figure 2: Quadrant framework for evaluating optimization attempts. The horizontal axis shows performance (good: beats or similar; bad: worse or failed). The vertical axis shows whether the agent targeted the correct bottleneck (correct: same or related target; wrong: different target or no optimization). Q1 True Success: correct target, good performance. Q2 Good Intent: correct target, bad performance. Q3 Lucky Win: wrong target, good performance. Q4 Total Failure: wrong target, bad performance.
Figure 5
Figure 6
AgentvLLMSGLang
Claude Code46.2%26.7%
TRAE (Sonnet)28.2%80.0%
TRAE (GPT-5)17.9%86.7%
Codex CLI20.5%80.0%
Table 3: True Success rates (%) by Project.
ProjectAgentHard Success (%)True Success (%)Gap (%)
vLLMClaude Code56.446.210.2
Codex CLI33.320.512.8
TRAE (Sonnet)33.328.25.1
TRAE (GPT-5)20.517.92.6
SGLangClaude Code46.726.720.0
Codex CLI80.080.00.0
Table 4: Hard Success vs. True Success rates. The Gap column quantifies cases where agents achieved performance improvements without targeting the correct bottleneck. A gap of zero indicates all hard successes genuinely addressed the specified optimization target.
ProjectAgentQ1Q2Q3Q4
vLLMClaude Code181542
Codex CLI82056
TRAE (Sonnet)112026
TRAE (GPT-5)72714
SGLangClaude Code4830
Codex CLI12300
Table 5: Distribution of outcomes across quadrants (Q1-Q4).
Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

StagevLLMSGLang
Total commits15,2348,421
Keyword matches892547
Scope filter ( << 10 files)341312
LLM classification186201
Manual curation3915
Appendix
Table 6: Commit filtering statistics for ISO-Bench.
Figure 7: Task specification schema defining the optimization challenge metadata, including repository information, runner requirements, and optimization constraints.
AgentModelBudget
Claude CodeClaude Sonnet 4.5120 min
Codex CLIGPT-5 (Reas. High)120 min
TRAE-Agent (Sonnet)Claude Sonnet 4.5120 min
TRAE-Agent (GPT-5)GPT-5 (Reas. High)120 min
Appendix
Table 7: Agent configurations for ISO-Bench evaluation.
Figure 8: Task prompt template provided to agents. The bottleneck description is customized per task based on PR analysis.
Figure 9: Patch from TRAE (Sonnet 4.5) on commit 98f47f2a , demonstrating Q1 (True Success). The agent correctly identified the bottleneck and achieved +21% throughput matching the human approach.
Figure 10: Patch from TRAE (Sonnet 4.5) on commit 015069b0 , demonstrating Q2 (Good Intent, Bad Execution). The agent targeted the correct bottleneck but produced non-working code.
Figure 11: Patch from TRAE (Sonnet 4.5) on commit fe66b347 , demonstrating Q3 (Lucky Win). The agent achieved speedup by computing incorrect values faster, causing 100% accuracy regression.
Figure 12: Patch from TRAE (Sonnet 4.5) on commit 2deb029d , demonstrating Q4 (Complete Failure). The agent produced complex code that completely missed the actual optimization target.
Figure 13: Execution log from MiniMax-M2.1 showing infinite verbalization loop. Despite 75 reasoning steps, the model made zero tool calls, demonstrating the understanding-execution gap.
Figure 14: File creation attempts from GPT-OSS-120B showing environment confusion. The model tried to implement mock PyTorch/Triton libraries instead of optimizing the actual vLLM scheduler code.
Figure 15: Execution trace from GLM-4.7 showing valid edits followed by task completion failure. The model made successful optimizations but could not recognize when to finalize the task.

Explore similar work

CardsList

  1. FormulaCode: Evaluating Agentic Optimization on Large Codebases

    Mar 16, 2026Atharva Sehgal, James Hou, Akanksha Sarkar +4AI Agent BenchmarksLarge Language Model-Guided Optimization

    The University of Texas at Austin · California Institute of Technology · Cornell University

  2. PerfAgent: Profiler-Guided Iterative Refinement for Repository-Level Code Optimization

    Jul 22, 2026Ryan Deng, Yuanzhe Liu, Bastian Lipka +4AI Coding Agents

    Massachusetts Institute of Technology, Cambridge, Massachusetts, USA · Rensselaer Polytechnic Institute, Troy, New York, USA · IBM, Ehningen, Germany +2

  3. Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?

    Jul 1, 2026Zhi Chen, Zhensu Sun, Yuling Shi +2Benchmark AuditingBenchmark Design

    Singapore Management University · Singapore, Singapore · Shanghai Jiao Tong University +1