FormulaCode: Evaluating Agentic Optimization on Large Codebases

Atharva Sehgal; James Hou; Akanksha Sarkar; Ishaan Mantripragada; Swarat Chaudhuri; Jennifer J. Sun; Yisong Yue

cs.SEMar 16, 2026

FormulaCode: Evaluating Agentic Optimization on Large Codebases

Authors: Atharva Sehgal, James Hou, Akanksha Sarkar, Ishaan Mantripragada, Swarat Chaudhuri, Jennifer J. Sun, Yisong Yue

Organizations: The University of Texas at Austin · California Institute of Technology · Cornell University

Abstract

Large language model (LLM) coding agents increasingly operate at the repository level, motivating benchmarks that evaluate their ability to optimize entire codebases under realistic constraints. Existing code benchmarks largely rely on synthetic tasks, binary correctness signals, or single-objective evaluation, limiting their ability to assess holistic optimization behavior. We introduce FormulaCode, a benchmark for evaluating agentic optimization on large, real-world codebases with fine-grained, multi-objective performance metrics. FormulaCode comprises 957 performance bottlenecks mined from scientific Python repositories on GitHub, each paired with expert-authored patches and, on average, 264.6 community-maintained performance workloads per task, enabling the holistic ability of LLM agents to optimize codebases under realistic correctness and performance constraints. Our evaluations reveal that repository-scale, multi-objective optimization remains a major challenge for frontier LLM agents. Project website at: https://formula-code.github.io

Figures & tables

Figure 1: FormulaCode is a continuously updating benchmark for evaluating the holistic ability of agents to optimize large codebases. Each task in FormulaCode comprises a problem description of a performance regression from GitHub, an environment containing a baseline repository snapshot, and multiple expert-written crowdsourced performance workloads, along with the tools to execute them. An agent’s performance improving edits are assessed based on their ability to outperform expert-written edits in optimizing multiple workloads while meeting multiple forms of correctness guarantees.

Figure 2: Overview of FormulaCode construction pipeline. FormulaCode follows a four stage pipeline to identify real-world performance optimization tasks. (1) Scrape compliant repositories (§ A.1 ). (2) Apply rule-based and LLM-based filters to identify candidate performance improvement pull requests (§ A.2 ). (3) Construct reproducible Docker environments for each candidate (§ A.5 ). (4) Validate each candidate for correctness and statistically significant performance improvement (§ A.6 ). The pipeline is fully automated and updates FormulaCode with new tasks every month.

		Mean	Max
Issue Text	Length (Tokens)	2718.03	15781
Gold Patch	# Lines edited	38.088	526
	# Files edited	3.93	34
	# Func. edited	6.06	54
Workloads	# Eval. Fns	264.58	1364
Workloads	% Coverage	41.24%	97.86%

Table 1: (Micro-averaged) statistics characterizing different attributes of a FormulaCode task instance. The average FormulaCode gold patch requires

5.2

more lines of code spread over

1.29\times

more files and

1.01\times

more functions than the average SWE-Bench ( Jimenez et al., 2024 ) patch.

Figure 3: Showing stratified advantage across hierarchy levels for each agent–model configuration. Each line traces the stratified advantage (

\texttt{Adv}_{\texttt{agent}}^{(\ell)}

) over

\ell\in\{1,2,3\}

, revealing whether a configuration prefers coarse module-level changes or fine-grained function-level edits.

Figure 4: Cost-Performance tradeoff of agent-model configurations. As most agents struggle on code optimizations tasks, the pareto set is primarily dominated by the most expensive model (Claude 4.0 Sonnet).

Figure 5: Multi-workload tradeoff performance of agent-model configurations. We quantify a model’s speedup performance as a function of its worst regression. The expert patch achieves the highest speedup while negotiating considerably high workload regressions.

Panel of Table 6 — Table 6: Comparing FormulaCode with related codebase benchmarks. FormulaCode is the only benchmark that satisfies the desired properties for evaluating LLM agents on real-world code optimization tasks. ++ denotes continually updating benchmarks. Data is sampled from real distributions like GitHub ( ), Leetcode ( ), AtCoder ( ), and Codeforces ( ); and LLM-generated or synthetic distributions ( ). An extended analysis is presented in § 5 .

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Distribution of tasks across repositories in FormulaCode till November, 2025. FormulaCode comprises of

957

tasks sampled from

70

diverse open source GitHub repositories. Most repositories are software tools used extensively within scientific communities. FormulaCode shows a strong long-tail pattern of bespoke repositories that are rarely covered in contemporary code-generation datasets. Table 10 presents a detailed overview.


1	INPUT SIGNATURE
2
3	problem_description : string
4	Problem statement and technical context from PR/issue.
5
6	git_patch : string

Appendix

Figure 7: Prompt template used by the LLM-based performance intent classifier described in A.3 . The prompt defines the input signature (problem description, git patch, and file change summary), the classifier module specifying decision criteria for identifying performance-motivated commits, and the output signature producing a reasoning trace and binary label (“YES”/“NO”).


1	Example PR
2
3	CLASSIFIER INPUT
4
5	problem_description : string
6	Labels: performance; Description: Fixes #14471.

Appendix

Figure 8: Example of performance intent classification for a real pull request (https://github.com/Qiskit/qiskit/pull/14782) , illustrating how the prompt in Fig. 7 is instantiated and how the classifier produces a reasoning trace and binary label.


1	Performance type & difficulty classifier
2
3	INPUT SIGNATURE PROMPT
4
5	problem_description : string
6	Problem statement and technical context from PR/issue.

Appendix

Figure 9: Prompt template used by the LLM-based classifier for assigning each performance task an optimization category and difficulty level ( B.3 ). The prompt defines the input signature, a taxonomy-driven classification module that maps code changes to optimization types, and an output schema that produces the predicted category, difficulty, and a brief reasoning trace.


1	INPUT SIGNATURE
2
3	owner_repo : string
4	The repository this commit belongs to (e.g., scikit-learn/scikit-learn).
5
6	sha : string

Appendix

Figure 10: Prompt structure for the docker build agent ( A.5 ), defining its input state, tool-calling interface, constraints, and executable script output.


1	Example PR
2
3	CLASSIFIER INPUT
4
5	problem_description : string
6	Labels: performance; Description: Fixes #14471.

Appendix

Figure 11: Example application of the optimization type and difficulty classifier (Figure 9 ), illustrating the predicted category, difficulty level, and reasoning for a real pull request ( https://github.com/Qiskit/qiskit/pull/14782)


1	Judge performance related PR prompt
2
3	INPUT SIGNATURE PROMPT
4
5	problem_description : string
6	Problem statement and technical context from PR/issue.

Appendix

Figure 12: Structured DSPy prompt used to judge whether a pull request is primarily intended to improve runtime or product performance. The prompt specifies the required inputs (problem description, code diff, and file-level change summary), explicit decision criteria and exclusions for performance-related changes, and an output format consisting of a justification and a binary YES/NO label. The design emphasizes conservative, evidence-based classification, prioritizing explicit runtime improvements over incidental or refactoring-only changes.


1	Problem Extractor Prompt description
2
3	INPUT SIGNATURE PROMPT
4
5	pr_title : string
6	The GitHub PR title

Appendix

Figure 13: Structured DSPy prompt used to extract the underlying problem and resolution context from a GitHub pull request. The prompt consumes the PR title, description, and discussion, and produces a structured summary capturing observed symptoms, triage steps, the implemented solution, and post-change observations. The design emphasizes near-verbatim extraction and separation of observations, investigation, and outcomes.

Figure 14: Overview of the pipeline for Docker environment synthesis. The system reuses chronologically adjacent build scripts when possible, otherwise invoking an LLM agent that generates and refines Docker scripts using build logs and repository context until a verifier confirms a successful, reproducible build.

Repository Name	#Stars	#Forks	Filter Stage 1	Filter Stage 2	Latest Task Date	Description
2 . scikit-learn/scikit-learn	63792	26359	2434	243	2025-10-31	scikit-learn: machine learning in Python
3 . pandas-dev/pandas	46922	19184	3298	560	2025-11-11	Flexible and powerful data analysis / manipulation library for Python, providing labeled data structures similar to R data.frame objects, statistical functions, and much more
4 . scipy/scipy	14120	5516	1454	209	2025-10-29	SciPy library main repository
5 . apache/arrow	16089	3884	1988	267	2025-07-22	Apache Arrow is the universal columnar format and multi-language toolbox for fast data interchange and in-memory analytics
6 . networkx/networkx	16277	3415	288	44	2025-09-16	NetworkX is a Python package for the creation, manipulation, and study of the structure, dynamics, and functions of complex networks.
7 . Qiskit/qiskit	6598	2659	717	212	2025-11-19	Qiskit is an open-source SDK for working with quantum computers at the level of pulses, circuits, and application modules.

Appendix

Table 9: Repositories and Tasks after applying rule-based filters (Filter Stage 1) and LLM-based filters (Filter Stage 2) as described in § A.2 . We also showcase the number of tasks, the date of creation of the latest task, and additional information about the functionality and popularity of the repository. Most repositories are software tools used extensively within scientific communities.

Repository	#Tasks	Latest Task	Avg. Difficulty	Avg. Patch Size (Tokens)	Avg. PR Size (Tokens)	Most Common Optimization
2 . pandas-dev/pandas	222	2025-10-21	0.77	1842.85	489.35	Micro Optimizations (26.6%)
3 . scikit-learn/scikit-learn	143	2025-10-31	1.0	2735.29	491.49	Micro Optimizations (23.1%)
4 . Qiskit/qiskit	142	2025-10-03	1.73	4438.38	505.02	Use Lower Level System (28.2%)
5 . xdslproject/xdsl	134	2025-10-09	1.36	3567.76	463.46	Remove Or Reduce Work (37.3%)
6 . optuna/optuna	94	2025-11-05	0.96	546.29	471.81	Use Better Algorithm (24.5%)
7 . pydata/xarray	69	2025-11-21	0.98	1929.9	474.04	Micro Optimizations (30.4%)

Appendix

Table 10: Repositories and Tasks represented in FormulaCode (as of November 30, 2025). We showcase a repository level breakdown of the number of tasks, the latest task (by PR merge date), the average difficulty (0-5, with 0 being easiest), the average number of tokens in the human patch and in the prompt instructions, and the most common optimization type of the human patch.

Figure 15: Illustration of Hierarchical Grouping of Pandas Workloads. By construction, each workload in FormulaCode is organized hierarchically based on three levels:

\ell=1

(Module),

\ell=2

(Class), and

\ell=3

(Function). Metrics (like

\texttt{speedup}_{\texttt{agent}}

and

\texttt{Adv}_{\texttt{agent}}

) are computed for each complete workload (leaf nodes). We can semantically aggregate workloads by stratification of workloads based on this hierarcy. For instance, in this example, the stratified speedup of pd.algorithms.Quantile.* can be calculated by computing the geometric mean of all leaf nodes that share the same the prefix string (depicted in the gray dotted box; pd.algorithms.Quantile.time_quantile(‘float’) , pd.algorithms.Quantile.time_quantile(‘int’) , and other complete workloads not shown.). The example also illustrates how highly localized optimizations are diluted by stratification, and underscores that, at higher levels of stratification, consistent speedups across a large number of workloads is required to achieve a significant stratified speedup.


1	OBJECTIVE
2	You are a performance optimization expert. Speed up the repository while maintaining correctness .
3
4	TOOLING
5	The micromamba environment includes Pytest for correctness testing and Airspeed Velocity (ASV) for benchmarking measurements and profiling.
6

Appendix

Figure 16: Example task in FormulaCode for Qiskit/qiskit (PR: https://github.com/Qiskit/qiskit/pull/14782 ). The prompt presents a complete optimization task, including the performance goal, the benchmarking and profiling tools (Pytest and ASV), a structured optimization workflow, and concrete repository context with motivating performance observations. The “Relevant Issues” section contains GitHub issues that are directly related to the performance problem addressed by the PR (describing the underlying bottlenecks the PR aims to fix). These issues provide important background context that mimics a real, human-authored PR setting. Issue discussions are truncated only in this figure for brevity, while the full issue content is provided to the agent during execution.


1	OBJECTIVE
2	You are a performance optimization expert. Speed up the repository while maintaining correctness .
3
4	TOOLING
5	The micromamba environment includes Pytest for correctness testing and Airspeed Velocity (ASV) for benchmarking measurements and profiling.
6

Appendix

Figure 17: Example task in FormulaCode for shapely/shapely (PR: https://github.com/shapely/shapely/pull/2283 ).


1	OBJECTIVE
2	You are a performance optimization expert. Speed up the repository while maintaining correctness .
3
4	TOOLING
5	The micromamba environment includes Pytest for correctness testing and Airspeed Velocity (ASV) for benchmarking measurements and profiling.
6

Appendix

Figure 18: Example task in FormulaCode for pandas-dev/pandas (PR: https://github.com/pandas-dev/pandas/pull/59608 ).

Figure 19: Timeline of FormulaCode tasks organized by the date the expert-patch was merged till November, 2025. Each box represents the number of expert-patch tasks merged during a particular month/year. FormulaCode is updated on the 25th of each month, and our most recent task is from 2025-11-21. The dataset grew by 27.00 tasks per month on average in 2025, facilitating contamination analyses for performance-optimization agents. Table 10 presents a detailed overview.

Parameter	Terminus 2	OpenHands
Decoding and sampling
Temperature	$0.7^{a}$	$0.0^{a}$
Top- $p$	provider default (unset) a	$1.0^{a}$
Top- $k$ / penalties	provider default	provider default
Reasoning effort	per model (Table 13 )	per model (Table 13 )
Max output tokens	model cap (Table 13 )	model cap (Table 13 )

Appendix

Table 12: Evaluation configuration for all agent–model runs. Entries marked a are framework defaults verified from source at the pinned versions (Terminus 2 at commit d71d8fc ; OpenHands v0.51.0 , run through the Terminal-Bench adapter); entries marked b are FormulaCode-specific changes. Per-model API limits, reasoning modes, and prices are in Table 13 .

Model	Adv	LogAdv	RP Rank
Claude 4.0 Sonnet	$-0.0112$	$-0.0130$	1
Qwen 3 Coder	$-0.0301$	$-0.0280$	2
GPT-5	$-0.0209$	$-0.0167$	3

Appendix

Table 15: Comparison of advantage metrics under linear and log-space for OpenHands configurations. Rankings are preserved under both formulations.

Model	$\|\texttt{Adv}\|\leq 0.05$	$\|\texttt{Adv}\|>0.05$
Claude 4.0 Sonnet	0.94	0.97
GPT-5	0.96	0.96
Gemini 2.5 Pro	0.98	0.95
Qwen 3 Coder	0.96	0.97

Appendix

Table 20: Mean normalized edit distance between agent and expert patches on FormulaCode-V , stratified by advantage proximity. Values close to 1 indicate low patch similarity. Across all configurations, agent and expert patches share only

\sim

4% of their content.

Figure 26: Visual intuition for Agent Advantage (

\texttt{Adv}_{\texttt{agent}}

; § 2 ). Each cross (✗) represents an individual workload using the expert-derived speedup (

\texttt{speedup}_{\texttt{expert}}

) and the agent-derived speedup (

\texttt{speedup}_{\texttt{agent}}

). The identity function line represents equal advantage (i.e.,

\texttt{speedup}_{\texttt{expert}}=\texttt{speedup}_{\texttt{agent}}

). Then, the agent advantage is the mean weighted deviation from the equal advantage line. The plot also showcases four optimization regions clockwise from top: (1) Super Optimization : workloads where an agent’s code performs better than the expert’s code and the baseline. (2) Under Optimization : workloads where the agent’s code and the expert’s code both deliver a positive speedup, but the expert outperforms the agent. (3) Performance Degredation : workloads where the expert discovers a speedup while the agent slows down the code. (4) Regression : workloads where neither the expert nor the agent slow down the code; usually an intentional tradeoff to optimize other workloads. Figure 26 showcases an example of workload distribution for various agents on FormulaCode .

Figure 27: Visualization of advantage for Terminus 2 Agents. Refer to Figure 26 for an explanation of each region. Each square represents the number of workloads in that region (within

0.5

units). A speedup of

1.0

indicates no deviation from baseline performance. The red dotted line represents equal advantage. This visualization is helpful to gauge the holistic behavior of models across the entire workload distribution. For instance, Claude 4.0 Sonnet (Top Left) achieves a better overall advantage than GPT-5 (Bottom Right) by making measured and surgical optimizations that align with the equal-advantage line, whereas optimizations proposed by GPT-5 are more volatile, with more workloads experiencing performance degradations and effectively bringing the overall advantage down.

Explore similar work

ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads?
Feb 23, 2026Ayush Nangia, Shikhar Mishra, Aman Gokrani +1AI Coding Agents AI Agent Benchmarks
PerfAgent: Profiler-Guided Iterative Refinement for Repository-Level Code Optimization
Jul 22, 2026Ryan Deng, Yuanzhe Liu, Bastian Lipka +4AI Coding Agents
Massachusetts Institute of Technology, Cambridge, Massachusetts, USA · Rensselaer Polytechnic Institute, Troy, New York, USA · IBM, Ehningen, Germany +2
SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads?
Nov 8, 2025Jeffrey Jian Ma, Milad Hashemi, Amir Yazdanbakhsh +5Software Engineering Agents Benchmark Design
Harvard University · Google DeepMind · Princeton University +2

Feb 23, 2026J/K move · Enter open · S save

ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads?

Ayush Nangia, Shikhar Mishra, Aman Gokrani, Paras Chopra

AI Coding Agents AI Agent Benchmarks Code Generation Evaluation LLM Inference Efficiency LLM Serving

Open paper AskPDF arXiv

We introduce ISO-Bench, a benchmark for coding agents to test their capabilities on real-world inference optimization tasks. These tasks were taken from vLLM and SGLang, two of the most popular LLM serving frameworks. Each task provides an agent with a codebase and bottleneck description, whereby the agent must produce an optimization patch evaluated against expert human solutions. We curated 54 tasks from merged pull requests with measurable performance improvements. While existing benchmarks heavily use runtime-based metrics, such approaches can be gamed to pass tests without capturing the actual intent of the code changes. Therefore, we combine both hard (execution-based) and soft (LLM-based) metrics to show that both are necessary for complete evaluation. While evaluating both closed and open-source coding agents, we find no single agent dominates across codebases. Surprisingly, agents often identify correct bottlenecks but fail to execute working solutions. We also show that agents with identical underlying models differ substantially, suggesting scaffolding is as important as the model.

Abstract

Explore similar work

ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads?

PerfAgent: Profiler-Guided Iterative Refinement for Repository-Level Code Optimization

SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads?