RSI-PDEbench: A Benchmark for Self-Improving Agents in PDE Solving
Authors: Binghang Lu, Lang Cao, Changhong Mou, Yige Sun, Cindy Xiangrui Kong, Zheyuan Deng, Tian Yuan, Bing Hu, Runyu Zhang, Na Lu, Guang Lin
Organizations: Purdue University · University of Illinois Urbana-Champaign · Utah State University · Independent researcher · Brown University · Massachusetts Institute of Technology · Arizona State University
Identifying and solving partial differential equations (PDEs) from sampled solution data often requires substantial expert effort in equation discovery, coefficient estimation, and numerical solver configuration, especially under noise and distribution shifts. Can large language model (LLM) agents automate this workflow accurately and efficiently? To investigate this question, we introduce \bench{}, a benchmark of 1,001 cases spanning 42 equation templates, including both classical and fractional PDEs. It provides three evaluation settings: Full evaluates performance on noise-free solution data; Noisy tests robustness to Gaussian noise with an RMS equal to 50% of the clean-signal RMS; and Hard-OOD tests challenging out-of-distribution generalization to unseen equation templates and shifted coefficient ranges. We further introduce RSI-PDE, a recursively self-improving agent harness that shortlists candidate equations by residual loss, uses relative L2 feedback to refine coefficients and numerical solvers, and applies a validation gate to policy and memory updates. We compare RSI-PDE with single-run baselines and native agent harnesses (Claude Code or Codex) across four frontier models. Across the three settings, RSI-PDE improves the pass rate averaged across the four models by 106.5--181.9% relative to single-run baselines and by 3.0--19.6% relative to native harnesses, while reducing total recorded model calls by 46.9% across the 12 model--setting pairs. These results highlight the potential of validated recursive self-improvement in LLM agents to advance scientific computing.
Figures & tables
Figure 1: Final accuracy and learning curves. Left: passed-case bars (gray: native; red: RSI-PDE). Right: learning curves (black: native; red: RSI-PDE). Both halves have Opus 5.5 above GPT-5.5, with Full, Noisy and Hard-OOD columns. Labels give final counts and RSI-minus-native case gains. Axes are locally zoomed; cropped bar baselines are marked. The native Opus Full curve whisker shows the observed 187–198 range over three runs. Curves preserve regressions and stopping rounds. Tables 3 and 4 give all models and baselines.
Figure 2: Examples of reference fields from public training cases. The benchmark includes diffusion and transport, fractional dynamics, wave propagation, fluid flow, phase-field systems and electromagnetics. The gallery is illustrative; Appendix B gives the complete 42-template catalog.
Resource
Cap
Development wall-clock
2 h
Model requests
400
Prompt tokens, including cache reads/writes
40 M
Output tokens
1.5 M
Validation submissions
30
Solver resources
1 CPU core + 1 H100 GPU
Table 1: Declared budget per development sub-run. All aggregate resource figures sum the three sub-runs before averaging repeated runs. Final hidden evaluation is outside the development budget.
Setting
Templates
Train
Validation
Test
Observation noise
Full
42
42
42
210
None
Noisy
24
22
18
160
50% RMS
Hard-OOD
39
20
20
278
1% RMS
Table 2: The three current evaluation settings. Noisy contains 200 selected cases in total, of which 160 are test cases. Hard-OOD separates 188 cases from unseen templates and 90 cases with shifted coefficients.
Figure 3: RSI-PDE harness. The frozen LLM reads the context (H1–H6), and its reply passes the supervisor checks (H7) before becoming a training experiment (H8), a public-data analysis (H9) or a proposal (H10). A proposal updates solver P , instruction policy π and memory M . The candidate passes the training self-test (H11) and the validation or public-pass gate (H12); the text describes the bounded training-regression waiver. Accepted state returns in the next round’s context; rejected proposals return diagnostic feedback. The supervisor and model weights remain fixed. Only the independent evaluator scores the frozen solver on hidden tests.
Figure 4: The residual-guided RSI-PDE workflow. (1) The agent starts without the equation: the family k and the coefficients θ are unknown, and it receives 256 samples, the known forcing, initial and boundary data, and the class library. (2) A frozen LLM, steered by its self-written policy π and memory M , writes the program P , whose router identifies the case and dispatches it to a class solver. (3) In the sandbox, residual ranking by minθLres(k,θ;ua) ranks candidate families and keeps the five best; forward solves and public relative L2 checks then refine the equation choice and its coefficients. (4) Public relative L2 errors, the self-test and the validation gate decide what is kept. Accepted (P,π,M) carry into the next round, so the policy and memory evolve with the solver; a rejection returns its L2 errors and reasons. The bottom row follows a toy viscous Burgers case ( ν⋆=0.01 ): the residual fit on a smoothed field gives ν=0.0171 , and gated L2 feedback refines it to ν=0.0101 , below τ . The later coefficient values are illustrative; the residuals, fields and errors are computed for this toy case, not taken from benchmark runs.
Model
Entrant
Passed /210
Rate (%)
Runs
Gain (pp)
Gap (pp)
Calls
Cost ($)
Opus 5.5
Single run
66
31.5
4
–
–
3
7.32
Claude Code
193
91.9
3
+60.4
+2.5
240
16.74
RSI-PDE
196
93.3
1
+61.8
-0.5
88
30.47 ∗
GPT-5.5
Single run
38
18.2
4
–
–
3
1.22
Codex
95
45.2
1
+27.0
+7.1
482
40.97
RSI-PDE
104
49.5
1
+31.3
+5.2
88
32.01
Table 3: Full results. Passes are out of 210; gain over the single-run baseline and validation-minus-test gap are percentage points. Bold marks the higher displayed harness count for a model. † : author-updated native total with earlier-run resource/diagnostic records. ∗ : retrospective stop-adjusted RSI cost, not the cost of the full-budget score shown.
Figure 5: Full passed-case trajectories for all four models. Black dashed squares: native harness; red circles: RSI-PDE. Local vertical ranges show recorded points from round one. Final labels give headline counts; red labels also give RSI-minus-native gains in cases. The native Opus 5.5 whisker shows the observed 187–198 range over three runs. GPT-6.1 and Sonnet 5.5 native count trajectories are approximate, converted from the author’s rounded percentages; the Sonnet endpoint therefore need not equal 183 before rounding. A round index is not a common wall-clock budget.
Model
Entrant
Noisy /160
Calls
Hard-OOD /278
Calls
Unseen /188
Shifted /90
Opus 5.5
Single run
37
3
133
3
88
45
Claude Code
76
187
245
240
161
84
RSI-PDE
120
278
254
81
165
89
GPT-5.5
Single run
51
3
42
3
33
9
Codex
83
332
81
374
44
37
RSI-PDE
93
104
102
91
57
45
Table 4: Noisy and Hard-OOD results for all four models. Bold marks the higher harness count, including ties. Unseen and shifted counts partition Hard-OOD. ‡ : corrected GPT-6.1 Noisy attribution; the mark applies to the Noisy identity and its corresponding calls. Missing single-run group breakdowns remain unavailable rather than inferred.
Figure 6: Recorded model calls by model and setting. Black: native harness; red: RSI-PDE. Calls are summed across development groups and are not replaced by stop-rule replay counts. The corresponding dollar figures have a different provenance, as explained in the text and Table 6 .
Figure 7: Each point pairs a website row’s median relative L2 error with its median post-hoc true-equation residual. Points are aggregate medians, not paired per-case observations; marker shapes identify models and colors identify harnesses. Coverage and reference runs vary, including historical Full native diagnostics. The plot illustrates complementary diagnostics and should not be read as a causal residual-to-accuracy law.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
ID
Template
Class
Dim.
Steady
Classical and fractional templates
1a
Poisson, manufactured
Poisson
1
yes
1b
Poisson, multiscale
Poisson
1
yes
1c
Poisson, interior layer
Poisson
1
yes
2a
Heat, manufactured
heat
1
2b
Heat, free decay
heat
1
Appendix
Table 5: The 42 templates. Dim.: spatial dimension (0 for the fractional ODE). Steady: no time dependence.
Figure 8: Noisy trajectories for all four models, with corrected GPT-6.1 identities: red RSI-PDE finishes at 97/160 and black Codex at 90/160. The curves retain the supplied numerical measurements and change only their corrected harness attribution. Round indices are validation-submission indices.
Figure 9: Hard-OOD trajectories. RSI-PDE is red and the native harness is black. Template novelty and coefficient shift are both included in the 278-case aggregate; Table 4 separates their endpoints.
Setting
Model
Entrant
Passed
Calls
Recorded rounds
Cost ($)
Full
Opus 5.5
Single run
66.3
3
0
7.32
Opus 5.5
Claude Code
193
240
20
16.74
Opus 5.5
RSI-PDE
196
88
46
30.47 ∗
GPT-5.5
Single run
38.3
3
0
1.22
GPT-5.5
Codex
95
482
87
40.97
GPT-5.5
RSI-PDE
104
88
64
32.01
Appendix
Table 6: Website resource records, summed over the development groups and averaged where repeated. Recorded rounds are summed group submissions, not the final horizontal coordinate of a combined curve. † : updated Full native total paired with historical resource records. ‡ : corrected Noisy GPT-6.1 attribution. ∗ : retrospective stop-rule replay cost; calls and accuracy remain full-budget values.
Model
Entrant
Ref. passes/cases
Median L2
Residual
Residual, passed
Residual, failed
ρ
Opus 5.5
Single run
68.3/100
6.4×10−4
6.2×10−4
6.2×10−4
7.1×10−4
0.70
Opus 5.5
Claude Code
193/210
4.5×10−4
3.9×10−4
3.4×10−4
1.9×10−3
0.71
Opus 5.5
RSI-PDE
196/210
4.3×10−4
4.6×10−4
4.3×10−4
1.5×10−3
0.71
GPT-5.5
Single run
39.8/190
2.1×10−2
1.9×10−2
7.9×10−4
4.3×10−2
0.87
GPT-5.5
Codex
95/210
2.2×10−3
4.7×10−3
1.2×10−3
1.7×10−2
0.78
GPT-5.5
RSI-PDE
104/210
1.8×10−3
4.5×10−3
7.4×10−4
1.5×10−2
0.85
Appendix
Table 7: Full reference-run diagnostics. † : historical native counts 199/210 for GPT-6.1 and 193/210 for Sonnet 5.5; the current headline totals are 189 and 183. Single-run coverage and sample means can also differ from the headline comparison. These rows are not new measurements of the updated totals.
Model
Entrant
Ref. passes/cases
Median L2
Residual
Residual, passed
Residual, failed
ρ
Opus 5.5
Single run
37/42
2.1×10−8
3.7×10−6
2.3×10−6
1.3×10−3
0.86
Opus 5.5
Claude Code
76/160
2.8×10−3
2.0×10−3
6.9×10−5
1.3×10−2
0.90
Opus 5.5
RSI-PDE
120/160
3.2×10−6
8.4×10−5
5.1×10−5
1.2×10−2
0.89
GPT-5.5
Single run
51.5/160
6.3×10−3
4.8×10−3
1.6×10−3
1.5×10−2
0.76
GPT-5.5
Codex
83/160
1.6×10−3
2.9×10−3
5.5×10−4
1.5×10−2
0.92
GPT-5.5
RSI-PDE
93/160
6.0×10−4
1.3×10−3
2.3×10−4
1.1×10−2
0.84
Appendix
Table 8: Noisy reference-run diagnostics. The GPT-6.1 harness identities follow the author’s correction; residual values and their original numerical coverage are preserved.
Model
Entrant
Ref. passes/cases
Median L2
Residual
Residual, passed
Residual, failed
ρ
Opus 5.5
Single run
133/133
4.3×10−4
4.3×10−4
4.3×10−4
–
0.60
Opus 5.5
Claude Code
245/278
4.2×10−4
4.1×10−4
3.4×10−4
2.4×10−2
0.66
Opus 5.5
RSI-PDE
254/278
4.1×10−4
3.9×10−4
3.6×10−4
3.6×10−2
0.62
GPT-5.5
Single run
42/278
3.8×10−2
3.0×10−2
7.7×10−4
5.3×10−2
0.82
GPT-5.5
Codex
81/278
1.7×10−2
7.3×10−2
4.6×10−4
1.5×10−1
0.87
GPT-5.5
RSI-PDE
102/278
3.7×10−3
8.8×10−3
6.0×10−4
1.0×10−1
0.90
Appendix
Table 9: Hard-OOD reference-run diagnostics, including the available single-run rows. The true-equation residual is an evaluator-only post-hoc measurement and does not alter the relative- L2 pass criterion.
Numerical solvers for partial differential equations (PDEs) are core computational tools in science and engineering. Building reliable PDE solvers requires not only executable code, but a numerical solver strategy, a set of decisions about discretization, stabilization, solver configuration, and resolution control, that matches the PDE structure. Recent LLM-based coding agents have begun to reduce the programming burden by generating and debugging solver implementations. However, they typically move directly from a PDE problem to solver code, leaving the solver strategy implicit in implementation details. Feedback from a failed solve is therefore routed back to code edits rather than to the underlying strategy, so numerical decisions remain hard to check before code is generated and hard to revise using numerical evidence when it fails. To address this limitation, we propose AutoPDE, a code agent that maintains the solver strategy as an explicitly represented object throughout the solving process: an independent, inspectable object that is built before any code is written and can be revised, using numerical evidence, whenever a solve fails. AutoPDE builds and maintains this object in three stages, all drawing from a library of reusable PDE-solving skills: PDE analysis identifies the equation type and algebraic structure; numerical method selection chooses a numerical method that matches the analysis result and commits to a discretization, stabilization, and linear solver accordingly; and adaptive tuning runs low-cost pilot solves to calibrate resolution and tolerances under the prescribed accuracy and runtime budget. We evaluate AutoPDE on the PDE Agent Bench, where experimental results show that AutoPDE achieves a pass rate of 54.5, improving over the strongest baseline by 14.2 percentage points.
Huanshuo Dong, Keyao Zhang, Hong Wang +6
University of Science and Technology of China · 2Zhejiang University · University of the Chinese Academy of Sciences +2
Partial differential equations (PDEs) are essential for modeling complex physical systems, and neural solvers have recently emerged as powerful data-driven tools for numerically solving them. However, existing neural solvers struggle with domain-specific challenges, such as varying parameters and high-speed flows, necessitating specialized architectures. Manually designing these specialized solver architectures is a highly iterative, time-consuming process requiring deep expertise, creating a significant bottleneck in scientific discovery. We propose leveraging autonomous AI research agents to automate the synthesis of specialized solvers. To support this, we introduce AutoPDEBench, a benchmark dedicated to LLM-driven automated research for PDE solver design. The benchmark includes 25 challenging datasets featuring both novel and actively studied physical scenarios. We evaluate a suite of general-purpose models (transformer, ROM, and graph-based) alongside a multi-agent instantiation of the iterative automated research pipeline, which serves as an agentic baseline. Empirical results show that the iterative automated research system significantly outperforms the general-purpose neural solver baselines. Our findings demonstrate the viability of using AI agents to automatically design neural solvers for complex physical systems. AutoPDEBench provides a foundational testbed to accelerate agent-driven scientific discovery in physics and engineering.
Ruoyan Li, Wei Wang, Yizhou Sun
Department of Computer Science University of California, Los Angeles
PDE-to-solver code generation aims to automatically synthesize executable numerical solvers from partial differential equation (PDE) specifications. This task requires not only understanding the mathematical structure of PDEs, but also selecting appropriate discretization schemes and solver configurations, and correctly implementing the resulting formulations in finite-element method (FEM) libraries. Existing code generation benchmarks mainly evaluate syntactic correctness, or success on predefined test cases. To our knowledge, there is currently no publicly available benchmark specifically for PDE-to-solver code generation, and general-purpose code benchmarks do not fully capture the unique challenges of numerical PDE solution, such as ensuring solver accuracy, efficiency, and compatibility with professional FEM libraries. We introduce PDEAgent-Bench, to the best of our knowledge, the first multi-metric, multi-library benchmark for PDE-to-solver code generation. PDEAgent-Bench contains 645 instances across 6 mathematical categories and 11 PDE families, with common FEM libraries for DOLFINx, Firedrake, and deal.II. Each instance provides an agent-facing problem specification, a reference solution on a prescribed evaluation grid, and case-specific accuracy and runtime targets. PDEAgent-Bench adopts a staged evaluation framework in which generated solvers must sequentially pass executability, numerical accuracy, and computational efficiency checks. Experiments with representative LLMs and code agents show that models can often produce runnable code, but their pass rate drops substantially once accuracy and efficiency requirements are enforced. These results indicate that current agents remain limited in producing numerically reliable and efficient PDE solvers, and that PDEAgent-Bench provides a reproducible testbed grounded in the practical requirements of numerical PDE solving.
Zhen Hang, Yushan Yashengjiang, Junhui Li +21
University of Science and Technology of China · Beijing University of Posts and Telecommunications · Tencent +10