RSI-PDEbench: A Benchmark for Self-Improving Agents in PDE Solving
Authors: Binghang Lu, Lang Cao, Changhong Mou, Yige Sun, Cindy Xiangrui Kong, Zheyuan Deng, Tian Yuan, Bing Hu, Runyu Zhang, Na Lu, Guang Lin
Organizations: Purdue University · University of Illinois Urbana-Champaign · Utah State University · Independent researcher · Brown University · Massachusetts Institute of Technology · Arizona State University
Identifying and solving partial differential equations (PDEs) from sampled solution data often requires substantial expert effort in equation discovery, coefficient estimation, and numerical solver configuration, especially under noise and distribution shifts. Can large language model (LLM) agents automate this workflow accurately and efficiently? To investigate this question, we introduce \bench{}, a benchmark of 1,001 cases spanning 42 equation templates, including both classical and fractional PDEs. It provides three evaluation settings: Full evaluates performance on noise-free solution data; Noisy tests robustness to Gaussian noise with an RMS equal to 50% of the clean-signal RMS; and Hard-OOD tests challenging out-of-distribution generalization to unseen equation templates and shifted coefficient ranges. We further introduce RSI-PDE, a recursively self-improving agent harness that shortlists candidate equations by residual loss, uses relative L2 feedback to refine coefficients and numerical solvers, and applies a validation gate to policy and memory updates. We compare RSI-PDE with single-run baselines and native agent harnesses (Claude Code or Codex) across four frontier models. Across the three settings, RSI-PDE improves the pass rate averaged across the four models by 106.5--181.9% relative to single-run baselines and by 3.0--19.6% relative to native harnesses, while reducing total recorded model calls by 46.9% across the 12 model--setting pairs. These results highlight the potential of validated recursive self-improvement in LLM agents to advance scientific computing.
Figures & tables
Figure 1: Final accuracy and learning curves. Left: passed-case bars (gray: native; red: RSI-PDE). Right: learning curves (black: native; red: RSI-PDE). Both halves have Opus 5.5 above GPT-5.5, with Full, Noisy and Hard-OOD columns. Labels give final counts and RSI-minus-native case gains. Axes are locally zoomed; cropped bar baselines are marked. The native Opus Full curve whisker shows the observed 187–198 range over three runs. Curves preserve regressions and stopping rounds. Tables 3 and 4 give all models and baselines.
Figure 2: Examples of reference fields from public training cases. The benchmark includes diffusion and transport, fractional dynamics, wave propagation, fluid flow, phase-field systems and electromagnetics. The gallery is illustrative; Appendix B gives the complete 42-template catalog.
Resource
Cap
Development wall-clock
2 h
Model requests
400
Prompt tokens, including cache reads/writes
40 M
Output tokens
1.5 M
Validation submissions
30
Solver resources
1 CPU core + 1 H100 GPU
Table 1: Declared budget per development sub-run. All aggregate resource figures sum the three sub-runs before averaging repeated runs. Final hidden evaluation is outside the development budget.
Setting
Templates
Train
Validation
Test
Observation noise
Full
42
42
42
210
None
Noisy
24
22
18
160
50% RMS
Hard-OOD
39
20
20
278
1% RMS
Table 2: The three current evaluation settings. Noisy contains 200 selected cases in total, of which 160 are test cases. Hard-OOD separates 188 cases from unseen templates and 90 cases with shifted coefficients.
Figure 3: RSI-PDE harness. The frozen LLM reads the context (H1–H6), and its reply passes the supervisor checks (H7) before becoming a training experiment (H8), a public-data analysis (H9) or a proposal (H10). A proposal updates solver P , instruction policy π and memory M . The candidate passes the training self-test (H11) and the validation or public-pass gate (H12); the text describes the bounded training-regression waiver. Accepted state returns in the next round’s context; rejected proposals return diagnostic feedback. The supervisor and model weights remain fixed. Only the independent evaluator scores the frozen solver on hidden tests.
Figure 4: The residual-guided RSI-PDE workflow. (1) The agent starts without the equation: the family k and the coefficients θ are unknown, and it receives 256 samples, the known forcing, initial and boundary data, and the class library. (2) A frozen LLM, steered by its self-written policy π and memory M , writes the program P , whose router identifies the case and dispatches it to a class solver. (3) In the sandbox, residual ranking by minθLres(k,θ;ua) ranks candidate families and keeps the five best; forward solves and public relative L2 checks then refine the equation choice and its coefficients. (4) Public relative L2 errors, the self-test and the validation gate decide what is kept. Accepted (P,π,M) carry into the next round, so the policy and memory evolve with the solver; a rejection returns its L2 errors and reasons. The bottom row follows a toy viscous Burgers case ( ν⋆=0.01 ): the residual fit on a smoothed field gives ν=0.0171 , and gated L2 feedback refines it to ν=0.0101 , below τ . The later coefficient values are illustrative; the residuals, fields and errors are computed for this toy case, not taken from benchmark runs.
Model
Entrant
Passed /210
Rate (%)
Runs
Gain (pp)
Gap (pp)
Calls
Cost ($)
Opus 5.5
Single run
66
31.5
4
–
–
3
7.32
Claude Code
193
91.9
3
+60.4
+2.5
240
16.74
RSI-PDE
196
93.3
1
+61.8
-0.5
88
30.47 ∗
GPT-5.5
Single run
38
18.2
4
–
–
3
1.22
Codex
95
45.2
1
+27.0
+7.1
482
40.97
RSI-PDE
104
49.5
1
+31.3
+5.2
88
32.01
Table 3: Full results. Passes are out of 210; gain over the single-run baseline and validation-minus-test gap are percentage points. Bold marks the higher displayed harness count for a model. † : author-updated native total with earlier-run resource/diagnostic records. ∗ : retrospective stop-adjusted RSI cost, not the cost of the full-budget score shown.
Figure 5: Full passed-case trajectories for all four models. Black dashed squares: native harness; red circles: RSI-PDE. Local vertical ranges show recorded points from round one. Final labels give headline counts; red labels also give RSI-minus-native gains in cases. The native Opus 5.5 whisker shows the observed 187–198 range over three runs. GPT-6.1 and Sonnet 5.5 native count trajectories are approximate, converted from the author’s rounded percentages; the Sonnet endpoint therefore need not equal 183 before rounding. A round index is not a common wall-clock budget.
Model
Entrant
Noisy /160
Calls
Hard-OOD /278
Calls
Unseen /188
Shifted /90
Opus 5.5
Single run
37
3
133
3
88
45
Claude Code
76
187
245
240
161
84
RSI-PDE
120
278
254
81
165
89
GPT-5.5
Single run
51
3
42
3
33
9
Codex
83
332
81
374
44
37
RSI-PDE
93
104
102
91
57
45
Table 4: Noisy and Hard-OOD results for all four models. Bold marks the higher harness count, including ties. Unseen and shifted counts partition Hard-OOD. ‡ : corrected GPT-6.1 Noisy attribution; the mark applies to the Noisy identity and its corresponding calls. Missing single-run group breakdowns remain unavailable rather than inferred.
Figure 6: Recorded model calls by model and setting. Black: native harness; red: RSI-PDE. Calls are summed across development groups and are not replaced by stop-rule replay counts. The corresponding dollar figures have a different provenance, as explained in the text and Table 6 .
Figure 7: Each point pairs a website row’s median relative L2 error with its median post-hoc true-equation residual. Points are aggregate medians, not paired per-case observations; marker shapes identify models and colors identify harnesses. Coverage and reference runs vary, including historical Full native diagnostics. The plot illustrates complementary diagnostics and should not be read as a causal residual-to-accuracy law.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
ID
Template
Class
Dim.
Steady
Classical and fractional templates
1a
Poisson, manufactured
Poisson
1
yes
1b
Poisson, multiscale
Poisson
1
yes
1c
Poisson, interior layer
Poisson
1
yes
2a
Heat, manufactured
heat
1
2b
Heat, free decay
heat
1
Appendix
Table 5: The 42 templates. Dim.: spatial dimension (0 for the fractional ODE). Steady: no time dependence.
Figure 8: Noisy trajectories for all four models, with corrected GPT-6.1 identities: red RSI-PDE finishes at 97/160 and black Codex at 90/160. The curves retain the supplied numerical measurements and change only their corrected harness attribution. Round indices are validation-submission indices.
Figure 9: Hard-OOD trajectories. RSI-PDE is red and the native harness is black. Template novelty and coefficient shift are both included in the 278-case aggregate; Table 4 separates their endpoints.
Setting
Model
Entrant
Passed
Calls
Recorded rounds
Cost ($)
Full
Opus 5.5
Single run
66.3
3
0
7.32
Opus 5.5
Claude Code
193
240
20
16.74
Opus 5.5
RSI-PDE
196
88
46
30.47 ∗
GPT-5.5
Single run
38.3
3
0
1.22
GPT-5.5
Codex
95
482
87
40.97
GPT-5.5
RSI-PDE
104
88
64
32.01
Appendix
Table 6: Website resource records, summed over the development groups and averaged where repeated. Recorded rounds are summed group submissions, not the final horizontal coordinate of a combined curve. † : updated Full native total paired with historical resource records. ‡ : corrected Noisy GPT-6.1 attribution. ∗ : retrospective stop-rule replay cost; calls and accuracy remain full-budget values.
Model
Entrant
Ref. passes/cases
Median L2
Residual
Residual, passed
Residual, failed
ρ
Opus 5.5
Single run
68.3/100
6.4×10−4
6.2×10−4
6.2×10−4
7.1×10−4
0.70
Opus 5.5
Claude Code
193/210
4.5×10−4
3.9×10−4
3.4×10−4
1.9×10−3
0.71
Opus 5.5
RSI-PDE
196/210
4.3×10−4
4.6×10−4
4.3×10−4
1.5×10−3
0.71
GPT-5.5
Single run
39.8/190
2.1×10−2
1.9×10−2
7.9×10−4
4.3×10−2
0.87
GPT-5.5
Codex
95/210
2.2×10−3
4.7×10−3
1.2×10−3
1.7×10−2
0.78
GPT-5.5
RSI-PDE
104/210
1.8×10−3
4.5×10−3
7.4×10−4
1.5×10−2
0.85
Appendix
Table 7: Full reference-run diagnostics. † : historical native counts 199/210 for GPT-6.1 and 193/210 for Sonnet 5.5; the current headline totals are 189 and 183. Single-run coverage and sample means can also differ from the headline comparison. These rows are not new measurements of the updated totals.
Model
Entrant
Ref. passes/cases
Median L2
Residual
Residual, passed
Residual, failed
ρ
Opus 5.5
Single run
37/42
2.1×10−8
3.7×10−6
2.3×10−6
1.3×10−3
0.86
Opus 5.5
Claude Code
76/160
2.8×10−3
2.0×10−3
6.9×10−5
1.3×10−2
0.90
Opus 5.5
RSI-PDE
120/160
3.2×10−6
8.4×10−5
5.1×10−5
1.2×10−2
0.89
GPT-5.5
Single run
51.5/160
6.3×10−3
4.8×10−3
1.6×10−3
1.5×10−2
0.76
GPT-5.5
Codex
83/160
1.6×10−3
2.9×10−3
5.5×10−4
1.5×10−2
0.92
GPT-5.5
RSI-PDE
93/160
6.0×10−4
1.3×10−3
2.3×10−4
1.1×10−2
0.84
Appendix
Table 8: Noisy reference-run diagnostics. The GPT-6.1 harness identities follow the author’s correction; residual values and their original numerical coverage are preserved.
Model
Entrant
Ref. passes/cases
Median L2
Residual
Residual, passed
Residual, failed
ρ
Opus 5.5
Single run
133/133
4.3×10−4
4.3×10−4
4.3×10−4
–
0.60
Opus 5.5
Claude Code
245/278
4.2×10−4
4.1×10−4
3.4×10−4
2.4×10−2
0.66
Opus 5.5
RSI-PDE
254/278
4.1×10−4
3.9×10−4
3.6×10−4
3.6×10−2
0.62
GPT-5.5
Single run
42/278
3.8×10−2
3.0×10−2
7.7×10−4
5.3×10−2
0.82
GPT-5.5
Codex
81/278
1.7×10−2
7.3×10−2
4.6×10−4
1.5×10−1
0.87
GPT-5.5
RSI-PDE
102/278
3.7×10−3
8.8×10−3
6.0×10−4
1.0×10−1
0.90
Appendix
Table 9: Hard-OOD reference-run diagnostics, including the available single-run rows. The true-equation residual is an evaluator-only post-hoc measurement and does not alter the relative- L2 pass criterion.