Solver to System: LLM Reasoning Across Representations and Execution in Scientific Simulation
Organizations: New York University
Abstract
Simulators based on partial differential equations (PDEs) are widely used to model physical systems, but generating and interpreting PDE solver code with large language models (LLMs) remains challenging. Scientific simulation code is a useful testbed for LLM scientific reasoning because (1) one system can be represented by code, equations, natural language, and numerical trajectories and (2) code can execute successfully while implementing invalid physics. We introduce Solver2System, a dataset of PDE solvers with multiple representations, controlled corruptions, and lexical perturbations, to evaluate whether LLMs can recover the underlying physical system through (1) cross-representation consistency and (2) static and agentic detection of physical invalidity. Across eight open-weight models, LLMs systematically favor some representations when evidence conflicts, and lexical perturbations to code alter interpretation of other unchanged representations. Execution improves physical-validity judgments, but weaker agents can judge correctly without identifying the causal defect. Thus, LLMs may reach correct judgments despite inconsistent representation reconciliation or causal misdiagnosis.
Figures & tables
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Source | PDE | Snippets | Avg Lines | Avg Chars | Methods Covered | Behaviors Covered |
| Human | Burgers | 4 | 00 67 (50–80) | 1,655 (1,332–2,102) | Explicit | Advection |
| Heat | 4 | 00 40 (36–43) | 0, 742 (631–784) | Explicit, Implicit | Diffusion | |
| Navier-Stokes | 4 | 0 128 (95–177) | 4,879 (3,356–6,459) | Explicit, Implicit, Spectral | Advection, Diffusion | |
| Wave | 4 | 00 66 (44–93) | 3,230 (998–7,666) | Explicit, Implicit, Spectral | Oscillation, Restoration | |
| Subtotal | 16 | 00 75 (36–177) | 2,626 (631–7,666) | |||
| Synthetic | Burgers | 4 | 00 36 (29–43) | 0, 983 (763–1,312) | Explicit | Advection, Diffusion |
| PDE | Code Snippet | Source |
|---|---|---|
| Heat Equation | heat_1 | Gilbert Francois - Heat Equations |
| heat_2 | Gilbert Francois - Heat Equations | |
| heat_3 | Gilbert Francois - Heat Equations | |
| heat_4 | Gilbert Francois - Heat Equations | |
| Wave Equation | wave_1 | Wave Equation Simulations [ 36 ] |
| wave_2 | Wave Equation Simulations [ 36 ] |
| Perturbation | Comments | Variables | Valid |
|---|---|---|---|
| Clean+Comment | ✓ | ✓ | ✓ |
| Clean, No Comment | ✓ | ✓ | |
| Corrupt Comment | ✗ | ✓ | ✓ |
| Obf Variable | ✗ | ✓ | |
| Invalid+Comment | ✓ | ✓ | ✗ |
| Invalid, No Comment | ✓ | ✗ |
| LLM | Model Version |
|---|---|
| R1-Distill-32B | deepseek-ai/DeepSeek-R1-Distill-Qwen-32B |
| QwQ-32B | Qwen/QwQ-32B |
| Qwen3-32B | Qwen/Qwen3-32B |
| Nemotron-3-Nano-30B | nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 |
| GLM-4.7-Flash | zai-org/GLM-4.7-Flash |
| Qwen3.5-27B | Qwen/Qwen3.5-27B |
| Factor | Levels | Description |
|---|---|---|
| Solver system | 32 | 4 PDE families: Burgers, Heat, Navier–Stokes, and Wave (8 each) |
| Corruption condition | 8 | 1 consistent condition and 7 corrupted conditions, including 4 trajectory corruptions |
| Identifier regime | 2 | informative identifiers or semantically meaningless identifiers |
| Presentation order | 2 | two orderings of the four views per system–condition–identifier combination |
| Arm | Model Version | Thinking Control | Setting |
|---|---|---|---|
| 2.5-flash-tb0 | gemini-2.5-flash | thinking_budget | (disabled) |
| 2.5-flash-tb8192 | gemini-2.5-flash | thinking_budget | tokens |
| 3.7-flash-low | gemini-3.7-flash | thinking_level | LOW |
| 3.7-flash-high | gemini-3.7-flash | thinking_level | HIGH |
| Tool | Purpose | Behavior |
|---|---|---|
| edit_source | Modify and execute the simulator to collect new evidence. | The agent provides a complete replacement source file, which is executed in the episode sandbox. Printed output is returned to the model, and numeric data may be saved to disk for later analysis. An empty call reruns the latest version of the simulator. Consumes one investigative action. |
| run_diagnostic | Analyze source code or evidence collected from previous executions. | Runs a standalone Python diagnostic provided by the agent without rerunning the simulator. It may read the current source files and numeric files previously saved by edit_source . Consumes one investigative action. |
| submit_final_answer | Submit the revised PDE, method, behavior, and physical-validity judgment. | The answer must concern the original solver and may cite collected execution evidence. The model reports a binary validity judgment, a 0–10 physical-validity score, and brief explanations. Does not consume the investigative budget. |
| Measure | Pooled ( ) | Gemini 2.5 Flash ( ) | Gemini 3.7 Flash ( ) |
|---|---|---|---|
| Exact category (none / some / all) | 18/20 (90%) [70–97%] | 9/10 | 9/10 |
| Named 1 real defect | 19/20 (95%) [76–99%] | 9/10 | 10/10 |
| Incorrect code claims | 18/20 (90%) [70–97%] | 8/10 | 10/10 |
| Named 1 real defect | False code claims | |||
|---|---|---|---|---|
| Expert | Judge: yes | Judge: no | Judge: yes | Judge: no |
| yes | 16 (6 / 10) | 1 (1 / 0) | 4 (4 / 0) | 0 |
| no | 0 | 3 (3 / 0) | 2 (2 / 0) | 14 (4 / 10) |
Scientific Reasoning in Language Models
Scientific reasoning in language models studies how AI systems use evidence, domain knowledge, and tools to solve problems and support discovery.
See all 167 papers