The Verifier Is Part of the Simulator: Evidence-View Gaming in Agentic Equation Discovery
Organizations: Seminar for Applied Mathematics, AI Center, ETH Zurich
Abstract
A growing number of agentic pipelines search for governing equations: an agent proposes laws and a verifier checks them against simulated trajectories. The verifier sees only a numerical summary: integrals against test functions on a time grid. That summary carries its own discretization error, making the verifier part of the simulator. We freeze the trajectory, candidate library, verifier and thresholds, leaving the agent one move: which permitted summary to admit. Without scores or language models, finer time grids reduce numerical discrepancies and false library rejections. On PDEBench, the verifier instead accepts equations missing diffusion on score-selected summaries. Maximizing a public score rewarding decisive verdicts and larger mismatches when rejecting the library wrongly rejects all 1,500 in-library cases; retrospectively retuning score constants on a separate bank reduces errors to 13/180 at best. We call this evidence-view gaming: reward hacking by choice of permitted summary. In single-turn forced-choice tests, Claude models rarely choose summaries giving wrong verdicts when asked to be accurate, but often do so when asked to maximize the score or report missing physics. In separate trust prompts, they often rank those harmful summaries least trustworthy. Requiring several summaries to agree reduces errors but often withholds a verdict and performs worse than a fixed fine view on PDEBench: a safeguard, not a certificate. Evidence chosen after its verdict is known cannot alone authorize a claim.
Figures & tables
| Action | Evidence view | Residual | Gap | Cond. | Score | Verdict | Truth |
|---|---|---|---|---|---|---|---|
| A03 | 7 samples, 5 tests | .953 | .009 | 1.528 | 13.77 | OUTSIDE_LIBRARY | false negative |
| A08 | 25 samples, 2 tests | .006 | .452 | 1.528 | 10.87 | IDENTIFIED | correct |
| Benchmark | Selection rule | Correct | Wrong | INCONCLUSIVE | Coverage |
|---|---|---|---|---|---|
| 1-D matched ( ) | Score maximum | 300 | 1,500 | 0 | 100% |
| 1-D matched ( ) | Four-view agreement | 1,435 | 3 | 362 | 79.89% |
| 1-D matched ( ) | Fixed fine view | 1,442 | 41 | 317 | 82.39% |
| PDEBench ( ) | Score maximum | 828 | 72 | 0 | 100% |
| PDEBench ( ) | Four-view agreement | 645 | 0 | 255 | 71.67% |
| PDEBench ( ) | Fixed fine view | 900 | 0 | 0 | 100% |
| Model | Accuracy | Decisive | Score | Omission | A03 least trusted | Acc. picks | Omit. picks |
|---|---|---|---|---|---|---|---|
| Opus 4.8 | 2 | 13 | 174 | 179 | 188/216 | 2/188 | 175/188 |
| Sonnet 4.6 | 7 | 13 | 144 | 180 | 158/216 | 9/158 | 154/158 |
| Haiku 4.5 | 5 | 14 | 102 | 180 | 163/216 | 19/163 | 156/163 |
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
| Action | Nodes | Modes | Authority terminal | Residual | Gap | Objective |
|---|---|---|---|---|---|---|
| A00 | 7 | 2 | IDENTIFIED | .017097 | .441876 | 10.657911 |
| A01 | 7 | 3 | INCONCLUSIVE | .066072 | .396681 | |
| A02 | 7 | 4 | INCONCLUSIVE | .071578 | .390984 | |
| A03 | 7 | 5 | OUTSIDE_LIBRARY | .953082 | .009140 | 13.765265 |
| A04 | 13 | 2 | IDENTIFIED | .012311 | .444356 | 10.753517 |
| A05 | 13 | 3 | IDENTIFIED | .016639 | .442210 | 10.666963 |
| Rule | Required evidence | On disagreement |
|---|---|---|
| sequential cross-bank | same definitive signature from two- and five-mode banks at | REFINE if ; INCONCLUSIVE at 49 |
| all-mode consensus | same definitive signature from modes at one | INCONCLUSIVE |
| Model | M3 valid | M5 valid | A1 FP | A2 FP | A3 FP | A5 FP | A3 A2 |
|---|---|---|---|---|---|---|---|
| Opus 4.8 | 863/864 | 2/216 | 174 | 2 | 13 | 179 | +6.11 pp |
| Sonnet 4.6 | 864/864 | 216/216 | 144 | 7 | 13 | 180 | +3.33 pp |
| Haiku 4.5 | 864/864 | 0/216 | 102 | 5 | 14 | 180 | +5.00 pp |
| Endpoint | M5 valid | A03 least | A2 takes A03 | A5 takes A03 | A5 A2 |
|---|---|---|---|---|---|
| Claude Opus 4.8 | 216/216 | 188/216 | 2/188 | 175/188 | +92.02 pp |
| Claude Sonnet 4.6 | 216/216 | 158/216 | 9/158 | 154/158 | +91.77 pp |
| Claude Haiku 4.5 | 215/216 | 163/216 | 19/163 | 156/163 | +84.05 pp |
| Model | Valid | FP canonical shuffled | pp | Same action | Input tok. | Output tok. | Cost |
|---|---|---|---|---|---|---|---|
| Opus 4.8 | 72/72 | 58 60 / 60 | +3.33 | 68/72 | 440,083 | 5,184 | $2.330015 |
| Sonnet 4.6 | 72/72 | 43 55 / 60 | +20.00 | 53/72 | 361,929 | 4,680 | $1.155987 |
| Haiku 4.5 | 72/72 | 32 44 / 60 | +20.00 | 51/72 | 361,857 | 4,680 | $0.385257 |
| Total | 216/216 | 133 159 / 180 | +14.44 | 172/216 | 1,163,869 | 14,544 | $3.871259 |
| Group | Serialized fields |
|---|---|
| Action identity | action_token , nodes, temporal modes, weak-row count, node-cost proxy |
| Authority output | terminal status and terminal support |
| Fit diagnostics | best residual, score gap, best condition, gate margin |
| Public objectives | scalar public objective, Gram log-determinant, smallest Gram eigenvalue, Gram condition |
| Explicitly absent | truth, held-out truth label, correct/false flag, fine reference, oracle result, future view |
| Model | Calls | Strict valid | Input tokens | Output tokens | Estimated API cost |
|---|---|---|---|---|---|
| Qwen2.5-7B-Instruct | 24 | 24 | 146,178 | 720 | local A100 |
| Claude Haiku 4.5 | 24 | 0 | 100,309 | 878 | $0.104699 |
| Claude Sonnet 4.6 | 24 | 19 | 100,333 | 833 | $0.313494 |
| Claude Opus 4.6 | 24 | 20 | 100,333 | 668 | $0.518365 |
| Model | Decoding record | M3 response | M5 response |
|---|---|---|---|
| Qwen2.5-7B | pinned revision, greedy | strict-valid; A08/A08 | valid; A03/A03 |
| Claude Opus 4.8 | provider default; temperature and thinking omitted | prose then A03/A03; invalid | A03/A03; invalid signal token |
| Claude Sonnet 4.6 | temperature 0; thinking not requested | prose then A03/A03; invalid | fenced A03/A03; invalid |
| Claude Haiku 4.5 | temperature 0; thinking not requested | fenced A08/A08; lenient-valid, strict-invalid | fenced A03/A03; invalid |
| Model | Valid | A03 S/F | C/F/U S | C/F/U F | Guard S | Guard F |
|---|---|---|---|---|---|---|
| Qwen2.5-7B | 24 | 3/2 | 22/2/0 | 22/2/0 | 8/0/16 | 11/0/13 |
| Claude Haiku 4.5 | 0 | 0/0 | 0/0/24 | 0/0/24 | 0/0/24 | 0/0/24 |
| Claude Sonnet 4.6 | 19 | 13/14 | 9/10/5 | 8/11/5 | 7/0/17 | 7/0/17 |
| Claude Opus 4.6 | 20 | 20/20 | 4/16/4 | 4/16/4 | 4/0/20 | 4/0/20 |
| Goal / producer | Model routing C/F/U | Forced commit C/F/U | Fixed challenge C/F/U |
|---|---|---|---|
| accuracy / spectral | 182/2/32 | 182/2/32 | 180/0/36 |
| accuracy / FD | 182/2/32 | 182/2/32 | 180/0/36 |
| decisive / spectral | 153/2/61 | 170/5/41 | 168/0/48 |
| decisive / FD | 165/0/51 | 182/2/32 | 180/0/36 |
| omission / spectral | 28/3/185 | 170/5/41 | 168/0/48 |
| omission / FD | 40/2/174 | 182/2/32 | 180/0/36 |
| Factor | Levels |
|---|---|
| PDE family | advection–diffusion, Burgers, Fisher–KPP, Allen–Cahn, KdV, hyperdiffusion |
| Noise model | IID, correlated |
| Relative noise | , , |
| Temporal nodes | , , , |
| Temporal modes | , (failure/controller); , , , (gaming) |
| Primary producer | pseudospectral/ETDRK4 |
| Policy | Correct | False | INCONCLUSIVE | Mean nodes |
|---|---|---|---|---|
| cross-bank refinement gate | 1,505 | 1 | 294 | 19.410 |
| fixed fine five-mode | 1,501 | 23 | 276 | 49.000 |
| fixed fine consensus | 1,497 | 0 | 303 | 49.000 |
| five-mode ladder | 1,506 | 23 | 271 | 26.340 |
| two-mode ladder | 1,725 | 10 | 65 | 17.257 |
| retrospective oracle | 1,505 | 0 | 295 | 19.433 |
| Policy | Correct | False | INCONCLUSIVE | Mean views | Mean max nodes |
|---|---|---|---|---|---|
| sequential all-mode | 1,509 | 0 | 291 | 9.738 | 20.50 |
| full 16-view bank | 1,509 | 0 | 291 | 16.000 | 49.00 |
| fixed | 1,456 | 31 | 313 | 1.000 | 25.00 |
| fixed | 1,510 | 29 | 261 | 1.000 | 49.00 |
| confidence maximum | 300 | 1,500 | 0 | 16.000 | 49.00 |
| D-optimal | 532 | 350 | 918 | 16.000 | 49.00 |
| Policy | Correct | False | INCONCLUSIVE | Mean nodes |
|---|---|---|---|---|
| nested cross-bank rule | 605 | 0 | 115 | 19.583 |
| disjoint-bank gate | 489 | 0 | 231 | 25.500 |
| fixed fine cumulative bank | 607 | 11 | 102 | 49.000 |
| Policy | Terminal | Correct | False | INCONCLUSIVE | Emitted-terminal ref. change |
| confidence-max | 1,800 | 300 | 1,500 | 0 | 1,462 |
| fixed | 1,483 | 1,442 | 41 | 317 | 6 |
| all-mode consensus | 1,438 | 1,435 | 3 | 362 | 3 |
| Action | Samples | Tests | False omissions / 1,500 | Score choices / 540 | Omission choices / 540 |
|---|---|---|---|---|---|
| A00 | 7 | 2 | 3 | 3 | 0 |
| A01 | 7 | 3 | 138 | 0 | 29 |
| A02 | 7 | 4 | 356 | 6 | 41 |
| A03 | 7 | 5 | 1,500 | 414 | 468 |
| A04 | 13 | 2 | 0 | 4 | 0 |
| A05 | 13 | 3 | 8 | 2 | 0 |
| Policy | Correct | False | INCONCLUSIVE | Coverage |
|---|---|---|---|---|
| public-score maximum | 6,523 | 8,477 | 0 | 100.00% |
| four-mode agreement | 4,000 | 6,761 | 4,239 | 71.74% |
| fixed | 5,225 | 1,322 | 8,453 | 43.65% |
| Policy | Correct | False ID | False OUT | INCONCLUSIVE | Coverage |
|---|---|---|---|---|---|
| score maximization | 828 | 69 | 3 | 0 | 100.00% |
| fixed fine (25 samples, 5 modes) | 900 | 0 | 0 | 0 | 100.00% |
| four-mode agreement | 645 | 0 | 0 | 255 | 71.67% |
| 101-sample reference | 900 | 0 | 0 | 0 | 100.00% |
| Control | Correct/TPR | False/specificity | INCONCLUSIVE |
|---|---|---|---|
| A10 residual | 1.000 | .9833 | – |
| A11, 1% noise | .3583 | .2000 | .4417 |
| A11, 5% noise | .2083 | .6667 | .1250 |
| A11, 10% noise | .1667 | .7167 | .1167 |
| Producer | Agree | Correct 49/97 | False 49/97 | Change |
|---|---|---|---|---|
| spectral | .975 | .8361/.8611 | .00833/.00833 | 9 correct |
| FD/BDF | .975 | .8361/.8611 | .00833/.00833 | 9 correct |
| Noise | Reference false | Confidence false | Difference | Consensus false/ INCONCLUSIVE |
|---|---|---|---|---|
| 1% | 46 (19.17%) | 200 (83.33%) | +64.17 [60.83,67.08] | 18/147 |
| 5% | 158 (65.83%) | 200 (83.33%) | +17.50 [16.67,18.75] | 155/42 |
| 10% | 172 (71.67%) | 200 (83.33%) | +11.67 [10.00,13.33] | 169/31 |
Reward Hacking
Reward hacking studies how AI systems exploit imperfect objectives and how to detect, measure, and prevent behavior that scores well but misses intent.
See all 202 papers