Stand-In Effects and Pooling Amplification: Lessons from Biological World Models for Testing Action Effects
Organizations: Department of Biology, Stanford University, Stanford, CA, USA · Institute of Health Informatics, University College London, London, UK
Abstract
Whether an action-conditioned world model responds to actions is usually checked by replacing or removing the actions and measuring the loss in accuracy. Taking biological experiments as the guiding case, we show that such diagnostics report action effects where there are none: under autocorrelated designs, future actions stand in for history the model does not see. When the design law is known, a conditional randomization test written as an e-value is valid in finite samples whatever the model. A second failure arises whenever many sites share one action sequence, as in world models: the pooled score turns small model errors along the action contrast into an e-value near zero, so a valid test loses its power although the effect is real. Tempering the score changes the bet, not the validity. The log e-value is concave in the tempering factor and zero at zero, so too small a factor costs little and too large a one can cost everything; every single factor we tried, estimated or tuned, failed in some setting, and mixtures over a fixed grid never did. Our own preregistered analysis of optogenetic ERK experiments failed by pooling; a tempered statistic, registered before the data were obtained, then rejected the null on a never-opened field (E = 1.1e10). In three registered tests on models trained by others, the pooled test rejected in 0 of 96 units of a diffusion model, and reached the power bar in 10 of 26 Atari games under IRIS's token likelihood and 1 of 6 control tasks under TD-MPC2; mixtures fixed in advance reached it in 96 units, 24 games and 6 tasks. On recorded robot video with iVideoGPT the mixture rejected in 155 of 159 trajectories; the pooled test did not collapse there (143). Under exact nulls the statistics made 14 rejections in 26,664 tests.
Figures & tables
| Name | Label | Definition | Status |
| Pooled | C0 | ( 1 ) | Registered statistic of M4 (zxar3); secondary in ctfxv, j75g2 and 4wj7k. |
| Kish-tempered | C1 | ( 6 ), ( 7 ) | Selected on synthetic data (§ 6 ); registered statistic of M5 (qezbx); secondary in ctfxv and, for Gaussian scores, in j75g2. |
| Field-mean | C2 | § 6 | Candidate in the selection; not selected. |
| Site-average | C3 | § 6 | Candidate in the selection; not selected. |
| Contrast-tempered | C1c | ( 6 ), ( 8 ) | Exploratory; declared in the simulation protocol before the run; secondary in ctfxv and, for Gaussian scores, in j75g2. It did not repair pooling amplification on DIAMOND or on IRIS’s decoded pixels. |
| Fractional Kelly | C4 | § 3.6 | Reference arm; simulation only. |
| Statistic | Pixel, larger effect | Pixel, smaller effect | ERK-like, cells at 16/16 |
|---|---|---|---|
| Contrast-tempered (C1c) | 180/192 | 15/192 | – |
| Kish-tempered (C1) | 54/192 | 0/192 | 12/12 |
| Tempering mixture (M) | 178/192 | 2/192 | 12/12 |
| Validation-tuned (V) | 185/192 | 11/192 | – |
| Rank e-value (R) | 80/192 | 1/192 | 0/12 |
| Rank e-value (R 1/2 ) | 176/192 | 12/192 | 0/12 |
| Slot | Data | Horizon | Decision | |
|---|---|---|---|---|
| 1 | ERK (positive control) | 10 min | 9576.45 | not rejected |
| 2 | ERK (positive control) | 30 min | 2974.53 | not rejected |
| 3 | CcaS, four-model average | 1 step | 1.68 | not rejected |
| 4 | CcaS, four-model average | 6 steps | 3466.13 | rejected |
| 5 | CcaS, published model | 12 steps | 17,572.64 | rejected |
| Statistic | 10 min | models | 30 min | models |
|---|---|---|---|---|
| C0 | 9,576 | 4 | 2,975 | 4 |
| C1 | 349 | 0 | 218 | 0 |
| M | 347 | 0 | 218 | 0 |
| M13 | 347 | 0 | 218 | 0 |
| V | 210 | 2 | 101 | 4 |
| W(1/4) | 346 | 0 | 218 | 0 |
| Statistic | 10 min | 30 min |
|---|---|---|
| Validation-tuned (V) | 210 | |
| V ρ | 350.5 | 206.1 |
| Kish (C1) | 349 | 217.5 |
| Mixture (M) | 346.6 | 218.2 |
| M ρ | 347.8 | 220.8 |
| W ρ (3/4) | 350.3 | 219.5 |
| Boxing | Pong | Breakout | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Statistic | A/32 | N/64 | A/32 | N/64 | A/32 | N/64 | |||
| Pooled (C0) | 0 | 222,582 | 0 | 0 | 21,805 | 0 | 0 | 5408 | 0 |
| Kish-tempered (C1) | 0 | 2698 | 0 | 0 | 383 | 0 | 0 | 1585 | 0 |
| Contrast-tempered (C1c) | 0 | 4012 | 0 | 0 | 975 | 0 | 1 | 95.3 | 0 |
| Validation-tuned (V) | 32 | 454 | 0 | 32 | 360 | 0 | 32 | 129 | 0 |
| Tempering mixture (M, primary) | 32 | 455 | 0 | 32 | 358 | 0 | 32 | 126 | 0 |
| Statistic | IRIS tokens (26) | med | NULL (1,664) | IRIS pixels units (96) | NULL (192) | TD-MPC2 (6) | med | NULL (384) |
|---|---|---|---|---|---|---|---|---|
| Pooled (C0) | 10 | 14 | 0 | 0 | 0 | 1 | 4 | 0 |
| Kish (C1) | – | – | – | 0 | 0 | 2 | 4 | 0 |
| Contrast (C1c) | – | – | – | 20 | 0 | 4 | 2 | 0 |
| Validation-tuned (V) | 24 | 0 | 0 | 96 | 0 | 6 | 0 | 0 |
| Mixture (M) | 22 | 0 | 1 | 96 | 0 | 5 | 0 | 0 |
| Mixture, 13 values (M13) | 22 | 0 | 2 | 96 | 0 | 5 | 0 | 0 |
| Statistic | NULL L-emp | NULL L-par | ACTIVE L-emp | Median | ACTIVE L-par | Median |
|---|---|---|---|---|---|---|
| Pooled (C0) | 0 | 0 | 143 | 13.9 | 142 | 13.8 |
| Validation-tuned (V) | 0 | 0 | 155 | 15.5 | 156 | 15.4 |
| Mixture (M) | 1 | 0 | 154 | 13.4 | 155 | 13.1 |
| W(3/4), primary | 0 | 0 | 155 | 15.3 | 156 | 15.1 |
| W(1/2) | 0 | 0 | 155 | 15.1 | 156 | 14.7 |
| W(1/4) | 1 | 0 | 155 | 14.4 | 156 | 14.1 |
| Game | Best factor | Break-even | Kish | Contrast |
|---|---|---|---|---|
| Boxing | 0.0016424 | 0.0046453 | 0.01812 | 0.024074 |
| Pong | 0.0046453 | 0.026278 | 0.048787 | 0.077295 |
| Breakout | 0.0039062 | 0.03125 | 0.32019 | 0.05298 |
| Statistic | Pixel world | ERK-like | ERK test fields | DIAMOND | IRIS tokens | IRIS pixels | TD-MPC2 | Robot video |
|---|---|---|---|---|---|---|---|---|
| Pooled (C0) | 2 | 1 | 9,576 / 2,975 | 0 | 10 | 0 | 1 | 143 |
| Kish (C1) | 54 | 12 | 349 / 218 | 0 | – | 0 | 2 | – |
| Contrast (C1c) | 180 | – | – | 1 | – | 20 | 4 | – |
| Validation-tuned (V) | 185 | – | 210 / 101 | 96 | 24 | 96 | 6 | 155 |
| Mixture (M) | 178 | 12 | 347 / 218 | 96 | 22 | 96 | 5 | 154 |
| W(3/4) | 185 | – | 345 / 217 | 96 | 24 | 96 | 6 | 155 |
Appendix figures & tables88 assets
Supplementary material from the paper’s appendix.
Appendix
| Document | Covers | SHA-256 |
|---|---|---|
| Core theory | P1(a)–(d), P1b, P2, P3, P4 | c726f679…c2c8c |
| Calibration supplement | P1(e), template HMM | 0a35b031…57081 |
| Pilot note | model averaging, ceilings | 81f39548…8c452a |
| Family | Status | Test / baseline | Minimum | Maximum |
|---|---|---|---|---|
| masked_shared | alternative | Test-exact | 0/256 | 97/256 |
| masked_shared | alternative | Test-MC | 0/256 | 86/256 |
| masked_shared | alternative | Batch-CRT | 0/256 | 87/256 |
| masked_shared | alternative | B-zero | 49/256 | 192/256 |
| masked_shared | alternative | B-swap | 65/256 | 132/256 |
| masked_shared | alternative | B-abl | 4/256 | 134/256 |
| Family | Verbatim description | Settings | Null k/K | CP95 upper |
|---|---|---|---|---|
| F01 | Semi-Markov / cubic memory | A, N | 0/200 | 1.5% |
| F02 | Time balance / threshold response | A, N | 0/200 | 1.5% |
| F03 | Full-history urn | A, N | 0/400 | 0.75% |
| F04 | Omitted-history proxy trap | A, N | 0/400 | 0.75% |
| F05 | Hidden-state nonlinear ODE | A, N | 0/200 | 1.5% |
| F06 | Heteroscedastic Student t3 | A, N | 0/200 | 1.5% |
| Cell | C0: k/K; log E | C1: k/K; log E | C2: k/K; log E | C3: k/K; log E | Best all / fixes |
|---|---|---|---|---|---|
| S1 null; rho=0; h=10 | 0/32; 33.3 | 0/32; 19.4 | 0/32; 18.4 | 0/32; 0.00 | C3 / C3 |
| S1 null; rho=0.243; h=10 | 0/32; 4101 | 0/32; 0.15 | 0/32; 0.15 | 0/32; 0.02 | C3 / C3 |
| S1; rho=0; h=10; signal =0.10 [hardest] | 0/16; 314 | 16/16; 132 | 16/16; 131 | 16/16; 83.6 | C1 / C1 |
| S1; rho=0.243; h=10; signal =0.10 | 0/16; 5992 | 16/16; 124 | 16/16; 124 | 16/16; 80.3 | C2 / C2 |
| S1; rho=0; h=10; signal =0.30 | 0/16; 1707 | 16/16; 133 | 16/16; 133 | 16/16; 127 | C2 / C2 |
| S1; rho=0.243; h=10; signal =0.30 | 0/16; 6415 | 16/16; 134 | 16/16; 134 | 16/16; 127 | C1 / C1 |
| Registration | Item | Recorded value / file | SHA-256 |
|---|---|---|---|
| zxar3 | Registration type | Open-Ended Registration (OSF registration page, 2026-10-28) | |
| zxar3 | Registered UTC | 2026-10-08T04:04:00Z | |
| zxar3 | Original display | Oct 7, 2026, 9:04 PM (as displayed by OSF) | |
| zxar3 | Approval evidence | OSF project activity log: approved 2026-10-10 05:00 UTC (displayed Oct 9, 10:00 PM PDT), after the M4 data access; manual or automatic approval not shown | 7ce2a1c36780 |
| zxar3 | Embargo end | 2027-01-31 (date only); ended early 2026-10-28 | |
| zxar3 | Linked project | https://osf.io/w37um |
| Event | UTC | Record / qualification | SHA-256 |
|---|---|---|---|
| Registration | 2026-10-08T04:04:00Z | User-supplied UTC mapping | |
| Published member extracted | 2026-10-08T04:16:54.887633Z | Archive member extraction; not model-load completion | 046df80cd770 |
| First preflight: stage_start | 2026-10-08T04:20:10.096072Z | Model architecture validation occurred during this preflight; exact load completion not separately timestamped | |
| First preflight: first_authorized_published_weight_hash | 2026-10-08T04:20:17.293604Z | Model architecture validation occurred during this preflight; exact load completion not separately timestamped | 046df80cd770 |
| First preflight: preflight_complete | 2026-10-08T04:20:54.493201Z | Model architecture validation occurred during this preflight; exact load completion not separately timestamped | 046df80cd770 |
| TRAIN start | 2026-10-08T04:21:35.395074Z |
| Event | UTC | Record / qualification | SHA-256 |
|---|---|---|---|
| M5 submitted / frozen | 2026-10-24T07:16:00Z | Created/submitted; OSF log initiation 07:16 UTC | |
| OSF approval (activity log) | 2026-10-24T07:20Z | Logged approval of the M5 registration, 4 min after submission | 7ce2a1c36780 |
| Approval evidence: browser visit | 2026-10-24T07:24:00Z | Self-reported; after the logged approval | 028b80f646c1 |
| GO record | NOT RECORDED | Authorization date 2026-10-24; no exact creation timestamp | 2e28b2c51a8e |
| GO verified / invocation | 2026-10-24T07:34:15.866396Z | PASS | 2e28b2c51a8e |
| Permanent marker claim | 2026-10-24T07:34:15.926548Z | REAL | 7361561f46f1 |
| stage | role | head |
|---|---|---|
| gate_20261004 | Initial feasibility gate | NOT RECORDED |
| gate2_20261005 | Design-law and novelty gate | NOT RECORDED |
| m1_designlaw_20261005 | M1 command/design gate | 9cee019748fb7344a64333e8954f891f37551aba |
| m1b_ccas_20261005 | M1b CcaS kernel diagnostics | a30eb829419b5be65516e31a0cf88e1872fa9b4f |
| m2_method | M2a method and DEV | b0ad5976ff954a3b85d7d1204ab60a341bec5907 |
| m2b_calibration | M2b semi-synthetic calibration | 604b13cfbf20f2c970b737e4023a500a9e90dc3b |
| directory | commit | file | sha256 |
|---|---|---|---|
| theory_review_20261026/claude | 8e608df1961ee2d117e28f74b0671d1e8542385e | REVIEW_REPORT.md | 10f0a48c88a3d875f3c8da7cf3be6e18e365dc7b0db3c64617ecfe8a5ff26832 |
| theory_review_20261026/claude | 8e608df1961ee2d117e28f74b0671d1e8542385e | FIXES.md | dbb8f114f2c3a0aafc088cf9d24bbb44ca7397265d80d06f66376e89b13b2186 |
| theory_review_20261026/claude | 8e608df1961ee2d117e28f74b0671d1e8542385e | outputs/SUMMARY.json | 6b46e3662f04afb91a498bbc7452ef197683dc8361c52c7910d1dc4d2efb86ba |
| theory_review_20261026/claude | 8e608df1961ee2d117e28f74b0671d1e8542385e | README.md | c31ab35b1d17c67741c12e04e9805e37c31e023db6ba04389c3af49baa66a66a |
| theory_review_20261026/claude | 8e608df1961ee2d117e28f74b0671d1e8542385e | review_claude.bundle | f80be8168165a12157dfbfc7083dac632b8df1f8997f86f1ae3fe13557399ae5 |
| refs_review_20261026/claude | 808ab6ad621b3a149a2f686df5db87c2179c5725 | REFS_REPORT.md | 7f4926922799caab5a17706cb0b9813c3c4d881a033ee8d2f65aae740f1adbfb |
| Field | Model | h | |||
|---|---|---|---|---|---|
| Pooled | LG | 10 | 0.227 | 0.197 | 0.143 |
| Pooled | GRU | 10 | 0.229 | 0.213 | 0.127 |
| Pooled | TF | 10 | 0.251 | 0.237 | 0.203 |
| Pooled | GB | 10 | 0.205 | 0.203 | 0.168 |
| Pooled | LG | 30 | 0.127 | 0.127 | 0.0966 |
| Pooled | GRU | 30 | 0.115 | 0.102 | 0.0657 |
| Field | Model | h | n mean | n min/max | ||
|---|---|---|---|---|---|---|
| Pooled | LG | 10 | 0.631 | 0.725 | 91.2 | 64 / 100 |
| Pooled | GRU | 10 | 0.554 | 0.598 | 91.2 | 64 / 100 |
| Pooled | TF | 10 | 0.806 | 0.854 | 91.2 | 64 / 100 |
| Pooled | GB | 10 | 0.818 | 0.828 | 91.2 | 64 / 100 |
| Pooled | LG | 30 | 0.759 | 0.760 | 90.9 | 63 / 100 |
| Pooled | GRU | 30 | 0.574 | 0.645 | 90.9 | 63 / 100 |
| Grid | Noise | R | K | C0 | C1 | C1c | C2 | C3 | C4 | NC | |||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| G2 | NONE | 16 | 16 | 64 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| G2 | NONE | 16 | 64 | 64 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| G2 | NONE | 16 | 256 | 64 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| G2 | GAIN | 16 | 16 | 64 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| G2 | GAIN | 16 | 64 | 64 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| G2 | GAIN | 16 | 256 | 64 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Grid | Noise | R | K | C0 | C1 | C1c | C2 | C3 | C4 | NC | |||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| G2 | NONE | 16 | 16 | 64 | 0 | 0 | 0 | 2 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| G2 | NONE | 16 | 64 | 64 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| G2 | NONE | 16 | 256 | 64 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| G2 | GAIN | 16 | 16 | 64 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| G2 | GAIN | 16 | 64 | 64 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| G2 | GAIN | 16 | 256 | 64 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| n | Effect | C0: median ; rate | C1: median ; rate | C1c: median ; rate | C2: median ; rate | C3: median ; rate | C4: median ; rate |
|---|---|---|---|---|---|---|---|
| 64 | d_lo | 6.11; 0% | 6.11; 0% | 0.53; 12.5% | 0.18; 0% | 0.09; 0% | 0.49; 12.5% |
| 64 | d_hi | 17.3; 6.2% | 17.3; 6.2% | 10.2; 100% | 0.18; 0% | 1.13; 0% | 5.61; 75% |
| 256 | d_lo | 83.0; 0% | 83.0; 0% | 0.59; 6.2% | 0.41; 0% | 0.07; 0% | 9.37; 0% |
| 256 | d_hi | 271; 0% | 271; 0% | 10.2; 100% | 0.57; 0% | 0.95; 0% | 8.72; 0% |
| 1024 | d_lo | 657; 0% | 657; 0% | 0.76; 6.2% | 0.68; 0% | 0.08; 0% | 21.2; 0% |
| 1024 | d_hi | 1430; 0% | 1430; 0% | 7.15; 81.2% | 0.73; 0% | 0.71; 0% | 21.6; 0% |
| n | Effect | C0: median ; rate | C1: median ; rate | C1c: median ; rate | C2: median ; rate | C3: median ; rate | C4: median ; rate |
|---|---|---|---|---|---|---|---|
| 64 | d_lo | 5.15; 0% | 0.47; 0% | 1.36; 6.2% | 0.03; 0% | 0.09; 0% | 0.32; 6.2% |
| 64 | d_hi | 11.2; 0% | 5.29; 87.5% | 10.3; 87.5% | 0.08; 0% | 0.94; 0% | 4.87; 75% |
| 256 | d_lo | 107; 0% | 0.47; 0% | 0.78; 12.5% | 0.07; 0% | 0.10; 0% | 9.39; 0% |
| 256 | d_hi | 228; 0% | 4.74; 100% | 9.75; 93.8% | 0.07; 0% | 0.88; 0% | 9.39; 6.2% |
| 1024 | d_lo | 609; 0% | 0.33; 0% | 0.63; 0% | 0.17; 0% | 0.07; 0% | 21.7; 0% |
| 1024 | d_hi | 1555; 0% | 4.03; 93.8% | 7.90; 87.5% | 0.07; 0% | 0.76; 0% | 22.0; 0% |
| n | Effect | C0: median ; rate | C1: median ; rate | C1c: median ; rate | C2: median ; rate | C3: median ; rate | C4: median ; rate |
|---|---|---|---|---|---|---|---|
| 64 | d_lo | 4.66; 0% | 4.66; 0% | 1.11; 0% | 0.11; 0% | 0.08; 0% | 0.32; 6.2% |
| 64 | d_hi | 12.8; 6.2% | 12.8; 6.2% | 8.86; 100% | 0.19; 0% | 0.91; 0% | 6.29; 75% |
| 256 | d_lo | 101; 0% | 101; 0% | 0.13; 6.2% | 0.64; 0% | 0.07; 0% | 9.80; 0% |
| 256 | d_hi | 285; 0% | 285; 0% | 7.16; 87.5% | 0.64; 0% | 0.79; 0% | 8.93; 0% |
| 1024 | d_lo | 620; 0% | 596; 0% | 0.21; 6.2% | 0.77; 0% | 0.05; 0% | 22.9; 0% |
| 1024 | d_hi | 1542; 0% | 1542; 0% | 9.25; 100% | 0.84; 0% | 0.86; 0% | 21.3; 0% |
| Cell | Model | CD-drop | CD-zero | CD-shuffle |
|---|---|---|---|---|
| G2_R16_NONE_N16 | M-CNN | 92.2% | 98.4% | 100% |
| G2_R16_NONE_N16 | M-LG | 64.1% | 65.6% | 100% |
| G2_R16_NONE_N64 | M-CNN | 92.2% | 98.4% | 100% |
| G2_R16_NONE_N64 | M-LG | 90.6% | 93.8% | 98.4% |
| G2_R16_NONE_N256 | M-CNN | 98.4% | 100% | 100% |
| G2_R16_NONE_N256 | M-LG | 87.5% | 87.5% | 100% |
| Cell (grid, R, noise, d/N) | CNN | CNN | CNN | LG | LG | LG |
|---|---|---|---|---|---|---|
| G2_R16_NONE_N16 | 0 | 1.00 | 14.4 | 0 | 1.00 | 9.20 |
| G2_R16_NONE_N64 | 0 | 1.00 | 15.8 | 0 | 1.00 | 12.4 |
| G2_R16_NONE_N256 | 0 | 1.00 | 13.9 | 0 | 1.00 | 12.5 |
| G2_R16_GAIN_N16 | 0 | 1.00 | 15.9 | 0 | 1.00 | 9.93 |
| G2_R16_GAIN_N64 | 0 | 1.00 | 13.9 | 0 | 1.00 | 11.9 |
| G2_R16_GAIN_N256 | 0 | 1.00 | 13.7 | 0 | 1.00 | 13.0 |
| Cell (grid, R, noise, d/N) | |||
|---|---|---|---|
| G2_R16_NONE_N16 | 1.10 | 1.00 | |
| G2_R16_NONE_N64 | 0 | 1.00 | 1.00 |
| G2_R16_NONE_N256 | 0 | 1.00 | 1.00 |
| G2_R16_GAIN_N16 | 0.00403 | 2.03 | 1.00 |
| G2_R16_GAIN_N64 | 0.00392 | 2.00 | 1.00 |
| G2_R16_GAIN_N256 | 0.00363 | 1.93 | 1.00 |
| Cell | Model | Frozen NC C1 | POST HOC wrong C1c | POST HOC comparison: sealed correct C1c |
|---|---|---|---|---|
| G2_R16_NONE_N16 | M-CNN | 0/64 | 63/64 | 0/64 |
| G2_R16_NONE_N16 | M-LG | 0/64 | 60/64 | 0/64 |
| G2_R16_NONE_N16 | AVERAGE | 0/64 | 63/64 | 0/64 |
| G2_R16_NONE_N64 | M-CNN | 0/64 | 63/64 | 0/64 |
| G2_R16_NONE_N64 | M-LG | 0/64 | 61/64 | 0/64 |
| G2_R16_NONE_N64 | AVERAGE | 0/64 | 63/64 | 0/64 |
| display_file | commit prefix | SHA prefix | commit_utc |
|---|---|---|---|
| protocol/P1_PROTOCOL_v1.md | 9dcbfa308cac | e9e232b6da79 | 2026-11-04T23:18:09+00:00 |
| protocol/P1_PROTOCOL_AMENDMENT_01.md | fb939b1ba2c7 | 6cd5f3a9d062 | 2026-11-05T11:32:44+00:00 |
| notes/ADAPTER_MAP_20261105.md | 1d35d7180165 | 4ef082324008 | 2026-11-05T07:00:11+00:00 |
| runs/smoke_20261105/UNIT_TESTS.txt | 1d35d7180165 | afef9f1d90a8 | 2026-11-05T07:00:11+00:00 |
| runs/full_20261106/GATE_SUMMARY.tsv | 70c059894e40 | 1f7a9eb42f42 | 2026-11-05T11:20:34+00:00 |
| runs/full_20261106/SUMMARY.json | 70c059894e40 | 240866741f55 | 2026-11-05T11:20:34+00:00 |
| item | value |
|---|---|
| primary R | Clearly weaker |
| primary M | Matches C1c |
| primary V | Matches C1c |
| secondary R | Clearly weaker |
| secondary M | Clearly weaker |
| secondary V | Similar |
| noise | frame | effect | C1 | C1c | M | V | R | R 1/2 | ORACLE_R |
|---|---|---|---|---|---|---|---|---|---|
| NONE | R8 | d0 | 0/32 | 0/32 | 0/32 | 0/32 | 0/32 | 0/32 | 0/32 |
| NONE | R8 | dlo | 0/16 | 2/16 | 0/16 | 1/16 | 0/16 | 1/16 | 16/16 |
| NONE | R8 | dhi | 1/16 | 16/16 | 16/16 | 16/16 | 10/16 | 16/16 | 16/16 |
| NONE | R16 | d0 | 0/32 | 0/32 | 0/32 | 0/32 | 0/32 | 0/32 | 0/32 |
| NONE | R16 | dlo | 0/16 | 1/16 | 0/16 | 1/16 | 0/16 | 2/16 | 16/16 |
| NONE | R16 | dhi | 0/16 | 16/16 | 15/16 | 16/16 | 9/16 | 15/16 | 16/16 |
| cell | arm | rejections | median_log_E |
|---|---|---|---|
| M2e_h10_f0.10_rho0 | C0 | 16/16 | 95.2 |
| M2e_h10_f0.10_rho0 | C1 | 16/16 | 124 |
| M2e_h10_f0.10_rho0 | M | 16/16 | 134 |
| M2e_h10_f0.10_rho0 | R | 0/16 | 111 |
| M2e_h10_f0.10_rho0 | R 1/2 | 1/16 | 6.09 |
| M2e_h10_f0.10_rho0.243 | C0 | 0/16 | 10,494 |
| item | value | utc |
|---|---|---|
| DIAMOND code commit | 5bcd1599755b4f2fae8e5e079e02f0728e174965 | |
| Boxing checkpoint revision | 5d4abca9af6ab1b3ab1c6fc228cacb7281130f7f | |
| Boxing checkpoint SHA-256 | fcb54ee18a7a19371800c86f18afcf78050bd453df99878601d3be9e0e7a5e07 | |
| Pong checkpoint revision | 5d4abca9af6ab1b3ab1c6fc228cacb7281130f7f | |
| Pong checkpoint SHA-256 | addd34bd91f8c8e9c5c7f954c873d4db9fb42440c9028e93bf5e434c5ecb9042 | |
| Breakout checkpoint revision | 5d4abca9af6ab1b3ab1c6fc228cacb7281130f7f |
| game | condition | statistic | rejections | median_log_E |
|---|---|---|---|---|
| Boxing | ACTIVE | C0 | 0 | 222,582 |
| Boxing | ACTIVE | C1 | 0 | 2698 |
| Boxing | ACTIVE | C1c | 0 | 4012 |
| Boxing | ACTIVE | V | 32 | 454 |
| Boxing | ACTIVE | M | 32 | 455 |
| Boxing | ACTIVE | M13 | 32 | 455 |
| Game | Cond. | Effects | Undef. | Empty | |||
| Boxing | ACTIVE | 0.0181 | 0.0241 | 0.00195 | 32 | none | none |
| Boxing | NULL | 0.0227 | 0.0178 | 0.0000610 | 3 | none | none |
| Pong | ACTIVE | 0.0488 | 0.0773 | 0.00391 | 32 | none | none |
| Pong | NULL | 0.0559 | 0.0375 | 0.0000610 | 1 | none | none |
| Breakout | ACTIVE | 0.320 | 0.0530 | 0.00391 | 32 | none | none |
| Breakout | NULL | 0.258 | 0.0472 | 0.000122 | 1 | none | none |
| item | probability | value |
|---|---|---|
| Boxing C_VAL | True | |
| Boxing C_COLLAPSE | True | |
| Boxing C_REPAIR | True | |
| Pong C_VAL | True | |
| Pong C_COLLAPSE | True | |
| Pong C_REPAIR | True |
| Game | K | c | V | Peak | V/peak | c/peak | Margin |
|---|---|---|---|---|---|---|---|
| Boxing | 0.0181 | 0.0241 | 0.00195 | 0.00164 | 1.19 | 14.7 | 451 |
| Breakout | 0.320 | 0.0530 | 0.00391 | 0.00328 | 1.19 | 16.1 | 76 |
| Pong | 0.0488 | 0.0773 | 0.00391 | 0.00465 | 0.84 | 16.6 | 29 |
| Event | UTC | Record |
|---|---|---|
| Protocol v0 committed | 2026-11-08T08:36:55Z | Draft protocol; hash given above. |
| Build pilot complete | 2026-11-08T10:23:04Z | NULL full-pipeline pilot and exact replay; ACTIVE predictions and validation quantities only, no e-value. |
| Protocol v1 committed | 2026-11-08T10:29:03Z | Protocol fixed before implementation. |
| Confirmatory code frozen | 2026-11-08T11:30:14Z | Commit c1f0e5f130c0674c29034517ffc7c733712d50a8. |
| Registration | 2026-11-09T03:33:44.537736Z | j75g2, registered from component 7mx6p of project w37um. |
| Beacon scheduled | 2026-11-09T03:35:00Z | Drand round 6,625,476. |
| Benchmark | Task | SHA-256 | Cap |
|---|---|---|---|
| DMControl | cartpole-balance (seed 3) | 0e2c0eada8f160dd65c8faaa5e4ca2f8f496104e21433e334d716b73257a52c2 | 500 |
| DMControl | reacher-easy | 4e415fc308fed6b46a944cbdbaf8f1551e73b43b2d6770c78f440bd36b61deb3 | 500 |
| DMControl | walker-walk | 6b22a70c41e1cf49a8875457afb8b1f643fcba0c6a1f9c3b2e289790e7d8a64b | 500 |
| MyoSuite | myo-pose | 2f584f4eea5ee4901ef76d0cee3ce749eeda3ff34a771da7916fde6493b1eaa9 | 100 |
| MyoSuite | myo-reach | f5db009da87c470d09c2b6680fccc9ecaffe969a8e6026176f0a8caea1218c2f | 100 |
| MyoSuite | myo-obj-hold | cee612ede3ac6d3755f586b279958d6ffc7724a4efeb0a1826b16c7ec4722a4e | 75 |
| Manifest entry | SHA-256 |
|---|---|
| Alien | 5e75195c53d33c64649f303836da97b7ad2e9f13d5f73b98c40fc21fa1352152 |
| Amidar | 7bac665810cca325deae7885458c675e4f70f7f68537d9740d0da35754aae819 |
| Assault | b2c0d61c253e57644206a6c1d2f8c483863b9fdb492b75ead8f6cf8af619877f |
| Asterix | da3175853640f62be5e0fff4fd2823f0139fcd62a35d091e1fbd993f972564e2 |
| BankHeist | 8a44a9151de1c2dbc42ddf9fe9600a7c47bc08e259adae28de609f3a77be5051 |
| BattleZone | 426da88aca45b59e3863c356d62b1cee6628eecfa6cdb51ec5887b662b4aa764 |
| Game/task | Condition | C0 | V | M | M13 |
|---|---|---|---|---|---|
| Alien | NULL | 0; 3,257 | 0; 0.03414 | 0; 0.782 | 0; 0.9827 |
| Alien | p1 | 26; 73.13 | 32; 266.8 | 32; 264.1 | 32; 264.2 |
| Amidar | NULL | 0; 1,312 | 0; 0.01363 | 1; 0.567 | 1; 0.6922 |
| Amidar | p1 | 1; 83.71 | 32; 80.82 | 32; 78.11 | 32; 78.26 |
| Assault | NULL | 0; 2,976 | 0; 0.09681 | 0; 0.7638 | 0; 0.9562 |
| Assault | p1 | 0; 110.4 | 32; 207.1 | 32; 204.4 | 32; 204.6 |
| Game/task | Condition | W0.25 | W0.5 | W0.75 | R |
|---|---|---|---|---|---|
| Alien | NULL | 0; 0.585 | 0; 0.3673 | 0; 0.187 | 0; 197.1 |
| Alien | p1 | 32; 265.4 | 32; 266.1 | 32; 266.5 | 29; 71.67 |
| Amidar | NULL | 1; 0.4255 | 0; 0.2596 | 0; 0.1137 | 0; 189.9 |
| Amidar | p1 | 32; 79.44 | 32; 80.13 | 32; 80.54 | 0; 60.65 |
| Assault | NULL | 0; 0.5964 | 0; 0.4007 | 0; 0.237 | 0; 176 |
| Assault | p1 | 32; 205.7 | 32; 206.4 | 32; 206.8 | 32; 97 |
| Game/task | Condition | R 1/2 |
|---|---|---|
| Alien | NULL | 0; 66.77 |
| Alien | p1 | 32; 100.6 |
| Amidar | NULL | 0; 64.16 |
| Amidar | p1 | 27; 15.07 |
| Assault | NULL | 0; 59.31 |
| Assault | p1 | 32; 107.5 |
| Game/task | Condition | C0 | C1 | C1c | V |
|---|---|---|---|---|---|
| Boxing | NULL | 0; 456,887 | 0; 97,043 | 0; 5,338 | 0; 0.2607 |
| Boxing | p1 | 0; 327,975 | 0; 61,526 | 0; 4,224 | 32; 173.7 |
| Breakout | NULL | 0; 27,543 | 0; 11,784 | 0; 1,807 | 0; 0.00004674 |
| Breakout | p1 | 0; 6,494 | 0; 3,107 | 6; 292.3 | 32; 102.6 |
| Pong | NULL | 0; 60,036 | 0; 34,046 | 0; 6,132 | 0; 0.04236 |
| Pong | p1 | 0; 4,662 | 0; 2,480 | 14; 75.26 | 32; 399.2 |
| Game/task | Condition | M | M13 | W0.25 | W0.5 |
|---|---|---|---|---|---|
| Boxing | NULL | 0; 2.675 | 0; 7.899 | 0; 1.571 | 0; 0.9236 |
| Boxing | p1 | 32; 171 | 32; 171.1 | 32; 172.3 | 32; 173 |
| Breakout | NULL | 0; 1.41 | 0; 2.062 | 0; 0.9255 | 0; 0.5184 |
| Breakout | p1 | 32; 99.89 | 32; 100 | 32; 101.2 | 32; 101.9 |
| Pong | NULL | 0; 1.69 | 0; 2.66 | 0; 1.089 | 0; 0.6124 |
| Pong | p1 | 32; 399.7 | 32; 399.8 | 32; 400.2 | 32; 400.4 |
| Game/task | Condition | W0.75 | R | R 1/2 |
|---|---|---|---|---|
| Boxing | NULL | 0; 0.5379 | 0; 204.4 | 0; 70.79 |
| Boxing | p1 | 32; 173.4 | 32; 136.4 | 32; 140.7 |
| Breakout | NULL | 0; 0.2265 | 0; 40.89 | 0; 13.57 |
| Breakout | p1 | 32; 102.3 | 31; 65.06 | 31; 47.81 |
| Pong | NULL | 0; 0.288 | 0; 136.3 | 0; 46.68 |
| Pong | p1 | 32; 400.2 | 32; 285 | 32; 195.6 |
| Game/task | Condition | C0 | C1 | C1c | V |
|---|---|---|---|---|---|
| cartpole-balance | NULL | 0; 61,915 | 0; 25,478 | 0; 324.1 | 0; 0.02645 |
| cartpole-balance | p1 | 0; 53,850 | 0; 23,635 | 0; 239.3 | 24; 6.289 |
| cartpole-balance | p0.5 | 0; 59,344 | 0; 24,746 | 0; 313.6 | 9; 1.877 |
| cartpole-balance | p0.2 | 0; 58,331 | 0; 25,108 | 0; 297 | 1; 0.2572 |
| cartpole-balance | p0.05 | 0; 59,259 | 0; 21,948 | 0; 333.1 | 0; 0.04316 |
| myo-obj-hold | NULL | 0; 2,122 | 0; 1,132 | 0; 21.42 | 0; 0.00295 |
| Game/task | Condition | M | M13 | W0.25 | W0.5 |
|---|---|---|---|---|---|
| cartpole-balance | NULL | 0; 1.616 | 0; 2.479 | 0; 1.047 | 0; 0.5823 |
| cartpole-balance | p1 | 19; 3.854 | 20; 3.989 | 22; 4.968 | 24; 5.618 |
| cartpole-balance | p0.5 | 5; 0.2186 | 5; 0.1571 | 6; 0.826 | 7; 1.309 |
| cartpole-balance | p0.2 | 0; 1.003 | 0; 1.466 | 0; 0.6001 | 0; 0.227 |
| cartpole-balance | p0.05 | 0; 1.415 | 0; 2.122 | 0; 0.9144 | 0; 0.4845 |
| myo-obj-hold | NULL | 0; 0.9085 | 0; 1.165 | 0; 0.6543 | 0; 0.3873 |
| Game/task | Condition | W0.75 | R | R 1/2 |
|---|---|---|---|---|
| cartpole-balance | NULL | 0; 0.2662 | 0; 208.1 | 0; 71.46 |
| cartpole-balance | p1 | 24; 6.008 | 0; 155.5 | 1; 38.38 |
| cartpole-balance | p0.5 | 9; 1.633 | 0; 175.9 | 0; 51.19 |
| cartpole-balance | p0.2 | 0; 0.04416 | 0; 199.2 | 0; 65.9 |
| cartpole-balance | p0.05 | 0; 0.1852 | 0; 204 | 0; 68.9 |
| myo-obj-hold | NULL | 0; 0.1768 | 0; 8.862 | 0; 3.132 |
| Game/task | Condition | ||||
|---|---|---|---|---|---|
| Alien | NULL | – | – | 0.0009766 | 10 |
| Alien | p1 | – | – | 0.25 | 2 |
| Amidar | NULL | – | – | 0.001953 | 9 |
| Amidar | p1 | – | – | 0.25 | 2 |
| Assault | NULL | – | – | 0.001953 | 9 |
| Assault | p1 | – | – | 0.25 | 2 |
| Game/task | Condition | ||||
|---|---|---|---|---|---|
| Boxing | NULL | 0.2149 | 0.01438 | 0.00006104 | 14 |
| Boxing | p1 | 0.1911 | 0.01684 | 0.0009766 | 10 |
| Breakout | NULL | 0.4308 | 0.06997 | 0.00006104 | 14 |
| Breakout | p1 | 0.4898 | 0.06458 | 0.001953 | 9 |
| Pong | NULL | 0.5706 | 0.1091 | 0.00006104 | 14 |
| Pong | p1 | 0.5679 | 0.09341 | 0.003906 | 8 |
| Game/task | Condition | ||||
|---|---|---|---|---|---|
| cartpole-balance | NULL | 0.4226 | 0.01158 | 0.00006104 | 14 |
| cartpole-balance | p0.05 | 0.3822 | 0.01206 | 0.0001221 | 13 |
| cartpole-balance | p0.2 | 0.4415 | 0.0117 | 0.0004883 | 11 |
| cartpole-balance | p0.5 | 0.4283 | 0.01197 | 0.0004883 | 11 |
| cartpole-balance | p1 | 0.4511 | 0.01212 | 0.0009766 | 10 |
| myo-obj-hold | NULL | 0.5514 | 0.02341 | 0.00006104 | 14 |
| No. | Dated prediction | Outcome | Evidence |
|---|---|---|---|
| 1 | C-VAL holds in every game and task of both arms (95%). | held | IRIS native: 26/26 C-VAL; TD-MPC2: 6/6 C-VAL |
| 2 | IRIS overall confirmation (75%). | held | IRIS: 26/26 C-VAL; 24/26 C-POWER, required 18 ; Alien: W(0.75)=32/32; Amidar: W(0.75)=32/32; Assault: W(0.75)=32/32; Asterix: W(0.75)=32/32; BankHeist: W(0.75)=16/32; BattleZone: W(0.75)=32/32; Boxing: W(0.75)=32/32; Breakout: W(0.75)=32/32; ChopperCommand: W(0.75)=32/32; CrazyClimber: W(0.75)=32/32; DemonAttack: W(0.75)=32/32; Freeway: W(0.75)=32/32; Frostbite: W(0.75)=32/32; Gopher: W(0.75)=32/32; Hero: W(0.75)=32/32; Jamesbond: W(0.75)=32/32; Kangaroo: W(0.75)=32/32; Krull: W(0.75)=32/32; KungFuMaster: W(0.75)=32/32; MsPacman: W(0.75)=26/32; Pong: W(0.75)=32/32; PrivateEye: W(0.75)=32/32; Qbert: W(0.75)=29/32; RoadRunner: W(0.75)=30/32; Seaquest: W(0.75)=32/32; UpNDown: W(0.75)=8/32 |
| 3 | TD-MPC2 overall confirmation (70%). | held | TD-MPC2: 6/6 C-VAL; 6/6 C-POWER, required 4 ; cartpole-balance: W(0.75)=24/32; myo-obj-hold: W(0.75)=32/32; myo-pose: W(0.75)=32/32; myo-reach: W(0.75)=32/32; reacher-easy: W(0.75)=32/32; walker-walk: W(0.75)=32/32 |
| 4 | IRIS native score at p = 1: C0’s median log E is below 0 in at least 2 of Boxing, Pong and Breakout (50%). | failed | 0/3 below zero, required 2 ; Boxing: C0 median log E=144.5; Pong: C0 median log E=428; Breakout: C0 median log E=121.4 |
| 5 | IRIS decoded-pixel score at p = 1: C0’s median log E is below 0 in at least 2 of the 3 games (85%). | held | 3/3 below zero, required 2 ; Boxing: C0 median log E= 327,975; Pong: C0 median log E= 4,662; Breakout: C0 median log E= 6,494 |
| 6 | TD-MPC2 at p = 1: C0’s median log E is below 0 in at least 3 of 6 tasks (50%). | held | 4/6 below zero, required 3 ; cartpole-balance: C0 median log E= 53,850; myo-obj-hold: C0 median log E=33.67; myo-pose: C0 median log E= 4,567; myo-reach: C0 median log E=58.22; reacher-easy: C0 median log E= 5,683; walker-walk: C0 median log E= 21,198 |
| Arm | Game/task | Condition | Effect reports | Undefined |
|---|---|---|---|---|
| IRIS pixels | Boxing | NULL | 2/64 | 0 |
| IRIS pixels | Boxing | p1 | 32/32 | 0 |
| IRIS pixels | Breakout | NULL | 5/64 | 0 |
| IRIS pixels | Breakout | p1 | 32/32 | 0 |
| IRIS pixels | Pong | NULL | 1/64 | 0 |
| IRIS pixels | Pong | p1 | 32/32 | 0 |
| Rule | Game | C0 | C1 | C1c | V | M | R | NULL max |
|---|---|---|---|---|---|---|---|---|
| Registered pixel | Boxing | 0; 222,582 | 0; 2,698 | 0; 4,012 | 32; 454.1 | 32; 454.5 | 32; 339 | 0 |
| Registered pixel | Pong | 0; 21,805 | 0; 382.5 | 0; 975 | 32; 360.3 | 32; 357.6 | 32; 174.8 | 0 |
| Registered pixel | Breakout | 0; 5,408 | 0; 1,585 | 1; 95.29 | 32; 128.9 | 32; 126.2 | 32; 56.32 | 0 |
| Floor 8/255 | Boxing | 0; 172,554 | 0; 19,681 | 0; 2,591 | 32; 487.9 | 32; 485.2 | 32; 359 | 0 |
| Floor 8/255 | Pong | 0; 6,065 | 0; 2,086 | 12; 34.52 | 32; 385.5 | 32; 382.8 | 32; 175.4 | 0 |
| Floor 8/255 | Breakout | 0; 700.5 | 0; 564.8 | 29; 85.29 | 32; 128.8 | 32; 129 | 32; 55.36 | 0 |
| Game | 1x1 (12,288) | 2x2 (3,072) | 4x4 (768) | 8x8 (192) |
|---|---|---|---|---|
| Boxing | 222,582 | 62,975 | 20,274 | 6,656 |
| Pong | 21,805 | 6,004 | 1,526 | 77.24 |
| Breakout | 5,408 | 1,683 | 312.7 | 66.59 |
| Prediction | Probability | Outcome |
|---|---|---|
| S0 reproduces DIAMOND study within protocol tolerances | 0.99 | supported |
| Collapse persists under S1a, S1b and S2 | 0.85 | supported |
| Collapse persists under S3a and S3b | 0.6 | supported |
| Weakens with k in at least 2 games | 0.7 | supported |
| Collapse persists at k=8, n=192 | 0.5 | supported |
| Collapse persists under S5 | 0.7 | supported |
| Game | C0 | C1 | C1c | V | M | R | R 1/2 | NULL max |
|---|---|---|---|---|---|---|---|---|
| Boxing | 0; 220,241 | 0; 2,645 | 0; 4,040 | 32; 463.1 | 32; 463.2 | 32; 343 | 32; 268.4 | 0 |
| Pong | 0; 22,354 | 0; 418.1 | 0; 969.1 | 32; 359.2 | 32; 357.8 | 32; 173.6 | 32; 153.6 | 0 |
| Breakout | 0; 5,553 | 0; 1,638 | 0; 103.6 | 32; 128.9 | 32; 126.2 | 32; 54.87 | 32; 49.49 | 0 |
| Game | Condition | |||
|---|---|---|---|---|
| Boxing | ACTIVE | 0.01811 | 0.0245 | 0.001953 |
| Boxing | NULL | 0.02288 | 0.01712 | 0.00006104 |
| Breakout | ACTIVE | 0.3204 | 0.05366 | 0.003906 |
| Breakout | NULL | 0.2583 | 0.04586 | 0.0001221 |
| Pong | ACTIVE | 0.04879 | 0.07486 | 0.003906 |
| Pong | NULL | 0.05585 | 0.03715 | 0.00006104 |
| Statistic | Simulator max null /64 | Baseline pixel max null /32 | Baseline ERK max null /32 | Baseline combined max null /32 |
|---|---|---|---|---|
| C0 | 0 | 0 | 0 | 0 |
| C1 | 0 | 0 | 0 | 0 |
| C1c | 0 | 0 | – | 0 |
| V | 0 | 0 | – | 0 |
| M | 0 | 0 | 0 | 0 |
| W(0.25) | 0 | 0 | – | 0 |
| Statistic | Mean simulator alternative rate | Simulator cells in mean | Worst decision- relevant rate | Defined decision- relevant cells |
|---|---|---|---|---|
| C0 | 0.2211 | 162 | 0 | 98 |
| C1 | 0.2224 | 162 | 0 | 98 |
| C1c | 0.4958 | 162 | 0 | 98 |
| V | 0.6348 | 162 | 0.8125 | 98 |
| M | 0.5604 | 162 | 0.5938 | 98 |
| W(0.25) | 0.588 | 162 | 0.625 | 98 |
| No. | Dated prediction | Outcome | Evidence |
|---|---|---|---|
| 1 | For > 0, C0’s median log E decreases with n in every ( , p, ) group (90%). | failed | 29 of 36 groups |
| 2 | With = 0, p = 1 and = 0, C0 rejects in at least 80% of test units at every n and level where the oracle does (80%). | held | 8 eligible cells |
| 3 | With = 0 and p 0.5, C0 collapses at n = 4,096 (70%). | held | 12 of 12 cells collapse |
| 4 | M’s deficit relative to V is at least 0.5·log 15 in at least half of the weak- cells (55%). | held | 28 of 54 cells |
| 5 | The worst-cell rate of W(0.5) is at least that of M (70%). | held | W(1/2): 0.75; M: 0.5938 |
| 6 | C1 rejects in fewer than 50% of test units in at least one alternative cell where M rejects in at least 80% (85%). | held | Stored witness cells in the readout |
| Statistic | Four-model | Linear Gaussian | GRU | Transformer | Boosting |
|---|---|---|---|---|---|
| C0 | 9,576 | 17,989 | 14,890 | 12,972 | 9,575 |
| C1 | 349 | 294.1 | 326.1 | 300.6 | 350.4 |
| M | 346.6 | 291.5 | 312.3 | 293.9 | 348 |
| M13 | 346.8 | 291.6 | 312.5 | 294 | 348.2 |
| V | 209.6 | 1,420 | 978.9 | 211 | 121.4 |
| W(1/4) | 346.4 | 291.3 | 312.1 | 293.7 | 347.8 |
| Statistic | Four-model | Linear Gaussian | GRU | Transformer | Boosting |
|---|---|---|---|---|---|
| C0 | 2,975 | 3,922 | 2,973 | 4,352 | 3,198 |
| C1 | 217.5 | 217.5 | 215 | 184.9 | 218.6 |
| M | 218.2 | 209.9 | 218.1 | 184.8 | 219.3 |
| M13 | 218.3 | 210 | 218.3 | 184.9 | 219.4 |
| V | 101.4 | 546.5 | 318.5 | 99.97 | 357.8 |
| W(1/4) | 218 | 209.7 | 217.9 | 184.6 | 219.1 |
| Min | Model | min | median | max | ||
|---|---|---|---|---|---|---|
| 10 | linear_Gaussian | 0.008769 | 0.009404 | 0.0195 | 0.125 | 3 |
| 10 | GRU | 0.008663 | 0.009291 | 0.01926 | 0.125 | 3 |
| 10 | Transformer | 0.007907 | 0.008481 | 0.0176 | 0.03125 | 5 |
| 10 | gradient_boosting | 0.009673 | 0.01037 | 0.02148 | 0.0625 | 4 |
| 30 | linear_Gaussian | 0.0155 | 0.01668 | 0.03418 | 0.25 | 2 |
| 30 | GRU | 0.0172 | 0.01851 | 0.03785 | 0.25 | 2 |
| Internal label | Name in the paper |
|---|---|
| P1 | baseline study |
| P2 | DIAMOND study (registration ctfxv) |
| P3 | theory review |
| P4 | IRIS and TD-MPC2 study (registration j75g2) |
| A1 | score-rule study |
| A1-G | sampling study |
| Score | Condition | Units | (a) pass | (b) pass | (c) pass | (d) fail |
|---|---|---|---|---|---|---|
| DIAMOND | ACTIVE | 96 | 96 | 96 | 96 | 0 |
| DIAMOND | NULL | 192 | 192 | 192 | 192 | 36 |
| IRIS pixels | NULL | 192 | 192 | 192 | 192 | 32 |
| IRIS pixels | p1 | 96 | 96 | 96 | 96 | 0 |
| IRIS tokens | NULL | 1664 | 1664 | 1664 | 1664 | 0 |
| IRIS tokens | p0.05 | 96 | 96 | 96 | 96 | 0 |
| Game/task | Condition | Units | Kish | Contrast | V | ||
|---|---|---|---|---|---|---|---|
| Boxing | ACTIVE | 32 | 0.0016424 | 0.0046453 | 0.01812 | 0.024074 | 0.0019531 |
| Boxing | NULL | 64 | 0.00012207 | 0.022728 | 0.017827 | ||
| Breakout | ACTIVE | 32 | 0.0039062 | 0.03125 | 0.32019 | 0.05298 | 0.0039062 |
| Breakout | NULL | 64 | 0.00058067 | 0.25775 | 0.047241 | 0.00012207 | |
| Pong | ACTIVE | 32 | 0.0046453 | 0.026278 | 0.048787 | 0.077295 | 0.0039062 |
| Pong | NULL | 64 | 0.00041059 | 0.055909 | 0.037484 |
| Game/task | Condition | Units | Kish | Contrast | V | ||
|---|---|---|---|---|---|---|---|
| Boxing | NULL | 64 | 0.00012207 | 0.2149 | 0.014383 | ||
| Boxing | p1 | 32 | 0.0011613 | 0.0023227 | 0.19113 | 0.016842 | 0.00097656 |
| Breakout | NULL | 64 | 0.0006356 | 0.43078 | 0.069967 | ||
| Breakout | p1 | 32 | 0.0023227 | 0.018581 | 0.48976 | 0.064584 | 0.0019531 |
| Pong | NULL | 64 | 0.00034527 | 0.57061 | 0.1091 | ||
| Pong | p1 | 32 | 0.0055243 | 0.074325 | 0.56788 | 0.093405 | 0.0039062 |
| Game/task | Condition | Units | Kish | Contrast | V | ||
|---|---|---|---|---|---|---|---|
| Alien | NULL | 64 | 0.0065695 | – | – | 0.00097656 | |
| Alien | p1 | 32 | 0.2973 | 1 | – | – | 0.25 |
| Amidar | NULL | 64 | 0.0017978 | 0.013139 | – | – | 0.0019531 |
| Amidar | p1 | 32 | 0.25 | 0.70711 | – | – | 0.25 |
| Assault | NULL | 64 | 0.0066684 | – | – | 0.0019531 | |
| Assault | p1 | 32 | 0.25 | 0.70711 | – | – | 0.25 |
| Game/task | Condition | Units | Kish | Contrast | V | ||
|---|---|---|---|---|---|---|---|
| cartpole-balance | NULL | 64 | 0.00037793 | 0.42256 | 0.011579 | ||
| cartpole-balance | p0.05 | 32 | 0.00010265 | 0.00029033 | 0.38223 | 0.012055 | 0.00012207 |
| cartpole-balance | p0.2 | 32 | 0.00029033 | 0.00082119 | 0.44146 | 0.011704 | 0.00048828 |
| cartpole-balance | p0.5 | 32 | 0.00069053 | 0.0013811 | 0.42835 | 0.011975 | 0.00048828 |
| cartpole-balance | p1 | 32 | 0.0011613 | 0.0023227 | 0.45107 | 0.012123 | 0.00097656 |
| myo-obj-hold | NULL | 64 | 0.0027621 | 0.55143 | 0.023407 |
| Game/task | Condition | Units | C0 | V | M | W(3/4) | O15 | O57 | O57h |
|---|---|---|---|---|---|---|---|---|---|
| Boxing | ACTIVE | 32 | 0 | 32 | 32 | 32 | 31 | 31 | 32 |
| Boxing | NULL | 64 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Breakout | ACTIVE | 32 | 0 | 32 | 32 | 32 | 25 | 26 | 27 |
| Breakout | NULL | 64 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Pong | ACTIVE | 32 | 0 | 32 | 32 | 32 | 31 | 31 | 31 |
| Pong | NULL | 64 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Game/task | Condition | Units | C0 | V | M | W(3/4) | O15 | O57 | O57h |
|---|---|---|---|---|---|---|---|---|---|
| Boxing | NULL | 64 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Boxing | p1 | 32 | 0 | 32 | 32 | 32 | 32 | 32 | 32 |
| Breakout | NULL | 64 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Breakout | p1 | 32 | 0 | 32 | 32 | 32 | 22 | 22 | 26 |
| Pong | NULL | 64 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Pong | p1 | 32 | 0 | 32 | 32 | 32 | 22 | 25 | 29 |
| Game/task | Condition | Units | C0 | V | M | W(3/4) | O15 | O57 | O57h |
|---|---|---|---|---|---|---|---|---|---|
| Alien | NULL | 64 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Alien | p1 | 32 | 26 | 32 | 32 | 32 | 32 | 32 | 32 |
| Amidar | NULL | 64 | 0 | 0 | 1 | 0 | 0 | 0 | 0 |
| Amidar | p1 | 32 | 1 | 32 | 32 | 32 | 32 | 32 | 32 |
| Assault | NULL | 64 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Assault | p1 | 32 | 0 | 32 | 32 | 32 | 32 | 32 | 32 |
| Game/task | Condition | Units | C0 | V | M | W(3/4) | O15 | O57 | O57h |
|---|---|---|---|---|---|---|---|---|---|
| cartpole-balance | NULL | 64 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| cartpole-balance | p0.05 | 32 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| cartpole-balance | p0.2 | 32 | 0 | 1 | 0 | 0 | 0 | 0 | 0 |
| cartpole-balance | p0.5 | 32 | 0 | 9 | 5 | 9 | 1 | 1 | 4 |
| cartpole-balance | p1 | 32 | 0 | 24 | 19 | 24 | 1 | 1 | 21 |
| myo-obj-hold | NULL | 64 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| No. | Prediction (probability) | Outcome | Record |
|---|---|---|---|
| 1 | Checks (a), (b), (c) and (d) of §2 hold in every unit of Part 1 (97%). | contradicted | Checks a,b,c: 4512/4512 passed each; d: 4430/4512 passed, 82 failed; largest d violation=3.0665170001627478 |
| 2 | In every group at p = 1 of TD-MPC2 and of the IRIS decoded pixels in which C1 or C1c rejects in fewer than 24 of 32 units while W(3/4) rejects in at least 24, the median of that factor exceeds 2 (85%). | supported | Eligible group-factor pairs=12; median kappa > 2 in 12; minimum median kappa=6.2594373676413 |
| 3 | W(3/4) rejects in 96 of 96 ACTIVE units and 0 of 192 NULL units of DIAMOND (95%). | supported | W(3/4): ACTIVE 96/96; NULL 0/192 |
| 4 | No online rule rejects in more than 2 of 64 NULL units in any group (97%). | supported | Largest NULL counts per group of 64: "O15": 0, "O57": 0, "O57h": 1 |
| 5 | O57 reaches 24 of 32 units at p = 1 in at least 22 of the 26 IRIS games and at least 5 of the 6 TD-MPC2 tasks (60%). | supported | O57 at p=1, >=24/32: IRIS tokens 22/26; TD-MPC2 5/6 |
| 6 | W(3/4) reaches the bar at p = 1 in at least as many IRIS games as each of the three online rules (55%). | supported | IRIS tokens at p=1, >=24/32: W(3/4)=24; online="O15": 22, "O57": 22, "O57h": 23 |
| Statistic | Four-model | Linear-Gaussian | GRU | Transformer | Boosting |
|---|---|---|---|---|---|
| Pooled (C0) | |||||
| Kish (C1) | 349.00 | 294.09 | 326.08 | 300.58 | 350.39 |
| Validation-tuned (V) | 209.64 | 211.02 | 121.40 | ||
| Mixture (M) | 346.64 | 291.48 | 312.32 | 293.87 | 348.03 |
| W(3/4) | 345.33 | 290.16 | 311.00 | 292.56 | 346.71 |
| V ρ | 350.54 | 36.18 | 176.92 | 300.69 | 351.93 |
| Statistic | Four-model | Linear-Gaussian | GRU | Transformer | Boosting |
|---|---|---|---|---|---|
| Pooled (C0) | |||||
| Kish (C1) | 217.52 | 217.49 | 215.04 | 184.90 | 218.60 |
| Validation-tuned (V) | |||||
| Mixture (M) | 218.18 | 209.89 | 218.13 | 184.78 | 219.30 |
| W(3/4) | 216.86 | 208.57 | 216.82 | 183.46 | 217.98 |
| V ρ | 206.08 | 207.46 | 189.62 | 179.75 | 177.62 |
| Min | Model | Min | Max | Frozen V | ||
|---|---|---|---|---|---|---|
| 10 | Linear-Gaussian | 0.0625 | 4 | 0.031068 | 0.067227 | 0.125 |
| 10 | GRU | 0.0625 | 4 | 0.031068 | 0.067227 | 0.125 |
| 10 | Transformer | 0.25 | 2 | 0.0079523 | 0.017699 | 0.03125 |
| 10 | Gradient boosting | 0.125 | 3 | 0.015779 | 0.034783 | 0.0625 |
| 30 | Linear-Gaussian | 0.0625 | 4 | 0.031068 | 0.067227 | 0.25 |
| 30 | GRU | 0.03125 | 5 | 0.060264 | 0.12598 | 0.25 |
| UTC | Record/event | Commit |
|---|---|---|
| 2026-11-10T07:13:41Z | Feasibility check committed; images of official test trajectories 0–15 only | 97ec197 |
| 2026-11-10T07:28:46Z | Protocol v0 committed | 83ac997 |
| 2026-11-10T09:01:17Z | Build pilot committed | fec685d |
| 2026-11-10T09:35:44Z | Protocol v1 committed | c30a2c0 |
| 2026-11-10T10:19:49Z | Frozen code committed after command-record check, tests, synthetic rehearsal and exact pilot replay | 68ad9b3 |
| 2026-11-10T10:20:57Z | Freeze evidence and rehearsal records committed | ed0cade |
| File | SHA-256 |
|---|---|
| P5_RESULTS_UPDATE_20261110.md | c1112eeb3d62b005bbab057368c48aee6663d3fb0c4d7285d92cdca952ff6f8a |
| P5_CONF_REPORT.md | cc434b700984181107831dcdf84b3157e3e1d663f965f2ea4e68db5477149718 |
| SUMMARY_P5.tsv | 3896016e0d8e939d3bc85f19e4d149d1309a7905776ddc7043563c49dba76135 |
| DECISION.json | 2cbbf20d8e986c2a442d948bd3f17b23c9c540d2ce81d91116152945d5006c29 |
| P5_BEACON_REPORT.md | 582969c3f050fa64b35b707768f9d60443bf79d0da62b139690f33021757374b |
| REGISTERED_RUN_PERMIT.json | 12e9d17c58b132fdb32dfe62fac8c0262fbf4a451db463d16cdbbd6dc56c73ac |
| File | SHA-256 |
|---|---|
| test/traj_0_to_255.tfrecords | 22ddae5a679656d923546763b5dd603ce5294ed3e4e29235f79ab9198c19cb7d |
| train/traj_256_to_511.tfrecords | 8be6d4eddd923d8880a5c59fd1b1b1f0f511859d3cd61d443aef4fbb78a6c04e |
| train/traj_14877_to_15132.tfrecords | 9fe51a24605a356192f0109c92f7e5e3710895a59bc832cb52896613bea742b7 |
| train/traj_29498_to_29753.tfrecords | e885d878532066522a541722c7a59095917f0fcd855d916724b78c1b635f72ad |
| train/traj_44119_to_44374.tfrecords | 9623951ed2e6d688f8005bd3ab49abc9475e5e11541f59ae736388f36ff71bd7 |
| Statistic | L-emp | L-par |
|---|---|---|
| Pooled (C0) | 0 | 0 |
| Validation-tuned (V) | 0 | 0 |
| Mixture (M) | 1 | 0 |
| W(3/4), primary | 0 | 0 |
| W(1/2) | 0 | 0 |
| W(1/4) | 1 | 0 |
| Law | Statistic | Rejected | Median | Minimum | Maximum |
|---|---|---|---|---|---|
| L-emp | C0 | 143/159 | 13.916 | 33.703 | |
| L-emp | V | 155/159 | 15.536 | 28.718 | |
| L-emp | M | 154/159 | 13.447 | 31.000 | |
| L-emp | W(3/4) | 155/159 | 15.305 | 29.873 | |
| L-emp | W(1/4) | 155/159 | 14.427 | 30.800 | |
| L-emp | W(1/2) | 155/159 | 15.053 | 30.440 |
| Law | Arm | ||
|---|---|---|---|
| emp | NULL | 9 | 0.001953125 |
| emp | ACTIVE | 1 | 0.5 |
| par | NULL | 8 | 0.00390625 |
| par | ACTIVE | 1 | 0.5 |
| Law | Arm | Split | Positive / units | Undefined |
|---|---|---|---|---|
| emp | NULL | validation | 2/80 | 0 |
| emp | NULL | test | 4/159 | 0 |
| emp | ACTIVE | validation | 79/80 | 0 |
| emp | ACTIVE | test | 151/159 | 0 |
| par | NULL | validation | 4/80 | 0 |
| par | NULL | test | 6/159 | 0 |
| Prediction | Probability | Outcome | Record |
|---|---|---|---|
| 1: C-VAL under both laws | 95% | held | Largest NULL counts: 2/159 emp, 1/159 par; both 4 |
| 2: C-POWER | 65% | held | 155/159 120 |
| 3: emp ACTIVE median C0 log E < 0 | 40% | failed | 13.915737557619188 < 0 is false |
| 4: emp ACTIVE C0 rejects fewer than W(3/4) | 65% | held | 143/159 < 155/159 |
| 5: emp ACTIVE lambda V < 1 | 75% | held | 0.5 < 1 |
| 6: ACTIVE W(3/4) counts differ by at most 15 | 80% | held | abs(155 - 156) = 1 15 |
| Law | Arm | Q1 | Median | Q3 | Best < 1 | Best = 1 | Best > 1 | L(1) < 0 |
|---|---|---|---|---|---|---|---|---|
| emp | NULL | 0.037162722 | 159 | 0 | 0 | 159 | ||
| emp | ACTIVE | 0.42044821 | 0.59460356 | 0.70710678 | 150 | 3 | 6 | 13 |
| par | NULL | 0.03125 | 159 | 0 | 0 | 159 | ||
| par | ACTIVE | 0.42044821 | 0.59460356 | 0.70710678 | 151 | 4 | 4 | 12 |
| Law | Arm | Break-even above 4 | No nonnegative grid value |
|---|---|---|---|
| emp | NULL | 0 | 86 |
| emp | ACTIVE | 6 | 1 |
| par | NULL | 0 | 87 |
| par | ACTIVE | 6 | 1 |
| Arm | Origins | C0 | V | M | W(3/4) |
|---|---|---|---|---|---|
| NULL | 1 | 1 | 0 | 0 | 0 |
| NULL | 2 | 2 | 0 | 0 | 0 |
| NULL | 3 | 1 | 0 | 0 | 0 |
| NULL | 4 | 0 | 0 | 0 | 0 |
| NULL | 5 | 0 | 0 | 0 | 0 |
| NULL | 6 | 0 | 0 | 0 | 0 |
| Arm | Origins | C0 | V | M | W(3/4) |
|---|---|---|---|---|---|
| NULL | 1 | 1 | 0 | 0 | 0 |
| NULL | 2 | 0 | 0 | 0 | 0 |
| NULL | 3 | 0 | 0 | 0 | 0 |
| NULL | 4 | 0 | 0 | 0 | 0 |
| NULL | 5 | 0 | 0 | 0 | 0 |
| NULL | 6 | 0 | 0 | 0 | 0 |
| Trajectory | C0 | V | M | W(3/4) | O57 | Best factor |
|---|---|---|---|---|---|---|
| 19 | 4.43 | 2.86 | 4.20 | 4.26 | 0.35355 | |
| 44 | 4.50 | 3.65 | 4.34 | 0.74 | 0.2973 | |
| 45 | 5.80 | 6.06 | 5.87 | 6.87 | 0.25 | |
| 67 | 2.64 | 6.34 | 4.03 | 6.06 | 5.24 | 0.42045 |
| 76 | 4.20 | 3.61 | 4.07 | 4.45 | 0.2973 | |
| 83 |
| Law | Origin group | n | Mean norm | Median norm | C2 count | C2 prop. | C3 count | C3 prop. |
|---|---|---|---|---|---|---|---|---|
| emp | ell(1) < | 37 | 0.059836846 | 0.062300014 | 10 | 0.27027027 | 7 | 0.18918919 |
| emp | All others | 2189 | 0.053252922 | 0.05490444 | 413 | 0.18867063 | 229 | 0.10461398 |
| par | ell(1) < | 39 | 0.060478934 | 0.064482291 | 10 | 0.25641026 | 6 | 0.15384615 |
| par | All others | 2187 | 0.053235451 | 0.054896909 | 413 | 0.18884316 | 230 | 0.1051669 |
| Law | Arm | Statistic | Units | Sum |
|---|---|---|---|---|
| emp | NULL | C0 | 159 | |
| emp | NULL | V | 159 | |
| emp | NULL | M | 159 | |
| emp | NULL | W0.75 | 159 | |
| emp | ACTIVE | C0 | 159 | 1795.3258 |
| emp | ACTIVE | V | 159 | 2367.691 |
World Model Evaluation
World Model Evaluation measures whether learned simulators predict consistent, physically plausible, action-responsive futures that help agents plan and act.
See all 245 papers