Patching the Protocol: A Taxonomy of Pre-Registration Repairs in LLM-Proposer Experiments
Organizations: Shenyang Institute of Technology, Shenyang, China
Abstract
Pre-registration is increasingly proposed as a trust mechanism for machine-learning research involving LLM-generated content, but its canonical form assumes a frozen protocol can be executed unchanged. Software-intensive studies violate this assumption: the protocol is partly implemented, and implementations carry defects. We report a three-case study of protocol repairs ("patches") from a program where an LLM proposes executable rules that a programmatic verifier gates, across production scheduling, incentive design, and task assignment. Each protocol required exactly one defect-repair patch: three events in total. None modified a numbered judgment criterion, its threshold, or the scoring rule, and none altered already collected data; two changed pre-registered design parameters before any data were collected, each with a recorded rationale. We propose a two-axis taxonomy of patches (defect class: environmental / feasibility / correctness, x timing: pre-data / post-data), introduce a verifiable non-impact claim for post-data repairs, and distill a nine-item checklist; all nine items were exercised by at least one patch event -- an observation of use, not a validation of sufficiency. The credibility of verifier-bound, LLM-proposer studies depends less on whether patches occur than on whether they are auditable.
Figures & tables
| Domain | Discovery context | Scale at patch time | Patch (key parameters) | Data at patch |
|---|---|---|---|---|
| A | run-time crash at pair 037 | batch study; 37 pairs written | redraw rule extended train holdout (seed ) | 37 pairs retained, re-validated |
| B | 1-pair smoke test | workers; budget | : ; participation constraint; baseline exemption | none (pre-data) |
| C | smoke debugging | capacity ; | deep-copy/idempotent run_match ; : ; deadlines – – min | none (pre-data) |
| Class | Locus | Symptom | Detection cost | Damage if missed |
|---|---|---|---|---|
| Environmental | pipeline boundary conditions | crash (loud) | low, stops the run | lost compute, recoverable |
| Feasibility | specification / parameters | structurally degenerate outcomes | low, one smoke cycle | misread artifact as finding |
| Correctness | measurement semantics | none (silent) | high, needs audit discipline | invalid dataset with plausible numbers |
| Checklist item | Domain A | Domain B | Domain C |
|---|---|---|---|
| 1 Freeze criteria | pre-reg v1–v3 freeze numbered criteria | v1.0 freeze | v1.0 freeze |
| 2 Append-only changelog | v1 v2 v3 v3.1 | 4 entries, v0.1 to v1.2 | 3 entries, v0.1 to v1.1 |
| 3 Negative-result clauses | failure branches pre-committed | failure branches pre-committed | failure branches pre-committed |
| 4 Leakage discipline | holdout evaluated once, counter-asserted | HoldoutGate | HoldoutGate |
| 5 Equal-budget controls | 20 random proposals per pair | 20 random proposals per pair | 20 random proposals per pair |
| 6 Log rejections verbatim | parse-fail rate archived; bad proposals kept | violation ledger fed back | violation ledger fed back |
ML Reproducibility
Machine learning reproducibility studies whether model results and scientific claims can be independently checked and reliably repeated.
See all 153 papers