Who Moved the Frontier? Paying Open Frontier Challenges with Free Identities
Organizations: Independent researcher. · Circle Research.
Abstract
In an open frontier challenge, anyone, person or AI agent, submits changes to a shared problem, and a change that beats the best measured score becomes the public baseline. How should it pay when identities are free? Rank prizes can be gamed: on a replay of one track of a recent challenge, one solver's steps under two identities would have taken first and second prize. We consider a prediction-market rule, proposed for machine-learning contests, that pays each accepted step for the progress it adds; with exact scores, among rules that see only a step's start and end, only these resist splitting and merging, and extra identities gain nothing. With noisy scores and independent normal runs, we prove that any payment that never takes money back and pays real improvements also pays something, on average, to a change that does nothing. We propose a fee per submission, at least what luck pays per attempt, that removes each attempt's expected profit if verifiers are honest and no submission is noisier than planned; nothing enforces that bound, and under noise the rule does not cap the total paid. If crediting another and faking credit to one's second account look the same, a rule that neither rewards fakes nor punishes honesty pays for credit at most what lying costs. We introduce a royalty on how much a later gain falls without the earlier change, which needs no claims but needs removable changes. For five public challenges, we name the rule our results would support.
Figures & tables
| Challenge | Scorer; who draws the noise | Open as run? | Rule in use | Rule our results would support |
|---|---|---|---|---|
| better.codes [ 22 ] | exact: a Lean proof checker | O1 ?, O2 yes, O3 no | discretionary awards | once pay is committed: per-step, no fee (Prop. 1 ); a target bound or a capped |
| NanoGPT speedrun [ 39 ] | noisy training time; the operator posts its own run logs | O1 ?, O2 yes, O3 no | none documented | if verifiers rerun and score each candidate: per-step on fresh runs with a fee, log price; the noisy loss check is outside our model |
| QSB pinning [ 76 ] | noisy; one fresh organiser run per candidate, against a stored best | O1 ?, O2 yes, O3 ? (payment fixed) | two top-3 periods + 3% raffle, $20,000 | with fresh runs of the best: per-step on the log score with a fee |
| Kernel challenge [ 59 ] | noisy; serving track: organiser reruns the top 25 on held-out data | O1 no, O2 no, O3 no | serving track: top-10 prizes, $100,000 | if opened: per-step on the mean of fresh reruns with a fee |
| ECDSA.Fail [ 51 ] | deterministic; the operator picks the test inputs by a 48-bit nonce | O1 ?, O2 yes, O3 ? | none documented in its paper | if inputs are drawn after submission: per-step with a fee; the sampled correctness check is outside our model |
| Most a solver gains, in $, by | |||||
|---|---|---|---|---|---|
| Rule (same $20,000) | Gini | top-3 share | fake identities | pooling | splitting a step |
| QSB’s published rule, replayed | 0.687 | 0.670 | 0 | ||
| Per-step rule (one price) | 0.574 | 0.483 | 0 | 0 | 0 |
| Winner-take-all | 0.958 | 1.000 | 0 | 0 | 0 |
| Fixed bonus per accepted step ($487.80) | 0.349 | 0.390 | 0 | 0 | |
| Rule | Non-exp. | Sybil | Split/merge | Packaging | Refinement | Credit | Total paid |
| Baseline rules | |||||||
| WTA | ✗ A.a | ✓ | ✓ A.e | – | – | none | |
| Top- , , | ✗ A.f | ✗ A.d | ✓ A.e | – | – | none | |
| Raffle, | ✗ A.f † | ✗ A.c | ✓ A.e | – | – | none | or |
| Bonus per step | ✓ A | ✓ A | ✗ A.b | – | – | none | unbounded |
| The per-step rule, alone and with a follower-financed royalty (Section 6 ) | |||||||
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Quantity | Estimate | 95% interval | Compared with noise |
|---|---|---|---|
| Noise: standard deviation for the same recipe (pooled) | 0.038 | ||
| Gain of depth | 0.353 | about | |
| Gain of batch | 0.121 | about | |
| Gain of lr | not resolved | ||
| Interaction of batch and depth | 0.094 | not distinguishable from 0 |
| Rule | fisher take | unpaid progress | fisher take | unpaid progress |
|---|---|---|---|---|
| Winner-take-all | ||||
| Top-3 by total gain | ||||
| Raffle | ||||
| Bonus, capped at the budget | ||||
| Per-step, deciding measurement | ||||
| pair | verdict | gain B | lines | changed | unk. | other | failing | outcome | |||
|---|---|---|---|---|---|---|---|---|---|---|---|
| 005 | B1 | B2 | dependence | 2 | 1 | 2 | 1 | 1 | 0 | A | failed |
| 043 | B3 | B4 | dependence | 1 | 5 | 36 | 3 | 4 | 0 | A | failed |
| 046 | B5 | B3 | dependence | 5 | 1 | 46 | 0 | 2 | 0 | A | failed |
| 066 | B6 | B7 | dependence | 2 | 7 | 1622 | 6 | 5 | 4 | A | failed |
| 084 | B8 | B3 | unresolved | 1 | 22 | 577 | 10 | 2 | 1 | A | failed |
| 104 | B3 | B6 | dependence | 1 | 7 | 613 | 4 | 11 | 1 | mixed | failed |
| NanoGPT speedrun [ 39 ] | better.codes [ 22 ] | QSB, pinning [ 76 ] | Muse Spark Open Kernel [ 59 ] | |
|---|---|---|---|---|
| Score | minutes to 3.28 val. loss on 8 H100 | soundness lower bound, bits (Lean-checked) | verified candidates/s on one GPU | kernel boards: speed-up over SGLang on one H200; serving track: throughput and interactivity on 8 H200 |
| Public frontier (O2) | yes: every record is the code to beat | yes: promoted submissions reset the floor | yes | not on the board we inspected: best result per account, no artifact |
| Record | 92 records, 2024-05 to 2026-08, 45 to 0.665 min | 81 promotions, 22 solvers, 2026-08-20 to 10-01, 53.12 to 68.15 bits | 44 promotions, 27 solvers, 2026-09-15 to 10-05, M to M | kernel board muse_bf16_proj : 10 rows captured, more paginated |
| Step size | median ; 32 of 91 under ; 6 over | median bits; 39 of 80 at the display minimum | median ; 10 of 41 rows under ; 2 over | — |
| Noise rule | over run logs that the mean loss is ; waived for systems-only speed-ups | exact (proof checker) | one fresh run per candidate on the organiser’s GPU, against the best’s stored score; hits re-derived on a CPU; noise , about | serving track: median of reruns of the top 25 on held-out data |
| Reward rule | none documented | discretionary (no committed rule) | two-period top-3 + equal-share raffle at ($20,000) | serving track: top-10 ranked prizes ($100,000); kernel boards: none |
| Rule (same $20,000) | Gini | top-3 share | Sybil split | pooling | step split |
|---|---|---|---|---|---|
| Published schedule, replayed (top-3 + raffle) | 0.687 | 0.670 | 0 | ||
| Per-step, one | 0.574 | 0.483 | 0 | 0 | 0 |
| Per-step, per period (QSB’s reset) | 0.781 | 0.671 | 0 | 0 | 0 |
| Per-step, concave | 0.738 | 0.690 | 0 | 0 | 0 |
| Winner-take-all | 0.958 | 1.000 | 0 | 0 | 0 |
| Per-submission bonus ($487.80) | 0.349 | 0.390 | 0 | 0 |
| Rule | paid | Gini | top-3 | first 10 | last 41 | fake identities | pooling |
|---|---|---|---|---|---|---|---|
| Per-step, per second | 52 | 0.92 | 0.81 | 0 | 0 | ||
| Per-step, log | 52 | 0.80 | 0.52 | 0 | 0 | ||
| Winner-take-all | 1 | 0.98 | 1.00 | 0 | 0 | ||
| Top-3 (pays ) | 3 | 0.95 | 1.00 | 3,333 | 1,333 | ||
| Raffle, 81 s (pays ) | 4 | 0.92 | 0.75 | 1,071 | 667 | ||
| Bonus per step | 52 | 0.55 | 0.35 | 0 | 0 |
Mechanism Design
Mechanism design studies rules for allocating resources and shaping incentives when participants hold private information or act strategically.
See all 149 papers