Articulation Upper Bounds for Language Models

Alexander Spangher; Daniel E. Ho; Diyi Yang; Noah Goodman; Sanmi Koyejo

Figure 7: Unit-mining process (Left-Hand Side of Figure 2 ) (A) N channels supply unit candidates (shown:

s_{1}

: units found online;

s_{2}

: units proposed directly by an LLM;

s_{3}

: units mined from the GEPA-induced prompt

p_{\text{GEPA}}

). Candidates (

u_{7}

, green; and

u_{44}

, red) all enter the pool

\Omega

; what follows orders and selects them. (B) As implemented: #1 one single-pass marginal against the starting prompt (

p_{\text{GEPA}}

’s shipped candidate) on the

100

-item selection panel, with

\Delta>0

making a unit eligible; #2 units are re-tested in combination only during selection (prefix sweep, delete pass), never as a pool filter; #3 multi-pass scoring is applied afterwards to shipped prompts and prefix curves, never to units; #4–#5 the shuffled- and foreign-unit controls are separate experiments on the frozen pool (Appendix A ) and remove nothing; #6 the delete pass runs during selection on the chosen set, on the same panel.

result	pool size(s)	selector	enumeration?	double greedy?	passes at selection
Table 2 , Table 3 , Table 4 (HotpotQA, HoVer, IFBench, PUPA, AIME)	$68$ , $164$ , $128$ , $120$ , $48$	single-pass paired marginals against GEPA’s shipped candidate, a cumulative prefix sweep (cap $48$ ), one drop-one pass, one add-back pass, single-pass confirmation on a disjoint panel	No	No	$T=1$ ; the shipped candidate is then scored with five test passes
Table 2 , Table 3 , Table 4 (LiveBench)	$48$	same selector, prefix cap $32$ (the earlier default, not $48$ ); the confirmation guard rejected every candidate unit	No	No	$T=1$ ; no unit compiled, so the shipped prompt is GEPA’s $600$ -call candidate rescored
HotpotQA no-GEPA-unit ablation (§ C.1 )	$60$	same selector, re-run on the pool with the $8$ trajectory-mined (GEPA-derived) units removed	No	No	$T=1$ at selection; the reported comparison used one scoring session at five passes
HoVer selection replicates (Table 4 , note $b$ ; § C.1 )	$164$ (same frozen pool, run twice)	same selector	No	No	$T=1$ ; five-pass test $.539$ and $.541$ against the row of record’s $.572$ (HB 226)
Screened robustness run, HoVer only (HB 225)	$164$ (same frozen pool)	lazy greedy on conditional marginals, restricted to units that clear a shuffled-unit null band after a $3$ -pass rescore	No	No	$T=3$ screening rescore, then $T=1$ fresh evaluations (at most $8$ per step); five-pass test $.520$ , $-.052$ $[-.077,-.027]$ against the row of record (HB 226)
Value-curve ceilings (Figure 10 left, § C.2 , § C.2 )	same pools as above	a curve fit to the prefix-sweep trajectory selection already produced (HotpotQA: the same session’s sweep extended to $k=68$ on the test split; the other five: the selection-panel sweep)	No	No	$T=1$

Appendix

Table 1: Selection procedure behind each reported result. The paper describes several selection procedures (ranking marginals once against the seed, recomputing the best marginal at every step, pruning and re-adding, double greedy, exhaustive enumeration); this table states which one produced which result. Every reported number in the paper traces to exactly one row of this table.

benchmark	$M_{\omega}-$ GEPA	$M_{\omega}-$ GEPA+Merge	$M_{\omega}-$ MIPROv2
HotpotQA	$+.077$ $[+.045,+.109]$	$+.222$ $[+.180,+.266]$	$+.208$ $[+.165,+.251]$
HoVer	$+.071$ $[+.043,+.099]$	$+.052$ $[+.025,+.079]$	$+.081$ $[+.052,+.110]$
LiveBench	$+.016$ $[-.010,+.043]$	$+.025$ $[-.002,+.054]$	$+.049$ $[+.021,+.080]$
IFBench	$+.025$ $[-.003,+.053]$	$+.023$ $[-.004,+.049]$	$+.060$ $[+.029,+.092]$
AIME	$+.035$ $[+.011,+.061]$	$+.027$ $[+.001,+.053]$	$+.083$ $[+.052,+.115]$
PUPA	$+.047$ $[+.018,+.076]$	$+.069$ $[+.041,+.098]$	$+.025$ $[-.006,+.057]$

Appendix

Table 2: Paired contrasts against every baseline : mean item-level difference with

95\%

bootstrap interval, same scoring sessions as Table 3 . The HotpotQA row is the equal-call session (HB 220; GEPA at

16{,}700

calls).

benchmark	GEPA	GEPA+Merge	MIPROv2	$\epsilon$ -est. ( $M_{\omega}$ )	$\Delta$ vs GEPA
HotpotQA	$.567$	$.421$	$.435$	$\mathbf{.643}$	$+.077^{***}$
HoVer	$.497$	$.515$	$.487$	$\mathbf{.567}$	$+.071^{***}$
LiveBench	$.695$	$.686$	$.661$	$.710$	$+.016$
IFBench	$.417$	$.419$	$.382$	$.442$	$+.025$
AIME	$.323$	$.331$	$.275$	$\mathbf{.357}$	$+.035^{**}$
PUPA	$.638$	$.616$	$.659$	$\mathbf{.685}$	$+.047^{**}$

Appendix

Table 3:

\epsilon

-estimation: prompt optimization comparisons . For every benchmark we score each candidate with five independent test passes (scores are the per-item means), using Qwen3-8B as the proposer and language model, to match Agrawal et al. (2025) . All benchmarks are run with the same number of steps as in Agrawal et al. (2025) ; on every benchmark the GEPA column is additionally the equal-call run, in which GEPA is re-run at the call budget that

\epsilon

-estimation spent (Appendix C.1 ). The comparison is therefore call-matched per benchmark, not budget-capped. On LiveBench the confirmation guard rejected every unit and shipped GEPA’s

600

-call prompt unchanged, so that row compares GEPA’s own prompts at two budgets and contains no unit-selection effect.

\Delta

is the mean paired item-level difference, with a

20{,}000

-draw bootstrap interval over items (each item’s score is its five-pass mean; the passes do not enlarge the sample); stars mark two-sided bootstrap

p<.01

( ∗∗ ) and

<.001

( ∗∗∗ ). Five passes measure scoring noise, not the variance between optimizer runs, which Appendix C.1 reports. Contrasts against GEPA+Merge and MIPROv2 are in Table 2 . Items per benchmark: HotpotQA 300, HoVer 300, IFBench 294, PUPA 221, AIME 150, LiveBench 126.

method	init / inherited	calls: prop/sel/val/test	total	chars	test score ( $\Delta$ vs GEPA)
HotpotQA ( $n_{\text{test}}{=}300$ , HB220/HB224, fp 2026-09-12T00:19:48Z)
GEPA (eq.-call) a	seed program	16,821/n/a/n/a/3,000	19,821	7,016	.567
GEPA+Merge	seed program	600/n/a/n/a/600	1,200	1,426	.421
MIPROv2	seed program	600/n/a/n/a/600	1,200	1,048	.435
$M_{\omega}$	GEPA-600 init $+$ 68u (60L/8T) $\to$ 28 compiled	0/16,000/700/600	17,300	4,619	.643 (+.077*** [+.045,+.109])
HoVer ( $n_{\text{test}}{=}300$ , HB212, fp 2026-08-20T07:01:05Z)

Appendix

Table 4: Per-procedure accounting. One block per benchmark; rows are the four Table 3 methods in its column order. Init/inherited : GEPA, GEPA+Merge and MIPROv2 all start from the benchmark’s seed program;

M_{\omega}

initializes from GEPA’s shipped 600-call candidate and mines a unit pool from optimizer trajectories plus reflection-LM suggestions (L=LLM-suggested, T=own-trajectory-mined, X=cross-task-LM-trajectory-mined, F=failure-grounded-mined; “u” units, “compiled” the number kept in the shipped prompt after drop-one/add-back and the no-regret guard). Calls: prop/sel/val/test : metric-evaluation calls by stage, summed from each run’s own proposals.jsonl (prop=proposal/mining-stage metric calls, i.e. failure-grounded sampling only, since LLM-suggested units cost reflection-LM calls rather than metric calls, see below; sel=select-init/select-marginal/prefix-sweep/drop-one/add-back; val=confirm-init/confirm-compiled, the no-regret guard; test=this run’s own final-test pass, own-session, never the certified score in the last column). GEPA/GEPA+Merge/MIPROv2 do not log a phase-separated selection/validation split, so their “prop” cell is their whole declared search budget ( result.json[’budget_calls’] , the max_metric_calls cap) except HotpotQA GEPA (16,821, realized, parsed from gepa_state.bin , HB199 §1) and PUPA GEPA (29,166 realized

=

98.8% of 29,526 declared before a disk failure, HB216/217); MIPROv2’s heavy mode has no cap, so its number is realized by construction. n/a marks a stage that does not apply to that optimizer; n/l marks a number not logged in this run’s own artifacts (IFBench and AIME’s GEPA column and HoVer’s MIPROv2 column retain no proposals.jsonl , only result.json , so their own-run test-stage calls cannot be recomputed from artifacts here; PUPA’s GEPA/Merge/MIPROv2/

M_{\omega}

directories and HoVer’s

M_{\omega}

directory retain none either, so every stage of those rows is n/l ). Total : realized calls summed across logged stages (LiveBench and AIME’s

M_{\omega}

directories mix rows from several restarted launch attempts; the number shown isolates the attempt whose own final-test score matches result.json , not the full file, which sums to 16,046 and 3,855 respectively including the failed restarts). Chars : shipped-candidate character count; the aggregate character mass proposed across an entire search is reconstructed only for HotpotQA (HB199 §1: GEPA

{\sim}

70M vs

M_{\omega}{\sim}

46.3M characters); elsewhere it is not logged. Test score : the certified same-session score from the row-of-record ledger entry named per block (never this run’s own-session number);

M_{\omega}

’s row adds the paired bootstrap

\Delta

and 95% CI against the GEPA column, stars as in Table 3 . a Three independent seeds (0,1,2); this row is seed 0 (declared before seeds 1-2 existed), which scored .567 here against .413/.402 for seeds 1/2 in the same certified session. b Two

M_{\omega}

selection replicates on the same frozen pool scored

.539

and

.541

in the HB226 session, against the row of record’s

.572

(HB221/HB226). c

M_{\omega}

’s compiled candidate never cleared the no-regret guard on LiveBench: the shipped prompt is byte-identical to GEPA’s shipped

600

-call candidate (the eq.-call row itself is the

18{,}708

-call run), so this row’s test score reflects that candidate re-scored under a different label, not a unit-recombination contribution. d GEPA’s own run reached 29,166 of 29,526 declared calls (98.8%) before an out-of-disk failure recovered by HB216; its own result.json left seed/best test null, so the certified score in this row comes only from the separate row-of-record session. Artifact: runs/arm_accounting_v1.json .

Figure 8: Missing mass. Good–Turing novelty

p_{\mathrm{unseen}}(n)=f_{1}/n

as mining proceeds, per task; the curves show data only, with no extrapolation. Accumulation curves are averaged over

150

random orderings. Left: five of the six supervised GEPA-suite benchmarks (all but PUPA, added after this sampling pass); species are semantic unit clusters. Right: unsupervised annotation tasks; species are normalized rubric items from each task’s task bank, and three tasks are labeled by name. Curves fall into saturating (the articulable space closes), plateauing (novelty persists at a steady rate), and climbing (open-ended drift) regimes.

Figure 9: Missing mass for five of the six supervised benchmarks (all but PUPA), from three independent mining replicates per benchmark (

3{\times}60

units, one declared sampler). Species are formed by deterministic word-overlap clustering at two granularities. Curves are averaged over

150

random orderings (the raw sequence arrives grouped by framing, so unaveraged

f_{1}/n

is non-exchangeable and can even rise). At the fine granularity (Jaccard

\geq.5

, grey) nearly every unit is its own species and novelty stays near

1

; the articulable space appears unbounded. At the coarse granularity (Jaccard

\geq.25

, blue) the same data yield bounded estimates: HoVer’s space is smallest and closes fastest (Chao1

\widehat{N}{\approx}191

), and IFBench’s remains the most open (

\widehat{N}{\approx}550

). Within every panel, the contrast between the two lines shows that whether a prompt-unit space saturates is a property of the granularity at which units are individuated, not of the task . Syntactic clustering is conservative for missing-mass claims (paraphrases a semantic clusterer would merge stay separate, inflating

f_{1}

and hence

p_{\mathrm{unseen}}

); these curves are upper bounds on novelty.

Figure 10: Missing value measurements. Left, supervised tasks: value at each # units

k=1,2...

(blue). A forward projection

\widehat{V}(k)

,

k>20

is made using

\epsilon

-estimation (orange dashed,

95\%

band widening with horizon), and the actual

k>20

drawn underneath. HotpotQA (fit

k\leq 20

, actual to

40

, projection to

60

) is the still-rising regime. Backtesting the projection procedure over all six benchmarks’ trajectories (Appendix C.2 ) gives pooled median prediction error

.018

at horizons of

1

–

3

units ahead,

.020

at

4

–

8

, and

.029

at

9{+}

. Right, unsupervised: per-task value (blue) and its missing-value ceiling (red), for

272

metrics across

8

creative domains, estimated by asking what a fresh draw from the same sampling process could still add.

benchmark	$k_{\max}$	$y(k_{\max})$	$+10$	$+20$	$+50$	power	sat. exp.	constrained	$\widehat{\epsilon}$ (point)	$\widehat{\epsilon}_{\mathrm{con}}$	viol. $4$ – $8$ / $9{+}$	$\widehat{N}$ pooled [block]
			power law $\hat{y}(k_{\max}{+}\Delta)$			$k{=}\infty$ ceiling $a$
HotpotQA	68	$.593$	$.602$	$.602$	$.604$	$.611$	$.600$	$.644$	$+.018$	$+.051$	$.03$ / $.08$	$337$ $[23,337]$
IFBench	48	$.855$	$.856$	$.856$	$.856$	$.856$	$.856$	$.892$	$+.001$	$+.037$	$.10$ / $.05$	$550$ $[42,550]$
HoVer	32	$.560$	$.592$	$.592$	$.592$	$.592$	$.570$	$.660$	$+.032$	$+.100$	$.00$ / $.00$	$191$ $[16,191]$
LiveBench	32	$.658$	$.718$	$.721$	$.726$	$.778$	$.723$	$.778$	$+.121$	$+.121$	$.00$ / $.00$	$369$ $[37,369]$
AIME	48	$.533$	$.524$	$.523$	$.520$	$.480$	$.445$	$.633$	$-.053$	$+.100$	$.00$ / $.00$	$177$ $[33,177]$

Appendix

Table 5: Projected and constrained ceilings , one row per benchmark. Values are in the value curves’ own units (validation-side greedy prefix means, as in Figure 10 and the backtest above), not Table 3 test scores; only the saturation verdict transfers from one to the other. The constrained asymptote, both

\widehat{\epsilon}

columns and the violation rates are power-law fits.

\widehat{\epsilon}_{\mathrm{con}}

is the constrained asymptote minus

y(k_{\max})

; the calibrated margin is zero. Violation = share of later prefixes above the window’s ceiling, held out. The block interval resamples whole sampling runs; its upper limit necessarily equals the pooled point (a duplicated run adds only doubletons), so the leave-one-run-out range above is the better measure of width. PUPA has a single sampling run. The constrained asymptotes replace the unconstrained ones wherever a ceiling is quoted. Artifacts: runs/ceiling_projection_v1.json and runs/epsilon_calibration_v1.json .

Figure 11: The receiver’s rule length against recovery, by regime. One point per criterion, three receivers overlaid: Llama-70B on the

130

-criterion set and the bank set (

338

distinct criteria), Qwen2.5-72B on the bank set, and Qwen2.5-14B as the mean over its four

30

-item rules per criterion. Lines are the mean recovery in each length quintile;

\rho

is the rank correlation of length with recovery.

message	model	$n$	median length (words)	Spearman with recovery or gain
transmitter’s definition, by regime	–	$1{,}270$	bounded $17$ , reaches $19$ , rising $19$	–
receiver’s rule, rising	Llama-70B / Qwen2.5-72B / Qwen2.5-14B	$196$ / $172$ / $28$	$424$ / $454$ / $272$	$-.04$ / $-.08$ / $+.18$
receiver’s rule, reaches	Llama-70B / Qwen2.5-72B / Qwen2.5-14B	$77$ / $53$ / $28$	$430$ / $461$ / $266$	$-.06$ / $-.14$ / $-.07$
receiver’s rule, bounded	Llama-70B / Qwen2.5-72B / Qwen2.5-14B	$65$ / $11$ / $56$	$413$ / $462$ / $260$	$+.05$ / $+.07$ / $-.24$
receiver’s rule, $30$ / $60$ / $90$ items	Qwen2.5-14B	$112$	bounded $263$ / $260$ / $256$ ; controls $269$ / $256$ / $239$	$-.04$
search unit, five pools	Qwen3-8B	$360$	$23$	$-.04$

Appendix

Table 6: Median message length in words and its rank correlation with recovery (rules) or marginal gain (units). The

30

,

60

and

90

-item row counts four rules per criterion and size. Artifact outputs/analyses/rule_length_v1/ .

task	$n$ (pool)	obs. $\rho$	perm. $p$	pool $\rho$	$r_{\mathrm{MI}}$	$r_{\mathrm{AUC}}$	corr. unstepped	corr. stepped	95% CI (stepped)	printed
peer-review	$21$ ( $42$ )	$.498$	$.0005$	$+.381$	$.660$	$.510$	$.86$	$.68$	$[.58,.95]$	$.68$ (s)
code-review	$31$ ( $31$ )	$.328$	$.0002$	—	$.708$	$.391$	$.62$	$.48$	$[.18,.71]$	$.62$ (u)
creative-writing	$15$	$.419$	$.0002$	—	$.809$	$.360$	$.78$	$.61$	$[.28,1.00]$	$.61$ (s)
math	$14$ ( $46$ )	$.352$	$.0005$	$+.029$	$.738$	$.310$	$.74$	$.56$	$[.20,.76]$	$.56$ (s)
press-releases	$23$ ( $23$ )	$.203$	$.0052$	—	$.789$	$.212$	$.50$	$.37$	$[-.04,.88]$	$.50$ (u)
humor	$21$ ( $106$ )	$.344$	$.0002$	$+.130$	$.816$	$.727$	$.45$	$.40$	$[.26,.54]$	$.45$ (u)

Appendix

Table 7: Reconstruction validity under both reliability conventions.

n

= metrics in the printed row; in parentheses, the full pool size where the row was restricted after the fact to metrics whose judge reaches mean AUC

\geq.55

against the mention labels; “pool

\rho

” is the correlation on the full pool (math:

+.029

,

p=.49

). Peer review and math are also the best configuration of a sweep over label variants (

1

of

14

and

1

of

8

). “printed” is the value an earlier version of this table printed, with its convention: (s) stepped, (u) unstepped. Permutation

p

-values shuffle within metric and stay valid after selection; the restricted effect sizes are not comparable to the pools. The 95% CI resamples metrics

2{,}000

times (seed

20260912

) and redraws a document or probe split for each reliability on every resample (artifact outputs/analyses/mi_silver_reliability_uncertainty_v1/ ); every row had the per-form data needed for

r_{\mathrm{MI}}

, so no sensitivity band was needed.

Figure 12: Articulation scaling in the observational-scaling presentation: existing models as scale points, plotted against a measured capability index rather than parameter count. Right, unsupervised (measured):

1{,}347

tasks evaluated by

14

executors spanning

1.2

–

73

B across five model families; each light trace is one task’s recovery curve, colored by its fitted verdict, with the binned mean and its

95\%

interval in black. Left, supervised (measured): the complete Qwen3 ladder (

4

rungs

\times

both benchmarks). Solid lines are the mean of

40

random unit-pool draws per rung, dashed lines the GEPA-shipped prompt. At

8

B the draws beat the shipped prompt in

40/40

cases on both benchmarks (HotpotQA

+.186

, HoVer

+.070

), and the gain grows with scale.

Figure 13: The three regimes on one axis : where each criterion’s truth conditions live (shared task-agnostic taxonomy, coarsened; regime-blinded labeling; two independent passes, Appendix G ). Rising and reaches are near-identical; bounded shifts sharply beyond the text; the numbers are in the text. The regime-by-task composition is Figure 4 in the main body.

executor	family	parameters	capability index $z$
Llama-3.2-1B	Llama	$1.2$ B	$-.24$
Mistral-7B	Mistral	$7.2$ B	$.28$
Gemma-2-27B	Gemma-2	$27.2$ B	$1.10$
Llama-3.2-3B	Llama	$3.2$ B	$1.17$
Qwen2.5-3B	Qwen2.5	$3.1$ B	$1.47$
Llama-3.1-8B	Llama	$8.0$ B	$1.56$

Appendix

Table 8: The executor set. The

14

instruction-tuned open models on which every regime curve is fitted, ordered by the capability index

z

(logit AUC on the

240

-item battery, Appendix E ). Qwen3.5-122B, a mixture of experts with

10

B active parameters, contributes only to the leave-one-executor-out consensus on the vertical axis and is not a curve point.

predictor of the held-out largest executor	median $\|$ error $\|$	median error / SE	within $2$ SE
fitted form (AICc choice, as in the paper)	$.062$	$2.4$	$42\%$
logistic always	$.051$	$2.1$	$49\%$
straight line always	$.075$	$3.0$	$35\%$
mean of the remaining executors	$.149$	$5.7$	$18\%$

Appendix

Table 9: Held-out prediction of the strongest executor ,

1{,}270

metrics. Each curve is refitted without its highest-

z

executor and that executor’s recovery is predicted; SE is the held-out point’s own standard error (

.025

at the median). Artifact outputs/analyses/heldout_baselines_v1/ .

printed regime	$n$	mean $P$ (own)	modal $=$ printed	$P$ (own) $<.6$	mean $P$ (own)	modal $=$ printed	$P$ (own) $<.6$	verdict changes without the top executor
		noise in recovery ( $2{,}000$ draws)			noise in the capability index ( $100$ draws)
rising	$884$	$.90$	$99\%$	$5\%$	$.91$	$98\%$	$7\%$	$3\%$ ( $10$ to reaches, $4$ to bounded, $13$ to noisy)
reaches	$321$	$.73$	$84\%$	$34\%$	$.82$	$95\%$	$13\%$	$36\%$ ( $106$ to rising, $8$ to bounded)
bounded	$65$	$.65$	$78\%$	$42\%$	$.82$	$92\%$	$12\%$	$23\%$ ( $7$ to reaches, $8$ to rising)

Appendix

Table 10: Membership uncertainty by regime.

P

(own) is the share of draws in which a metric keeps its printed verdict. Columns 3–5: each executor’s recovery redrawn from its standard error (

2{,}000

draws). Columns 6–8: each executor’s capability index redrawn from its split-half standard error, with the planted reference recomputed (

100

draws). Last column: refit without the highest-

z

executor. Artifacts outputs/analyses/regime_uncertainty_v1/ and osl_pca_v1/ .

		number of bounded criteria
domain	criteria	printed	noise in recovery, $95\%$ of draws	noise in the index, expected	without the top executor	battery’s first component	equally spaced ranks
humor	$278$	$41$	$30$ – $46$	$44.0$	$41$	$3$	$129$
news homepages	$132$	$13$	$6$ – $15$	$11.0$	$10$	$4$	$28$
peer review	$84$	$9$	$6$ – $12$	$8.6$	$9$	$3$	$29$
creative writing	$367$	$2$	$2$ – $8$	$2.4$	$2$	$2$	$67$
math	$137$	$0$	$0$ – $3$	$0.4$	$0$	$0$	$9$

Appendix

Table 11: Bounded criteria by domain under each check. Noise in recovery: each executor’s recovery redrawn from its standard error (

2{,}000

draws). Noise in the index: each executor’s capability index redrawn from its split-half standard error, expected count over

100

draws. Without the top executor: each curve refit without its highest-

z

executor. The last two columns refit every curve on a differently built capability axis (Table 13 ). Artifacts outputs/analyses/regime_uncertainty_v1/ and osl_pca_v1/ .

contrast (regime / type)	share in regime vs. other regimes	permutation $q$	$q$ , recovery noise	$q$ , capability-index noise	$q$ , without the top executor	log-odds, domain effects only	$+H(m)$ and reliability
reaches / local patterns	$34\%$ vs. $17\%$	$.001$	$.0004$	$.0004$	$.001$	$.92$ $[.62,1.22]$	$.77$ $[.46,1.09]$
rising / whole-text structure	$28\%$ vs. $19\%$	$.009$	$.0004$	$.0004$	$.09$	$.46$ $[.17,.78]$	$.45$ $[.14,.76]$
bounded / identity and social group	$23\%$ vs. $6\%$	$.03$	$.011$	$.008$	$.19$	$.92$ $[.23,1.57]$	$.80$ $[.08,1.46]$
rising / verifiable features	$29\%$ vs. $22\%$	$.31$	$.24$	$.13$	$.50$	$.18$	$.16$
bounded / beyond-text features	$15\%$ vs. $10\%$	$.42$	$.40$	$.41$	$.44$	$.27$	$.12$

Appendix

Table 12: The five regime-by-type contrasts under the checks.

q

from within-domain permutation with false-discovery control, on the printed verdicts, weighted by membership probabilities under recovery noise, weighted by membership probabilities under capability-index noise, and on the verdicts refit without each curve’s most capable executor. Log-odds with

95\%

domain-clustered bootstrap intervals from the nuisance regression (intervals omitted for the two contrasts that are null in every model). Artifacts outputs/analyses/nuisance_regression_v1/ , regime_uncertainty_v1/ and osl_pca_v1/ .

capability axis	rank corr. with $z$	rising	reaches	bounded	verdicts changed	contrasts surviving (of five)
battery $z$ (the paper)	$1.00$	$884$	$321$	$65$	—	$3$ : reaches/local, rising/whole-text, bounded/identity
$z$ redrawn from its measurement error ( $100$ draws)	—	$674$ – $1{,}013$	$192$ – $484$	$46$ – $100$	$3\%$ (modal)	the same $3$ (probability-weighted)
first component of the battery’s five sub-scores	$.97$	$859$	$373$	$12$	$30\%$	$1$ : reaches/local
first component of the recovery matrix (circular)	$.94$	$1{,}012$	$198$	$47$	$27\%$	$0$
executor ranks, equally spaced	$1.00$	$269$	$728$	$273$	$56\%$	$3$ : reaches/local, bounded/identity, bounded/beyond-text

Appendix

Table 13: The regime verdicts under other capability axes. Every curve is refitted with the shipped rule and the planted reference is recomputed on each axis; noisy verdicts are omitted from the counts (

0

;

2

–

28

;

26

;

13

;

0

). Contrasts are the five regime-by-type contrasts of Section 3 , within-domain permutation with false-discovery control at

q<.05

. Artifact outputs/analyses/osl_pca_v1/ .

Figure 14: OSL patterns for tasks, grouped into

k=12

(top) and

k=8

(bottom) categories : the full codebook underlying the main text’s

6

-category view (Fig. 5 ), which is a nested merge of these labels.

examples added	chosen by; label shown	criteria	models	change in recovery vs. definition (bits)	vs. arbitrary examples
$0$ – $5$ items that move the transmitter toward the labels two judges agree on	Llama-3.1-8B; label chosen with the item	$236$	Llama-70B, Qwen-72B	pooled $+.002$ ; $3$ criteria above the noise bound at both judges ( ${\sim}0.6$ expected)	arbitrary sets set the bound
$3$ YES and $3$ NO items the definition’s reconstruction labels most differently	each judge; its own label	$153$ / $146$	Llama-70B, Qwen-72B	pooled $+.001$ ; $0$ above the bound at both judges	–
$3$ near-identical pairs labeled oppositely	each judge; its own label	$125$ / $112$	Llama-70B, Qwen-72B	pooled $+.004$ ; $2$ above the bound at both judges	–
six items chosen for the receiver’s decoding	Llama-3.1-8B with Qwen2.5-14B; label chosen with the item	$130$	Llama-70B	bounded $-.001$ $[-.017,+.016]$ ; controls $+.002$ $[-.016,+.019]$	$+.008$ $[-.006,+.021]$ ; $-.001$ $[-.022,+.019]$
the same, with fixed labels	the same pair; transmitter’s own label	$130$	Llama-70B	bounded $+.002$ $[-.019,+.020]$ ; controls $+.019$ $[-.001,+.039]$	$-.013$ $[-.029,+.003]$ ; $+.005$ $[-.009,+.021]$
six items chosen at the judge	Llama-70B; its own label	$48$	Llama-70B	bounded $+.028$ $[-.002,+.062]$ ; controls $+.028$ $[+.005,+.055]$	$+.014$ $[-.016,+.048]$ ; $+.026$ $[-.007,+.059]$

Appendix

Table 14: Every version of the examples test. Changes are means over criteria, with

95\%

bootstrap intervals where they were computed. “Bounded” and “controls” are the

65

bounded criteria with a bank definition and one matched reaches or rising criterion each. Where two counts are given, they are the criteria covered at Llama-70B and at Qwen-72B. The last column compares the chosen examples with arbitrary examples carrying the same kind of label.

transmitter	definition	own examples	headings swapped	random labeled items	own minus random	own minus swapped	labeled as written
		change against the definition
Qwen2.5-3B	$.001$	$.000$ $[-.001,+.000]$	$-.001$ $[-.001,-.000]$	$-.001$ $[-.001,-.000]$	$+.001$ $[+.000,+.001]$	$.000$ $[-.000,+.001]$	$.51$
Llama-3.1-8B	$.008$	$-.001$ $[-.002,-.001]$	$-.002$ $[-.003,-.002]$	$-.001$ $[-.002,-.001]$	$.000$ $[-.001,+.001]$	$+.001$ $[+.000,+.002]$	$.79$
Qwen2.5-7B	$.042$	$-.005$ $[-.009,-.002]$	$-.010$ $[-.013,-.007]$	$-.005$ $[-.008,-.002]$	$.000$ $[-.004,+.003]$	$+.004$ $[+.002,+.007]$	$.77$
Phi-4	$.035$	$-.008$ $[-.012,-.005]$	$-.012$ $[-.015,-.008]$	$-.001$ $[-.005,+.002]$	$-.007$ $[-.010,-.004]$	$+.004$ $[+.001,+.006]$	$.80$
Qwen2.5-14B	$.036$	$.000$ $[-.003,+.004]$	$-.012$ $[-.015,-.008]$	$+.005$ $[+.002,+.008]$	$-.004$ $[-.008,-.001]$	$+.012$ $[+.008,+.016]$	$.74$

Appendix

Table 15: Examples the transmitter writes itself ,

449

criteria from eight tasks; Phi-4 ran where the press-release items were unavailable, so its row covers the other seven tasks,

389

criteria. Definition-only recovery in bits, then mean changes over criteria with

95\%

bootstrap intervals. “Random labeled items” are four real design items labeled by the transmitter under the definition alone. The last column is the share of its own four examples that the transmitter, given only the definition, labels the way it wrote them. Each model is its own transmitter and receiver.

task	Llama-70B	Qwen2.5-14B
math answers	$-.010$ $[-.018,-.002]$	$-.008$ $[-.019,+.003]$
public comments	$-.012$ $[-.025,-.001]$	$-.007$ $[-.014,-.002]$
creative writing	$-.003$ $[-.010,+.003]$	$-.014$ $[-.023,-.005]$
humor	$-.007$ $[-.022,+.007]$	$+.005$ $[-.002,+.012]$

Appendix

Table 16: Written examples minus real labeled items , by task: mean change in recovery in bits, with

95\%

bootstrap intervals over the criteria of each task (

60

,

60

,

60

and

82

).

	Qwen2.5-14B			Llama-70B
receiver output (executor sees only this)	bounded ( $65$ )	controls ( $65$ )	all ( $449$ )	bounded ( $65$ )	controls ( $65$ )	all ( $449$ )
rule (existing channel, $R=2$ )	$.034$ $[.023,.047]$	$.044$ $[.029,.060]$	$.036$ $[.031,.041]$	$.076$ $[.053,.100]$	$.078$ $[.053,.106]$	$.043$ $[.037,.050]$
$4$ items, random	$.013$ $[.009,.019]$	$.021$ $[.014,.030]$	$.022$ $[.019,.025]$	$.048$ $[.033,.065]$	$.049$ $[.034,.066]$	$.030$ $[.026,.035]$
$4$ items, receiver-chosen	$.011$ $[.007,.015]$	$.021$ $[.014,.029]$	$.022$ $[.019,.025]$	$.047$ $[.032,.063]$	$.055$ $[.038,.074]$	$.030$ $[.026,.035]$
$8$ items, receiver-chosen	$.024$ $[.015,.033]$	$.031$ $[.020,.042]$	$.026$ $[.023,.030]$	$.055$ $[.037,.074]$	$.060$ $[.040,.082]$	$.035$ $[.030,.041]$
all $30$ items	$.031$ $[.021,.043]$	$.038$ $[.025,.053]$	$.028$ $[.024,.032]$	$.056$ $[.040,.074]$	$.059$ $[.040,.080]$	$.036$ $[.031,.042]$

Appendix

Table 17: The receiver answers with examples instead of a rule (each of Qwen2.5-14B and Llama-70B as transmitter, receiver and executor;

449

criteria). Recovery

I(m;m^{\prime})

in bits, mean over criteria with

95\%

bootstrap intervals. “Bounded” and “matched controls” are the

65

bounded criteria with a bank definition and their matched reaches or rising criteria. Noise floor: two standard deviations of run-to-run reconstruction noise is

.104

bits per criterion at Qwen2.5-14B and

.110

at Llama-70B.

Figure 15: What each process writes: concept vs. measurement content (unsupervised tasks; two-pass labeling; the two-way collapse shown here has

\kappa=.93

, against

\kappa=.68

for the full 8-way codebook, whose distribution this bar chart summarizes, with dominant subtypes named in each bar). Tacit decompression writes concept content exclusively; optimization, whether reflective or recombinative, writes measurement content almost exclusively. The divide is crossed only for esoteric concepts that the judge does not already hold.

Figure 16: Tacit-knowledge isomorphism (humor bank, criterion “Concise Wording and clarity”). Recovery is

.049

/

.050

/

.048

bits, with overlapping bootstrap

95\%

intervals (

[.027,.082]

/

[.026,.083]

/

[.011,.106]

), for a name to Llama-70B, a name-plus-definition to Llama-8B, and a name-plus-definition-plus-explanation to Llama-3B. The figure illustrates the pattern that Section 3 measures at scale; the three values are copied by hand from an earlier freeze. The 8B condition’s reference executor is self-consistency-advantaged.

	bounded	reaches	rising
form reliability, all executors (median)	$.95$	$.95$	$.94$
form reliability, top three executors	$.98$	$.97$	$.96$
pairwise Spearman among the top three	$.22$	$.35$	$.36$
Spearman with the consensus, top three	$.40$	$.63$	$.52$
median YES rate, top three	$.15$	$.36$	$.29$
share with YES rate outside $[.2,.8]$ , top three	$.56$	$.32$	$.51$

Appendix

Table 18: Four checks on the shipped panel labels, by regime. Medians over (executor, metric) points; “top three” are the three highest-

z

executors. The refit rows recount the

1{,}270

fitted metrics with the alternative

y

(planted reference recomputed with the same

y

); metrics that no longer fit either curve form are counted as noisy and omitted here (

43

,

42

and

32

respectively).

transmitter	criteria	one-class	short-class	rule recovery, all	rule recovery, two-class criteria
Qwen2.5-14B	bounded ( $65$ )	$.40$	$.69$	$.005$	$.055$ ( $n{=}32$ )
	matched controls ( $65$ )	$.42$	$.54$	$.004$	$.080$ ( $n{=}36$ )
	other ( $319$ )	$.31$	$.50$	$.009$	$.028$ ( $n{=}193$ )
Llama-70B	bounded ( $65$ )	$.28$	$.38$	$.013$	$.106$ ( $n{=}43$ )
	matched controls ( $65$ )	$.29$	$.35$	$.022$	$.088$ ( $n{=}43$ )
	other ( $319$ )	$.24$	$.39$	$.011$	$.023$ ( $n{=}219$ )

Appendix

Table 19: One-sided labels by stratum. Share of criteria whose transmitter labels every demonstration item one way (one-class) or leaves fewer than two items on one side among the thirty shown (short-class), and the median rule-channel recovery in bits, over all criteria and over those with at least two items per class. Computed from the per-criterion rows of outputs/exp_examples_shown/summary_{qwen25-14b,llama70b}.json .

stratum	$30$ items	$60$ items	$90$ items	change, $30$ to $90$
bounded ( $56$ )	$.042$ $[.028,.057]$	$.026$ $[.017,.036]$	$.027$ $[.018,.038]$	$-.014$ $[-.023,-.006]$
matched controls ( $56$ )	$.051$ $[.032,.070]$	$.043$ $[.027,.061]$	$.040$ $[.026,.058]$	$-.011$ $[-.019,-.003]$

Appendix

Table 20: Rule-channel recovery by the number of labeled items the receiver sees. Mean

I(m;m^{\prime})

in bits with

95\%

bootstrap intervals over criteria, Qwen2.5-14B throughout; the interaction (bounded change minus controls’ change) is

-.004

[-.015,+.008]

, and no criterion gains more than two noise standard deviations.

	$n$	$y$ at GPT-5.6	$y$ at the panel’s top three	$y$ at the panel’s best	above the planted reference
bounded	$65$	$.31$	$.52$	$.61$	$0/65$
matched controls	$65$	$.31$	$.62$	$.70$	$1/65$
planted rules	$40$	$.58$	$.67$	$.76$	–

Appendix

Table 21: A frontier executor on the scaling instrument. Medians over metrics with a finite value of the reliability-corrected agreement

y

with the panel consensus: at GPT-5.6 (

z=4.78

), at the panel’s three highest-

z

executors (mean of the three), and at the single panel executor with the highest

y

on that metric. The last column counts metrics whose frontier

y

exceeds the planted reference of their domain. Planted-rule accuracy against the code truth on the same probes:

.93

at GPT-5.6,

.86

at the panel’s strongest executor; battery AUC

.992

.

pair (base vs. post-trained)	$y$ agreement (scaffold vs. chat)	min. planted-rule AUC ratio	usable metrics	$z$ : base / post-trained scaffold / post-trained chat	family ladder
Qwen2.5-3B	$.55$	$.85$	$.92$	$1.14$ / $1.23$ / $1.47$
Qwen2.5-7B	$.80$	$.86$	$.78$	$1.76$ / $2.03$ / $1.66$	not monotone
Qwen2.5-14B	$.80$	$.83$	$.89$	$1.57$ / $1.61$ / $2.00$
Qwen2.5-32B	$.87$	$.96$	$.94$	$2.78$ / $3.00$ / $2.55$
Llama-3.2-3B	$.77$	$.61$	$.56$	$0.41$ / $1.36$ / $1.17$
Llama-3.1-8B	$.86$	$.76$	$.68$	$1.60$ / $1.72$ / $1.56$	monotone

Appendix

Table 22: Function gates for the base models on all

1{,}387

metrics. Preregistered thresholds after the addendum:

y

agreement

\geq.8

, planted-rule ratio

\geq.85

, usable share

\geq.75

,

z

increasing with size within a family. The Qwen2.5-7B pair passed all four on the

131

-metric gate sample and misses the first by

.001

on the full set. The Qwen2.5-32B pair passes the three function gates on the full set.

pair	bounded	reaches	rising	bounded minus others
Qwen2.5-3B	$-.009$ $[-.042,+.022]$	$+.006$ $[-.011,+.024]$	$-.019$ $[-.032,-.006]$	$+.003$ $[-.032,+.037]$
Qwen2.5-7B	$+.017$ $[-.003,+.039]$	$+.040$ $[+.029,+.051]$	$+.035$ $[+.028,+.044]$	$-.020$ $[-.041,+.004]$
Qwen2.5-14B	$+.024$ $[-.005,+.055]$	$+.039$ $[+.026,+.052]$	$+.051$ $[+.041,+.062]$	$-.023$ $[-.053,+.008]$
Qwen2.5-32B	$-.040$ $[-.069,-.014]$	$-.013$ $[-.022,-.005]$	$-.011$ $[-.017,-.004]$	$-.028$ $[-.059,-.002]$
Llama-3.2-3B	$+.015$ $[-.039,+.065]$	$+.077$ $[+.054,+.098]$	$+.052$ $[+.039,+.065]$	$-.042$ $[-.099,+.009]$
Llama-3.1-8B	$+.044$ $[+.020,+.067]$	$+.108$ $[+.091,+.125]$	$+.123$ $[+.113,+.133]$	$-.076$ $[-.100,-.051]$

Appendix

Table 23: Post-trained minus base recovery under one scaffold , by regime: mean over metrics of

y_{\text{post-trained}}-y_{\text{base}}

with

95\%

bootstrap intervals, metrics with unusable base labels excluded. Positive means the post-trained sibling recovers more. Artifact: outputs/exp_posttrain_regimes/ladder_v3/summary_7pairs.json .

Criterion	What was measured	Agreement	Verdict	Citation
Absurdism and non sequitur technique	Absurdity humor present in TV scenes	$\alpha$ = .82; 3 trained coders, 10% of 107 series	high (close)	( van der Wal et al., 2020 )
Accumulation and format subversion	Nearest: joke mechanism, e.g. exaggeration	$\kappa$ = .71; 3 annotators, 600 sentences	none found	( Zhang et al., 2019 )
Balancing humor with pathos	Regular vs dark humor in short videos	$\alpha$ = .72; 3 annotators, 200 videos	moderate (loose)	( Johns et al., 2026 )
Character and impersonation craft	Satirical vs regular news sentences	$\kappa$ = .40; 6 annotators, 13,873 sentences	moderate (loose)	( Smădu et al., 2025 )
Comedic narrative structure and closure	Which stand-up transcript lines are funny	86.7% raw agreement; 11 raters, 6 transcripts	moderate (close)	( Romanowski et al., 2025 )
Concrete imagery and vivid specificity	Word concreteness ratings (single words)	$r$ = .92 with older norms (agg.); 37,058 words	high (loose)	( Brysbaert et al., 2014 )

Appendix

Table 24: Human agreement on the 41 bounded humor criteria in the closest published study: 6 high, 15 moderate, 6 low and 14 none found (all 65 criteria: 8 high, 20 moderate, 9 low, 28 none found).

Criterion	What was measured	Agreement	Verdict	Citation
Creative writing
Epistolary form: structure, timing, authenticity	Nearest: crowd genre labels for web pages	Fleiss $\kappa$ = .874, 88.2% agreement; 42 annotators	none found	( Asheghi et al., 2014 )
News homepages
AI use disclosure and fidelity	Nearest: spotting AI-generated news images	$\alpha$ = .22; 3 raters per item	none found	( Huang et al., 2024 )
Accuracy, verification and transparency	Fact-checkers rate the same articles or claims	$\kappa$ = .38 (3 checkers, 207 articles); $\kappa$ = .52 (2 outlets)	moderate (close)	( Allen et al., 2021 ; Lim, 2018 )
Agenda setting and attribute salience	Media issue emphasis vs voters’ issue salience	$r$ = +.967 / +.979 (agg.); 100 voters, 1968	high (close)	( McCombs & Shaw, 1972 )

Appendix

Table 25: Human agreement on the 24 bounded creative-writing, news-homepage and peer-review criteria in the closest published study: 2 high, 5 moderate, 3 low and 14 none found (all 65 criteria: 8 high, 20 moderate, 9 low, 28 none found).

Family	$n$	What was measured	Agreement (statistic, raters, items)	Verdict (match)	Citation
Creative writing
Figurative language; exposition; structure and pacing; setting	22	Expert pass/fail tests (TTCW)	$\kappa$ : devices .37, exposition .27, structure .39, coherence .27, setting .33; 3 of 10 experts, 48 stories	low (close)	( Chakrabarty et al., 2024 )
Query, pitch, market fit	5	Reviewer agreement on manuscripts	mean $\kappa$ = .17, ICC = .34; 48 studies	low (loose)	( Bornmann et al., 2010 )
Humor and satire	3	Crowd funniness ratings of tweets	$\alpha$ = .16; 27,282 tweets	low (loose)	( Castro et al., 2018 )
Translation fidelity	1	MT adequacy and fluency ratings	$\kappa$ = .23, .25; WMT 2007 judges	low (close)	( Callison-Burch et al., 2007 )
Endings; theme; dialogue and voices; originality	17	Expert pass/fail tests (TTCW)	$\kappa$ : ending .48, theme .66, voice .44, originality .40–.41; 48 stories	moderate (close)	( Chakrabarty et al., 2024 )

Appendix

Table 26: Human agreement on the 321 reaches criteria by family: a study was found for 185 (13 high, 98 moderate, 74 low), none for 136. Families of one task with the same study and verdict share a row.

Family	$n$	What was measured	Agreement (statistic, raters, items)	Verdict (match)	Citation
Creative writing
Plot structure; causality; twists; pacing; character; exposition; sensory detail; figurative language; setting	101	Expert pass/fail tests (TTCW)	$\kappa$ : structure .39, coherence .27, pacing .39, character .31, exposition .27, setting .33, devices .37; 3 of 10 experts, 48 stories	low (close)	( Chakrabarty et al., 2024 )
Query, pitch, market fit	14	Reviewer agreement on manuscripts	mean $\kappa$ = .17, ICC = .34; 48 studies	low (loose)	( Bornmann et al., 2010 )
Motivation, conflict, stakes	13	Crowd labels of character motives	$\alpha$ = .24–.31; crowd, ROCStories	low (close)	( Rashkin et al., 2018 )
Emotional impact	9	Sentence emotion labels in tales	$\alpha$ = .39; 3 annotators	low (close)	( Christ et al., 2022 )
Argument and exposition	4	Expert argument-quality ratings	$\alpha$ $\leq$ .22 (7 raters); best pair .35–.39	low (close)	( Wachsmuth et al., 2017 )

Appendix

Table 27: Human agreement on the 884 rising criteria by family, part 1 (creative writing, humor, mathematical exposition). Over all rising criteria a study was found for 494 (15 high, 196 moderate, 283 low) and none for 390. Columns and verdict thresholds as in Table 24 .

Family	$n$	What was measured	Agreement (statistic, raters, items)	Verdict (match)	Citation
News homepages
Inclusive language; harm	11	Hate speech, offensiveness ratings	$\alpha$ = .18–.29; 56 raters, 20 tweets	low (close)	( Roß et al., 2016 )
News vs opinion; argument	5	Adults sort fact from opinion	near chance, splits by party; 5,035 adults	low (close)	( Mitchell et al., 2018 )
Sensationalism and clickbait	3	Clickbait strength of headlines	$\kappa$ = .21; 5 raters, 38,517 tweets	low (close)	( Potthast et al., 2018 )
Storytelling; surprise	3	Story surprise and empathy ratings	ICC2k = .28, .34 (agg.); 1,056 stories	low (loose)	( Chhun et al., 2022 )
Framing and context	2	Annotators mark policy frames	$\alpha_{U}$ = .08–.23 by issue	low (close)	( Card et al., 2015 )

Appendix

Table 28: Human agreement on the rising criteria by family, part 2 (news homepages, peer review, press releases, notice and comment). Counts as in part 1.

Articulation Upper Bounds for Language Models

Abstract

Figures & tables

Appendix

LLM Evaluation

On the Limits of Prompt-Conditioned Language Models as General-Purpose Learners

The Communication Bottleneck: A Round-Trip Study of Tree-Structured Expression Serialization in Language Models

LLMs Can Better Capture Human Judgments--With the Right Prompts

Abstract

LLM Evaluation

Explore similar work

On the Limits of Prompt-Conditioned Language Models as General-Purpose Learners

The Communication Bottleneck: A Round-Trip Study of Tree-Structured Expression Serialization in Language Models

LLMs Can Better Capture Human Judgments--With the Right Prompts