The AI Evaluation Ecosystem

Yash Dave; Sang T. Truong; Serena Wang; Sanmi Koyejo

cs.AIOct 7, 2026

The AI Evaluation Ecosystem

Authors: Yash Dave, Sang T. Truong, Serena Wang, Sanmi Koyejo

Organizations: Stanford University · University of British Columbia

Abstract

AI evaluation shapes the decisions of model providers, users, funders, and regulators. We argue that designing valid benchmarks requires contextualizing design choices in the dynamics of this ecosystem of actors. We develop a simulation architecture that combines rule-based market dynamics with LLM-driven strategic actors, building on advances in Generative Agent-Based Modeling (GABM). We model benchmarks, consumer needs, and provider capabilities as vectors over a six-dimensional capability space (reasoning, coding, knowledge, safety, communication, agentic), with structural information partitions across actors. As a case study, we apply this stylized simulation to explore benchmark holdout design. We find that moving from public benchmarks to private holdout benchmarks shrinks the gap between benchmark scores and user satisfaction on most benchmarks but widens it on a few, depending on where holdout weights shift scoring credit. We stress-test our findings at both the instrument and case-study level, drawing on the V&V framework of Sargent (2013) and GABM-specific evidence criteria. Beyond holdout design, our simulation is a hypothesis-generating sandbox for studying how evaluator and policy choices, in turn, reshape the ecosystem.

Figures & tables

Figure 1: The AI evaluation ecosystem: six modeled actor types and the feedback loops between them, along three pathways. Fainter arrows are softer institutional channels. Grey dashed boxes mark actors and adjacent supply chains [ Hopkins et al., 2025 ] that the simulation leaves out.

Figure 2: One 40-round LLM-driven run, calibrated baseline , Claude Opus 4.6, seed

43

( core_privacy/llm/claude-opus-4-6/baseline/seed_43 ). (a) Stacked market share. (b) Per-provider incident markers (shape = severity) and regulator interventions (vertical dashed lines, color = action on the escalation ladder). (c) Portfolio allocations: R&D (solid) and safety (dotted) per provider, share-weighted mean in bold. (d) End-of-run per-benchmark score (colored) vs matched consumer satisfaction (gray); red = benchmark overpromises, blue = underpromises.

Figure 3: (a) Per-benchmark gap

g

(published score

-

matched consumer satisfaction, where matched satisfaction is ground-truth capability scored against consumer need-weights and averaged over segments, each weighted by its size and by its alignment with the benchmark’s dimension weights) across the five privacy conditions, for five representative benchmarks (one per primary capability dimension; full

13

-benchmark version in Appendix F.2 ). Foreground: LLM Sonnet 4.6 (

N{=}10

) with

95\%

CIs. Background: heuristic mode (

N{=}50

, all five conditions).

\Delta g

per row

=

gap@ private_only

-

gap@ public_only . (b)

\Delta g

vs. population capability surplus on each benchmark’s primary dimension (capability on the highest-weighted dimension minus the cross-dimension mean, computed from public_only seeds), over

11

benchmarks; Agentic Tasks and Function Calling are excluded because the agentic dimension starts far below the other five (initial capability

0.08

0.12

against

0.25

0.55

, Table 5 ), which makes their surplus values extreme. The exclusion is conservative: Appendix F.2 reports the fit on all

13

, where the slope tightens.

	Instrument checks (design & calibration)	Case study validation (holdout design)
Approach	Sargent V&V activities; each structural assumption rated calibrated, anchored, or stipulated.	Mechanism-isolation tests on the per-benchmark $\Delta g$ result.
Evidence	Three-tier visibility enforced structurally (Appendix B ); parameter grounding ( $K{=}3$ calibrated, cosine targets $0.85/0.95$ anchored, need-weights anchored in sector-level evidence; Appendices C.1 and C.2 ).	Cross-mode sign agreement on 12 of 13 benchmarks; cross-model replication on Opus 4.6 and GPT-5.5 ( $N{=}3$ seeds each; Appendix F.7 ); iid_holdout channel isolation; within-benchmark rescoring (Appendix F.2 ); fitted null recovers $97\%$ of $\Delta g$ variance (Appendix F.5 ).

Table 1: Validation evidence by task. The instrument column records design guarantees and calibration provenance; case-study validation stress-tests the holdout-design

\Delta g

result.

Question (policy framing)	Simulation slot-in	Strategic angle
2. Conflicts of interest (Arena; Scale SEAL). Does monetizing submissions redirect capital toward aggressive submitters at consumer cost?	Small-sample comparison in Appendix H.1 . Per-submission fees (R&D budget); best-of- $N$ selection; paid early access.	Larger-budget providers capture more submissions; funders read inflated scores as quality signals; capital flows to aggressive submitters rather than providers best serving users.
3. Transparency mandates (EU AI Act Art. 53; CA SB-53; NY RAISE Act). Do forced allocation or capability disclosures reduce gaming?	New regulator lever: exposes part of provider private state (R&D mix, internal eval results) to public.	Providers face disclose-vs-window-dress tradeoff; tests whether visibility of the R&D mix changes Goodhart pressure or just changes the narrative around it.
4. Audit & verification (UK AISI pre-deployment agreements; WH 2023 voluntary commitments). How effective are audits at enforcing voluntary safety commitments?	Regulator lever: provider commits to a forward safety allocation at $t$ ; regulator verifies at $t{+}k$ with penalty on violation.	Game-theoretic: provider can over-promise, under-promise, or credibly commit. Tests whether self-binding works when verification is credible, and how it interacts with incident pressure and market share.

Table 2: Additional case studies the ecosystem simulation can host. Case 1 (holdout design) is the worked example in § 5.1 and case 2 (conflicts of interest) has a small-sample comparison in Appendix H.1 ; cases 3 and 4 are presented as architectural slot-ins rather than worked studies. Further slots (media shadow, benchmark sponsorship) in Appendix H .

Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Order of events in one round, with the phase numbers of Algorithm 1 . In Phases 1 to 4, providers plan with the public signals of the previous round, the evaluator scores them, providers update their beliefs, and incidents and media coverage follow. In Phase 5, consumers, the regulator and funders respond in that order to the round’s public state.

Dimension	Represents	Benchmark anchor	Investment pathway
Reasoning	Problem solving, logic, multi-step inference	MMLU, GPQA	Reasoning post-training, CoT RLHF
Coding	Software engineering, debugging	HumanEval, MBPP	Code-specific data, correctness RL
Knowledge	Factual recall, domain expertise	MMLU subsets	Domain corpus, retrieval fine-tuning
Safety	Harmlessness, adversarial robustness	TruthfulQA	RLHF for harmlessness, red-teaming
Communication	Instruction following, fluency	MT-Bench, IFEval	Instruction-following RLHF
Agentic	Tool use, multi-step task execution	SWE-bench, WebArena	Trajectory RL, tool-use fine-tuning

Appendix

Table 3: Capability dimensions, real-world anchors, and primary investment pathways.

Benchmark	Tier	Reas.	Code	Know.	Safe.	Comm.	Agent.	$\cos_{\text{part.}}$	$\cos_{\text{priv.}}$	Real analog
General Capability	public	0.35	0.08	0.30	0.05	0.20	0.02	0.97	0.89	MMLU
Coding Evaluation	public	0.10	0.78	0.03	0.01	0.02	0.06	0.99	0.93	HumanEval/MBPP
Safety Evaluation	partial	0.03	0.01	0.05	0.80	0.10	0.01	0.99	0.92	TruthfulQA/BBQ
Instruction Following	public	0.08	0.02	0.05	0.04	0.80	0.01	0.98	0.86	MT-Bench/IFEval
Scientific Reasoning	partial	0.78	0.02	0.15	0.00	0.04	0.01	0.98	0.92	GPQA/MMLU-Pro
Clinical Reasoning	public	0.20	0.01	0.65	0.08	0.05	0.01	0.98	0.89	Med-PaLM/MedQA

Appendix

Table 4: The default suite of 13 benchmarks in order of introduction, with each benchmark’s hidden dimension weights and privacy tier. Four benchmarks are active at round 0, and the evaluator introduces the other nine one at a time ( benchmark_introduction_cooldown

=4

). The dynamic evaluator mode draws from a pool of 22 benchmarks that covers the same six dimensions. The privacy tier ( public , partial or private ) determines how scores are reported (Appendix C.4 ). The split of 8 public, 3 partial and 2 private benchmarks is a modeling choice. The partial tier is patterned on SEAL-style leaderboards and the private tier on FrontierMath-style holdouts. The columns

\cos_{\text{part.}}

and

\cos_{\text{priv.}}

give the realized cosine between the benchmark’s public weights and its holdout weights when the benchmark is partial and when it is private (Appendix C.4 ). The last column names the real benchmarks that each row is patterned on. The four round-0 benchmarks are patterned on MMLU [ Hendrycks et al., 2021 ] , HumanEval [ Chen et al., 2021 ] and MBPP [ Austin et al., 2021 ] , TruthfulQA [ Lin et al., 2022 ] , and MT-Bench and IFEval.

Provider	Reas.	Code	Know.	Safe.	Comm.	Agent.	R&D	Safety	Prod.	Notes
Orion Labs	0.52	0.48	0.50	0.42	0.52	0.12	55%	15%	30%	GPT-3.5 frontier
Apex AI	0.48	0.40	0.46	0.55	0.48	0.10	60%	30%	10%	Safety lead (CAI)
Genesis Sys.	0.50	0.38	0.52	0.40	0.42	0.12	70%	15%	15%	Largest compute
Mirage AI	0.44	0.42	0.44	0.32	0.40	0.10	80%	10%	10%	Scale-first
Spark AI	0.36	0.40	0.32	0.28	0.34	0.10	65%	10%	25%	Specialization startup
OpenCore	0.38	0.42	0.35	0.25	0.30	0.08	75%	10%	15%	Open-source

Appendix

Table 5: Initial provider profiles: capability in each dimension, and the starting portfolio split across R&D, safety and product. Capability values range from 0.08 to 0.55, and agentic capability is low for every provider. We set the values by hand, informed by 2023 frontier-model benchmark results [ Maslej et al., 2024 ] .

Archetype	LB trust	Switch cost	Switch thresh.	Cost sens.
Leaderboard follower	0.85	0.05	0.10	0.15
Experience-driven	0.35	0.08	0.06	0.30
Cautious	0.50	0.20	0.18	0.20
Enterprise cautious	0.25	0.35	0.20	0.10
Enterprise growth	0.45	0.25	0.15	0.15
Enterprise established	0.35	0.40	0.18	0.08

Appendix

Table 6: Consumer archetypes and behavioral parameters.

Profile	Reas.	Code	Know.	Safe.	Comm.	Agent.	Pop.
software_dev	0.22	0.55	0.03	0.02	0.03	0.15	0.14
content_writer	0.10	0.02	0.20	0.03	0.63	0.02	0.08
legal	0.30	0.01	0.35	0.22	0.10	0.02	0.04
healthcare	0.12	0.02	0.28	0.48	0.08	0.02	0.05
finance	0.35	0.08	0.22	0.25	0.05	0.05	0.05
educator	0.15	0.02	0.28	0.08	0.45	0.02	0.07

Appendix

Table 7: Consumer need weights by use-case profile. Bold indicates the dominant dimension. Need weights are assigned to each profile from the sector-level evidence listed in Appendix C , together with occupation-level adoption rates [ Bick et al., 2024 ] . Population weights are adoption-weighted shares assigned from the same adoption data, with organizational adoption surveys [ Maslej et al., 2025 , Singla et al., 2025 ] as context. The Pop. column holds base shares that sum to 1.04; the simulation rescales them within each consumer type so that organizational segments hold 0.25 of the market at round 0, rising to 0.55 by round 40 (Appendix B.7 ). The Pop. avg row gives the resulting values at round 0.

Lever	Real analog	Cool.	Effect
Voluntary commitment	WH/Seoul pledges	3 r	Safety public_comms boost (+0.15, decaying)
Publish advisory	AISI summaries, NIST RMF	4 r	Enterprise and cautious archetypes reduce LB trust ( $\times$ 0.90, floor 0.15)
Safety disclosure	EU AI Act Art. 53	6 r	Safety comms boost (+0.20); rnd_efficiency $\times$ 0.95
Commission audit	AISI pre-deployment eval	8 r	rnd_efficiency $\times$ 0.85 for 3 rounds
Impose sanction	EU AI Act fines (7%)	10 r	rnd_efficiency penalty + funder allocation $\times$ 0.90
Emergency investigation	Cruise robotaxi shutdown	0 r	Critical-incident override; immediate intervention regardless of cooldowns

Appendix

Table 8: Regulator intervention levers with real-world analogs, cooldowns, and mechanical effects. The cooldowns apply in heuristic mode; in LLM mode the code enforces only a three-round gap after any intervention.

Type	Pattern	Funds OS?	Scoring formula
VC	Concentrated	No	share_growth $\times$ $m$ $\times$ (1 $-$ incident_risk) $\times$ (1 $-$ share) $\times$ diversification
Corporate	Multi-relationship	Yes	share $\times$ $m$ $\times$ (1 $-$ incident_risk)
Government	Spread, safety-oriented	Yes	(1 $-$ incident_rate) $\times$ share_growth
Foundation	Ecosystem health	Yes	(1 $-$ incident_rate) $\times$ share_growth

Appendix

Table 9: Funder types and allocation logic.

Parameter	Value	Parameter	Value
Ecosystem structure		Provider initialization
Providers	6	Capability dimensions	6
Benchmark pool (total)	22 (max 13 active)	Init. capability (range)	$[0.08,\;0.55]$
Consumer segments	51	Benchmark orientation	0.80 (fixed)
Funder types	4	Belief learning rate	0.10 / 0.15 / 0.20 (by profile)
Simulation window	40 rounds ( $\approx$ 3.3 yr)

Appendix

Table 10: Summary of simulation hyperparameters.

Parameter / Design Choice	Related literature
Parameter values
Market growth rate, 0.03/mo	S&P Global Market Intelligence [2025] (40% CAGR)
Consumer population weights, per-archetype (assumed)	Bick et al. [2024] (occupation-level adoption)
Leaderboard trust differs across users, 0.25–0.85 (assumed)	Hardy et al. [2025]
Incident base rates, 20% / provider / round (assumed)	Responsible AI Collaborative [2021]
Structural design choices

Appendix

Table 11: Related literature for simulation parameters and design choices. Each row gives the parameter or design choice and the literature it draws on; a value not stated in a cited source is a modeling choice.

Assumption	Status	Support and sensitivity
Six actor types cover the ecosystem, with heterogeneity inside each type	Stipulated	Tractability; excluded actors and levers in Appendix G.2 .
Consumers choose through a mix of leaderboard signal and experienced quality	Anchored (trust differs across users); stipulated (trust values, mixture weights)	Hardy et al. [2025] ; values in Appendix B.7 .
Gaps are measured against the need weights of a mixed consumer population	Anchored (which needs dominate each profile, from sector evidence); stipulated (the exact values)	Sector evidence listed at the start of this appendix and Bick et al. [2024] ; need-weights set the starting gap and its sign, so which shifts count as widening (§ 5.3 , item 4).
Six capability dimensions with uneven benchmark coverage	Anchored	§ 5.3 , item 2; the ontology is text-only (see below).
Benchmark dimension loadings (Table 4 )	Stipulated	Author-assigned, guided by each benchmark’s real analog. Every score and every gap is computed against them. Suite composition is discussed below.
Holdout weights	Anchored (target cosines); stipulated (directions and realized distances)	Targets anchored to retro-holdout score inflation in Haimes et al. [2024] and to Epoch data [ Epoch AI, 2024 ] ; the vectors are hand-authored, their realized cosines differ across benchmarks (Appendix C.4 ), and which named benchmarks widen depends on them (§ 5.1 ).

Appendix

Table 12: Status of the assumptions behind the holdout-design result.

Dimension	$n_{p}$	$n_{b}$	$n_{c}$	Simple mean	$n$ -weighted mean	Median of medians
Knowledge	8	2	12	3.86	3.71	3.05
Reasoning	8	4	17	2.95	2.97	2.47
Math	8	5	33	4.27	3.43	3.03
Coding	4	3	10	3.72	3.22	3.07
Agentic	5	4	10	4.56	3.85	3.25
Safety	3	1	3	2.97	3.11	2.50

Appendix

Table 13: Pooled

K_{\text{advance}}

(months) by analytical dimension, across the eight-lab frontier set.

n_{p}

= providers with at least one advancing (provider, benchmark) cell;

n_{b}

= benchmarks in the dimension;

n_{c}

= cells contributing.

Provider	$n_{b}$	$n_{a}$	Mean $K$	Median $K$	SD
OpenAI	17	127	4.39	2.57	4.92
Anthropic	17	117	3.08	2.88	1.14
Google	17	74	3.86	3.58	1.43
Meta	9	47	5.21	4.75	3.30
xAI	8	23	2.94	3.03	0.59

Appendix

Table 14: Per-provider cadence (

K_{\text{advance}}

in months) across the primary benchmark set, tier-1 labs.

n_{b}

= benchmarks with at least one advance;

n_{a}

= total advances observed.

Factor	Formula	Rationale
Safety investment	$1-(\text{portfolio[safety]}\times 1.5)$	Higher safety $\to$ lower probability
Market share (exposure)	$(0.3+\text{share}\times 1.5)\times\sqrt{\text{total\_market}}$	Larger market $\to$ higher probability
Incident history	$+0.02$ per prior major/critical in last $15$ rounds, cap $+0.10$	Past harm $\to$ elevated future risk; aging window lets safety culture recover
Active sanction	$\times 0.75$ while sanctioned	Regulatory oversight reduces probability

Appendix

Table 15: Incident probability scaling factors.

Figure 5: Market share of Orion Labs across

N{=}50

LLM runs (

5

privacy conditions

\times

seeds

42

–

51

; Claude Sonnet 4.6). Gray lines are individual runs; the two colored lines are the runs at the ends of the range ( private_dominant , seed 45, final share 0.07; iid_holdout , seed 46, 0.81). Markers:

\times

= critical incident,

\triangle

= major incident.

	Observable system	Non-observable system
Subjective approach	None (no observable counterfactual ecosystem)	Explore model behavior: exogenous shock (Appendix E.6 ); an ablation that removes incidents (Appendix F.4 ). Comparison to other models: cross-mode heuristic $\leftrightarrow$ LLM (Appendix F.2 ); cross-model Sonnet/Opus/GPT-5.5 (Appendices E.7 and F.7 ).
Objective approach	None (not pursued)	Comparison to other models, statistical: fitted $\Delta g$ null model (Appendix F.5 ).

Appendix

Table 16: Operational-validity techniques used in this paper, mapped to the operational-validity classification of Sargent [2013] . Cells marked None hold no technique, because real ecosystem trajectories under counterfactual privacy regimes are not observable, so all evidence sits in the right column.

Figure 6: Reasoning-frame distribution across matched 3-way triples. Each stacked bar shows the share of planning entries for that (provider, model) tagged with each frame. Privacy framing (purple) is roughly half-rate in GPT-5.5 relative to both Anthropic models; incident framing (red) shows the steepest cross-model gradient (Sonnet

\gg

Opus

>

GPT-5.5). Tagging uses fixed keyword and regular-expression patterns.

Activity	Evidence	Location
Data validity	Empirical anchors: $K{=}3$ cadence, target cosines $0.85$ and $0.95$ , 2023 capability vectors, need weights anchored in sector-level evidence and occupation-level adoption in Bick et al. [2024] , incident base rate informed by the AI Incident Database	Appendices C and C.1
Conceptual model validation	Face validity on real-world incentive structure (media, funder, regulator); pattern-oriented check (three design targets, two emergent patterns); PIMMUR Profile, Interaction and Realism	Appendices E.2 and E.4
Computerized model verification	Three-tier visibility partition enforced by the architecture; rounds.jsonl instrumentation; JSON retry and per-actor fail-safe; PIMMUR Memory, Minimal-Control and Unawareness	Appendices E.3 and E.4
Operational validation	Exogenous-shock probe (explores model behavior under a designed perturbation); an ablation that removes incidents; cross-mode heuristic $\leftrightarrow$ LLM; cross-model Sonnet/Opus/GPT-5.5; fitted $\Delta g$ null model	Appendices E.5 , E.6 , E.7 , F.2 , F.4 , F.5 and F.7

Appendix

Table 17: Coverage summary. Sargent V&V activity

\times

evidence presented

\times

location.

Figure 7: Extended version of Figure 3 : same two-panel layout, but both panels now show all

13

benchmarks in the active suite, including the two agentic-calibration outliers (Agentic Tasks, Function Calling) excluded from the main-body forest and panel-b regression. Visual conventions identical to the main-body figure. Direction of

\Delta g

agrees across LLM and heuristic modes on

12

13

benchmarks; the lone disagreement is Adversarial Robustness, within sampling noise of zero in both modes. Panel (b) ’s regression is fit on all

13

points; the two outliers anchor the high-

|\Delta g|

end and tighten the slope substantially relative to the 11-benchmark main-body version.

	LLM Sonnet 4.6 ( $N{=}10$ )		Heuristic ( $N{=}50$ )
Benchmark	private_only	iid_holdout	private_only	iid_holdout
General Capability	0.00	0.40	0.00	0.44
Coding Evaluation	0.00	0.70	0.00	0.42
Safety Evaluation	0.00	0.70	0.08	0.50
Instruction Following	1.00	0.70	1.00	0.68
Scientific Reasoning	0.00	0.40	0.04	0.56

Appendix

Table 18: Fraction of seeds in which the absolute gap

|g|

widens relative to public_only at the same seed, per benchmark. Under private_only , widening is near-deterministic for the benchmarks that widen and absent for most others; under the iid_holdout null (same weights, private reporting) every benchmark sits near a coin flip. Computed by scripts/plots/paper/holdout_widening_table.py .

Figure 8: Heuristic privacy ladder: market-leader benchmark-score vs. matched consumer-satisfaction reliability, pooled across $87{,}617$ leader-rows from $N{=}50$ seeds per condition $\times$ 5 privacy conditions. Diagonal $=$ perfect calibration; below-diagonal $=$ benchmark overpromises. (a) colored by benchmark with per-benchmark overpromise rate; (b) colored by privacy condition with per-condition overpromise rate. The reliability is consistent with the aggregated cloud in Figure 3 and the dimensional structure of § 5 .

Figure 9: Heuristic privacy ladder, market-leader scoring rows colored by provider identity ( $N{=}50$ seeds per condition). The reliability pattern (deviation from the diagonal) holds across the providers that hold the market-leader position; the aggregate finding is not driven by a single provider’s heuristic response.

Model	$r$	$r^{2}$	RMSE from $y{=}x$	MAE
Null (round- $0$ initial conditions)	$0.987$	$0.974$	$0.0097$	$0.0072$
Sim ( $40$ -round end-of-run capability)	$0.988$	$0.976$	$0.0091$	$0.0070$
Sim $-$ Null	$+0.001$	$\mathbf{+0.002}$	$-0.0006$	$-0.0002$

Appendix

Table 19: Null-vs-sim

\Delta g

prediction on the LLM cross-sequence dataset (Claude Sonnet 4.6,

n{=}208

paired cells,

5

-fold CV by seed). The null model is fit on round-

0

capability + portfolio per (provider, dim); the sim model uses the simulation’s

40

-round end-of-run capability. Both feed the same

\Delta g=\mathbf{c}\cdot(\mathbf{w}_{\mathrm{holdout}}-\mathbf{w}_{\mathrm{public}})

identity.

Figure 10: Null vs. sim prediction of $\Delta g$ on the cross-sequence LLM dataset ( $n{=}208$ ). Left: null model fit on round- $0$ initial conditions. Right: sim model using the simulation’s $40$ -round end-of-run capability. Both panels track the $y{=}x$ diagonal closely, and both achieve $r^{2}\approx 0.97$ . The simulation’s $40$ -round trajectory adds $+0.2$ pp $r^{2}$ over the fitted null.

Figure 11: Path A variance decomposition. Left: per-benchmark cross-provider

\sigma^{2}

under regular baseline (

x

) vs. initial_uniform_capability ablation (

y

). The ablation has

n{=}2

seeds, so each

y

value is the variance of two runs. Points at

y{=}0

would mean full null dominance; points on

y{=}x

would mean sim contributes everything. Most benchmarks cluster low along

y

, with Adversarial Robustness as an outlier. Right: aggregate cross-provider

\sigma^{2}

comparison (regular vs. uniform-cap), showing the

\sim

13\times

collapse in variance under ablation (equivalently,

\sim

3.6\times

collapse in

\sigma

). Aggregate ratio

\sigma^{2}_{\mathrm{uniform}}/\sigma^{2}_{\mathrm{regular}}=0.075

Test condition	$n$	$r^{2}_{\mathrm{null}}$	$r^{2}_{\mathrm{sim}}$	$\Delta r^{2}$ (sim $-$ null)
Reference: heterogeneous caps (S0+s5+s8)	$208$	$0.974$	$0.976$	$+0.002$
Held-out: uniform caps ( initial_uniform_capability )	$39$	$\mathbf{0.955}$	$0.975$	$+0.020$

Appendix

Table 20: Path B held-out null replication. The null model is fit on the

62

heterogeneous-cap LLM runs of the primary dataset, then evaluated on a held-out test set of

6

new LLM runs under the initial_uniform_capability ablation. The null’s

r^{2}

drops only

\sim

2

pp despite test-set initial-capability heterogeneity being structurally zero.

Figure 12: Path B held-out null replication (

n{=}39

test cells under initial_uniform_capability ). Null model is trained on

62

heterogeneous-cap runs and predicts the held-out uniform-cap test set. Both null and sim track the

y{=}x

diagonal closely; null

r^{2}=0.955

, sim

r^{2}=0.975

Figure 13: Per-benchmark clean gap by privacy condition, one panel per model. Benchmarks sorted by Sonnet public_only mean descending so the panels share a reading axis. Error bars are

95\%

CIs across seeds (

n{=}10

Sonnet seeds 42–51,

n{=}3

Opus and GPT-5.5 seeds 42–44). The privacy-condition ordering (public at the largest gap, private_only most compressed on high-gap benchmarks) replicates under all three planners.

Figure 14: Paired per-benchmark clean gap at seed 42. One panel per privacy condition with a matched 3-way Sonnet+Opus+GPT-5.5 triple; circle = Sonnet, diamond = Opus, square = GPT-5.5. Dots are provider-mean gap per benchmark. All five privacy conditions have a complete 3-way matched triple at seed 42.

		gap			score			HHI
Condition	Seed	S	O	G	S	O	G	S	O	G
public_only	42	$+0.000$	$+0.001$	$+0.021$	$0.705$	$0.706$	$0.695$	$0.514$	$0.500$	$0.486$
public_only	43	$+0.019$	$+0.071$	$+0.044$	$0.685$	$0.730$	$0.661$	$0.321$	$0.469$	$0.342$
public_only	44	$+0.008$	$+0.043$	$+0.009$	$0.689$	$0.711$	$0.694$	$0.540$	$0.403$	$0.588$
baseline	42	$-0.009$	$-0.008$	$-0.006$	$0.657$	$0.701$	$0.667$	$0.410$	$0.468$	$0.472$
baseline	43	$+0.073$	$+0.043$	$+0.060$	$0.708$	$0.730$	$0.671$	$0.528$	$0.480$	$0.374$

Appendix

Table 21: Endpoint metrics for the

15

matched 3-way triples: all five privacy conditions at seeds

42

43

, and

44

. All values are share-weighted means over the final

5

rounds except HHI (market Herfindahl). gap = score

-

consumer satisfaction. The prose above gives the per-model range of mean safety allocation, which we drop from the table to keep it within the page width. S = Sonnet 4.6, O = Opus 4.6, G = GPT-5.5.

Figure 15: Cross-model market-share path dependence under baseline for Orion Labs. Highlighted: seeds

\{42,43,44\}

\times

three models (color = model, linestyle = seed); all nine cells are populated. Background grey: every other full-40-round LLM run in the core privacy set, across all conditions, seeds and models. Critical incidents marked on every line; major incidents on highlighted lines only. Figure 5 shows the same provider under one model across five privacy conditions; this figure holds the condition fixed and varies the model.

Actor	Modeled	Not modeled
Model providers	• R&D / safety / product portfolio (sums to 1.0) • Per-benchmark focus levels and weight beliefs • Public communications (announcements, press releases) • Optional best-of- $N$ submissions (evaluator-capture) • Six-dim capability vector with sqrt-budget capability gains	• Distillation or model merging across providers • Internal (non-published) benchmark suites that gate release decisions • Training-data vendor relationships and selective contamination of public benchmarks • Multi-model portfolios (one provider shipping several model sizes simultaneously) • Internal red-team findings that never become observable incidents
Consumers	• 51 segments (archetype $\times$ use-case) • Need weights assigned from sector-level evidence on AI use and concerns, with occupation-level adoption data • Switching cost + post-switch cooldown; dynamic enterprise-share growth • Incident-weighted satisfaction with sector-matched 2 $\times$ penalty • Media-attention-driven exploration churn	• Multi-provider routing (one consumer using several models in a portfolio) • Internal use-case-specific evaluation by organizational consumers (private benchmarks they trust above the public leaderboard) • Regulatory-mandated provider choice for specific deployment contexts (healthcare-grade certification, government procurement)
Evaluator	• 22-benchmark pool with three modes: fixed sequence, randomized pool, dynamic LLM-driven • Three benchmark types (public, partial with $h{=}0.3$ , private with $h{=}1.0$ ) and a $K{=}3$ reporting lag • Saturation detection from score deltas; hand-authored holdout weights per benchmark • Optional best-of- $N$ + early-access toggles (evaluator-capture)	• Coalitions or standards bodies (HELM-like consortia, MLCommons) • Multiple competing evaluators with different weightings on the same measurement axis • Benchmark retraction or integrity audits after release • Data-vendor / data-labeling supply chains feeding the benchmark • Conflict-of-interest disclosure requirements and evaluator reputation dynamics (track record affecting which scores are trusted)
Regulator	• Single regulator with a graduated lever ladder (voluntary commitment $\to$ advisory $\to$ disclosure $\to$ audit $\to$ sanction) • Three parameter sets (US light-touch, EU precautionary, balanced); every reported run uses balanced • Per-lever cooldowns in heuristic mode, a three-round gap on the LLM path; incident-rate and concentration triggers • Mandatory safety floor under audit, sanction or emergency investigation	• Industry-specific gatekeeping regulators (FDA for healthcare AI, FAA for aviation AI) • Multiple jurisdictions with mobility or arbitrage between them • Whistleblower channels surfacing hidden conduct • Safe-harbor provisions and self-certification pathways
Funders	• Four types (VC, corporate, government, foundation) with type-specific cooldowns, informed by the cadence of AI funding events between 2020 and 2026 • $193B capital pool with a geometric 7% per month availability curve • Per-funder identity blocks and peer-funder visibility (LLM mode) • Allocation against leaderboard, market shares, regulator interventions, recent funding history	• Compute-credit financing (cloud-provider equity-for-credits arrangements, e.g., Microsoft–OpenAI, AWS–Anthropic) • Strategic vs. purely financial corporate-investor distinction • Exit events (acquisition, IPO) and their behavioral feedback into remaining providers
Media	• Single MediaActor (TechPress) with multi-round narrative state inertia (OPTIMISM / SKEPTICISM / CRISIS) • Per-provider attention vector and sentiment; headline budget per round • Weighted coverage of leaderboard movement, incidents, regulatory actions, provider announcements • Rule-based (not LLM-driven)	• Differentiated media types (trade press vs. mainstream vs. research press) • Social-media amplification (TikTok, Twitter/X, Reddit) and the influencer economy • Engagement-driven coverage (algorithmic feedback between virality and what gets covered next)

Appendix

Table 22: Per-actor action-space choices: scope modeled in the simulation vs. extensions not implemented. Each row lists the strategy levers exercised by the actor alongside concrete real-world levers the architecture would need to extend to host an adjacent question.

Seed	$\Delta$ HHI	$\Delta$ leader share	$\Delta$ mean safety	$\Delta$ Apex share	$\Delta$ Apex safety	$\Delta$ gap
$42$	$-0.019$	$-0.032$	$-0.009$	$+0.069$	$-0.022$	$+0.011$
$43$	$-0.039$	$-0.041$	$-0.012$	$+0.093$	$-0.022$	$+0.008$
Mean	$-0.029$	$-0.037$	$-0.011$	$+0.081$	$-0.022$	$+0.010$

Appendix

Table 23: Paired deltas ( evaluator_capture minus baseline ) in LLM mode at seeds

42

and

43

N{=}2

. Apex share and Apex safety are provider-specific; the remaining columns are ecosystem-level.

\Delta

gap uses the per-benchmark gap of § 5 , which is the published score minus matched consumer satisfaction, averaged over the

13

benchmarks and the six providers in the final round.

Explore similar work

Interactive Evaluation Requires a Design Science
May 18, 2026Keyang Xuan, Peiyang Song, Pan Lu +10Scientific Discovery
University of Texas Austin · California Institute of Technology · Carnegie Mellon University +5
AI Cartography: Mapping the Latent Landscape of AI Benchmark Ecosystems
May 24, 2026Michael Hardy, Anka Reuel, Lijin Zhang +6Artificial Intelligence Benchmarks Leaderboard
Stanford University, Stanford, CA, USA · BroadMetrics, Edison, NJ, USA
BEAMS: Benchmarking and Evaluating AI for Modeling and Simulation
May 27, 2026Sara Metcalf, William SchoenbergArtificial Intelligence Evaluation User Simulation
ISEE Systems

May 18, 2026, cs.AIJ/K move · Enter open · S save

Interactive Evaluation Requires a Design Science

Keyang Xuan, Peiyang Song, Pan Lu, Pengrui Han+9

University of Texas Austin · California Institute of Technology · Carnegie Mellon University+5

Scientific Discovery

Open paper AskPDF arXiv

AI evaluation is undergoing a structural change. Large language models (LLMs) are increasingly deployed as systems that act over time through tools, environments, users, and other agents, while many evaluation practices still inherit assumptions from response-centered benchmarks (e.g., fixed inputs, isolated outputs, and outcome judgments that can be made from a single response). The field has begun to build interactive benchmarks, but the resulting landscape is fragmented: benchmarks differ in what interaction artifacts they admit, how trajectories are scored, and what claims their results support. This position paper argues that interactive evaluation should be treated as a principled evaluation paradigm, not merely a new family of agent benchmarks. Simply adopting previous evaluation paradigms does not suffice. We define evaluation as an autonomous mapping from evidence to judgments, and show that interactive evaluation changes both sides of this mapping: the evidence becomes interaction-generated trajectories, while the evaluation procedure must assess process, recoverability, coordination, robustness, and system-level performance. Building on this definition, we propose a two-axis taxonomy, derive design principles and reporting standards, examine representative scenarios, and analyze how longstanding evaluation challenges reappear at the trajectory level.

Abstract

Explore similar work

Interactive Evaluation Requires a Design Science

AI Cartography: Mapping the Latent Landscape of AI Benchmark Ecosystems

BEAMS: Benchmarking and Evaluating AI for Modeling and Simulation