Evaluating a stochastic AI system often requires many repeated samples per instance, especially for metrics defined over long horizons. Recent work has shown that modeling instance-level success probabilities with a Beta distribution recovers Pass@k from far fewer samples than direct estimation. We show that this framework supports a broader range of evaluation targets. We fit a mixture of Beta distributions by marginal maximum likelihood, with the number of components chosen by BIC. From that one fit, we compute Passk and AUC@k at any horizon in closed form. We also derive delta-method confidence intervals for these metrics and for Pass@k, which require no additional model calls. Treating configurations rather than tasks as instances extends the same model to sensitivity analysis. For repeated success metrics, we find that mixture priors yield more accurate estimators than naive estimation across competition mathematics and graduate-level science benchmarks when extrapolating beyond the observed number of trials, and the resulting intervals cover the full-data reference. When prompts are treated as instances, it estimates the distribution of prompt sensitivity more accurately than a comparable item-response baseline without external covariates. Direct estimators remain preferable once trials reach the target horizon.
Figures & tables
Figure 1: From 5 trials per task, Mixture tracks the 80-trial reference at every horizon. One GPQA group; band: 95% interval (Appendix D ).
Figure 2: Once k>m (vertical line), naive error grows faster than Mixture’s. 104 groups, m=10 .
Figure 3: On AIME 2025 (30 tasks, m=5 ), all Mixture intervals cover the reference.
Figure 4: Without external covariates, Mixture beats pIRT at every budget. Mean W1 error of the prompt score distribution by number of observed cells B .
Dataset
Tasks
Naive
Mixture
Win rate (%)
Trace
30
12.90
4.84
96
Math
30–53
14.77
6.55
95
GPQA
3,600
13.65
1.42
100
Table 1: Mixture cuts error 2–10 × in every dataset. MAE at k=60 , m=5 , mean of Pass 60 and AUC@60.
k≤m
k>m
Dataset
Metric
Naive
Mixture
Naive
Mixture
Trace
Pass k
1.58
1.85
7.07
3.51
Trace
AUC@ k
0.83
1.46
3.24
2.21
Math
Pass k
1.93
2.34
9.46
4.97
Math
AUC@ k
0.66
1.38
2.38
2.06
GPQA
Pass k
1.19
0.16
5.95
0.83
Table 2: Mixture wins whenever k>m ; naive wins on Trace and Math when k≤m . MAE by regime.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Short name
Collections
AceReason-Nemotron-1.1-7B
AceReason-7B
Trace
Bespoke-Stratos-7B
Bespoke-7B
Trace
DeepSeek-R1-Distill-Qwen-1.5B
R1-Distill-1.5B
Trace
EXAONE-4.0-1.2B
EXAONE-1.2B
Trace
FuseO1-DeepSeekR1-QwQ-SkyT1-Flash-32B-Preview
FuseO1-32B
Trace
gpt-oss-20b_high
gpt-oss-high
Trace, Math, GPQA
Appendix
Table 3: Models, short names, and the collections that use them.
Figure 5: All 60 intervals cover the 80-trial reference. Same as Figure 3 for all 20 Trace models.
Trace
Math
GPQA
All
Tol. (pp)
Mix.
Naive
Fewer (%)
Mix.
Naive
Fewer (%)
Mix.
Naive
Fewer (%)
Mix.
Naive
Fewer (%)
2
30
50
62
45
60
44
5
55
94
30
50
62
3
15
35
81
22.5
40
72
5
37.5
94
15
35
81
4
10
25
88
17.5
32.5
78
5
27.5
94
10
25
88
5
5
20
94
10
20
88
5
22.5
94
5
20
94
6
5
15
94
7.5
20
91
5
20
94
5
15
94
Appendix
Table 4: Mixture reaches every tolerance with fewer trials than naive; at 3 points it needs 15 trials per task instead of 35. Median trials per task needed to reach each error tolerance at k=60 .
Figure 6: Both errors fall as m grows; on Trace and Math, naive overtakes Mixture for Pass@60 and AUC@60 at large m . MAE at k=60 .
k≤m
k>m
Dataset
Metric
Naive
Comb.
Mixture
Naive
Mixture
Trace
Pass@ k
1.19
0.88
1.72
4.89
2.91
Trace
Pass k
1.58
1.03
1.85
7.07
3.51
Trace
AUC@ k
0.83
0.65
1.46
3.24
2.21
Math
Pass@ k
0.95
0.68
1.68
3.70
2.79
Math
Pass k
1.93
1.10
2.34
9.46
4.97
Appendix
Table 5: Combinatorial is the most accurate when k≤m ; Mixture is the most accurate when k>m . MAE by regime, including the combinatorial estimator.
Metric
m
Naive
Comb.
Mixture
Pass@60
5
12.29
–
4.57
Pass@60
10
8.03
–
3.75
Pass@60
20
4.46
–
2.96
Pass@60
40
2.29
–
2.12
Pass@60
60
1.61
1.16
1.69
Pass@60
80
1.27
0.00
1.37
Appendix
Table 6: Trace: MAE at k=60 by m . Combinatorial needs m≥60 .
Metric
m
Naive
Comb.
Mixture
Pass@60
5
8.82
–
3.77
Pass@60
10
5.85
–
3.38
Pass@60
20
3.25
–
2.96
Pass@60
40
1.98
–
2.08
Pass@60
60
1.35
0.92
1.82
Pass@60
80
1.07
0.00
1.45
Appendix
Table 7: Math: MAE at k=60 by m . Combinatorial needs m≥60 .
Metric
m
Naive
Comb.
Mixture
Pass@60
5
17.93
–
1.59
Pass@60
10
11.67
–
1.24
Pass@60
20
6.64
–
0.61
Pass@60
40
3.45
–
0.38
Pass@60
60
2.32
0.09
0.20
Pass@60
80
1.74
0.00
0.13
Appendix
Table 8: GPQA: MAE at k=60 by m . Combinatorial needs m≥60 .
Dataset
Metric
Estimator
Seed 0
Seed 1
Seed 2
Mean
SD
Trace
Pass@ k
Naive
4.96
5.09
4.63
4.89
0.24
Trace
Pass@ k
Mixture
2.93
2.77
3.01
2.91
0.12
Trace
Pass k
Naive
6.79
6.99
7.44
7.07
0.34
Trace
Pass k
Mixture
3.34
3.55
3.64
3.51
0.15
Trace
AUC@ k
Naive
3.31
3.40
3.03
3.24
0.19
Trace
AUC@ k
Mixture
2.24
2.11
2.27
2.21
0.08
Appendix
Table 9: Mixture beats naive for every seed, and the seed standard deviation is at most 0.34 points. Extrapolation MAE ( k>m ) for each sampling seed.
Table 16
Figure 7: Mixture beats pIRT at almost every quantile, most clearly in the tails. MAE at five quantiles of the prompt score distribution.
Figure 8: BIC almost always selects H=1 ; GPQA supports a second component as m grows. Share of fits in which BIC selects each H .
Figure 9: Naive wins for k near or below m ; Mixture wins as k grows past m . Trace MAE as k grows, one column per m .
Figure 10: Same pattern as Trace. Same as Figure 9 for Math.
Figure 11: Mixture is more accurate at every m and k . Same as Figure 9 for GPQA.