Each run of an AI-driven research system (ADRS) is an expensive search over a vast solution space, and dependable evaluation requires many runs, making run data both costly to produce and valuable to retain for large-scale analysis. Yet this data remains fragmented: teams operate in isolation, ADRS frameworks emit results in different formats, and no shared infrastructure exists to aggregate or compare runs across problems and systems. We present ROAR, a solution for systematically unifying and analyzing heterogeneous ADRS outputs. ROAR addresses two challenges: reconciling heterogeneous ADRS outputs and enabling analytics across runs with different objectives and scoring functions. We achieve this through a relational schema and parsing layer that normalize heterogeneous ADRS outputs while preserving data lineage and temporal structure, and accommodating new systems without requiring schema modifications. From building a corpus of more than 900 runs from multiple ADRS, we show how pooled data can reveal properties of problem landscapes that are difficult to observe. Consistent with prior work, runs with identical configurations may converge to different scores. We find that many runs realize most gains early, and that the effectiveness of different strategies for incorporating prior solutions into the search process varies across problems. We further show that the pooled corpus is actionable and not merely analytical by using ROAR to configure ADRS runs. Together, these results illustrate how pooled ADRS data can expose problem-dependent structure in search behavior that is difficult to detect from any single system, team, or benchmark. Such cross-cutting insights are difficult to obtain while runs remain siloed; ROAR is the first infrastructure designed to unify them.
Figures & tables
Figure 1: Basin structure for Polynomino Packing for the 9 configurations with the most runs; the full set of 14 configurations with n≥10 appears in Figure 4 (Appendix F ). Color indicates classification: magenta = confirmed bimodal, orange = inconclusive due to CI, blue = unimodal. Labels report sample size, point-estimate BC, and bootstrapped 95% CI.
Figure 2: Calibration of rule-of-3 across three problems. The dashed line at 63% is the ceiling of the heuristic. Observed escape rates fall far below it.
Figure 3: Factor importance for predicting score improvement, by problem. Bars are log-odds of improvement per 1 SD of each factor; positive (green) indicates the factor is associated with more improvements, negative (red) with fewer. ∣C↑∣ , ∣C↓∣ give the improving and non-improving candidate counts.
Problem
Cfgs
Campaigns
Gain
95% CI
Polynomino
9
115
+18.9
[+11.6,+25.6]
Knapsack
3
37
+16.9
[−1.0,+33.6]
Table 1: RQ4: gain of the ROAR budget policy over a single 50-iteration campaign, per problem. Campaigns is the number of campaigns in the included configurations; each is held out once and scored against the policy fit on the others. Gain is the mean difference in best score. Intervals are 95% bootstrap confidence intervals over campaigns, with the policy refit on each resample.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Metric
Count
Campaigns (runs)
928
Candidates
48,210
Distinct generators
14
Distinct mechanisms
6
Appendix
Table 2: Dataset summary statistics.
Problem
Runs
Cands
Mechs
Models
Palindrome Ham. Path
290
17,181
3
10
Polynomino Packing
353
16,293
3
12
Bounded 2D Knapsack
271
13,857
3
12
CUDA Attention
3
481
2
—
CUDA Layer Norm
4
269
2
—
Dispatcher Cold Path
6
68
5
1
Appendix
Table 3: Problem coverage in the dataset. The three combinatorial problems dominate the corpus; CUDA kernel problems provide signal on continuous landscapes; single-campaign problems are excluded from statistical analyses.
Table 11: Data processing performance across pipeline stages.
Figure 4: Basin structure for Polynomino Packing for all 14 configurations with n≥10 runs; 16 configurations with fewer runs omitted. Color indicates classification: magenta = confirmed bimodal, orange = CI inconclusive, blue = unimodal. Labels report sample size, point-estimate BC, and bootstrapped 95% CI. Of 14 configurations, 7 are confirmed bimodal, 6 are inconclusive, and 1 is unimodal.
Figure 5: Basin structure for Bounded 2D Knapsack at the configuration level. Each violin represents runs from a single configuration. Color indicates classification: magenta = confirmed bimodal, orange = CI inconclusive, gray = deterministic. Of 10 configurations, 5 are confirmed bimodal, 3 are inconclusive, and 2 are deterministic.
Figure 6: Escape rate vs. stagnation length by mechanism for Bounded 2D Knapsack.
Figure 7: Escape rate vs. stagnation length by mechanism for Polynomino Packing.
Figure 8: Early diversity (first 25% of iterations) vs. final score for Bounded 2D Knapsack (left), Polynomino Packing (center), and CUDA Attention (right). Only Polynomino Packing shows a significant relationship, which is positive; Bounded 2D Knapsack shows no early-diversity signal, and CUDA Attention is inconclusive given the limited sample size. Each panel title reports the Spearman rank correlation ρ between early diversity and final score and its two-sided p -value.
Figure 9: Early diversity (first 25% of iterations) vs. final score by mechanism for Bounded 2D Knapsack.
Figure 10: Early diversity (first 25% of iterations) vs. final score by mechanism for Polynomino Packing.
Figure 11: Early diversity vs. final score by model for Bounded 2D Knapsack.
Figure 12: Early diversity vs. final score by model for Polynomino Packing.
Figure 13: Within-decile code diversity across score deciles. Runs are binned into ten equal-sized groups by best-candidate score (decile 1 = lowest, decile 10 = highest). Lower diversity indicates greater convergence among solutions in that performance tier.
Figure 14: Factor importance by model and mechanism for Bounded 2D Knapsack. Cells are GLMM log-odds per SD. Each panel shows one mechanism; rows are models, columns are factors. Hatched cells are not statistically significant ( p≥.05 , Wald test). The negative direction is consistent across most model–mechanism combinations, with Gemini-2.5-Flash as a notable outlier.
Figure 15: Factor importance by model and mechanism for Polynomino Packing. Cells are GLMM log-odds per SD. Each panel shows one mechanism; rows are models, columns are factors. Hatched cells are not statistically significant ( p≥.05 , Wald test). An overall more positive trend is observed.
Figure 16: Per-configuration gain of the ROAR budget policy over current practice (one campaign using the full budget) at an identical 50-iteration budget, for the 12 tested configurations. Points are leave-one-out estimates; bars are 95% out-of-bag refit bootstrap intervals (1,000 draws), blue where the interval excludes zero. The right-hand column gives each configuration’s fitted policy (kstop,n) .