Kernel Autoresearch for Open-Ended Model Discovery

Richard Cornelius Suwandi; Feng Yin; Kevin Murphy

cs.LGOct 7, 2026

Kernel Autoresearch for Open-Ended Model Discovery

Authors: Richard Cornelius Suwandi, Feng Yin, Kevin Murphy

Organizations: CUHK-Shenzhen · University of British Columbia

Abstract

Kernels encode the inductive bias of a wide range of machine learning models, yet automated kernel design faces a fundamental dilemma. A fixed grammar of base kernels and operators guarantees validity but limits the search to structures expressible by those building blocks. Conversely, unrestricted programs remove this limitation but no longer guarantee validity. In our stress tests, 22-58% of LLM-generated kernels that pass numerical checks on random inputs fail when evaluated at different scales or dimensions. We propose Kernel Autoresearch (Kernaut), which treats kernel design as open-ended model discovery. Coding agents write kernels as programs, while construction contracts ensure that every accepted kernel is valid. A quality-diversity archive retains high-performing kernels with distinct behaviors, and novelty screening steers agents toward functionally new candidates. Our experiments demonstrate that the discovered kernels encode reusable inductive biases that generalize to unseen tasks. On held-out black-box optimization families, a discovered kernel outperforms a meta-learned deep kernel trained on the same episodes. Furthermore, kernels discovered from ten enzyme-kinetic rate laws achieve lower error than tuned ARD and deep kernel baselines on five unseen mechanisms. The discovered kernels are also interpretable programs that human researchers can refine: a human-refined version of one further reduces the held-out predictive error by 5.7% and optimization regret by 7.8%.

Figures & tables

Figure 1: The Kernaut framework. The coding agent (blue) writes kernel components under one of four construction contracts. The deterministic backend (orange) applies three acceptance tiers, T0 to T2, and only T2 programs enter the quality-diversity archive, where each cell keeps the best candidate for one behavior. Archived kernels and structured feedback guide later proposals. The grey module computes the posterior and tunes parameters.

Figure 2: Synthetic regression: search archives. (a) Validation continuous ranked probability score (CRPS, Equation ( 11 ), lower is better) of every accepted program in 14 campaigns. E1–E7 use the three-model ensemble, and F1–F7 use GPT-6 Astra as the sole proposer. Stars mark validation-selected programs, and the dashed line marks the best tuned baseline. (b) Lineage of the first ensemble campaign (E1). Edges join each parent to its revisions, and the starred node is the selected program.

Figure 3: Frozen BBO results on 60 held-out test tasks (lower is better). Left: CRPS. Right: regret AUC, the mean normalized simple regret over BO steps. Points are means. Bars are paired 95% bootstrap intervals for the gap to DWF, where each line ends. Rows are ordered by average rank across both metrics. Fixed kernels use the setting selected on training tasks. Grid spectral mixtures (GSM), input warping, and random Fourier features (RFF) average five fitted variants.

Figure 4: Forecasting: held-out greenhouse-gas records. The discovered kernel is the ensemble-selected residual period-bank kernel. Each panel shows, for one record, the window with its median CRPS. Values are in parts per trillion (ppt). SF 6 is forecast forward. The chlorofluorocarbon records CFC-12 and CFC-11 are backcast, so their held-out stretch precedes the dotted line. Bands are 95% intervals. RBF misses the turn in both CFC records. Its smooth posterior mean follows the broader trend of the training window and starts above the last observation.

Figure 5: Enzyme kinetics and glucose dynamics: transfer to unseen mechanisms and patients. (a) CRPS reduction (%) relative to Matérn- $5/2$ on held-out enzyme mechanisms, with each campaign’s validation CRPS below. Purple means lower error and brown means higher error. Validation selected E1 and F2, shown in bold. (b) Normalized mean squared error (MSE, Equation ( 25 ) of Appendix E.2 , log scale) for adult continuous glucose monitor (CGM) forecasts after a 45-minute prefix, against training episodes per patient. Validation selected the discovered kernels. References are tuned ARD-Matérn, per-patient type-II maximum-likelihood (ML-II) fitting, and the best fixed kernel.

Appendix figures & tables42 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: The archive fills niche by niche over a campaign. MAP-Elites archive cells after runs 3, 6, 9, and 12 for the first ensemble campaign (E1, top) and the first GPT-6 Astra campaign (F1, bottom). Rows are niches and columns are novelty bands. Each filled cell shows its current elite (best search score), colored by validation CRPS (darker is better) and labeled with the run that produced it. A ring marks the validation-selected program, which is DWF in the first run of E1. Cells are indexed by niche and novelty band only, so the feature-growth descriptor of Equation ( 4 ) is omitted.

Weighting	Weights	DWF rank	Elites kept
Original	$(.55,.30,.10,.05)$	1	9/9
AUC-heavy	$(.30,.55,.10,.05)$	1	9/9
Equal	$(.25,.25,.25,.25)$	1	9/9
CRPS-only	$(1,0,0,0)$	1	9/9

Appendix

Table 1: Sensitivity of the first BBO archive to the selection weights. Each row re-scores the nine programs under one weighting of

(\Delta_{\mathrm{C}},\Delta_{\mathrm{AUC}},\Delta_{\mathrm{F}},\Delta_{\mathrm{W}})

in Equation ( 5 ), using the four stored components. “DWF rank” is the position of DWF among the nine by search score, where 1 is best. “Elites kept” counts the programs that remain elites of their cells.

T	Label	What is established
0	Executable	Sandboxed execution succeeds and the output matches the interface
1	Empirical	Randomized Gram-matrix tests pass on sampled input sets
2	Contract certified	Checked interface conformance and trusted PSD-preserving assembly under the stated component assumptions

Appendix

Table 2: Implemented evidence ladder.

Relation	What must match
Determinism	The same input, replayed twice
Permutation	The Gram of a permuted batch equals the permuted Gram
Subset	The Gram of a random subset equals the corresponding submatrix
Extension	Appending then truncating points reproduces the original block
Duplicate	A duplicated point reproduces its own row

Appendix

Table 3: Tier 2 contract-conformance relations. See Equation ( 9 ).

Field	Purpose
mathematical_form	Kernel expression and construction
psd_argument	Proof sketch for positive semidefiniteness
novelty_claim	What is new relative to known kernels
closest_known_kernel	Nearest baseline and how this differs
expected_bo_behavior	Predicted BBO performance characteristics
construction_niche	Declared construction category

Appendix

Table 4: Formulation fields registered before code submission (§ 3.4 ).

Figure 7: Finite numerical screening admits invalid programs. Outcome of each cohort of kernel programs, split into programs that fail to run, fail the public Tier-1 screen, pass it but are falsified on the hidden suite, and survive the hidden suite. Labels give the number falsified among the programs that passed the public screen. Canaries were written to pass the public screen and fail the hidden suite. Certified controls pass Tier 2. Unrestricted programs that survive the hidden suite are empirically unfalsified, not certified.

Family	$d$	Split(s)
Branin	2	train, val
Ackley	2	train, val
Cosine	8	train, val
Hartmann-6	6	train, val
Levy	6	train, val
Bukin	2	test

Appendix

Table 5: Task-family instantiation and split (§ 4.1 , § 4.3.1 ). Meta-training and meta-validation use the same five families with disjoint seed offsets (

0

and

10^{4}

). Meta-test uses six families never seen during search, at offset

2\times 10^{4}

. Each family contributes 10 seeded episodes per split, giving 50 meta-training, 50 meta-validation, and 60 meta-test tasks. Targets are clean function values.

	CRPS $\downarrow$		AUC $\downarrow$
Candidate	val	test	val	test
Linear	0.783	0.791	0.913	0.872
Periodic ( $\ell{=}1$ , $p{=}2$ )	0.549	0.618	0.816	0.678
Rational quadratic (RQ; $\ell{=}0.25$ , $\alpha{=}1$ )	0.559	0.618	0.785	0.706
Matérn-5/2 ( $\ell{=}0.5$ )	0.551	0.612	0.797	0.693
RBF ( $\ell{=}0.25$ )	0.565	0.641	0.823	0.756

Appendix

Table 6: Frozen-benchmark candidates and learned-feature references, with validation and test CRPS and regret AUC. Lower is better. Bold marks the best value within each block.

Figure 8: FSBO needs about two orders of magnitude more meta-training data to match DWF. Test CRPS (left) and regret AUC (right) on the 60 BBO test tasks, lower is better. FSBO is meta-trained on the 518 points that search scored (matched budget) or on 64,000 points (rich budget). Solid lines show native FSBO, which is fine-tuned on every episode. Dotted lines show frozen FSBO, which refits only amplitude and noise. DWF (blue) is archived with 518 points, and the dashed line is fixed Matérn-5/2. Lines join the two measured FSBO budgets and do not interpolate.

Figure 9: Seven ensemble and seven frontier campaigns on the 60 BBO test tasks. The frontier campaigns use GPT-6 Astra. Each dot is one campaign’s validation-selected kernel, bars mark the mean per proposer, and the dashed line is fixed Matérn-5/2 (

\ell{=}0.5

, test CRPS

0.6117

, regret AUC

0.6931

). The black diamond is one Qwen-only campaign (test CRPS

0.616

, regret AUC

0.709

). Lower is better for both metrics. Campaigns that beat Matérn: frontier 7 of 7 on CRPS and 6 of 7 on regret AUC, ensemble 6 of 7 on CRPS and 4 of 7 on regret AUC.

Search	Test CRPS	Mean AUC	Failed calls
Full archive	$\mathbf{0.571\pm 0.031}$	0.681	28
Independent proposals	$0.617\pm 0.037$	0.695	15
Closure grammar	$0.618\pm 0.021$	0.706	22
CKS greedy grammar, depth $\leq 3$	$0.625$	0.684	–

Appendix

Table 7: Search-ablation results on the BBO domain. The first three rows are 12 seeded campaigns per arm, reported as mean

\pm

sample SD. CKS is one deterministic greedy search. Failed calls are model or tool calls that did not yield an accepted response, summed over each arm’s 12 campaigns.

Figure 10: A stronger proposer starts ahead and keeps improving. Best validation CRPS among the programs accepted after each discovery run (lower is better). Thin lines are the 14 individual campaigns. Thick lines are the means over the seven ensemble and seven frontier (GPT-6 Astra) campaigns. The dashed line is the best tuned baseline. A run that accepts no program leaves its curve unchanged.

Figure 11: Validation regret of the validation-CRPS leader. At each discovery run, the plotted value is the validation regret AUC of the program with the best validation CRPS among those accepted so far. Thin lines are the 14 campaigns. Thick lines are means over seven ensemble and seven frontier campaigns. The score is validation regret.

Figure 12: DWF outside the search families. CRPS change relative to Matérn-5/2, where positive values favor DWF. Each synthetic point averages ten episodes for one function and domain variant. Color gives the empirical reflection-symmetry index

S

, defined in the paragraph below. Circles use the standard domain, and triangles use an off-center subdomain. The refit panel clips eight losses below

-50\%

, with a minimum of

-369\%

. Real-regression points average 20 splits for one dataset and training size.

vs. fixed baselines
	Validation ( $n{=}50$ )			Test ( $n{=}60$ )
Competitor	wins	med. $\Delta$ CRPS	$p$	wins	med. $\Delta$ CRPS	$p$
Periodic ( $\ell{=}1$ , $p{=}2$ )	34/50	$-0.009$	$0.013$ *	45/60	$-0.018$	$7.0\times 10^{-5}$ ***
Matérn-5/2 ( $\ell{=}0.5$ )	31/50	$-0.015$	$0.069$	39/60	$-0.013$	$0.023$ *
RQ ( $\ell{=}0.25$ , $\alpha{=}1$ )	39/50	$-0.018$	$3.5\times 10^{-5}$ ***	45/60	$-0.019$	$2.2\times 10^{-5}$ ***
vs. discovered kernels

Appendix

Table 8: Paired Wilcoxon tests of DWF’s per-task CRPS against each competitor (

n{=}50

validation,

n{=}60

test). “Wins” counts tasks where DWF is lower. Med.

\Delta

CRPS is the median signed difference, and negative favors DWF. The

p

-values are unadjusted. Holm correction over the five competitors within each split leaves every 0.05-level conclusion unchanged.

{}^{*}p<0.05

{}^{**}p<0.01

{}^{***}p<0.001

Figure 13: Frozen DWF geometry, where

u

is a coordinate in

[0,1]

. Left: the warp stays near the identity, and the fold is one tent peaking at

u=0.513

(dotted line). Right: covariance with a reference point at

u=0.2

and the dotted line at its fold mirror

u=0.825

. Joint DWF matches Matérn-

5/2

at the mirror, while the additive split stays near

0.70

Figure 14: Prior samples under the discovered kernel. Four draws from unit-variance GPs on

[0,1]

. Panels (a)–(c) share the same standard normals, so the draws differ only through the kernel: (a) Matérn-

5/2

with

\ell=0.5

, (b) joint DWF at its frozen parameters, (c) the refinement

k_{\mathrm{add}}=\tfrac{1}{2}k_{w}+\tfrac{1}{2}k_{a}

. The dotted line marks the fold peak

c=1/(2\nu)=0.513

, and arrows join the mirror pair

u=0.2

and

1/\nu-u=0.825

. (d) Finite-difference slopes near

c

. They jump for DWF (solid) and stay continuous for Matérn (dashed), apart from small ripples from the

10^{-8}

jitter. (e) Prior correlation between

f(u)

and

f(1/\nu-u)

. Joint DWF is within

0.003

of Matérn, the additive kernel stays above

0.57

, and the fold branch alone fixes it at

1

. (f) One DWF draw on

[0,1]^{2}

, with contours bending along

x_{j}=c

. ‡ Human researcher-defined refinement, not search output.

Construction	Test CRPS (repeat mean $\pm$ SD)	Selected val.	Selected test	Selected AUC
Identity Matérn	$0.6198\pm 0.0010$	0.5546	0.6189	0.6824
Warp only	$0.6245\pm 0.0028$	0.5590	0.6223	0.6896
Fold only	$0.6339\pm 0.0283$	0.5519	0.6147	0.6901
Additive branches ‡	$0.5668\pm 0.0000$	0.5020	0.5668	0.6190
Joint branches	$0.6131\pm 0.0071$	0.5404	0.6011	0.6712
Rational quadratic	$0.6123\pm 0.0049$	0.5489	0.6090	0.6896

Appendix

Table 9: Matched predictive-objective controls (tuning protocol described above). Test CRPS mean

\pm

SD is over five tuning repeats on the same 60 episodes, not over five discovery campaigns. Selected columns report the validation-chosen repeat. Archived DWF is a frozen anchor with a different historical search budget. ‡ Human researcher-defined refinement, not search output.

Construction	Test CRPS	Regret AUC
Joint	0.601146	0.671159
Joint, smooth	0.601162	0.671159
Additive ‡	0.566832	0.618955
Additive, smooth ‡	0.566879	0.618955

Appendix

Table 10: BBO sharp/smooth ablation. Entries use validation-selected configurations evaluated on the same 60 test episodes. Lower is better. ‡ Human researcher-defined refinement, not search output.

Per discovery run	BBO	Time-series
Rounds	1–10	8–10
Tool calls	0–11	7–10
Input tokens	4k–190k	69k–184k
Output tokens	0.3k–10k	3k–9.6k

Appendix

Table 11: Agent-loop cost per discovery run, min–max over the 12 runs of each reported search.

Panel of Figure 15 — Figure 15: Share of tool calls by workflow phase, pooled across the 12 runs of each reported search (97 BBO calls, 109 time-series calls). Each phase groups these tools. Formulation: propose_formulation . Draft/verify: stage_candidate and verify_staged_candidate . Parameter search: optimize_parameters and evaluate_parameters . Submission: submit_staged_candidate . Archive query: query_archive and inspect_candidate . Final evaluation: evaluate_candidate .

Method	val	SF 6
Linear	0.826	0.959
Periodic ( $p{\approx}$ annual)	0.241	1.799
RQ ( $\ell{=}0.5$ )	0.095	0.067
Matérn-5/2 $\ell{=}1$	0.091	0.050
Matérn-3/2 $\ell{=}1$	0.099	0.075
RBF $\ell{=}0.5$	$\mathbf{0.086}$	$0.029$

Appendix

Table 12: Frozen greenhouse-gas candidates, fitted reference families, and the closed-grammar CAKE winner: mean CRPS over validation episodes and over ten held-out SF 6 windows (lower is better).

Kernel	All CRPS	Forward CRPS	Reversed CRPS	Forward coverage	Forward width
Residual period-bank (selected)	0.0456	0.0488	0.0391	0.918	0.334
Harmonic–trend root	0.0194	0.0174	0.0235	0.925	0.121
RBF ( $\ell=0.5$ )	0.0264	0.0284	0.0223	0.880	0.153
Uniform ARD-RBF ( $\ell=1$ )	0.0184	0.0178	0.0195	0.916	0.117

Appendix

Table 13: Frozen forecasting diagnostics on 30 SF 6 windows (20 forward, 10 reversed). Coverage and width refer to nominal 95% marginal predictive intervals on forward windows. Targets are clean held-out values, and variance includes fitted observation noise, matching the benchmark CRPS. Widths are in training-standardized units. The results are descriptive, from one record with overlapping windows.

Method	SF 6	CFC-12	CFC-11	Mean	Fwd.
Fixed (RBF)	$\mathbf{0.026}\pm 0.003$	$0.268\pm 0.061$	$0.320\pm 0.055$	$0.205\pm 0.027$	$0.098\pm 0.007$
Best fixed ∗	$0.026\pm 0.003$	$0.182\pm 0.053$	$0.134\pm 0.034$	$0.114\pm 0.021$	$0.024\pm 0.002$
GSM	$1.473\pm 0.047$	$0.812\pm 0.173$	$0.633\pm 0.105$	$0.973\pm 0.069$	$0.690\pm 0.017$
Input warping	$0.103\pm 0.010$	$0.143\pm 0.039$	$0.129\pm 0.026$	$0.125\pm 0.016$	$0.057\pm 0.004$
RFF	$0.052\pm 0.008$	$0.252\pm 0.067$	$0.237\pm 0.033$	$0.181\pm 0.025$	$0.105\pm 0.012$
CAKE	$0.027\pm 0.003$	$0.220\pm 0.063$	$0.189\pm 0.051$	$0.145\pm 0.027$	$\mathbf{0.022}\pm 0.002$

Appendix

Table 14: Mean CRPS

\pm

SEM over 30 held-out windows per record (lower is better). Each unstarred frozen row is selected on validation data. The overall mean weights the three records equally, and Fwd. uses forward-time windows only. ∗ Best fixed selects a fixed baseline separately for each test record. Harmonic–trend selects a discovery by SF 6 test performance only, then evaluates that same discovery on both CFC records. † AutoGP infers a new structure for every window and averages eight predictions.

Figure 16: The same 90 windows, split by time direction. Each point is the equal-record mean CRPS (mean within SF 6 , CFC-12, and CFC-11, then the mean of those three). Forward and reversed windows are separate summaries inside the 48-month benchmark. The final-120-month diagnostic is a different protocol and is not included. Frontier F1–F3 are the validation-selected kernels from the three GPT-6 Astra forecasting campaigns.

Comparator	SF 6	CFC-12	CFC-11	Pooled
Fixed (RBF, $\ell{=}0.5$ )	7/30 ( $7.1\times 10^{-5}$ )	26/30 ( $2.0\times 10^{-6}$ )	25/30 ( $2.4\times 10^{-5}$ )	58/90 ( $7.5\times 10^{-6}$ )
Input warping	26/30 ( $2.6\times 10^{-7}$ )	12/30 ( $0.98$ )	15/30 ( $0.42$ )	53/90 ( $8.7\times 10^{-5}$ )
Random Fourier features	14/30 ( $0.52$ )	21/30 ( $0.0066$ )	27/30 ( $3.9\times 10^{-7}$ )	62/90 ( $4.0\times 10^{-7}$ )
Grid spectral mixture	30/30 ( $1.9\times 10^{-9}$ )	30/30 ( $1.9\times 10^{-9}$ )	29/30 ( $8.0\times 10^{-8}$ )	89/90 ( $2.7\times 10^{-16}$ )
CAKE	2/30 ( $9.3\times 10^{-9}$ )	16/30 ( $0.052$ )	16/30 ( $0.24$ )	34/90 ( $0.92$ )
Best fixed per record ∗	7/30 ( $7.1\times 10^{-5}$ )	17/30 ( $0.20$ )	14/30 ( $0.72$ )	–

Appendix

Table 15: Paired comparisons of the validation-selected discovery (residual period-bank + poly) against each validation-selected reference, over 30 windows per record. Entries give windows won by the discovery and the two-sided Wilcoxon

p

-value. Pooled combines all 90 windows. Windows overlap within records, so

p

-values describe variation within records and not independent replications. Holm correction over the five references within each column leaves every 0.05-level conclusion unchanged.

Figure 17: Posterior forecasts of the validation-selected kernels on the median-CRPS window of each record (rows are methods, columns are records). Black dots are the observed prefix, hollow markers are the held-out 48 months, and light grey dots are the rest of the record. Lines and bands are the predictive mean and 95% interval including observation noise, in parts per trillion (ppt). Both CFC windows are reversed, so the forecast runs backward from the dotted origin. Panel CRPS is on the standardized scale of Table 14 .

Figure 18: Forward forecasts of the final 120 months of each record, a horizon 2.5 times longer than the main benchmark (rows are methods, columns are records). Black dots are the observed prefix, hollow markers are the held-out final ten years, and the dotted line is the forecast origin. Lines and bands are the predictive mean and 95% interval including observation noise, in ppt. Panel CRPS is on the training-standardized scale, and bold marks the lowest value in each column.

Figure 19: Additive posterior decomposition of the selected discovery on CFC-12 (peak-origin split, 120-month horizon). Top row: observed and held-out points with the posterior mean and its two-standard-deviation band. Next rows: the Matérn, constant, and combined sinusoid-bank components, then the training residuals. All rows are deviations from the training mean in ppt, so the component means sum to the full posterior mean. The dashed line is the forecast origin, and the split CRPS is

0.070

Figure 20: The discovered ChemBench kernel is assembled from interpretable biochemical features. Top: substrate saturation, Arrhenius temperature dependence, and the pH window, each normalized to compare shape. At their frozen weights, the temperature and pH features vary by less than $10^{-5}$ and $0.02$ across the input box (Appendix F.2 ). Bottom: the inhibitor–substrate coupling $\xi_{I}\,i\,s(a)$ and the bisubstrate coupling $\tfrac{1}{2}s(a)s(b)$ , the fourth and eighth features of Equation ( 24 ), on a log color scale with white contours at each decade. Both grow with substrate concentration because the saturation midpoint lies above the $100$ mM bound.

Figure 21: Matched ChemBench references and discovery campaigns on the same 75 test episodes. Squares mark each reference’s validation-selected initialization, and bars give mean

\pm

SD over five initializations. Each campaign point is its validation-selected frozen discovery, green if validation CRPS is below

0.70

and grey otherwise. Blue rings mark the campaign selected overall. The dotted line is fixed Matérn-5/2. Historical campaigns keep their original budgets.

Reference (test CRPS)	Ensemble E1 ( $0.555$ )	Frontier F2 ( $0.312$ )
Optimized ARD ( $0.567$ )	$-0.012$ $[-0.033,0.010]$ , $p=0.87$	$-0.256$ $[-0.326,-0.193]$ , 73/75, $p=6\times 10^{-13}$
Coordinate-aware DKL ( $0.581$ )	$-0.025$ $[-0.054,0.001]$ , $p=0.32$	$-0.269$ $[-0.346,-0.202]$ , 71/75, $p=2\times 10^{-12}$
Domain library ( $0.588$ )	$-0.032$ $[-0.054,-0.011]$ , $p=0.13$	$-0.276$ $[-0.347,-0.215]$ , 75/75, $p=3\times 10^{-13}$
Monotone warping ( $0.655$ )	$-0.100$ , $p=3\times 10^{-11}$	$-0.344$ , 71/75, $p=6\times 10^{-13}$
Isotropic Matérn ( $0.712$ )	$-0.156$ , $p=8\times 10^{-13}$	$-0.400$ , 74/75, $p=3\times 10^{-13}$
Shared-coordinate DKL ( $0.783$ )	$-0.228$ , $p=1\times 10^{-12}$	$-0.472$ , 74/75, $p=3\times 10^{-13}$

Appendix

Table 16: Paired comparison of the validation-selected ChemBench kernels with the matched references over 75 held-out episodes. Entries give the mean CRPS difference (discovery minus reference, negative favors the discovery), its within-mechanism paired-bootstrap 95% interval where computed, episodes won, and the Holm-adjusted two-sided Wilcoxon

p

-value across the six references.

Figure 22: Validation CRPS predicts held-out CRPS across ChemBench campaigns. Validation and test CRPS of the validation-selected kernel of each search campaign, nine ensemble and three GPT-6 Astra, on 75 held-out episodes from five mechanisms never seen during search (lower is better, Spearman

\rho=0.95

over the ensemble campaigns). Rings mark the campaign that validation selects within each proposer, and the dashed line is fixed Matérn-5/2 (

\ell{=}1

, test CRPS

0.721

Panel of Figure 23 — Figure 23: Local predictive behavior on two held-out biochemical mechanisms. In each panel, shading shows $\pm 1.96$ predictive standard deviations. The lower strip shows the absolute error of each posterior mean, tinted where the discovered kernel is closer.

Figure 24: GlucoseBench adult test error for every method and setting. CGM normalized MSE (arithmetic mean over queries within each patient, then geometric mean over 10 patients), colored on a log scale where darker is lower. In the column codes, b $k$ means $k$ training episodes and c $m$ means an $m$ -minute observed prefix. Columns show b2 c45 , b3 c45 , b4 c45 , and b6 c45 , followed by prefix lengths of 0, 120, and 180 minutes at six training episodes. Rows are the discovered kernels, the tuned references, and the fixed-kernel and persistence baselines. Rel.-time uses time since the meal and bolus as inputs, and ML-II is per-patient marginal-likelihood fitting. A box marks the best colored cell in each column. Persistence is printed under the heatmap and is left off the color scale.

Discovery	b2 c45	b3 c45	b4 c45	b6 c0	b6 c45	b6 c120	b6 c180
Frontier F1	10, .002 $-67\%$	9, .004 $-48\%$	9, .027 $-31\%$	10, .002 $-26\%$	10, .002 $-23\%$	10, .002 $-18\%$	6, 1.0 $-3\%$
Frontier F2	8, .014 $-38\%$	10, .002 $-42\%$	7, .027 $-25\%$	10, .002 $-26\%$	8, .020 $-22\%$	9, .014 $-17\%$	6, .92 $-2\%$
Ensemble c2	9, .004 $-64\%$	10, .002 $-50\%$	8, .037 $-27\%$	8, .020 $-20\%$	7, .049 $-15\%$	8, .014 $-12\%$	5, .62 $-3\%$
Ensemble c1	10, .002 $-47\%$	6, .28 $-19\%$	5, .77 $-10\%$	8, .027 $-16\%$	7, .23 $-10\%$	7, .56 $-3\%$	2, .020 $+12\%$
Ensemble c3	10, .002 $-50\%$	9, .010 $-33\%$	7, .19 $-20\%$	5, 1.0 $+2\%$	5, .77 $+0\%$	5, .77 $+4\%$	2, .037 $+28\%$

Appendix

Table 17: Paired per-patient comparison with the strongest reference, relative-time ARD-Matérn. In each column, b

k

denotes

k

training episodes and c

m

denotes an

m

-minute observed prefix. Each cell gives patients won out of 10, the raw two-sided exact Wilcoxon

p

-value, and the percentage change in geometric-mean nMSE. Negative percentages are lower error. A win count of 5 is an even split. Holm adjustment over these 35 cells leaves every adjusted

p

-value above

0.05

(minimum

0.068

Figure 25: The selected glucose kernel encodes a delayed meal effect and a fading insulin effect. The two gates of the ensemble campaign-2 feature map after tuning, as functions of time since the meal and since the bolus. Dotted lines mark the 46-minute midpoint of the meal gate and the 23-minute start of the insulin gate. After that start, insulin decays with a time constant of 127 minutes.

Figure 26: With two training episodes the reference undershoots the meal response, with six the gap closes. One adult and one held-out query, forecast after the 45-minute prefix (dotted line) from 2 versus 6 training episodes. Curves are posterior means with 95% intervals for the validation-selected Astra kernel (campaign 1) and relative-time ARD-Matérn. Lower panels show the delivered meal and bolus. The patient is a median case for the ensemble kernel.

Figure 27: Cross-domain reuse of validation-selected kernels. Rows are source kernels and columns are target domains. Each cell gives the target test CRPS change (%) relative to that domain’s fixed reference and the number of evaluation units won. Negative CRPS changes mean lower error and better performance. Purple favors the transferred kernel and brown favors the reference. Stars mark unadjusted paired tests with $p<0.05$ . † The chemistry-source cells use the fixed coordinate conversion described below, so read them as interface-compatibility checks.

Kernel	Report	What tuning settled
DWF (BBO)	The objective varies smoothly, as under a Matérn-5/2 prior, along a slightly rescaled copy of each coordinate. Values at points mirrored about the middle of a coordinate, $u$ and $1.025-u$ , are encouraged but not forced to agree. The prior has a kink at the mirror line $u=0.513$ .	Tuned geometry: the fold frequency settled at $\nu=0.975$ , so the fold is a single tent peaking at $0.513$ that reflects each coordinate once, and the warp stays within $0.027$ of the identity (Appendix C.2 ). The registration had anticipated multi-scale periodicity and boundary sensitivity, which the tuned kernel does not use. The kink is Proposition 4 .
Residual period-bank + poly (forecasting)	The record follows a smooth trend, as under a Matérn-5/2 prior with lengthscale $0.5$ of the normalized span, plus a constant offset, plus a second smooth component built from 36 sinusoids. Every sinusoid has a period between 7 and 47 times the length of the record, so within one record this component acts as a record-wide smooth trend rather than a cycle.	Tuned behavior: the sinusoid frequencies settled at $0.021$ – $0.143$ cycles per normalized span, so the bank models slow curvature across the whole record. The registration described an annual cycle, but it took the annual period (about $0.03$ of the span) as a frequency. An annual cycle would need 25–47 cycles per span. The “poly” feature is a constant.
ChemBench, ensemble E1 (enzyme kinetics)	The rate depends smoothly on a monotone transform of log substrate concentration that is steepest at high concentration, the same transform of the second substrate, their product, and log enzyme loading. Inhibitor and product concentrations matter only in proportion to substrate saturation.	Tuned relevance: as in ARD, tuning decides which inputs matter. Enzyme loading and the two substrates matter most, while inhibitor and product matter little (Figure 28 ), and the saturation midpoint lies above the input range (Figure 20 ). The program includes Arrhenius and pH-window features, but tuning set their weights so that the correlation is $1.000000$ at every base point. These probes show little sensitivity to temperature and pH, without establishing exact invariance.

Appendix

Table 18: Plain-language reports of the headline discoveries. Each report is condensed from the agent’s registered formulation and checked against the frozen program with the trusted interpreter. The right column states what tuning settled.

Kernel	Report	What tuning settled
ChemBench, frontier F2 (enzyme kinetics)	An explicit 248-feature map treats the rate as a linear combination of ten substrate-response shapes modeled on the training mechanisms, each at three substrate scales, their log-substrate sensitivities, and a small linear backbone. The response features are multiplied by enzyme, temperature, and pH factors, so posterior coefficients can select both mechanism shape and environmental modulation.	Tuned relevance: enzyme loading and substrate dominate, product, inhibitor, and temperature matter less, and pH is nearly ignored (Figure 28 ). The frozen kernel therefore retains strong enzyme and substrate effects, some product and inhibitor response, and almost no pH dependence.
GlucoseBench, frontier F1	An ARD-Matérn-5/2 residual is augmented by 36 intervention features. At three response scales, gamma envelopes encode causal meal and bolus transients, long-period wave packets encode response age, and cumulative channels retain dose exposure after the transients fade. Signed and complementary channels allow meal and bolus responses to oppose or covary.	Tuned behavior: signed channels dominate because their mixing parameter is $\rho=0.20$ , while cumulative memory remains active with gain $0.56$ . The base kernel’s time lengthscale is $3.21$ normalized spans and its four intervention lengthscales are $10^{6}$ . On public training inputs, endpoint-sweep correlations are $0.90$ for time, $0.94$ for meal size, $0.96$ for meal start, and $0.98$ for bolus size and start.

Appendix

Table 19: Plain-language reports of the headline discoveries, continued.

Figure 28: The two ChemBench kernels respond to different inputs. Mean kernel correlation as one normalized input moves from

0

v

with the others fixed, over 100 base points. A curve near

1

indicates similar normalized kernel features along the sampled sweeps. Left: ensemble campaign E1 depends mainly on enzyme loading and the two substrates, responds little to inhibitor and product, and shows almost no temperature or pH sensitivity in these sweeps. Right: frontier campaign F2, which uses GPT-6 Astra, depends on enzyme loading and substrate, responds mildly to the product, inhibitor, and temperature, and shows almost no pH sensitivity in these sweeps.

Explore similar work

May 18, 2026cs.LG

Automated Kernel Discovery Towards Understanding High-dimensional Bayesian Optimization

Gaussian Process (GP) kernels are central to Bayesian optimization (BO), yet designing effective kernels for high-dimensional problems still relies on extensive manual engineering. Existing automated approaches struggle in high dimensions for two bottlenecks: their kernel search space is limited to additions and multiplications of base kernels, and LLM-based approaches require conditioning on raw observations, which becomes infeasible due to context-length limits and the difficulty of extracting meaningful patterns. We introduce \textbf{Kernel Discovery}, a LLM-driven evolutionary framework for high-dimensional BO that searches a broader kernel space beyond predefined composition rules and does not require conditioning on observations. Motivated by the observation that directly prompting an LLM to generate kernel code yields syntactically varied but functionally identical kernels, we adopt a two-stage approach: an LLM first proposes novel mathematical forms, then a second LLM call converts each form into validated, executable code. We also propose a leave-one-out continuous ranked probability score (LOO-CRPS) as a selection criterion that penalizes overfitted kernels. On five high-dimensional BO benchmarks, our method achieves an average rank of \textbf{1.2 out of 17}, outperforming competitive baselines. We further analyze the discovered kernels to identify which kernels lead to improvements in high-dimensional BO.

Taeyoung Yun, Woocheol Shin, Inhyuck Song +2

1Korea Advanced Institute of Science and Technology (KAIST)

Bayesian Optimization Gaussian Process Kernel Method Graphics Processing Unit Kernels Evolutionary Optimization Methods

Ask PDF

Jul 22, 2026cs.AI

KernelGenBench: Can LLMs and Agents Write Efficient Kernels Across Operator Sources and Hardware Platforms?

Modern AI systems depend on specialized accelerator kernels, whose development is complicated by increasingly diverse operators and hardware. LLMs and agentic systems promise to automate this work, but existing evaluations do not show whether their performance transfers across operator sources and hardware platforms, or what such transfer costs. We present KernelGenBench, the first unified multi-source and multi-chip infrastructure for evaluating LLM- and agent-generated Triton kernels. With a common Triton target spanning six hardware platforms, it provides the broadest cross-vendor hardware coverage among existing kernel-generation benchmarks. We report two controlled analytical views: KernelGenBench-MS (Multi-Source) covers 210 operators from PyTorch ATen, production vLLM operators, and proprietary cuBLAS routines, while KernelGenBench-MC (Multi-Chip) evaluates a semantically stable 110-operator subset across six hardware platforms. Our evaluation consumed over 15 billion tokens. Agentic execution improved correctness, but no method dominated across sources and platforms: vLLM posed the strongest correctness challenge, cuBLAS set the highest performance ceiling, and AutoKernel accuracy fell from 87% on NVIDIA to 25% on Iluvatar CoreX. These improvements were costly: specialized agents averaged 4.99 million tokens per successful operator, rising to 6.25 million for CUDA Optimized Skill. The results establish operator source, hardware platform, and agentic scaffold as distinct dimensions of kernel-generation capability, and show that success in a familiar source-hardware setting is not a reliable proxy for deployment readiness.

Peiyu Zang, Jian Tao, Jialing Zhang +4

Beijing Normal University · BAAI · Beijing Jiaotong University +2

Graphics Processing Unit Kernels Cuda Large Language Model Benchmarks

Ask PDF

Apr 17, 2026cs.CL

AdaExplore: Failure-Driven Adaptation and Diversity-Preserving Search for Efficient Kernel Generation

Recent large language model (LLM) agents have shown promise in using execution feedback for test-time adaptation. However, robust self-improvement remains far from solved: most approaches still treat each problem instance independently, without accumulating reusable knowledge. This limitation is particularly pronounced in domain-specific languages such as Triton, which are underrepresented in LLM pretraining data. Their strict constraints and non-linear optimization landscape further make naive generation and local refinement unreliable. We propose AdaExplore, an agent framework that enables self-improvement via accumulated execution feedback for performance-critical kernel code generation through two complementary stages: failure-driven adaptation and diversity-preserving search, jointly improving correctness and optimization performance without additional fine-tuning or external knowledge. In the adaptation stage, the agent synthesizes tasks and converts recurring failures into a reusable memory of validity rules, helping subsequent generations remain within the feasible set. In the search stage, the agent organizes candidate kernels as a tree and alternates between small local refinements and larger structural regeneration, allowing it to explore the optimization landscape beyond local optima. Experiments on kernel runtime optimization benchmarks validate these gains: AdaExplore achieves 3.12x and 1.72x speedups on KernelBench Level-2 and Level-3, respectively, within 100 steps, and continues to improve with additional computation.

Weihua Du, Jingming Zhuo, Yixin Dong +9

1Carnegie Mellon University · University of Washington · 3Arm Ltd.

Agentic Optimization Code Optimization Domain-Specific Language

Ask PDF