ConflictGuide: AutoResearch Improves When Competing Behaviors Are Made Visible

Binqian Xu; Qiran Zou; Xiangbo Shu; Dianbo Liu

cs.AISep 30, 2026

ConflictGuide: AutoResearch Improves When Competing Behaviors Are Made Visible

Authors: Binqian Xu, Qiran Zou, Xiangbo Shu, Dianbo Liu

Organizations: National University of Singapore · Nanjing University of Science and Technology

Abstract

When designing machine learning models, desirable properties are often in tension: improving one behavior can impair another, so task progress can depend on alleviating the conflict. LLM-based AutoResearch systems, which iteratively edit model code and retain edits based on scalar task-performance feedback, have largely ignored this trade-off. We find that scalar feedback supports broad exploration early in search, but it does not reveal how edits affect competing behaviors. In matched-budget experiments, introducing competing-behavior feedback as task gains diminish increases the share of proposals that improve both behaviors and sustains progress beyond scalar-only plateaus. Obtaining this feedback for a given model requires identifying its competing behaviors and designing probes to measure them. To make competing-behavior feedback actionable, we introduce ConflictGuide. Its reusable ConflictGuide-Skill combines a literature-grounded taxonomy with model-specific evidence to identify competing behaviors and specify probes for a code agent to implement as metrics. Evolution proceeds in two stages: Stage I explores with task feedback; Stage II uses probe feedback to steer proposals toward conflict alleviation and retains marginal-gain edits only when probes indicate sufficient alleviation. Across five diverse model families, ConflictGuide reduces task and conflict-related errors by up to 28% and 14%, respectively, relative to scalar-only AutoResearch, with gains extending to other code agents.

Figures & tables

Figure 1: Comparison of AutoResearch (scalar-only feedback) and ConflictGuide (competing-behavior feedback). (a) Feedback visible to the code agent. (b) Proposal-level behavioral outcomes across models. ConflictGuide increases joint improvements from 7.2% to 13.1% and reduces joint degradations from 29.0% to 20.7%. (c) From common branch points under matched continuation budgets, ConflictGuide continues to lower the task metrics, whereas AutoResearch plateaus.

Figure 2: Overview of ConflictGuide. (a) ConflictGuide-Skill uses a literature-grounded taxonomy to identify a model-specific conflict and operationalize its competing behaviors through qualified probes. (b) Both stages share the evolution loop: Stage I uses scalar feedback for exploration; once task-metric gains diminish, Stage II adds conflict-specific feedback to guide proposals and retention.

Model	Domain / Task	Conflict mechanism (Axis)	Primary Probe metric(s)
SpecB–FNO	Scientific ML / PDE operator learning	R2.M2 ( space ) Frequency-structure mismatch: dominant-mode fit vs. non-dominant-mode utilization	$Z_{\mathrm{ND}}(x)=\dfrac{\sum_{n,t}\sum_{k\in\mathcal{N}}w_{k}\lvert R^{(x)}_{n,t,k}\rvert^{2}}{\sum_{n,t}\sum_{k\in\mathcal{N}}w_{k}\lvert Y_{n,t,k}\rvert^{2}+\epsilon_{\mathrm{spec}}}\downarrow$
SNGP	Uncertainty-aware classification / OOD detection	R3.M4 ( objective ) Compression vs. uncertainty: predictive fit vs. distance-aware feature geometry	$\begin{aligned} Z_{\mathrm{mar}}&=Q_{q_{\mathrm{mar}}}\!\left[\widetilde{z}_{i}^{\!\top}\widetilde{\mu}_{y_{i}}-\max_{c\neq y_{i}}\widetilde{z}_{i}^{\!\top}\widetilde{\mu}_{c}\right]\uparrow\\[-1.0pt] Z_{\mathrm{dist}}&=\max_{g\in\{\mathrm{s},\mathrm{c}\}}Q_{q_{\mathrm{dist}}}\!\left[\left\|r_{ij}-\widetilde{r}\right\|\right]_{(i,j)\in\mathcal{P}_{g}}\downarrow\end{aligned}$
ESN	Reservoir computing / time-series prediction	R2.M4 ( time ) Temporal-component mismatch: memory retention vs. nonlinear processing	$\begin{aligned} Z_{\mathrm{mem}}&=\mathbb{E}_{q,t,d,j}\!\left[\min\!\left\{\sigma_{j}\!\left(P_{q,t,d}\right),1\right\}\right]\uparrow\\[-1.0pt] Z_{\mathrm{nl}}&=\mathbb{E}_{q}\!\left[\frac{\mathbb{E}_{t}\left\lVert K_{q,t}-\overline{K}_{q}\right\rVert_{F}^{2}}{\mathbb{E}_{t}\left\lVert K_{q,t}\right\rVert_{F}^{2}+\epsilon_{\mathrm{nl}}}\right]\uparrow\end{aligned}$
TCM–Lite	Learned image compression / rate–distortion coding	R3.M10 ( space ) Compact representation damaging fine structure: latent rate vs. detail preservation	$\begin{aligned} Z_{\mathrm{rate}}(x)&=-N_{\mathrm{pix}}^{-1}\sum\nolimits_{u\in\widehat{\mathcal{U}}_{x}}\log_{2}\widetilde{p}_{x}(u)\;\downarrow\\[-1.0pt] Z_{\mathrm{detail}}(x)&=\mathbb{E}_{s}\left[\frac{\lVert\nabla L_{s}-\nabla\widehat{L}_{x,s}\rVert_{1}}{\lVert\nabla L_{s}\rVert_{1}+\lVert\nabla\widehat{L}_{x,s}\rVert_{1}+\epsilon}\right]\;\downarrow\end{aligned}$
GCNII	Graph learning / node classification	R2.M5 ( space ) Relational-structure mismatch: useful aggregation vs. incompatible messages	$Z_{\mathrm{dis}}(x)=\mathbb{E}_{k}\left[\mathcal{L}_{\mathrm{NLL}}\!\left(x;A_{\mathrm{dis}}^{(k)},X,\mathcal{V}\right)-S(x)\right]\downarrow$

Table 1: Five experimental model–conflict settings and primary Probe metrics (Appendix F ).

Round	Method	Main Cumulative Edit	NRMSE $\downarrow$	ND-NMSE $\downarrow$	Params (M) $\downarrow$
0	Reference	None (Original)	$0.0458\pm 0.0014$	$0.0861\pm 0.0086$	328.5
	Autoresearch	Spatial gate	$0.0382\pm 0.0015$	$0.0617\pm 0.0050$	328.9
	ConflictGuide	Mode refinement	$\mathbf{0.0344\pm 0.0003}$	$\mathbf{0.0530\pm 0.0007}$	333.5
1	Reference	None (Original)	$0.0458\pm 0.0014$	$0.0861\pm 0.0086$	328.5
	Autoresearch	Energy match	$0.0445\pm 0.0007$	$0.0766\pm 0.0035$	403.9
	ConflictGuide	Multi-scale residual	$\mathbf{0.0442\pm 0.0014}$	$\mathbf{0.0760\pm 0.0064}$	407.2

Table 2: SpecB–FNO on 2D incompressible Navier–Stokes (

\nu\!=\!10^{-5}

;

64\!\times\!64

). NRMSE is the task metric; ND-NMSE the conflict-specific Probe; Params in millions. Mean

\pm

s.d. over three paired seeds; bold = lowest error per round; “None (Original)” denotes the unmodified reference.

Figure 3: Validation trajectories for one search round on SpecB-FNO, SNGP, and GCNII. Both methods continue from a common branch point with equal budgets; lower is better in all panels.

Round	Method	Main Cumulative Edit	NLL $\downarrow$	OOD AUPR $\uparrow$
Round	Method	Main Cumulative Edit	NLL $\downarrow$	SVHN	CIFAR-10
0	Reference	None (Original)	$1.0434\pm 0.0111$	$0.7538\pm 0.0322$	$0.7304\pm 0.0029$
	Autoresearch	Multi-scale RFF	$1.0578\pm 0.0004$	$0.7259\pm 0.0298$	$0.7339\pm 0.0045$
	ConflictGuide	LN-residual RFF	$\mathbf{0.8508\pm 0.0061}$	$\mathbf{0.7970\pm 0.0090}$	$\mathbf{0.7487\pm 0.0022}$
1	Reference	None (Original)	$1.0434\pm 0.0111$	$0.7538\pm 0.0322$	$0.7304\pm 0.0029$
	Autoresearch	Adaptive normalized RFF	$0.9421\pm 0.0008$	$0.7711\pm 0.0122$	$0.7455\pm 0.0024$

Table 3: SNGP on CIFAR-100. Clean ID NLL is the task metric; OOD AUPR (DS) is the corresponding held-out conflict-related outcome. Mean

\pm

s.d. over three paired seeds; bold = best per round; “None (Original)” denotes the unmodified reference.

Round	Method	Main Cumulative Edit	Accuracy $\uparrow$	NLL $\downarrow$
Round	Method	Main Cumulative Edit	Accuracy $\uparrow$	Clean	Contaminated
0	Reference	None (Original)	$0.7255\pm 0.0539$	$0.7838\pm 0.1681$	$0.8083\pm 0.1744$
	Autoresearch	Disagreement gate	$0.7255\pm 0.0555$	$0.7854\pm 0.1686$	$0.8075\pm 0.1690$
	ConflictGuide	Residual–neighbor mixer	$\mathbf{0.7686\pm 0.0605}$	$\mathbf{0.7618\pm 0.2422}$	$\mathbf{0.7893\pm 0.2399}$
1	Reference	None (Original)	$0.7255\pm 0.0539$	$0.7838\pm 0.1681$	$0.8083\pm 0.1744$
	Autoresearch	Disagreement self-gate	$0.7922\pm 0.0648$	$0.6583\pm 0.1526$	$0.6710\pm 0.1294$

Table 4: GCNII on Wisconsin. Clean accuracy and NLL measure performance; contaminated NLL measures performance under contamination (

\rho=0.3

). Mean

\pm

s.d. over ten splits, retraining per split, averaging five frozen contaminations; bold = best per round; “None (Original)” = unmodified.

Round	Method	Main Cumulative Edit	NRMSE $\downarrow$	MSE $\downarrow$	$R^{2}$ $\uparrow$
0	Reference	None (Original)	$0.4261\pm 0.0417$	$0.0094\pm 0.0018$	$0.8167\pm 0.0356$
	AutoResearch	Gated leak	$0.3173\pm 0.0262$	$0.0052\pm 0.0008$	$0.8986\pm 0.0165$
	ConflictGuide	State–drive coupling	$\mathbf{0.2285\pm 0.0308}$	$\mathbf{0.0027\pm 0.0007}$	$\mathbf{0.9468\pm 0.0138}$
1	Reference	None (Original)	$0.4261\pm 0.0417$	$0.0094\pm 0.0018$	$0.8167\pm 0.0356$
	AutoResearch	State–dependent leak	$0.3246\pm 0.0264$	$0.0054\pm 0.0009$	$0.8939\pm 0.0171$
	ConflictGuide	Dual state coupling	$\mathbf{0.2561\pm 0.0424}$	$\mathbf{0.0034\pm 0.0011}$	$\mathbf{0.9326\pm 0.0207}$

Table 5: ESN on Mackey–Glass long-horizon forecasting (

\tau,h\!=\!17,84

), testing transfer of probe-guided edits. NRMSE, MSE, and

R^{2}

are task metrics. Mean

\pm

s.d. over 200 paired runs (20 sequence

\times

10 reservoir seeds); bold = best per round; “None (Original)” = unmodified reference.

Round	Method	Main Cumulative Edit	Actual bpp $\downarrow$	MS-SSIM $\uparrow$	$z_{\mathrm{detail}}$ $\downarrow$
0	Reference	None (Original)	$0.5727\pm 0.0650$	$0.8981\pm 0.0269$	$0.1918\pm 0.0295$
	Autoresearch	Cross-branch gating	$0.5672\pm 0.0188$	$0.9186\pm 0.0029$	$0.1744\pm 0.0076$
	ConflictGuide	Intra-encoder skip	$\mathbf{0.5442\pm 0.0149}$	$\mathbf{0.9209\pm 0.0003}$	$\mathbf{0.1709\pm 0.0010}$
1	Reference	None (Original)	$0.5727\pm 0.0650$	$0.8981\pm 0.0269$	$0.1918\pm 0.0295$
	Autoresearch	Cross-scale gating	$0.5245\pm 0.0149$	$0.9271\pm 0.0009$	$0.1550\pm 0.0003$
	ConflictGuide	Channel recalibration	$\mathbf{0.5159\pm 0.0096}$	$\mathbf{0.9273\pm 0.0007}$	$\mathbf{0.1535\pm 0.0010}$

Table 6: TCM–Lite on Kodak at

\lambda\!=\!0.013

. Actual bpp reports realized rate, MS-SSIM task quality, and

z_{\mathrm{detail}}

the conflict-specific detail Probe. Mean

\pm

s.d. over three paired formal-training seeds; bold = better method per completed round; “None (Original)” denotes the unmodified reference.

Configuration	Guidance	Retention	SpecB-FNO		SNGP
Configuration	Guidance	Retention	NRMSE $\downarrow$	ND-NMSE $\downarrow$	NLL $\downarrow$	OOD AUPR $\uparrow$
Baseline	✗	✗	$0.0382_{\pm.0015}$	$0.0617_{\pm.0050}$	$1.1777_{\pm.0210}$	$0.7890_{\pm.0389}$
Guidance only	✓	✗	$0.0348_{\pm.0015}$	$0.0555_{\pm.0059}$	$1.0381_{\pm.0406}$	$0.7914_{\pm.0047}$
ConflictGuide	✓	✓	$\mathbf{0.0344}_{\pm.0003}$	$\mathbf{0.0530}_{\pm.0007}$	$\mathbf{0.8455}_{\pm.0076}$	$\mathbf{0.8131}_{\pm.0294}$

Table 7: Ablation of ConflictGuide components. Guidance only introduces conflict-specific guidance; full ConflictGuide adds auxiliary retention. OOD AUPR on SVHN. Mean

\pm

s.d.; bold = best.

Figure 4: Probe-guided versus unguided evolution across four coding agents (a–d). Left: test NRMSE and non-dominant relative NMSE (3-seed mean; dashed = reference; lower is better). Right: validation NRMSE from shared step-100 winner. Blue = unguided; orange = Probe-guided.

Figure 5: Budget sensitivity. (a) Round-0 validation trajectories for three Stage I budgets. (b) Test NRMSE under three 200-iteration allocations; 100/100 balances performance and search efficiency.

Figure 6: Conflict-specific feedback on SNGP and GCNII proposals. (a) Conflict-relief rates. (b.1–b.2) Edit-category prevalence; dot area encodes relief rate, and lines/labels show changes from Baseline (

\Delta

, pp). Overall, feedback increases relief rates and shifts the distribution of code edits.

Configuration	GCNII		SNGP
Configuration	Clean NLL $\downarrow$	Contaminated NLL $\downarrow$	NLL $\downarrow$	OOD AUPR $\uparrow$
Multi-metric feedback	$0.5662\pm 0.0909$	$0.5679\pm 0.0929$	$0.8918\pm 0.0087$	$0.7292\pm 0.0023$
Trade-off prompt only	$0.5694\pm 0.0791$	$0.5682\pm 0.0824$	$1.1959\pm 0.0234$	$0.7317\pm 0.0024$
ConflictGuide	$\mathbf{0.5540\pm 0.0847}$	$\mathbf{0.5488\pm 0.0863}$	$\mathbf{0.8455\pm 0.0076}$	$\mathbf{0.7357\pm 0.0010}$

Table 8: Comparison of multi-metric, prompt-only, and ConflictGuide feedback on GCNII and SNGP (search round 2). Values are formal-evaluation means

\pm

s.d.; bold marks the best per metric.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Mechanism label	Included code changes
Frequency-specific spectral	Fourier-mode weighting, selection, filtering, or spectral operators.
Spatial/local pathway	Spatial or pointwise feature extraction and local helper paths alongside the spectral path.
Normalization/activation/channel	Normalization, nonlinear activation, channel width, or channel projection.
Stage fusion/residual scale	Fusion between stages or branches and scaling of residual corrections.
Optimization/regularization	Optimizer, schedule, loss regularization, or training-time stabilization.

Appendix

Table 9: Operational mechanism labels used in the SpecB–FNO proposal audit.

Figure 7: Distribution of code-edit mechanisms in the SpecB–FNO exploratory searches. (a) Feedback from the outset, with 94 valid Scalar-only proposals and 85 valid Probe-visible proposals. (b) Delayed feedback from a common Stage-I branch point, with 81 valid Scalar-only proposals and 83 valid ConflictGuide proposals. Percentages are computed within each arm. Mechanism labels are non-exclusive; shares therefore need not sum to 100%.

Configuration	Test NRMSE $\downarrow$	ND-NMSE $\downarrow$
Reference	$0.045822\pm 0.001432$	$0.086148\pm 0.008565$
Scalar-only	$\mathbf{0.037916\pm 0.001519}$	$\mathbf{0.061121\pm 0.005326}$
Probe from outset	$0.046471\pm 0.002050$	$0.086498\pm 0.009130$

Appendix

Table 10: Full-scale evaluation with feedback provided from the outset. Mean

\pm

standard deviation over three training seeds; lower is better.

Configuration	Test NRMSE $\downarrow$	ND-NMSE $\downarrow$
Reference	$0.045822\pm 0.001432$	$0.086148\pm 0.008565$
Scalar-only	$0.038224\pm 0.001466$	$0.061698\pm 0.004965$
ConflictGuide	$\mathbf{0.034447\pm 0.000342}$	$\mathbf{0.053048\pm 0.000724}$

Appendix

Table 11: Full-scale evaluation after delayed feedback. Mean

\pm

standard deviation over three paired training seeds; lower is better.

Stage	Operation	Main evidence	Output
0	Model context	Architecture, task, data, objective, $S$ , editable scope, and runtime budget	Normalized Model Context Card
1	Taxonomy screening	Target-model code and configuration, interpreted through the fixed Root–Axis–Mechanism taxonomy	Taxonomy selection or explicit abstention
2	Conflict instantiation	Selected mechanism, task relevance, shared component or resource, and bidirectional interference	Model-specific competing behaviors and an observability analysis of $S$
3	Probe design	Existing outputs, deterministic diagnostics, and lightweight read-only observables	Fixed Conflict Probe Card and implementation specification
4	Probe qualification	Null replicates and pre-specified parent–candidate replay pairs	Qualified, rejected, or inconclusive Probes

Appendix

Table 12: ConflictGuide-Skill workflow. Each stage must be completed before its output is used by the next stage.

Condition	Skill response
Insufficient model, task, data, objective, or scalar-metric context	Abstain. The taxonomy selection cannot be grounded in target-specific evidence.
No two independently desirable behaviors	Reject. The proposed pair does not constitute a task-relevant behavioral conflict.
No identifiable shared component, representation, parameter path, or budget	Reject. The proposed behaviors lack a defensible coupling mechanism.
Only one interference direction is plausible	Reject. The case describes a one-sided failure rather than competing behaviors.
Behavior-specific observables are unavailable or incomparable across candidates	Abstain or reject. The conflict cannot be operationalized reliably under the current evaluation setting.
A Probe duplicates the scalar task metric, leaks unavailable information, or depends on post-evolution tuning	Reject. The proposed measurement does not provide valid fixed feedback.

Appendix

Table 13: Main abstention and rejection conditions used by ConflictGuide-Skill.

Field	Description
Target behavior	The declared competing behavior measured by the Probe.
Observable and computation	Source value, formula or algorithm, sampling, and aggregation.
Direction and normalization	Whether an increase or decrease is desirable and how values are made comparable across candidates.
Evaluation setting	Frozen dataset, split, references, perturbations, and randomness policy.
Additional requirements	Whether extra training, labels, hooks, checkpoints, or model outputs are required.
Relation to scalar task metric	What behavior-specific information the Probe adds beyond the scalar task metric.

Appendix

Table 14: Core fields of a Conflict Probe specification.

Card component	Recorded information
Model context	Architecture, task, data, objective, scalar task metric, evolution boundary, budget, and evidence sources.
Taxonomy selection	Selection status, Root, axis, mechanism, causal links, optimization signatures, and abstention reason when applicable.
Behavioral conflict	Competing behaviors, desirable directions, shared coupling, interference directions, confidence, and rejection reason when applicable.
Feedback observability	Definition and direction of the scalar task metric, conflict visibility, and behavior-specific information it aggregates away.
Probes	Fixed Probe specifications, information added beyond the scalar task metric, computational costs, and failure modes.
Qualification	Null calibration, scalar-tied informativeness test, direction test, decision state, and rejection conditions.

Appendix

Table 15: Inspectable outputs in a Conflict Probe Card.

Root	Conflict is against	Compute test	Primary discriminator	Typical causal role
R1	Compute, memory, latency, parameter count, or bandwidth	Partially resolvable	The conflict substantially eases with greater capacity, resolution, or evaluation budget.	May induce downstream information loss or structural compromise.
R2	Mismatch between an inductive bias or structural constraint and the signal	Not resolvable	The imposed prior is too weak, too strong, or incompatible with the signal structure; architectural change is required.	Often upstream of shortcut learning or failure under distribution shift.
R3	Information loss imposed by compression or abstraction	Not resolvable	Required information has been discarded or incompatible information demands must share a limited representation.	Often downstream of resource limits and upstream of uncertainty or generalization failures.
R4	The gap between the training distribution and deployment conditions	Not resolvable	Performance relies on non-causal or distribution-specific evidence that fails under domain, group, temporal, or intervention shifts.	Usually a downstream symptom; upstream causes should be identified.
R5	New adaptation competing with retained knowledge or behavior	Partially resolvable	The conflict arises sequentially across training stages rather than simultaneously among objectives.	May follow distribution, label-process, or normative changes.
R6	A fixed search budget divided between coverage and solution quality	Partially resolvable	The conflict concerns exploration of solutions, actions, or hypotheses, not coverage of demographic populations.	May expose resource limits or aggravate structural and normative violations.

Appendix

Table 16: Root-level semantics and selection boundaries. Figure 9 shows the admissible axes and mechanism ranges; this table provides the complementary criteria used to distinguish and causally relate the Roots.

ID	Mechanism	Core diagnostic definition
R1: Resource allocation vs fidelity
R1.M1	Insufficient fidelity	Does the model systematically underfit the target because the allocated resolution, step count, or capacity is too low?
R1.M2	Uniform allocation mismatch	Does a uniform allocation waste resources on easy cases while remaining insufficient for hard cases, long-tail cases, or complex samples?
R1.M3	Insufficient expressive bandwidth	Is the current positional encoding, basis, feature map, latent grid, channel rank, codebook, recurrent width, or local representation insufficient to express the required rate of variation? - Primarily space and time . May be module when a specific branch’s rank is the limit, or modality when a specific modality’s encoder is the limit. - If the loss is informational rather than budgetary (i.e. more capacity would not help because the representation deliberately discards the factor), use R3 instead.
R1.M4	Added freedom causing overfitting	After adding local parameters, high-frequency encoding, extra steps, extra depth, or extra experts, does the model become more likely to fit noise, label errors, spurious patterns, or training-set-specific details?
R1.M5	Cross-partition consistency issue	Do outputs become inconsistent across the partitions created by the chosen axis?

Appendix

Table 17: Complete Root-specific mechanism inventory.

ID	Artifact	Diagnostic definition
OPT.M1	Gradient scale imbalance	Gradients from some objectives are persistently larger, so shared training primarily serves them although the substantive demands are compatible.
OPT.M2	Convergence rhythm conflict	Objectives converge at different speeds, so later updates damage early-converged objectives or slow objectives drag down shared training.
OPT.M3	Weighted-sum masking	Aggregation in the training objective hides degradation of individual objectives while the weighted sum appears normal.

Appendix

Table 18: Standalone optimization artifacts, used only when the conflict disappears with independent parameters.

Model	Evolution training and auxiliary data	Online keep-validation	Post-search retraining and formal evaluation
SpecB–FNO	Navier–Stokes official training pool: 200 training examples and 48 disjoint mask-construction examples; the remaining 704 examples are unused online.	48 fixed examples, disjoint from training and mask construction.	Retrain: 800/100/100 train/mask/validation partition of the official training pool. Test: 200 official examples (indices 1000–1199).
SNGP	CIFAR-100 training set, stratified per class into 400 training and 20 temperature-calibration images. A further 50 images/class are reserved for search-inaccessible audit.	30 fixed images/class, disjoint from training, calibration, and audit.	Retrain: all 50,000 CIFAR-100 training images. Test: CIFAR-100; balanced OOD comparisons with SVHN, and CIFAR-10.
GCNII	Chameleon public split 0: its 60% per-class training partition; five frozen, label-free disagreement graphs are constructed without test nodes.	The corresponding 20% validation partition; the 20% test partition is inaccessible online.	Retrain/test: the frozen source is trained independently on each of ten public 60/20/20 splits of Cora, Citeseer, Pubmed, Chameleon, Cornell, Texas, and Wisconsin.
ESN	One frozen NARMA-30 realization: 200 washout and 5,000 readout-fit samples, evaluated with reservoir seeds 1103, 2207, and 3301.	The subsequent 2,500 samples from the same realization.	Refit/test: independent NARMA-30 and Mackey–Glass sequences crossed with ten independent reservoir seeds; a fresh ridge readout is fitted for every pair.
TCM–Lite	An image-disjoint DIV2K+Flickr2K patch bank: 32,768 training patches and four separate 2,048-patch panels for mechanism development, hidden audit, qualification discovery, and qualification confirmation.	2,048 fixed $64\times 64$ RGB patches from a fifth source-image-disjoint panel.	Retrain: all 3,450 DIV2K+Flickr2K source images. Test: full-resolution Kodak images using actual arithmetic coding.

Appendix

Table 19: Search-time data isolation and post-search evaluation. Reduced evolution substrates support search throughput, whereas formal results are obtained only after source freezing and full-scale retraining.

Dataset	Layers	Hidden	Dropout	$\alpha$	$\lambda$	Weight decay
Chameleon	8	64	0.5	0.2	1.5	$5\times 10^{-4}$
Cornell	16	64	0.5	0.5	1.0	$10^{-3}$
Texas	32	64	0.5	0.5	1.5	$10^{-4}$
Wisconsin	16	64	0.5	0.5	1.0	$5\times 10^{-4}$
Cora	64	64	0.5	0.2	0.5	$10^{-4}$
Citeseer	64	64	0.5	0.5	0.5	$5\times 10^{-6}$

Appendix

Table 20: GCNII configurations used for formal evaluation.

Model	Task-gain condition	Required Probe improvement	Guard condition
SpecB–FNO	Direct: $G_{t}^{S}>5\times 10^{-4}$ . Probe route: $0<G_{t}^{S}\leq 5\times 10^{-4}$ .	$\frac{Z_{\mathrm{ND}}(p)-Z_{\mathrm{ND}}(c)}{Z_{\mathrm{ND}}(p)+\epsilon}\geq 10^{-2}.$	$Z_{\mathrm{D}}(c)\leq 1.01Z_{\mathrm{D}}(a)$ and $Z_{\mathrm{late}}(c)\leq 1.02Z_{\mathrm{late}}(a).$
SNGP	Direct: $G_{t}^{S}>10^{-3}$ . Probe route: $0<G_{t}^{S}\leq 10^{-3}$ .	$Z_{\mathrm{mar}}(c)-Z_{\mathrm{mar}}(a)\geq 10^{-2}$ and $Z_{\mathrm{dist}}(a)-Z_{\mathrm{dist}}(c)\geq 5\times 10^{-3}.$	$\operatorname{Acc}(c)\geq\operatorname{Acc}(a)-10^{-2}.$
GCNII	Direct: $G_{t}^{S}>\epsilon_{S}$ . Probe route: $0\leq G_{t}^{S}\leq\epsilon_{S}$ , where $\epsilon_{S}=1.01\times 10^{-3}$ .	$Z_{\mathrm{dis}}(p)-Z_{\mathrm{dis}}(c)>\tau_{P}.$	No additional model-specific guard; the non-negative task-gain requirement prevents clean-NLL regression.
ESN	Direct: $G_{t}^{S}>\epsilon_{S}$ . Probe route: $0\leq G_{t}^{S}\leq\epsilon_{S}$ , where $\epsilon_{S}=4.18\times 10^{-2}$ .	Either $G_{\mathrm{mem}}>\tau_{\mathrm{mem}},\quad G_{\mathrm{nl}}\geq-\tau_{\mathrm{nl}},$ or $G_{\mathrm{nl}}>\tau_{\mathrm{nl}},\quad G_{\mathrm{mem}}\geq-\tau_{\mathrm{mem}}.$	$\tau_{\mathrm{mem}}=8.85\times 10^{-3}$ and $\tau_{\mathrm{nl}}=2.63\times 10^{-4}$ , frozen from the 95th percentiles of paired no-edit variation.
TCM–Lite	Direct: $G_{t}^{S}>0$ . Probe route: $G_{t}^{S}=0$ .	Either $G_{\mathrm{rate}}>\tau_{R},\;G_{\mathrm{detail}}\geq-\tau_{D},$ or $G_{\mathrm{detail}}>\tau_{D},\;G_{\mathrm{rate}}\geq-\tau_{R}.$	$\tau_{R}=5.89\times 10^{-2}$ and $\tau_{D}=4.44\times 10^{-2}.$

Appendix

Table 21: Model-specific ConflictGuide retention conditions. The task column shows the direct-retention condition followed by the Probe-route task band. No retained proposal may have negative task gain.

Model	Search seeds	Stage I	Stage-II AutoResearch	Stage-II ConflictGuide
SpecB–FNO	0, 1, 2	100	100	100
SNGP	0, 1, 2	100	100	100
GCNII	0, 1, 2	100	100	100
ESN	0, 1, 2	20	50	50
TCM–Lite	0, 1, 2	100	100	100

Appendix

Table 22: Executed main-search budgets. ESN uses a shorter matched budget because of its more restricted editable space.

Aspect	Upstream AutoResearch	Our controlled harness	Reason for the modification
Loop ownership	The coding agent edits the program, invokes training, interprets the output, and performs Git-based keep or reset operations.	The agent only proposes source edits. A protected external controller owns validation, training, evaluation, retention, rollback, and accounting.	Prevents evaluator modification and information leakage, and applies the same frozen decision rule to every candidate.
Session and history	The agent follows an iterative experiment loop and records a compact results.tsv .	Every proposal uses a fresh session supplied with a deterministic, source-hash-bound structured history view; a separate complete ledger archives all attempts.	Removes unrecorded conversational state and permits exact reconstruction of the information available at each iteration.
Task substrate	The public implementation optimizes one language-model training program under a fixed wall-clock budget and a scalar validation metric.	We instantiate the loop for five model families with frozen model-specific training budgets and scalar task metrics $S(x)$ .	A single training duration and metric are not meaningful across PDE, classification, graph, reservoir, and compression tasks.
Feedback	Each experiment is summarized primarily by the scalar task metric.	The scalar-only arm receives only $S(x)$ , whereas ConflictGuide additionally receives registered competing-behavior Probes and Probe-aware decision information.	This is the experimental intervention whose effect we study.
Search design	The public loop follows a single sequential trajectory.	Each search seed runs its own Stage-I scalar trajectory, after which matched AutoResearch and ConflictGuide continuations begin from the same branch point.	Creates a paired comparison that controls initialization and all pre-branch search outcomes.
Editable boundary	train.py is editable while supporting infrastructure is fixed.	We retain the one-file editable boundary, but impose model-specific API, shape, determinism, runtime, memory, and parameter constraints.	Preserves evaluator semantics and prevents edits from changing the experimental task or resource budget.

Appendix

Table 23: Relationship between the upstream AutoResearch implementation and our controlled multi-model evolution harness. The modifications isolate the effect of competing-behavior feedback while supporting heterogeneous training substrates.

Model	Proposal generation	Candidate evaluation
SpecB–FNO	1,800 s	1,500 s
SNGP	1,800 s	1,024 s
GCNII	1,800 s	168 s
ESN	1,800 s	1,800 s
TCM–Lite	660 s	300 s

Appendix

Table 24: Administrative watchdogs used during online evolution. These limits determine failure handling and do not replace the fixed model-specific training budgets.

Model	Scalar-only evaluator	ConflictGuide evaluator	Incremental-cost accounting
SpecB–FNO	Task metric and hidden spectral Probes	Identical computation, with Probes exposed to the agent	The realized difference is $0.0$ s. Across eight matched calibration evaluations, the spectral Probe pass required a median of $3.60$ s, corresponding to $0.88\%$ of the complete candidate-evaluation time.
SNGP	Training, GP-state reconstruction, temperature fitting, and task metrics	The same computation plus representation collection and geometry Probes	Across three matched no-edit branch anchors, the median paired increase was $7.68$ s, or $2.57\%$ of the scalar-only evaluation time. This is a descriptive wall-clock estimate because the paired jobs were not executed simultaneously.
GCNII	Median candidate-evaluation time of $10.4$ s	Probe-normalized median candidate-evaluation time of $11.7$ s	The estimated incremental cost is $1.3$ s per candidate, corresponding to a $12.8\%$ increase over the scalar-only evaluator.
ESN	Median total candidate-evaluation time of $82.4$ s, including hidden Jacobian Probes	The same median total time of $82.4$ s, with Probe measurements exposed to the agent	The three reservoir-seed Jacobian Probe passes together required approximately $77.9$ s, corresponding to $94.5\%$ of the complete candidate-evaluation time.
TCM–Lite	Median total candidate-evaluation time of $258.5$ s, including hidden rate and detail measurements	The same median total time of $258.5$ s, with the measurements exposed to the agent	The separately timed multiscale-detail pass required approximately $0.04$ s, or less than $0.1\%$ of the total. The rate measurement adds no separate pass because it is already required by the rate–distortion task metric.

Appendix

Table 25: Evaluator-side cost of Probe feedback. Times are wall-clock seconds. A zero realized difference means that the scalar-only arm computed the same hidden Probe quantities under the matched evaluator workload. Probe-pass times isolate the computation attributable to those measurements.

Model	Multi-metric feedback	Trade-off-only prompting	ConflictGuide
SNGP	Fifteen-bin calibration error and development OOD Dempster–Shafer AUPR.	A generic qualitative reminder, without additional metrics or model-specific conflict information.	Low-quantile centroid margin $Z_{\mathrm{mar}}$ , input–feature distance distortion $Z_{\mathrm{dist}}$ , their desirable directions, and Probe-aware retention feedback.
GCNII	Mean NLL and accuracy under five frozen, randomly rewired degree-matched graphs.	A generic qualitative reminder, without additional metrics or model-specific conflict information.	Disagreement-contamination sensitivity $Z_{\mathrm{dis}}$ , its desirable direction, and Probe-aware retention feedback.

Appendix

Table 26: Additional information supplied by the three feedback strategies. Trade-off-only prompting supplies a qualitative reminder without additional quantitative measurements.

Setting	Code generation	Post-execution bug review	Search structure
Claude Code	Claude Opus 4.6	Not used	Sequential retained-parent search
Kimi Code	Kimi K2.7 Code	Not used	Sequential retained-parent search
AIDE (OpenAI)	o4-mini	GPT-4.1-mini	Solution tree with improve and debug operations
AIDE (GLM)	GLM-5.3 (low reasoning effort)	GLM-5.3-Flash	Solution tree with improve and debug operations

Appendix

Table 27: Agent and search configurations in the SpecB–FNO transfer experiments. Numerical candidate metrics are computed by the fixed evaluator in all settings. The post-execution review model in AIDE only identifies implementation or runtime failures.

Figure 10: Post-hoc competing-behavior trajectories for TCM–Lite during the task-only Stage I of search round 1. The behavior metrics were hidden from the proposal agent. Changes are direction-aligned and normalized by the frozen Probe thresholds; positive values indicate improvement.

Figure 11: Scalar ambiguity: similar task gains can correspond to different behavior outcomes. Positive-gain scalar-only proposals are grouped by within-model task-gain magnitude to assess whether scalar improvement identifies directional behavior outcomes. Multiple outcome categories appear in every group, showing that similar scalar gains remain compatible with materially different behavior changes.

Explore similar work

May 7, 2026cs.MA

Auto Research with Specialist Agents Develops Effective and Non-Trivial Training Recipes

We study auto research as a closed empirical loop driven by external measurement. Each submitted trial carries a hypothesis, an executable code edit, an evaluator-owned outcome, and feedback that shapes the next proposal. The output is not a generated paper or a single model checkpoint, but an auditable trajectory of proposals, code diffs, experiments, scores, and failure labels. We instantiate this loop with specialist agents that partition recipe surfaces and share measured lineage across trials. The central empirical finding is that lineage feedback lets agents turn evaluator outcomes, including crashes, budget overruns, size failures, and accuracy-gate misses, into later program-level recipe edits rather than one-shot suggestions. Across 1,197 headline-run trials plus 600 Parameter Golf control trials after one-time setup and launch, humans did not choose proposals, edit recipes, override scores, or repair failed trials during the search. In the three headline runs, the same submitted-trial loop reduces Parameter Golf validation bpb by

0.81\%

, raises NanoChat-D12 CORE by

38.7\%

, and reduces CIFAR-10 Airbench96 wallclock by

4.59\%

, with each task measured by its own external evaluator and legality checks. The trace includes a strict architecture-domain audit of 157 headline-run submissions and program rewrites such as a NanoChat attention-kernel path change. Within this scope the loop autonomously writes code, submits experiments, absorbs feedback, applies and combines known techniques inside each environment, and improves public starting recipes.

Jingjie Ning, Xiaochuan Li, Ji Zeng +2

School of Computer Science, Carnegie Mellon University

Auto Research Loop Single Trajectory Lineages

Ask PDF

Jul 19, 2026cs.MA

Auto Research for Materials: Auditable AI-Scientist Workflows with Held-Out Transfer

Auto Research uses language-model agents to propose, implement, and evaluate machine-learning changes in a closed loop, but is usually judged by its terminal pipeline. A terminal score cannot reveal which technical decision produced a gain or distinguish a reusable discovery from a change adapted to development feedback. We introduce intervention-centered Auto Research, which validates research decisions rather than only final artifacts and makes their reliability measurable. Feature, Model, Representation, and Data axes are searched independently with inner five-fold feedback. Each axis winner is frozen before an outer-holdout matrix compares all alternatives on evidence the loop never sees. Across 701 agent-executed attempts spanning ten Matbench endpoints, outer evidence confirms the selected intervention on nine of ten endpoints and preserves 89.3% of non-tied intervention orderings. It also rejects an aggregate Representation gain that inner feedback endorsed. The resulting matrix reveals an information-dependent hierarchy. Composition-only tasks support several routes to improvement, whereas structure-informed tasks favor local geometry features and complementary tree ensembles. A subsequent compatibility test combines already frozen Feature and Model code without further search or tuning and raises mean outer-holdout improvement from 19.0% to 26.3%. By validating decisions rather than only artifacts, this design turns adaptive search into reusable evidence wherever agents propose executable alternatives against a fixed evaluator.

Jingjie Ning, Xiaochuan Li, Shanshan Zhong +2

School of Computer Science, Carnegie Mellon University · 2DP Technology

Research Automation Auto Research Reusable Knowledge Feedback Materials

Ask PDF

Aug 26, 2026cs.LG

TraceML: What Auto-Research Agents Miss in Long-Horizon ML Development

Auto-research agents now run machine-learning development unattended for hours, revising data pipelines, models, and validation from their own feedback, yet on most competitions they still finish below strong human competitors. Outcome-based benchmarks record this gap but not its cause, because they grade the final submission and discard the development process behind it. We introduce TraceML, which pairs human and agent work on the same competitions under one version-level schema: 4{,}465 human Kaggle trajectories across 134 competitions, seven of which are also worked by two agent scaffolds, giving 430 paired human and 207 agent trajectories. Every code version carries its score, its timestamp, and labels for the action taken, its intent, the edit size, and the score effect. Read this way, the gap becomes concrete. Experts alternate data work, validation, model changes, and ensembling, and return to approaches they had set aside. Each agent scaffold instead collapses into a narrow loop: Codex spends its steps re-weighting ensembles and tuning submissions, MLEvolve mutates its model in place, and neither pivots at the human rate nor reopens abandoned work. A short planning prompt distilled from human practice moves the behaviors it names toward the human profile and lifts scores, but the effort profile stays agent-shaped: instruction closes only the part of the gap that reduces to instructions. We release the corpus, the schema, and the labeling models, together with an open-source toolkit that turns a run from any command-line agent into a TraceML trajectory and reads it against the human cohorts.

Jiarui Yan, Weiwei Sun, Sijie Li +2

Carnegie Mellon University

Mle-Bench Lite Traces Long-Horizon Task Planning

Ask PDF