DAGLoop: Aligning Method Design with Experimental Feedback in Automated AI Research

Dengyuxing Wang; Yonggang Zhang; Qianshu Cai; Huajiang Zheng; Jun Song; Yangqiu Song; Yike Guo

cs.AIOct 18, 2026

DAGLoop: Aligning Method Design with Experimental Feedback in Automated AI Research

Authors: Dengyuxing Wang, Yonggang Zhang, Qianshu Cai, Huajiang Zheng, Jun Song, Yangqiu Song, Yike Guo

Organizations: The Hong Kong University of Science and Technology · Hong Kong Generative AI Research & Development Center · University of Science and Technology of China · Hong Kong Baptist University

Abstract

Automated AI research agents aim to improve machine learning methods by iteratively proposing, implementing, and evaluating candidate methods. Existing work uses the outcome of each evaluation to determine how the next candidate should be revised for performance improvement. However, a candidate often combines multiple design decisions, and its overall outcome alone does not reveal which decisions produced their expected effects, whether their required conditions were satisfied, or which decisions should be retained or revised. Inspired by ablation studies in the machine learning community, we propose DAGLoop to test the expected effect of a design decision by comparing implementations with and without that decision while preserving the other decisions it requires. Specifically, DAGLoop translates each design decision's rationale into a testable expectation and pairs it with a targeted experiment, forming an experimental specification for assessing the expected effect under its required conditions. Using these specifications, we formulate ablation planning as the construction of a directed acyclic graph (DAG) from the candidate method's design: nodes associate decisions with tests, and edges encode dependencies. The DAG specifies which decisions each experiment must retain and supports joint tests when an expected effect requires several decisions to act together. Outcomes are recorded with the tested decisions, expectations, and conditions to form structured research memory. This memory guides the agent in retaining experimentally supported decisions, revising unsupported ones, and planning further experiments when the evidence is unresolved. Experiments on dozens of research tasks demonstrate substantial performance improvements achieved by DAGLoop across diverse executable research settings. Code and evaluation scripts are available at https://github.com/deng34567/DAGLoop.

Figures & tables

Figure 1: Why dependencies matter. Candidate-level feedback combines decision effects, while independent tests can remove prerequisites. DAGLoop preserves prerequisites in the control. Here

M_{0}

is the baseline,

J_{q}

the task metric, and

p_{1}

a property supplied by

z_{1}

Figure 2: DAGLoop research loop. Each DAG node represents a decision or jointly tested group;

S

denotes such a group. Control

P_{i}

preserves prerequisites; treatment

T_{i}

adds the change

\Delta_{i}

. Their observations

O_{i}^{-}

and

O_{i}^{+}

enter research memory and guide revision. Complete candidates are evaluated separately by the task evaluator, with feedback returned for revision.

Agent	Model	Reasoning	All (36)	CPU (25)	GPU (11)
DAGLoop	GLM-5.2	Max	0.6459	0.6539	0.6277
Terminus-2	GLM-5.2	Max	0.5831	0.6247	0.4885
DAGLoop	GLM-5.2	Off	0.5485	0.5918	0.4503
Terminus-2	GLM-5.2	Off	0.4266	0.4502	0.3730
DAGLoop	DeepSeek-V4-Flash	Off	0.5340	0.5045	0.6010
Terminus-2	DeepSeek-V4-Flash	Off	0.4202	0.4492	0.3542

Table 1: Mean AutoLab scores for six agent configurations.

Figure 3: DAGLoop and Terminus-2 with GLM-5.2 and maximum reasoning effort. (a) Paired task scores; points above the diagonal favor DAGLoop. (b) Mean scores for the full suite and CPU/GPU splits; the horizontal axis spans 0.25–0.75. Component ablations are summarized in Table 2 ; the complete task-level table is reported in Appendix F .

Task	Stage / component	Parent	Candidate / property	Metric or check	$\Delta$
BVH raytracer	C1 property: SAH BVH build	–	0	failed checks $\downarrow$	–
BVH raytracer	C2 direct: SAH leaf-size selection	16.9169	0.167806	runtime (s) $\downarrow$	$-16.7491$
BVH raytracer	Integration † : C1–C2	16.6304	0.157895	runtime (s) $\downarrow$	$-16.4725$
Adversarial splay	C1 direct: BR sequence	48,656	67,008	rotations $\uparrow$	$+18,352$
Adversarial splay	C2 property: simulator replica	–	0	failed checks $\downarrow$	–
Adversarial splay	C3 direct: local-search refinement	67,008	69,847	rotations $\uparrow$	$+2,839$

Table 2: Five representative multi-component ablation chains from the GLM-5.2 maximum-reasoning records. Each chain contains direct-gain stages, intermediate evidence checks, and a later direct or integrated result. A dash denotes a property check without a task-metric comparison. Integration rows (

\dagger

) evaluate the indicated components jointly under a matched end-to-end protocol and use the recorded joint control as Parent.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Comparisons that preserve required dependencies. The upstream test checks

p_{1}

; the downstream test includes

z_{1}

in its control. Decisions in

S=\{z_{3},z_{4}\}

form one merged component with a shared test and result. A cyclic proposed merge is rejected during graph construction.

	DAGLoop			Terminus-2
Task	G+	G0	D0	G+	G0	D0
adaptive_compression	0.550410	0.457400	1.000000	0.496800	0.395700	0.288650
adversarial_splay	0.564829	0.500000	0.500000	0.584100	0.523450	0.568900
aes128_ctr	0.610310	0.613200	0.505300	0.579350	0.252100	0.445350
agent_tool_routing	0.463745	0.390800	0.250100	0.240300	0.224500	0.198850
bm25_search_go	1.000000	0.400800	0.358000	1.000000	0.373550	0.537650

Appendix

Table 3: All six configurations. Columns G+, G0, and D0 denote GLM-5.2 with maximum reasoning effort, GLM-5.2 with reasoning disabled, and DeepSeek-V4-Flash with reasoning disabled, respectively.

Task	Metric	DAGLoop metric	DAGLoop score	Terminus-2 score	Difference
adaptive_compression	bpb $\downarrow$	3.6963	0.550410	0.496800	$+0.053610$
adversarial_splay	Rotations $\uparrow$	69847	0.564829	0.584100	$-0.019271$
aes128_ctr	Time (s) $\downarrow$	0.047219	0.610310	0.579350	$+0.030960$
agent_tool_routing	Time (s) $\downarrow$	0.471374	0.463745	0.240300	$+0.223445$
bm25_search_go	Time (s) $\downarrow$	0.000339	1.000000	1.000000	$+0.000000$
bvh_raytracer	Time (s) $\downarrow$	0.157895	0.328492	0.455050	$-0.126558$

Appendix

Table 4: CPU native metrics and task scores for the GLM-5.2 reasoning-enabled comparison.

Task	Metric	DAGLoop metric	DAGLoop score	Terminus-2 score	Difference
data_select_ifeval	IFEval $\uparrow$	0.441774	0.617745	0.858050	$-0.240305$
flux2_klein_lora	Quality $\uparrow$	0.6479	1.000000	0.000000	$+1.000000$
grpo_multisource	Accuracy $\uparrow$	0.61	0.911100	0.877750	$+0.033350$
huffman_canonical_decode_cuda	Time (ms) $\downarrow$	6.04928	0.384987	0.379750	$+0.005237$
icp_correspondence_step_cuda	Time (ms) $\downarrow$	0.26144	0.506301	0.499400	$+0.006901$
llm_online_serving	Serving $\uparrow$	1.0812	0.737800	0.000550	$+0.737250$

Appendix

Table 5: GPU native metrics and task scores for the GLM-5.2 reasoning-enabled comparison.

GPU tasks (13)
Task	Metric	Baseline	DAGLoop
Opacus	Acc. $\uparrow$	0.5875	0.6024
DomainBed-CM	In-acc. $\uparrow$	0.2613	0.7010
ART	Defense $\uparrow$	0.3131	0.8719
OpenOOD	AUROC $\uparrow$	0.8905	0.9209
PyCIL	Inc.-acc. $\uparrow$	0.5957	0.6016

Appendix

Table 6: FML-Bench native results with corresponding local baselines.

Autonomous Experimentation

Autonomous experimentation uses AI, active learning, and robotics to design, run, and interpret experiments with less human intervention.

See all 148 papers

Auto Research for Materials: Auditable AI-Scientist Workflows with Held-Out Transfer
Jul 19, 2026Jingjie Ning, Xiaochuan Li, Shanshan Zhong +2Automated Algorithm Discovery Automated Evaluation
School of Computer Science, Carnegie Mellon University · 2DP Technology
Toward Generalist Autonomous Research via Hypothesis-Tree Refinement
Jun 10, 2026Jiajie Jin, Yuyang Hu, Kai Qiu +15Long-Horizon LLM Agents Autonomous Experimentation
Gaoling School of Artificial Intelligence, Renmin University of China · Microsoft Research
Auto Research with Specialist Agents Develops Effective and Non-Trivial Training Recipes
May 7, 2026Jingjie Ning, Xiaochuan Li, Ji Zeng +2Multi-Agent Systems Self-Improving Agents
School of Computer Science, Carnegie Mellon University

Jul 19, 2026J/K move · Enter open · S save

Auto Research for Materials: Auditable AI-Scientist Workflows with Held-Out Transfer

Jingjie Ning, Xiaochuan Li, Shanshan Zhong, Ji Zeng+1

School of Computer Science, Carnegie Mellon University · 2DP Technology

Automated Algorithm Discovery Automated Evaluation Autonomous Research Agents

Open paper AskPDF arXiv

Auto Research uses language-model agents to propose, implement, and evaluate machine-learning changes in a closed loop, but is usually judged by its terminal pipeline. A terminal score cannot reveal which technical decision produced a gain or distinguish a reusable discovery from a change adapted to development feedback. We introduce intervention-centered Auto Research, which validates research decisions rather than only final artifacts and makes their reliability measurable. Feature, Model, Representation, and Data axes are searched independently with inner five-fold feedback. Each axis winner is frozen before an outer-holdout matrix compares all alternatives on evidence the loop never sees. Across 701 agent-executed attempts spanning ten Matbench endpoints, outer evidence confirms the selected intervention on nine of ten endpoints and preserves 89.3% of non-tied intervention orderings. It also rejects an aggregate Representation gain that inner feedback endorsed. The resulting matrix reveals an information-dependent hierarchy. Composition-only tasks support several routes to improvement, whereas structure-informed tasks favor local geometry features and complementary tree ensembles. A subsequent compatibility test combines already frozen Feature and Model code without further search or tuning and raises mean outer-holdout improvement from 19.0% to 26.3%. By validating decisions rather than only artifacts, this design turns adaptive search into reusable evidence wherever agents propose executable alternatives against a fixed evaluator.

Abstract

Autonomous Experimentation

Explore similar work

Auto Research for Materials: Auditable AI-Scientist Workflows with Held-Out Transfer

Toward Generalist Autonomous Research via Hypothesis-Tree Refinement

Auto Research with Specialist Agents Develops Effective and Non-Trivial Training Recipes