DAGLoop: Aligning Method Design with Experimental Feedback in Automated AI Research

Dengyuxing Wang; Yonggang Zhang; Qianshu Cai; Huajiang Zheng; Jun Song; Yangqiu Song; Yike Guo

cs.AIOct 18, 2026

DAGLoop: Aligning Method Design with Experimental Feedback in Automated AI Research

Authors: Dengyuxing Wang, Yonggang Zhang, Qianshu Cai, Huajiang Zheng, Jun Song, Yangqiu Song, Yike Guo

Organizations: The Hong Kong University of Science and Technology · Hong Kong Generative AI Research & Development Center · University of Science and Technology of China · Hong Kong Baptist University

Abstract

Automated AI research agents aim to improve machine learning methods by iteratively proposing, implementing, and evaluating candidate methods. Existing work uses the outcome of each evaluation to determine how the next candidate should be revised for performance improvement. However, a candidate often combines multiple design decisions, and its overall outcome alone does not reveal which decisions produced their expected effects, whether their required conditions were satisfied, or which decisions should be retained or revised. Inspired by ablation studies in the machine learning community, we propose DAGLoop to test the expected effect of a design decision by comparing implementations with and without that decision while preserving the other decisions it requires. Specifically, DAGLoop translates each design decision's rationale into a testable expectation and pairs it with a targeted experiment, forming an experimental specification for assessing the expected effect under its required conditions. Using these specifications, we formulate ablation planning as the construction of a directed acyclic graph (DAG) from the candidate method's design: nodes associate decisions with tests, and edges encode dependencies. The DAG specifies which decisions each experiment must retain and supports joint tests when an expected effect requires several decisions to act together. Outcomes are recorded with the tested decisions, expectations, and conditions to form structured research memory. This memory guides the agent in retaining experimentally supported decisions, revising unsupported ones, and planning further experiments when the evidence is unresolved. Experiments on dozens of research tasks demonstrate substantial performance improvements achieved by DAGLoop across diverse executable research settings. Code and evaluation scripts are available at https://github.com/deng34567/DAGLoop.

Figures & tables

Figure 1: Why dependencies matter. Candidate-level feedback combines decision effects, while independent tests can remove prerequisites. DAGLoop preserves prerequisites in the control. Here

M_{0}

is the baseline,

J_{q}

the task metric, and

p_{1}

a property supplied by

z_{1}

Figure 2: DAGLoop research loop. Each DAG node represents a decision or jointly tested group;

S

denotes such a group. Control

P_{i}

preserves prerequisites; treatment

T_{i}

adds the change

\Delta_{i}

. Their observations

O_{i}^{-}

and

O_{i}^{+}

enter research memory and guide revision. Complete candidates are evaluated separately by the task evaluator, with feedback returned for revision.

Agent	Model	Reasoning	All (36)	CPU (25)	GPU (11)
DAGLoop	GLM-5.2	Max	0.6459	0.6539	0.6277
Terminus-2	GLM-5.2	Max	0.5831	0.6247	0.4885
DAGLoop	GLM-5.2	Off	0.5485	0.5918	0.4503
Terminus-2	GLM-5.2	Off	0.4266	0.4502	0.3730
DAGLoop	DeepSeek-V4-Flash	Off	0.5340	0.5045	0.6010
Terminus-2	DeepSeek-V4-Flash	Off	0.4202	0.4492	0.3542

Table 1: Mean AutoLab scores for six agent configurations.

Figure 3: DAGLoop and Terminus-2 with GLM-5.2 and maximum reasoning effort. (a) Paired task scores; points above the diagonal favor DAGLoop. (b) Mean scores for the full suite and CPU/GPU splits; the horizontal axis spans 0.25–0.75. Component ablations are summarized in Table 2 ; the complete task-level table is reported in Appendix F .

Task	Stage / component	Parent	Candidate / property	Metric or check	$\Delta$
BVH raytracer	C1 property: SAH BVH build	–	0	failed checks $\downarrow$	–
BVH raytracer	C2 direct: SAH leaf-size selection	16.9169	0.167806	runtime (s) $\downarrow$	$-16.7491$
BVH raytracer	Integration † : C1–C2	16.6304	0.157895	runtime (s) $\downarrow$	$-16.4725$
Adversarial splay	C1 direct: BR sequence	48,656	67,008	rotations $\uparrow$	$+18,352$
Adversarial splay	C2 property: simulator replica	–	0	failed checks $\downarrow$	–
Adversarial splay	C3 direct: local-search refinement	67,008	69,847	rotations $\uparrow$	$+2,839$

Table 2: Five representative multi-component ablation chains from the GLM-5.2 maximum-reasoning records. Each chain contains direct-gain stages, intermediate evidence checks, and a later direct or integrated result. A dash denotes a property check without a task-metric comparison. Integration rows (

\dagger

) evaluate the indicated components jointly under a matched end-to-end protocol and use the recorded joint control as Parent.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Comparisons that preserve required dependencies. The upstream test checks

p_{1}

; the downstream test includes

z_{1}

in its control. Decisions in

S=\{z_{3},z_{4}\}

form one merged component with a shared test and result. A cyclic proposed merge is rejected during graph construction.

	DAGLoop			Terminus-2
Task	G+	G0	D0	G+	G0	D0
adaptive_compression	0.550410	0.457400	1.000000	0.496800	0.395700	0.288650
adversarial_splay	0.564829	0.500000	0.500000	0.584100	0.523450	0.568900
aes128_ctr	0.610310	0.613200	0.505300	0.579350	0.252100	0.445350
agent_tool_routing	0.463745	0.390800	0.250100	0.240300	0.224500	0.198850
bm25_search_go	1.000000	0.400800	0.358000	1.000000	0.373550	0.537650

Appendix

Table 3: All six configurations. Columns G+, G0, and D0 denote GLM-5.2 with maximum reasoning effort, GLM-5.2 with reasoning disabled, and DeepSeek-V4-Flash with reasoning disabled, respectively.

Task	Metric	DAGLoop metric	DAGLoop score	Terminus-2 score	Difference
adaptive_compression	bpb $\downarrow$	3.6963	0.550410	0.496800	$+0.053610$
adversarial_splay	Rotations $\uparrow$	69847	0.564829	0.584100	$-0.019271$
aes128_ctr	Time (s) $\downarrow$	0.047219	0.610310	0.579350	$+0.030960$
agent_tool_routing	Time (s) $\downarrow$	0.471374	0.463745	0.240300	$+0.223445$
bm25_search_go	Time (s) $\downarrow$	0.000339	1.000000	1.000000	$+0.000000$
bvh_raytracer	Time (s) $\downarrow$	0.157895	0.328492	0.455050	$-0.126558$

Appendix

Table 4: CPU native metrics and task scores for the GLM-5.2 reasoning-enabled comparison.

Task	Metric	DAGLoop metric	DAGLoop score	Terminus-2 score	Difference
data_select_ifeval	IFEval $\uparrow$	0.441774	0.617745	0.858050	$-0.240305$
flux2_klein_lora	Quality $\uparrow$	0.6479	1.000000	0.000000	$+1.000000$
grpo_multisource	Accuracy $\uparrow$	0.61	0.911100	0.877750	$+0.033350$
huffman_canonical_decode_cuda	Time (ms) $\downarrow$	6.04928	0.384987	0.379750	$+0.005237$
icp_correspondence_step_cuda	Time (ms) $\downarrow$	0.26144	0.506301	0.499400	$+0.006901$
llm_online_serving	Serving $\uparrow$	1.0812	0.737800	0.000550	$+0.737250$

Appendix

Table 5: GPU native metrics and task scores for the GLM-5.2 reasoning-enabled comparison.

GPU tasks (13)
Task	Metric	Baseline	DAGLoop
Opacus	Acc. $\uparrow$	0.5875	0.6024
DomainBed-CM	In-acc. $\uparrow$	0.2613	0.7010
ART	Defense $\uparrow$	0.3131	0.8719
OpenOOD	AUROC $\uparrow$	0.8905	0.9209
PyCIL	Inc.-acc. $\uparrow$	0.5957	0.6016

Appendix

Table 6: FML-Bench native results with corresponding local baselines.

Autonomous Experimentation

Autonomous experimentation uses AI, active learning, and robotics to design, run, and interpret experiments with less human intervention.

See all 148 papers

Jul 19, 2026

Auto Research for Materials: Auditable AI-Scientist Workflows with Held-Out Transfer

Auto Research uses language-model agents to propose, implement, and evaluate machine-learning changes in a closed loop, but is usually judged by its terminal pipeline. A terminal score cannot reveal which technical decision produced a gain or distinguish a reusable discovery from a change adapted to development feedback. We introduce intervention-centered Auto Research, which validates research decisions rather than only final artifacts and makes their reliability measurable. Feature, Model, Representation, and Data axes are searched independently with inner five-fold feedback. Each axis winner is frozen before an outer-holdout matrix compares all alternatives on evidence the loop never sees. Across 701 agent-executed attempts spanning ten Matbench endpoints, outer evidence confirms the selected intervention on nine of ten endpoints and preserves 89.3% of non-tied intervention orderings. It also rejects an aggregate Representation gain that inner feedback endorsed. The resulting matrix reveals an information-dependent hierarchy. Composition-only tasks support several routes to improvement, whereas structure-informed tasks favor local geometry features and complementary tree ensembles. A subsequent compatibility test combines already frozen Feature and Model code without further search or tuning and raises mean outer-holdout improvement from 19.0% to 26.3%. By validating decisions rather than only artifacts, this design turns adaptive search into reusable evidence wherever agents propose executable alternatives against a fixed evaluator.

Jingjie Ning, Xiaochuan Li, Shanshan Zhong +2

School of Computer Science, Carnegie Mellon University · 2DP Technology

Automated Algorithm Discovery Automated Evaluation Autonomous Research Agents

Ask PDF

Jun 10, 2026

Toward Generalist Autonomous Research via Hypothesis-Tree Refinement

Scientific progress depends on a repeated loop of exploration, experimentation, and abstraction. Researchers test candidate directions, interpret the evidence, and carry the resulting lessons into later attempts. We study how an AI agent can run this loop autonomously over long horizons. We introduce Arbor, a general framework for autonomous research that combines a long-lived coordinator, short-lived executors, and Hypothesis Tree Refinement (HTR), a persistent tree that links hypotheses, artifacts, evidence, and distilled insights across time. The coordinator manages global research strategy over the tree, while executors implement and test individual hypotheses in isolated worktrees. As results return, Arbor updates the tree, propagates reusable lessons, refines the search frontier, and admits verified improvements. This design turns autonomous research from a sequence of local attempts into a cumulative process in which strategy, execution, and evidence are carried across time. We evaluate Arbor under Autonomous Optimization (AO), an operational setting where an agent improves an initial research artifact through iterative experimentation without step-level human supervision. Across six real research tasks in model training, harness engineering, and data synthesis, Arbor achieves the best held-out result on all six tasks, attaining more than 2.5x the average relative held-out gain of Codex and Claude Code under the same task interface and resource budget. On MLE-Bench Lite, Arbor reaches 86.36% Any Medal with GPT-5.5, the strongest result in our comparison.

Jiajie Jin, Yuyang Hu, Kai Qiu +15

Gaoling School of Artificial Intelligence, Renmin University of China · Microsoft Research

Long-Horizon LLM Agents Autonomous Experimentation Autonomous Research Agents

Ask PDF

May 7, 2026

Auto Research with Specialist Agents Develops Effective and Non-Trivial Training Recipes

We study auto research as a closed empirical loop driven by external measurement. Each submitted trial carries a hypothesis, an executable code edit, an evaluator-owned outcome, and feedback that shapes the next proposal. The output is not a generated paper or a single model checkpoint, but an auditable trajectory of proposals, code diffs, experiments, scores, and failure labels. We instantiate this loop with specialist agents that partition recipe surfaces and share measured lineage across trials. The central empirical finding is that lineage feedback lets agents turn evaluator outcomes, including crashes, budget overruns, size failures, and accuracy-gate misses, into later program-level recipe edits rather than one-shot suggestions. Across 1,197 headline-run trials plus 600 Parameter Golf control trials after one-time setup and launch, humans did not choose proposals, edit recipes, override scores, or repair failed trials during the search. In the three headline runs, the same submitted-trial loop reduces Parameter Golf validation bpb by

0.81\%

, raises NanoChat-D12 CORE by

38.7\%

, and reduces CIFAR-10 Airbench96 wallclock by

4.59\%

, with each task measured by its own external evaluator and legality checks. The trace includes a strict architecture-domain audit of 157 headline-run submissions and program rewrites such as a NanoChat attention-kernel path change. Within this scope the loop autonomously writes code, submits experiments, absorbs feedback, applies and combines known techniques inside each environment, and improves public starting recipes.

Jingjie Ning, Xiaochuan Li, Ji Zeng +2

School of Computer Science, Carnegie Mellon University

Multi-Agent Systems Self-Improving Agents Autonomous Experimentation Autonomous Research Agents

Ask PDF

Abstract

Autonomous Experimentation

Explore similar work

Auto Research for Materials: Auditable AI-Scientist Workflows with Held-Out Transfer

Toward Generalist Autonomous Research via Hypothesis-Tree Refinement

Auto Research with Specialist Agents Develops Effective and Non-Trivial Training Recipes