Camyla: Scaling Autonomous Research in Medical Image Segmentation

Yifan Gao; Haoyue Li; Feng Yuan; Xin Gao; Weiran Huang; Xiaosong Wang

cs.AIApr 12, 2026

Camyla: Scaling Autonomous Research in Medical Image Segmentation

Authors: Yifan Gao, Haoyue Li, Feng Yuan, Xin Gao, Weiran Huang, Xiaosong Wang

Organizations: University of Science and Technology of China · Shanghai Innovation Institute · Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory

Abstract

We present Camyla, a system for fully autonomous research within the scientific domain of medical image segmentation. Camyla transforms raw datasets into literature-grounded research proposals, executable experiments, and complete manuscripts without human intervention. Autonomous experimentation over long horizons poses three interrelated challenges: search effort drifts toward unpromising directions, knowledge from earlier trials degrades as context accumulates, and recovery from failures collapses into repetitive incremental fixes. To address these challenges, the system combines three coupled mechanisms: Quality-Weighted Branch Exploration for allocating effort across competing proposals, Layered Reflective Memory for retaining and compressing cross-trial knowledge at multiple granularities, and Divergent Diagnostic Feedback for diversifying recovery after underperforming trials. The system is evaluated on CamylaBench, a contamination-free benchmark of 31 datasets constructed exclusively from 2025 publications, under a strict zero-intervention protocol across two independent runs within a total of 28 days on an 8-GPU cluster. Across the two runs, Camyla generates more than 2,700 novel model implementations and 40 complete manuscripts, and surpasses the strongest per-dataset baseline selected from 14 established architectures, including nnU-Net, on 22 and 18 of 31 datasets under identical training budgets, respectively (union: 24/31). Senior human reviewers score the generated manuscripts at the T1/T2 boundary of contemporary medical imaging journals. Relative to automated baselines, Camyla outperforms AutoML and NAS systems on aggregate segmentation performance and exceeds six open-ended research agents on both task completion and baseline-surpassing frequency. These results suggest that domain-scale autonomous research is achievable in medical image segmentation.

Figures & tables

Figure 1: Main results of Camyla across two independent runs on CamylaBench . (a) Per-dataset Dice improvement over the strongest baseline for both runs ( Camyla

{}_{\text{D}}

and Camyla

{}_{\text{S}}

). Datasets marked with

\dagger

are validation; the remainder are blind-test. (b) Comparison with six open-ended research agents on 26 blind-test datasets: Camyla achieves 100% task completion with zero proposal drift. (c) Manuscript quality evaluation: senior human reviewers and AI evaluators both score Camyla ’s manuscripts at the T1/T2 boundary of contemporary medical image segmentation venues. Tier-defining journals: T1 = IEEE Transactions on Medical Imaging and Medical Image Analysis (2 venues); T2 = IEEE Journal of Biomedical and Health Informatics, et al. (7 venues); T3 = Biomedical Physics & Engineering Express, et al. (9 venues).

ID	Dataset	Region	Modality	N	C	Cfg	Baseline	Dice	ID	Dataset	Region	Modality	N	C	Cfg	Baseline	Dice
1 †	GDMRI-CT	Brain	MRI	50	1	3D	STU-Net	70.6	17	Fundus-AVSeg	Ophthal.	Fundus	100	3	2D	U-Mamba	77.6
2 †	MRE-BSA	Abdomen	MRI	114	10	3D	U-Mamba	71.8	18	HRUS-MBT	Soft Tiss.	Ultrasound	34	1	3D	nnU-Net	50.1
3 †	PNPC	Head&Neck	MRI	277	1	3D	nnU-Net	60.6	19	LapGC-KVAD	Abdomen	Laparoscopy	1252	4	2D	U-Mamba	52.8
4 †	AMSMC-HTM	Head&Neck	MRI	24	4	3D	nnU-Net	81.4	20	LongCIU	Lung	CT	90	2	2D	nnU-Net	73.5
5 †	NLSTseg	Lung	CT	604	1	3D	nnU-Net	23.2	21	MSLesSeg	Brain	MRI	115	1	3D	U-Mamba	69.6
6	SMRI-FB	Brain	MRI	70	8	3D	MedNeXt	85.3	22	MU-Glioma	Brain	MRI	594	4	3D	nnU-Net	71.6

Table 1: Complete inventory of the 31 datasets in CamylaBench , sourced from Scientific Data 2025.

\dagger

= validation. Cfg. = 2D/3D. Cls. = foreground classes. Baseline = best of 14 pretrained architectures.

Figure 2: Architecture of the Camyla system. The four-stage pipeline transforms a raw segmentation dataset into a complete research manuscript. Stage 1 establishes baselines via the precomputed baseline bank in CamylaNet . Stage 2 executes experimental discovery through literature-grounded proposal generation and three coupled mechanisms for long-horizon experimentation: Quality-Weighted Branch Exploration (QWBE), Layered Reflective Memory (LRM), and Divergent Diagnostic Feedback (DDF). Stage 3 conducts ablation verification of the winning method. Stage 4 synthesizes the final manuscript through a multi-agent writing pipeline.

Cycle	Trial	Dice $\uparrow$	Status	Diagnostic	Modification Record (abridged)
Cycle 1: HCP-Net — Hierarchical Token Diffusion replacing skip connections
1	1	0.7142	Baseline	—	Precomputed UKAN baseline.
1	2	0.6835	Underperf.	impl. shortcut	HTD with cross-attention; hierarchy embedding omitted.
1	3 $\star$	0.7682	Success	—	HTD at 64 ${\times}$ 64; learnable hierarchy embeddings as query bias.
1	6	0.7610	Success	—	Reverted to query-bias (guided by Trial 3 in cycle memory).
1	7	0.6976	Underperf.	regression	Added Sobel edge-aware refinement; hurt Dice.

Table 2: Complete experimental trajectory across three research cycles on optic disc segmentation (Dataset 16, baseline UKAN Dice = 0.7142). Each cycle explores a different research proposal; the best artifact and a compressed global memory are relayed to the next cycle. Trials marked with

\star

achieved the cycle-best result. The system progressively improves from 0.7142 to 0.8191 (+10.5 Dice points) as global-level insights accumulate and guide subsequent cycles.

	Camyla ${}_{\text{D}}$	Camyla ${}_{\text{S}}$
Idea generator model	DeepSeek V3.2	Claude Sonnet 4.6
Success rate (all 31)	18 / 31 (58.1%)	22 / 31 (71.0%)
Blind-test (26)	14 / 26 (53.8%)	18 / 26 (69.2%)
Validation (5)	4 / 5	4 / 5
Win via Dice	16	11
Win via HD95 tiebreak	2	11

Table 3: Summary of two independent Camyla runs on CamylaBench (31 datasets). Success is defined as exceeding the baseline Dice by more than 0.5 pp, or achieving lower HD95 when Dice is within 0.5 pp. HD95 is measured in millimeters.

	All (31)	Blind-test (26)
Both runs succeed	16	12
Camyla ${}_{\text{D}}$ only	2	2
Camyla ${}_{\text{S}}$ only	6	6
Neither	7	6
Union	24 (77.4%)	20 (76.9%)

Table 4: Cross-run consistency between Camyla

{}_{\text{D}}

and Camyla

{}_{\text{S}}

. The union of both runs covers substantially more datasets than either run alone.

Figure 3: Per-dataset performance change over the best baseline for both independent runs. Four panels show (a)

\Delta

Dice for Camyla

{}_{\text{D}}

, (b)

\Delta

Dice for Camyla

{}_{\text{S}}

, (c)

\Delta

HD95 for Camyla

{}_{\text{D}}

, and (d)

\Delta

HD95 for Camyla

{}_{\text{S}}

. Orange bars indicate improvement; blue bars indicate regression. For HD95, positive values indicate reduced boundary error (improvement).

Method	Type	Search Space	Paper
AutoNNUNet	AutoML	Closed	No
DiNTS	NAS	Closed	No
AutoSeg3D	Ensemble	Closed	No
autoresearch	Research agent	Open	No
AI Scientist	Research agent	Open	Yes
AI Scientist-v2	Research agent	Open	Yes

Table 5: Characterization of automated baselines compared against Camyla . “Search space” indicates whether the method explores a predefined space (closed) or can propose arbitrary code modifications (open). “Paper output” indicates whether the method produces a manuscript.

Method	Datasets	Mean Dice	Pareto vs Camyla ${}_{\text{D}}$
AutoNNUNet	31	64.27%	9/31
DiNTS (retrained)	31	58.03%	1/31
AutoSeg3D	18	63.43%	5/18
Camyla ${}_{\text{D}}$	31	65.93%	—
Camyla ${}_{\text{S}}$	31	65.22%	—

Table 6: Aggregate comparison between Camyla and automated segmentation baselines. For each method, we report the mean Dice across evaluated datasets and the number of datasets on which the method is Pareto-superior to Camyla

{}_{\text{D}}

. AutoSeg3D is evaluated only on the 18 volumetric (3D) datasets.

Family	Configuration	Completed	$>$ Baseline	Proposal Drift	Mean Dice	Mean HD95	Median Dice	Median HD95
autoresearch	Claude Code + Opus 4.6	18 / 26	10 / 26	4 / 26	63.14%	34.72	67.25%	20.83
autoresearch	Claude Code + MiniMax 2.5	15 / 26	12 / 26	14 / 26	62.58%	36.41	66.43%	22.15
autoresearch	Claude Code + GLM 4.7	16 / 26	10 / 26	12 / 26	62.21%	37.06	65.89%	23.04
autoresearch	Codex + GPT-5.4 ( xhigh )	12 / 26	8 / 26	10 / 26	61.35%	38.92	64.71%	24.67
AI Scientist	Default workflow	18 / 26	5 / 26	–	60.24%	40.55	63.18%	26.31
AI Scientist-v2	BFTS workflow	12 / 26	3 / 26	–	58.73%	43.18	61.52%	28.94

Table 7: Comparison with open-ended research agents on the 26 blind-test datasets. “Completed” denotes successful experiment execution; “

>

Baseline” counts datasets on which the system exceeds the baseline (Dice improvement

>

0.5 pp, or HD95 improvement when Dice is within 0.5 pp). “Proposal Drift” denotes clear deviation from the original plan. Dice and HD95 are summarized over completed runs only. Camyla

{}_{\text{D}}

= DeepSeek V3.2; Camyla

{}_{\text{S}}

= Claude Sonnet 4.6.

Tier	Journals	Papers	Representative venues
T1	2	10	IEEE Transactions on Medical Imaging; Medical Image Analysis
T2	7	35	IEEE Journal of Biomedical and Health Informatics; Artificial Intelligence in Medicine; et al.
T3	9	45	International Journal of Computer Assisted Radiology and Surgery; Biomedical Physics & Engineering Express; et al.
Total	18	90

Table 8: Tier composition of the external evaluation benchmark (90 papers across 18 journals, five papers per journal). Representative venues are listed; see Appendix 15.7 for the complete list.

Source	T1	T2	T3	Internal	Ext. Overall
Junior reviewers	3.208	3.109	2.433	3.053	2.954
Senior reviewers	3.364	3.217	2.792	3.311	3.038
All human	3.257	3.145	2.558	3.137	2.972
AI average (5 models)	3.644	3.051	2.492	3.610	2.862

Table 9: Aggregate overall-recommendation scores by reviewer group and paper tier on the structured-summary track. Human rows correspond to the provisional reviewer matrix; the AI row reports the average over five frontier model evaluators.

	T1	T2	T3	Int. Blind	Int. Val.	Int. All	Ext. All
Stanford Agentic	5.690	4.603	4.422	4.766	4.363	4.685	4.633

Table 10: Stanford Agentic Reviewer results on full manuscripts.

Figure 4: Manuscript quality scores across evaluation tracks. (a) Structured-summary track: mean overall-recommendation scores from senior human reviewers and the five-model AI average, stratified by paper tier (T1, T2, T3) and Camyla -generated internal papers. The dashed line marks the external overall mean for senior reviewers. (b) Full-manuscript track: Stanford Agentic Reviewer scores for external tiers and Camyla internal blind-test papers.

Configuration	Wins ( / 5)	Mean $\Delta$ Dice	Mean FSP $\downarrow$	Mean Nodes
Full Camyla	4	+1.85	4.8	21.4
$-$ QWBE	3	+1.31	9.5	26.8
$-$ LRM	2	+0.47	6.3	24.2
$-$ DDF	3	+1.02	7.3	23.6

Table 11: Aggregate ablation results on the five validation datasets (1, 2, 3, 4, 5). Wins : datasets exceeding the best baseline. Mean

\Delta

Dice : average Dice improvement (pp). Mean FSP : average first-success position (lower is better; computed over winning datasets only). Mean Nodes : average total nodes expanded.

Figure 5: Cumulative win rate as a function of node budget

N

. A dataset counts as won at budget

N

if any node with index

\leq N

exceeds the baseline. Two independent runs and their union are shown.

Figure 6: Full page layouts of two representative manuscripts generated by Camyla . (a) Dataset 8: a 17-page paper on neonatal brain lesion segmentation proposing a hierarchical context gating architecture. (b) Dataset 11: a 17-page paper on liver segmentation from cirrhotic MRI proposing a selective state space modeling framework. Both manuscripts are produced end-to-end without human intervention.

Figure 7: Node-by-node exploration trajectory for Dataset 2 (MRE-BSA, Proposal 1: SFP-DiffNet). Baseline: U-Mamba, Dice = 0.7179. Each row shows one experimental node with its parent derivation, architectural description, and Dice score (red: below baseline; green: above). All 10 nodes fail to surpass the baseline. The system explores local vs. global gating, multi-head voting, and iterative diffusion variants of the Anatomical Flow Field Generator, but never escapes the “flow field + diffusion” framework. The best attempt (N1, Dice = 0.6758) falls short by 4.2 pp.

Figure 8: Node-by-node exploration trajectory for Dataset 3 (PNPC, Proposal 2: DDSR-Net). Baseline: nnU-Net, Dice = 0.6063. Eight root branches explore diverse DDAM variants (attention-based, deformable, multi-scale), all below baseline. The turning point occurs at the N7 $\to$ N9 transition: the system identifies N7 as the strongest root (Dice = 0.5515) and performs an architecture-level redesign—replacing spatial attention with deformable sampling and CBAM-style attention—jumping to 0.6088. N10 adds multi-scale context and axial attention, reaching 0.6134 and surpassing the baseline (+0.71 pp). The chain 7 $\to$ 9 $\to$ 10 demonstrates QWBE shifting from broad exploration to focused refinement.

Figure 9: Node-by-node exploration trajectory for Dataset 9 (BTXRD, Proposal 1: HCF-CCN). Baseline: UTNet, Dice = 0.4445. Nodes 1–6 explore the CCAN+MRFA direction (Dice 0.19–0.26, far below baseline). At N7, the system abandons CCAN entirely and introduces Multi-Scale Gated Axial Attention (MGAA)—decomposing 2D attention into sequential 1D axial operations—jumping to 0.4711 and surpassing the baseline for the first time. The subsequent chain 7 $\to$ 8 $\to$ 9 $\to$ 10 refines MGAA to 0.5017 (+5.7 pp over baseline). This pivot is not incremental but a fundamentally different attention paradigm, exemplifying the behavior DDF is designed to produce.

ID	Dataset	BL	Best	$\Delta$	FSP	N	ID	Dataset	BL	Best	$\Delta$	FSP	N
Full Camyla							$-$ QWBE
1	GDMRI-CT	72.33	73.87	+1.53	12	30	1	GDMRI-CT	72.33	72.98	+0.65	18	30
2	MRE-BSA	71.79	70.40	$-$ 1.39	–	30	2	MRE-BSA	71.79	69.87	$-$ 1.92	–	30
3	PNPC	60.63	61.34	+0.71	19	30	3	PNPC	60.63	61.08	+0.45	22	30
4	AMSMC-HTM	81.41	82.69	+1.29	1	30	4	AMSMC-HTM	81.41	82.25	+0.84	6	30
5	NLSTseg	23.20	41.78	+18.58	1	30	5	NLSTseg	23.20	36.41	+13.21	3	30

Table 12: Per-dataset ablation results on the five validation datasets. For each configuration we report the best Dice achieved ( Best ), the Dice improvement over baseline (

\Delta

), the first-success position ( FSP ; “–” if never succeeded), and the total nodes expanded ( N ). ∗ Mean FSP is computed over winning datasets only.

			Camyla ${}_{\text{D}}$ (DeepSeek V3.2)				Camyla ${}_{\text{S}}$ (Claude Sonnet 4.6)
ID	Dataset	BL (%)	Best	$\Delta$ BL	FSP	Mech.	Best	$\Delta$ BL	FSP	Mech.	$\|\Delta_{\text{D-S}}\|$
1 †	GDMRI-CT	72.33	73.87	+1.54	12	D	72.49	+0.16	1	H	1.38
3 †	PNPC	60.63	61.34	+0.71	19	H	61.95	+1.32	18	D	0.61
4 †	AMSMC-HTM	81.41	82.69	+1.28	1	D	82.89	+1.48	6	H	0.20
5 †	NLSTseg	23.20	41.78	+18.58	3	D	41.26	+18.06	11	D	0.52
6	SMRI-FB	85.30	86.44	+1.14	2	H	85.74	+0.44	6	H	0.70

Table 13: Per-dataset comparison of the two independent runs on the 16 co-won datasets.

\dagger

= validation. Mech. : D = Dice

>

0.5 pp; H = HD95 tiebreak.

			AutoNNUNet		Camyla ${}_{\text{D}}$		Camyla ${}_{\text{S}}$		Baseline
ID	Dataset	Cfg.	Dice	HD95	Dice	HD95	Dice	HD95	Dice	HD95
1 †	GDMRI-CT	3d	72.83%	7.52	73.87%	5.58	72.49%	7.11	72.33%	8.45
2 †	MRE-BSA	3d	70.73%	16.38	70.40%	16.21	69.13%	17.00	71.79%	17.75
3 †	PNPC	3d	61.20%	52.45	61.34%	21.05	61.95%	27.73	60.63%	45.28
4 †	AMSMC-HTM	3d	82.77%	2.60	82.69%	2.60	82.89%	2.62	81.41%	2.89
5 †	NLSTseg	3d	47.33%	111.85	41.78%	165.10	41.26%	183.73	23.20%	183.22

Table 14: Detailed comparison between AutoNNUNet and Camyla (two independent runs) on the 31-dataset benchmark.

\dagger

= validation. Best values among the three methods are bolded .

ID	Dataset	Cfg.	Search	Retrained	Camyla ${}_{\text{D}}$	Camyla ${}_{\text{S}}$	Baseline
			DiNTS
1 †	GDMRI-CT	3d	65.85%	66.15%	73.87%	72.49%	70.29%
2 †	MRE-BSA	3d	57.92%	63.45%	70.40%	69.13%	69.72%
3 †	PNPC	3d	48.21%	45.79%	61.34%	61.95%	60.63%
4 †	AMSMC-HTM	3d	24.95%	79.36%	82.69%	82.89%	81.41%
5 †	NLSTseg	3d	46.78%	24.45%	41.78%	41.26%	23.20%

Table 15: Detailed comparison between DiNTS and Camyla (two independent runs) on the benchmark.

\dagger

= validation. Best values among DiNTS, Camyla

{}_{\text{D}}

, and Camyla

{}_{\text{S}}

are bolded .

			AutoSeg3D		Camyla ${}_{\text{D}}$		Camyla ${}_{\text{S}}$		Baseline
ID	Dataset	Cfg.	Dice	HD95	Dice	HD95	Dice	HD95	Dice	HD95
1 †	GDMRI-CT	3d	72.03%	7.91	73.87%	5.58	72.49%	7.11	72.33%	8.45
2 †	MRE-BSA	3d	69.78%	16.86	70.40%	16.21	69.13%	17.00	71.79%	17.75
3 †	PNPC	3d	58.43%	50.30	61.34%	21.05	61.95%	27.73	60.63%	45.28
4 †	AMSMC-HTM	3d	82.50%	2.72	82.69%	2.60	82.89%	2.62	81.41%	2.89
5 †	NLSTseg	3d	41.84%	122.56	41.78%	165.10	41.26%	183.73	23.20%	183.22

Table 16: Detailed comparison between AutoSeg3D and Camyla (two independent runs) on the 18 volumetric benchmark datasets.

\dagger

= validation. Best values among the three methods are bolded .

Item	Value
Benchmark datasets	31
Validation datasets	5
Blind-test datasets	26
External papers	90
Internal papers	40
Total papers	130

Table 17: Evaluation design used in the current benchmark analysis.

Group	Venue / Source	Definition	# Papers	Role in Benchmark
T1	IEEE Transactions on Medical Imaging	Top-tier medical image analysis venue	5	External benchmark
T1	Medical Image Analysis	Top-tier medical image analysis venue	5	External benchmark
T2	IEEE Journal of Biomedical and Health Informatics	JCR Q1 specialty journal	5	External benchmark
T2	Computerized Medical Imaging and Graphics	JCR Q1 specialty journal	5	External benchmark
T2	Biomedical Signal Processing and Control	JCR Q1 specialty journal	5	External benchmark
T2	Computer Methods and Programs in Biomedicine	JCR Q1 specialty journal	5	External benchmark

Table 18: Detailed composition of the evaluation benchmark. Each retained public journal contributes five papers, yielding 90 public papers in total. The internal comparison pool contributes 40 papers.

Figure 10: Per-dimension manuscript quality scores across evaluation tracks. Four panels show (a) clarity of problem formulation and motivation, (b) methodological novelty, (c) experimental completeness, and (d) overall recommendation. Each panel reports mean scores stratified by paper tier (T1, T2, T3) and Camyla -generated internal papers, for three reviewer groups: junior human reviewers, senior human reviewers, and the five-model AI average. Error bars indicate standard error of the mean.

Architecture	2D	3D	Family
nnU-Net	✓	✓	Convolutional
SwinUNETR	✓	✓	Transformer
SegResNet	✓	✓	Convolutional
U-Net++	✓	✓	Convolutional
U-Mamba	✓	✓	State-space
MedNeXt		✓	Convolutional

Table 19: Baseline bank architecture compatibility. Each architecture is marked with the configurations it supports.

Configuration	Wins	$p$ ( $p_{0}{=}0.5$ )	$p$ ( $p_{0}{=}0.3$ )	Wilson 95% CI
Camyla ${}_{\text{S}}$ (all 31)	22/31	0.015	$3.0\times 10^{-6}$	[53.4%, 83.9%]
Camyla ${}_{\text{D}}$ (all 31)	18/31	0.237	0.001	[40.8%, 73.6%]
Camyla ${}_{\text{S}}$ (blind 26)	18/26	0.038	$2.4\times 10^{-5}$	[50.0%, 83.5%]
Camyla ${}_{\text{D}}$ (blind 26)	14/26	0.423	0.007	[35.5%, 71.2%]
Union (all 31)	24/31	0.002	—	[60.2%, 88.6%]
Union (blind 26)	20/26	0.004	—	[57.8%, 89.4%]

Table 20: Binomial test results (one-sided,

H_{1}:p>p_{0}

) and Wilson 95% confidence intervals for win rates.

ID	Dataset	Cfg	$N$	BL Dice	Best Dice	$\Delta$ Dice	$p$ (Dice)	Sig	$p$ (HD95)	Sig
1 $\dagger$	GDMRI-CT	3D	44	0.7233	0.7387	+0.0153	5.07e-01	n.s.	9.75e-01	n.s.
3 $\dagger$	PNPC	3D	325	0.6063	0.6134	+0.0071	1.38e-01	n.s.	3.85e-01	n.s.
4 $\dagger$	AMSMC-HTM	3D	251	0.8141	0.8298	+0.0157	6.26e-09		1.28e-06
5 $\dagger$	NLSTseg	3D	1,446	0.2320	0.4686	+0.2365	1.29e-66		3.04e-11
6	SMRI-FB	3D	1,242	0.8530	0.8667	+0.0136	4.52e-40		1.89e-22
7	LMD-BM	3D	383	0.4619	0.5151	+0.0532	1.58e-15		4.10e-04

Table 21: Per-sample Wilcoxon signed-rank tests for Camyla

{}_{\text{D}}

winning experiments.

\dagger

= validation.

N

= paired samples. ***

p{<}0.001

, **

p{<}0.01

, *

p{<}0.05

, n.s. = not significant.

ID	Dataset	Cfg	$N$	BL Dice	Best Dice	$\Delta$ Dice	$p$ (Dice)	Sig	$p$ (HD95)	Sig
1 $\dagger$	GDMRI-CT	3D	44	0.7233	0.7277	+0.0044	9.52e-01	n.s.	3.20e-01	n.s.
3 $\dagger$	PNPC	3D	325	0.6063	0.6195	+0.0133	1.64e-01	n.s.	4.43e-01	n.s.
4 $\dagger$	AMSMC-HTM	3D	251	0.8141	0.8289	+0.0148	7.47e-06		1.15e-07
5 $\dagger$	NLSTseg	3D	1,446	0.2320	0.4126	+0.1805	$<$ 1e-300		4.95e-07
6	SMRI-FB	3D	1,242	0.8530	0.8590	+0.0059	6.77e-14		1.97e-07
7	LMD-BM	3D	383	0.4619	0.5101	+0.0482	1.05e-08		2.56e-04

Table 22: Per-sample Wilcoxon signed-rank tests for Camyla

{}_{\text{S}}

winning experiments. Notation follows Table 21 .

	Run D (18)	Run S (22)	Total (40)
Dice significant	12 (67%)	12 (55%)	24 (60%)
HD95 significant	11 (61%)	13 (59%)	24 (60%)
Either metric	13 (72%)	15 (68%)	28 (70%)
Both metrics	10 (56%)	10 (45%)	20 (50%)

Table 23: Aggregate per-sample significance counts across both runs.

Metric	Value
Total experiments	62
Total GPU-hours	5,461.9 h
Total GPU-days	227.6 d
Mean GPU-hours / experiment	88.1 h
Median GPU-hours / experiment	34.2 h
Baseline-beating experiments (40)	2,595.2 h (mean 64.9 h)

Table 24: Computational cost summary across both runs (62 experiments total). GPU-hours are measured on NVIDIA RTX 4090 48 GB GPUs under the unified early-stop policy.

ID	Dataset	Att.1	Att.2	Tot.	Sav.	ID	Dataset	Att.1	Att.2	Tot.	Sav.
1 †	GDMRI-CT	33.6	9.6	43.2	52.3	17	Fundus-AVSeg	6.5	31.2	37.8	33.0
2 †	MRE-BSA	81.3	33.5	114.8	0.0	18	HRUS-MBT	243.5	9.5	253.0	144.3
3 †	PNPC	34.7	22.8	57.4	42.0	19	LapGC-KVAD	188.1	235.9	423.9	153.9
4 †	AMSMC-HTM	45.0	21.4	66.4	109.6	20	LongCIU	42.7	9.4	52.1	50.0
5 †	NLSTseg	26.5	32.6	59.1	61.0	21	MSLesSeg	51.9	12.4	64.3	80.7
6	SMRI-FB	48.4	23.1	71.5	78.9	22	MU-Glioma	474.7	256.1	730.8	0.0

Table 25: Per-dataset GPU-hours under the unified early-stop policy. Each dataset has two attempts (one per run).

\dagger

= validation. Tot. = sum across both attempts. Sav. = GPU-hours saved by early stopping. Total: 5,461.9 h (saved 1,930.5 h).

Dimension	Count
Datasets covered	31
Independent experiments	62
Experimental-discovery directories	147
Agent event files (.jsonl)	2,865
Agent session summaries (.md)	3,953
Generated experiment code files (.py)	1,343

Table 26: Scale of the CamylaTrace-232k dataset. All counts are restricted to experimental-discovery directories from the 62 main experiments.

Dataset	Domain	Scope	Events	Code
SWE-bench	Software eng.	Bug fixes	Patch-level	Yes
ML-bench	Machine learning	Task completion	Limited	Yes
CamylaTrace-232k	Scientific research	Iterative research	232K	1,343

Table 27: Comparison of CamylaTrace-232k with existing agent trajectory datasets.

#	Category	Priority	Suggestion
1	Proposal_Gap	High	Implement the learnable hierarchy embedding ( $\xi_{h}$ ) in the HTD module as described in the proposal. Currently the cross-attention only uses projected features; adding the hierarchy embedding to the Key ( $K=T_{e}^{l}+\xi_{h}$ ) will allow the model to learn the spatial/structural relationship between the optic disc and cup.
2	Architecture	Medium	Replace the fixed $32{\times}32$ spatial pooling in the HTD module with a higher resolution ( $64{\times}64$ ). The current implementation significantly downsamples high-resolution fundus features before attention, likely losing fine boundary details.
3	Code_Fix	High	Fix the SADCB fusion logic. The proposal specifies a channel-wise gating mechanism ( $\alpha=\sigma(\mathrm{Conv}_{1\times 1}(X))$ ) to fuse parallel branches, but the current code uses a simplified selection or summation.
4	Architecture	Medium	Add a residual connection around the HTD module in the decoder. A residual path for the original skip-connection information alongside the diffused tokens would stabilize training.
5	Hyperparameter	Medium	Increase base_channels from 32 to 64 and adjust embed_dim in the HTD module to 128.

Table 28: DDF diagnostic report for Dataset 16, Cycle 1, Trial 2 (Dice = 0.6835 vs. baseline 0.7142).

#	Category	Priority	Suggestion
1	Proposal_Gap	High	Implement the Complexity Estimator in RATF using a $3{\times}3$ Sobel filter or local variance to generate a complexity map, then use this map to weight a Softmax over different pooling kernel sizes to achieve true adaptive tokenization.
2	Architecture	High	Convert the CATA block into a bottleneck style where the multi-scale attention output is concatenated with a $3{\times}3$ depthwise-separable convolution branch before the final projection, ensuring local spatial consistency.
3	Code_Fix	Medium	Reduce the token_reduction factor from 4 to 2 and increase the initial hidden_sizes from 64 to 96.
4	Architecture	Medium	Replace the learned Positional Embedding with Conditional Positional Encoding (CPE) using a $3{\times}3$ depthwise convolution applied to the token grid.
5	Proposal_Gap	Medium	Implement the Size-Biased Attention formula: $\mathrm{Attn}=\mathrm{Softmax}(QK^{\top}/\sqrt{d}+\log(\text{token\_sizes})\cdot W_{s})$ , directly injecting scale information into the attention map as proposed.

Table 29: DDF diagnostic report for Dataset 16, Cycle 3, Trial 2 (Dice = 0.6095 vs. baseline 0.7142).

Module	Status	Description
HTD Tokenization	Simplified	Uses AdaptiveAvgPool2d instead of strided projection
HTD Hierarchy Embedding	Simplified	Additive vector vs. conditional attention bias
SADCB Gating	Simplified	Softmax over branches vs. per-channel adaptive gating

Table 30: DDF optimization-mode audit after Trial 3 beats the baseline (Dice = 0.7682).

Cycle	Trial	W. Dice	L. Dice	Status	Diagnostic	Key Modification
Cycle 1: HCP-Net — Hierarchical Token Diffusion replacing skip connections
Seeded from: Baseline (UKAN, Dice = 0.7142)
1	1	0.7142	—	Baseline	—	Precomputed UKAN baseline.
1	2	0.6835	Err	Underperf.	code_issue	HTD with patch embedding + cross-attention; hierarchy embedding omitted, spatial pooling too aggressive ( $32{\times}32$ ).
1	3 $\star$	0.7682	0.6753	Success	—	HTD at higher resolution ( $64{\times}64$ ); learnable hierarchy embeddings as query bias in cross-attention.
1	4	0.6916	0.6053	Underperf.	regression	Moved hierarchy embedding from query to key; mathematical formulation change degraded performance.

Table 31: Complete experimental trajectory for Dataset 16 (optic disc segmentation, UKAN baseline Dice = 0.7142). W./L. Dice : winning/losing agent’s Dice.

\star

= cycle-best. Global memory at cycle transitions shown in italics.

Cycle	Proposal	Best Dice	$\Delta$ BL	Best Trial	Trials	Key Insight
1	HCP-Net	0.7682	+5.40	3	11	Token diffusion with query-bias works; boundary modules hurt.
2	BAFNet	0.7829	+6.87	8	11	Boundary-aware normalization works if numerically stable; multi-scale detection hurts.
3	ARTNet	0.8191	+10.49	6	11	Adaptive tokenization via complexity estimation; simplicity $>$ complexity confirmed.

Table 32: Cross-cycle summary for Dataset 16. Each cycle explores a distinct proposal, seeded from the previous cycle’s best artifact with compressed global memory.

Autonomous Research Agents

Autonomous research agents use language models and tools to search literature, design and run experiments, analyze results, and support scientific discovery.

See all 318 papers

Jun 1, 2026

AutoMedBench: Towards Medical AutoResearch with Agentic AI Models

Autonomous agents are increasingly expected to support end-to-end medical-AI research workflows, moving beyond isolated prediction tasks or short-form clinical question answering. However, existing medical agent benchmarks primarily evaluate final outputs, providing limited visibility into agent behavior within the research process. To address this gap, we present AutoMedBench, a workflow-aware benchmark for autonomous medical-AI research across diverse medical imaging and multimodal inference tasks, organizing agent execution into a unified five-stage workflow (S1-S5): Plan, Setup, Validate, Inference, and Submit. It comprises long-horizon tasks with each run averaging 33 agent turns, spanning five research tracks: segmentation, image enhancement, visual question answering (VQA), report generation, and lesion detection. Each task is evaluated under two difficulty tiers, Lite and Standard, which use the same data and metrics but differ in the amount of task-brief scaffolding, and each run is scored using both final task performance and S1-S5 stage scores, enabling stage-level analysis from the initial task brief to the final submitted artifact. Across thousands of recorded runs, stage-level scoring reveals that Validate is the weakest workflow stage on average, whereas Setup is the strongest, suggesting that current agents are better at making pipelines executable than at verifying their reliability. Post-run error analysis further shows that verification and submission failures dominate tagged errors, accounting for 37.7% and 38.1% of fired codes respectively, whereas task-understanding errors are rare at 0.9%, and runs with one fired error code have a 48% lower overall score than runs with no error code on average.

Junqi Liu, Selena Song, Yuhan Wang +12

University of California, Santa Cruz · 2NVIDIA

AI Agent Evaluation AI Agent Benchmarks Scientific Workflow Automation Autonomous Research Agents Medical AI Agents

Ask PDF

Oct 20, 2026

A Multi-Agent Framework for Automated Machine Learning with Evidence-based Planning for Medical Image Analysis

Deep learning has become the default tool for medical image analysis, driven by the open-source movement. However, achieving the best performance on a given task remains difficult, with architecture selection and hyperparameter optimization (HPO) still considered "more of an art than a science". Automated machine learning (AutoML) automates parts of model training, yet helps little in medical imaging: general-purpose approaches lack medical architectures and preprocessing, and medical-focused ones demand engineering expertise and remain confined to narrow tasks such as 3D segmentation. We introduce MedicalMLOmni, a multi-agent AutoML framework that automates model training for diverse single- and multi-modal medical tasks through a natural language interface. Specifically, MedicalMLOmni performs evidence-based planning by retrieving published results to guide model selection and configuration, and employs parallel HPO where multiple agents explore adjustment strategies concurrently. MedicalMLOmni achieves the best score on 19 of 20 benchmarks; PMC-VQA is the exception, where a vision-language model with broader pre-training leads. Backbone-matched controls attribute the gains to the framework, not the backbone. By removing coding requirements, MedicalMLOmni lowers the barrier for domain experts with rich datasets but limited engineering expertise.

Yiqing Shen, Mathias Unberath

Department of Computer Science, Johns Hopkins University

Automated Machine Learning Medical Image Analysis

Ask PDF

Aug 29, 2026

Towards Fully Automated Medical Imaging Code Generation via Validation-based Context Engineering

Large language models (LLMs) have demonstrated considerable promise in program generation for small-scale and conventional application development; however, they remain limited when applied to complex, domain-specific tasks such as medical image processing. General-purpose models lack explicit domain knowledge and robust validation mechanisms to ensure correctness, often requiring substantial human intervention to produce reliable processing pipelines. To address these limitations, we propose AutoMedImg, a multi-agent framework for fully automated medical image processing code generation. AutoMedImg orchestrates specialised agents across two phases: a Planning Phase that performs dataset analysis and architecture design with semantic and formal verification, and a Coding Phase that generates modules in parallel with static checking, execution testing, and assembly validation. This multi-stage validation mitigates error propagation throughout generation, while comprehensive auto-context engineering combining domain-specific knowledge bases, shared memory, and validation feedback automates context construction without manual prompting. A cross-project adaptive pipeline synthesis mechanism further accumulates validated pipelines and retrieves proven components for new tasks based on project similarity, enhancing generation efficiency through cross-project learning. Extensive evaluation across six diverse and well-established medical imaging datasets with five backbone LLMs demonstrates that AutoMedImg achieves zero human intervention, with Dice scores of up to 0.90 for segmentation tasks and 99% accuracy for classification.

Zixiao Zhao, Jing Sun, Zhe Hou +5

University of Auckland Auckland, New Zealand · Griffith University Brisbane, Australia · Suzhou Industrial Park Monash Research Institute of Science and Technology Suzhou, China +2

LLM Code Generation

Ask PDF

Abstract

Autonomous Research Agents

Explore similar work

AutoMedBench: Towards Medical AutoResearch with Agentic AI Models

A Multi-Agent Framework for Automated Machine Learning with Evidence-based Planning for Medical Image Analysis

Towards Fully Automated Medical Imaging Code Generation via Validation-based Context Engineering