Applying machine learning systems to new domains introduces bottlenecks in both labeled data acquisition and manual pipeline engineering. While recent MLE agents show promise in automating pipeline design, they have been studied almost exclusively in fully supervised settings, leaving them untested in realistic regimes where labeled data is scarce but unlabeled data is abundant. In this work, we introduce Agentic Semi-Supervised Learning (AgentSSL), framing SSL as a program search problem and studying how agents navigate this space under limited supervision. We find that even under weak feedback and matched compute, agents can synthesize code that combines and adapts established SSL techniques, exceeding the strongest human-engineered baselines in 8 of 12 settings by up to +21.8%. However, our results identify program selection as a critical bottleneck in AgentSSL, where performance drops by as much as 47% when selecting based on the agent's feedback metric. To systematically study this bottleneck, we introduce ProgramSelectBench, a large-scale benchmark with a combined total of 16,977 agent-generated and human-engineered SSL programs, and evaluate both unsupervised and active program selection methods. We further evaluate AgentSSL on real-world ecology detection tasks, highlighting the opportunities and challenges of transferring agent-discovered strategies beyond controlled benchmarks.
Figures & tables
Figure 1: MLE Agent research is focused on reducing engineering cost, but relies on human labeled data. SSL research seeks to reduce annotation effort but requires human engineering to deploy. We study whether AgentSSL can reduce both costs.
Figure 2: Overview of AgentSSL. Given a small labeled set and a larger unlabeled corpus, an LLM-based agent generates SSL programs that are evaluated using either unsupervised SSL metrics or noisy validation feedback, with the resulting fitness score guiding iterative refinement. Following search, we must select the best program either using unsupervised metrics or active model selection. We evaluate AgentSSL on both image classification SSL benchmarks and real-world ecological detection tasks.
DTD
SUN397
RESISC45
RETINO
CLEVR
KITTI
Method
3
6
3
6
1
2
40
80
10
20
5
10
Supervised
61.47
70.23
63.35
70.54
55.71
70.98
50.66
51.48
39.96
46.62
41.54
45.66
FixMatch
60.71
70.12
63.10
70.90
62.13
81.60
34.93
39.70
38.93
56.32
39.05
42.76
SoftMatch
62.59
69.80
69.68
73.29
73.17
86.04
44.14
54.37
46.48
66.83
40.04
41.30
FineSSL
65.83
73.00
68.41
72.33
76.13
86.93
27.47
27.43
34.21
47.54
40.74
37.22
PET
65.43
71.97
63.94
68.19
60.48
78.31
30.45
32.77
33.34
34.31
39.01
42.80
Table 1: Oracle test set performance across datasets and shot settings. The best SSL baseline engineered by humans is shown in yellow per dataset/shot. The best agent-generated result is shown in purple . We see that the best agent solution ( Abest ) is generally comparable to the best SSL human baseline ( Hbest ) and outperforms it on 8 of 12 datasets. N (noisy validation) and U (unsupervised) denote the feedback metric.
DTD
SUN397
RESISC45
RETINO
CLEVR
KITTI
Method
3
6
3
6
1
2
40
80
10
20
5
10
Humanall
− 3.39
− 2.87
− 4.69
− 1.69
− 14.63
− 1.93
− 23.92
− 42.66
− 15.59
− 20.28
− 9.10
− 17.53
AgentSSLN
− 0.67
− 2.34
− 0.13
− 0.13
–
− 10.38
− 38.12
− 47.29
0.00
− 2.66
− 8.20
− 19.83
AgentSSLU
− 3.81
− 10.74
− 1.90
− 3.60
− 14.43
− 2.25
− 24.12
− 18.62
− 13.56
− 3.23
− 10.50
− 20.96
Table 2: Unsupervised selection regret averaged across seeds. Human row selects across all human algorithms and hyperparameters using the rank Zhang et al. (2025) metric. Agent rows select across all programs from a single trajectory using the feedback metric. Darker red denotes larger selection regret.
Method
DTD
SUN397
RESISC45
RETINO
CLEVR
KITTI
ActiveTesting
-2.35
-1.17
-2.61
+25.59
-5.41
+10.46
VMA
-1.86
-1.47
-7.68
+26.07
-4.49
+14.43
CODA
-2.50
-2.09
-0.87
+21.22
-29.09
+14.63
IID
-4.71
-0.83
-3.47
+26.07
-5.75
+10.01
ModelSelector
-2.32
-0.87
-13.21
+18.70
-2.05
+0.00
Table 3: Comparison in final selection accuracy when using labels for training versus selecting on AgentSSL unsupervised metric runs. Each cell reports the mean across active-selection seeds for train (50%) / active selection (50%) minus train (100%) / unsupervised selection (0%). IID, ActiveTesting, VMA and ModelSelector are averaged over 10 runs; CODA is deterministic.
Figure 3: Performance of AgentSSL programs as the amount of unlabeled data varies while the labeled set remains fixed.
Figure 4: Agents discover diverse and complementary SSL strategies. Left: The distribution of SSL method categories from evolutionary search rollouts on DTD. Right: Different high-performing programs produce confident pseudolabels on complementary subsets of the unlabeled data.
Figure 5: Cross-dataset generalization of agent-discovered SSL programs. Rows indicate the source dataset from which the best program is selected, and columns indicate the target dataset. Values show the change in accuracy relative to in-domain performance.
DTD (k=6)
RESISC45 (k=2)
KITTI (k=5)
Method
A
D
A
D
A
D
GPT-5.4
0.72
0.37
0.78
0.37
0.47
0.42
Gemini-3.1
0.71
0.46
0.76
0.59
0.46
0.52
Table 4: LLM comparison with accuracy (A) and diversity (D).
DTD
KITTI
RESISC45
Method
A
D
A
D
A
D
MCTS
0.70
0.57
0.46
0.66
0.82
0.58
Evolutionary
0.74
0.66
0.48
0.65
0.77
0.68
Greedy
0.72
0.37
0.47
0.42
0.78
0.37
Table 5: Comparison of search strategies in terms of accuracy (A) and diversity (D).
Figure 6: Agent performance on ecological detection tasks. On WildFin, AgentSL and AgentSSL are comparable and approach the full-data expert pipeline. On BuckTales, both agents exhibit high variance and underperform the expert YOLOv8 baseline.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Correlation between the agent’s feedback metric and test accuracy for (a) the unsupervised feedback metric and (b) noisy validation. Each point is one non-buggy program generated during search, pooled across both shot settings and all three search seeds for each dataset. The x-axis is the program’s fitness score or feedback metric, the numerical score the agent received during search as feedback. The y-axis is its test accuracy averaged over three evaluation seeds. Red lines are least-squares fits, and r is the Pearson correlation coefficient.
Baseline Method
Models trained
Supervised
144
FixMatch
18
SoftMatch
18
FineSSL
18
PET
6
V-PET
6
Appendix
Table 6: Number of model configurations evaluated per dataset and shot setting. PET and V-PET additionally require predictions from 144 LoRA supervised and 144 AdaptFormer [ 7 ] supervised models.
Algorithm
DTD
SUN397
RESISC45
RETINO
CLEVR
KITTI
Total
Supervised
56.7
272.1
74.1
194.2
165.3
30.3
792.6
FixMatch
27.9
486.6
168.2
473.9
430.9
37.5
1,625.1
SoftMatch
24.7
494.0
163.9
543.8
451.7
58.3
1,736.3
FineSSL
23.9
800.4
154.8
1,293.1
400.8
42.7
2,715.7
PET
5.7
158.0
43.7
91.2
83.4
9.6
391.6
V-PET
7.6
139.5
42.0
98.4
70.8
9.2
367.4
Appendix
Table 7: Total train compute in GPU-hours. Baseline compute is aggregated across seeds, shot settings, models, adaptation configurations, and learning rates. Agent compute includes all trajectories using unsupervised feedback ( AgentSSLU ) and noisy-validation feedback ( AgentSSLN ).
CLEVR
DTD
KITTI
RESISC45
RETINO
SUN397
TOTAL
10
20
3
6
5
10
1
2
40
80
3
6
Baseline
354
354
354
354
354
354
354
354
354
354
354
354
4,248
Noisy-Val
375
243
1,401
498
603
537
–
348
195
477
375
309
5,361
Unsupervised
1,113
456
1,251
309
798
1,317
333
261
372
237
537
384
7,368
Total
1,842
1,053
3,006
1,161
1,755
2,208
687
963
921
1,068
1,266
1,047
16,977
Appendix
Table 8: ProgramSelectBench tasks and number of programs in each task.
Figure 8: Accuracy Distribution for each ProgramSelectBench task.
Metric
IID
ActiveTesting
VMA
CODA
ModelSelector
Simple regret
3.80
4.14
4.78
0.00
1.73
Cumulative regret
70.07
83.76
90.60
0.00
12.50
Appendix
Table 9: Mean per-task standard deviation for cumulative regret and simple regret across active-selection seeds. Simple regret is regret at the final iteration. Cumulative regret sums the simple regrets across all 25 iterations.
Figure 9: Selection regret for both unsupervised selection metrics and active selection metrics across datasets and shot counts, for agent noisy validation tasks (left), agent unsupervised tasks (right), and baseline tasks (bottom).
Figure 10: Regret vs. Iterations plot for all active selection methods on every ProgramSelectBench task.
Figure 11: High-level SSL method distributions across datasets for greedy rollouts using the unsupervised metric as feedback.
DTD
RESISC45
KITTI
Method
3
6
1
2
5
10
AgentSLavgU
65.15
69.73
66.09
78.86
43.27
46.65
AgentSSLavgU
65.44
72.32
71.53
78.10
47.35
47.40
AgentSLavgN
65.53
72.55
-
73.71
44.16
40.23
AgentSSLavgN
65.57
71.45
-
76.06
42.71
45.85
AgentSLmaxU
65.83
72.23
74.83
79.56
37.26
40.50
Appendix
Table 10: Comparison between AgentSL and AgentSSL across datasets and label settings. Superscripts indicate the feedback signal used during search: U denotes unsupervised metrics and N denotes noisy validation feedback. Subscripts indicate seed aggregation: avg reports average performance across seeds, while max reports the average best-discovered program from each seed.
Figure 16: Comparison of rollout behavior under different feedback signals across datasets. Top row shows noisy validation feedback and bottom row shows unsupervised feedback. Fitness score denotes the scalar feedback metric provided to the agent during search.