LLM critics increasingly steer iterative search in agentic AutoML, prompt optimization, and self-refinement--evaluated for whether they help, but rarely for whether they can hurt. We isolate the critic's marginal effect through a four-arm audit and characterize the loop's tolerance through controlled fault injection. We show a naive 8B critic drives search below the no-critic random-search floor on 7 of 8 tabular datasets, losing up to 0.245 AUC in the worst case; the magnitude varies strongly with dataset, frontier, and budget. When: stronger critics attenuate the loss on the evaluated suite, without establishing a universal capability threshold. Why: controlled fault injection shows that non-directional errors are nearly costless under best-so-far search, whereas directional errors consume finite search opportunities; an idealized effective-sample calculation yields a necessary condition, q(m−1)>β, not a deployment certificate. How: a simple observable validation gate and a learned detector can reduce realized loss, but neither is guaranteed to preserve the full-budget no-critic baseline. We will release CRITFAULT, to our knowledge the first benchmark for detecting harmful critic feedback in AutoML. A critic is not a free addition: audit before trusting, and gate when capability or verification is insufficient.
Figures & tables
Figure 1: A weak LLM critic actively harms search. Left: worst case on adult —an 8 B critic (coral, AUC 0.68 ) pins search far below no-critic random search (teal, 0.93 ) and an independently seeded second random search (gray). Right: the below-floor sign holds on 7/8 datasets. Bars show the No-Critic Δ , the critic-guided minus no-critic final test AUC, so negative means below floor (used throughout).
Figure 2: The iterative LLM-guided search loop we study.
Figure 3: When . (a) Capability ladder (No-Critic Δ , 8 seeds): adult falls from −0.245 (2024-gen 8B) to ∣Δ∣≤0.011 at every ≥ 14B tier (Table 1 ). (b) The same 8B critic on the 96-configuration frontier with T=8 (coral) and, for three datasets, on a 250-configuration frontier with T=20 (teal); frontier and horizon change together, so the contrast does not isolate either factor.
3B 25
8B 24
8B 25
14B
32B
70B
H4.5
adult
−.210
−.245
+.001
+.001
+.001
−.011
+.002
electr.
−.064
−.072
−.012
+.012
+.008
−.031
+.006
jm1
−.008
−.080
−.003
−.002
−.000
−.011
+.002
kc1
+.000
−.028
−.000
−.001
−.001
−.023
−.000
bank-mkt
−.007
+.000
+.000
+.001
−.007
+.001
credit-g
+.006
−.005
−.012
−.007
+.006
−.001
Table 1: Capability ladder: No-Critic Δ (AUC, 8 seeds) across seven tiers spanning four providers and two generations (2024–2025). The 3B model was evaluated on four of the six datasets.
Setting (space size)
Δ
95% CI
Verdict
Tabular AutoML (96)
−0.245
[−0.27,−0.22]
harm
Credit scoring (96)
+0.000
[−0.001,+0.002]
neutral
20NG text-pipe (36)
−0.003
[−0.008,+0.001]
neutral
20NG weak → strong (50)
+0.003
[−0.005,+0.010]
neutral
M4 forecasting (13)
+0.001
[+0.001,+0.002]
helps
RAG retrieval (54)
+0.007
[+0.006,+0.008]
helps
Table 2: Cross-domain test of the mismatch hypothesis (same 8B critic, four-arm protocol, 8 seeds; No-Critic Δ , 95% bootstrap CI). Credit scoring is an in-domain control (same domain, same 96 -config space) yet neutral: only the original tabular pipeline-search setting is harmed.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Analysis
Datasets
Purpose
Real-critic audit
8 binary
Below-floor harm
Fault injection
9 (see text)
Error structure
Capability ladder
6 targeted
Capability attenuation
CritFault
12 total
Feedback detection
Appendix
Table 3: Experimental scope by analysis. Exact dataset names and model settings are reported in the appendix.
Nine datasets (mean 0.982 , all ≥0.96 )
bank-marketing
0.990
letter
0.983
credit-g
0.989
churn
0.980
phoneme
0.988
jm1
0.974
adult
0.987
kc1
0.966
electricity
0.985
Near-zero headroom (mean 0.851 )
Appendix
Table 4: Bad-feedback detector, leave-one-dataset-out AUC (train on 11 datasets, test on the held-out one), injection labels, all five features (including the two offline structural features). Mean 0.982 on the nine datasets outside the near-zero-headroom group (every one ≥0.96 ) and 0.851 on the three near-zero-headroom datasets (segment/mfeat/splice), where nearly every configuration scores near the oracle.
Dataset
3B 25
8B 24
8B 25
14B
32B
70B
H4.5
adult
−.210
−.245
+.001
+.001
+.001
−.011
+.002
electricity
−.064
−.072
−.012
+.012
+.008
−.031
+.006
jm1
−.008
−.080
−.003
−.002
−.000
−.011
+.002
kc1
+.000
−.028
−.000
−.001
−.001
−.023
−.000
bank-marketing
−.007
+.000
+.000
+.001
−.007
+.001
credit-g
+.006
−.005
−.012
−.007
+.006
−.001
Appendix
Table 5: Capability ladder across a seven-tier, four-provider span: real-critic No-Critic Δ (AUC, 8 seeds, identical frozen splits) on six discriminating datasets (full per-dataset version of Table 1 ). The 2024-generation 8 B critic (Meta) is negative on 5/6 ; the larger critics are within 0.031 of zero but not uniformly non-negative (Llama-3.3- 70 B reaches −0.031 on electricity and −0.023 on kc1 ). The pattern is not monotone in size, provider, or release date.
Dataset
no-critic
oracle
AutoGluon
electricity
.9411
.9542
.9316
credit-g
.8017
.8085
.7849
churn
.9412
.9463
.9264
adult
.9287
.9302
.9236
bank-marketing
.9297
.9328
.9309
kc1
.8279
.8314
.8378
Appendix
Table 6: Competitive AutoML setup. Test ROC-AUC on identical frozen public splits: our no-critic floor and in-space oracle vs. AutoGluon-Tabular [ 10 ] (real SOTA, 120 s matched budget, medium_quality ). The in-space oracle matches or beats AutoGluon on 5 of 8 datasets; the harm we measure is thus relative to a SOTA-competitive baseline.
Dataset
headroom
Δ @96
Δ @250
electricity
.056
−30.8 [ −43.6,−18.9 ]
−36.0 [ −48.1,−24.4 ]
credit-g
.042
−22.2 [ −29.0,−15.1 ]
−28.3 [ −38.8,−19.7 ]
kc1
.030
−14.3 [ −19.7,−8.9 ]
−25.4 [ −36.3,−15.9 ]
jm1
.028
−14.9 [ −19.6,−10.2 ]
−15.9 [ −24.0,−8.6 ]
adult
.017
−5.8 [ −9.0,−3.2 ]
−13.2 [ −18.1,−8.9 ]
Appendix
Table 7: Enlarging the space does not protect against a modeled systematically-adversarial critic. Wide weak-to-strong space with real headroom ( 1.7 – 5.6% oracle-over-mean AUC); enlarging the space 2.6× ( 96→250 configs) with a modeled systematically-adversarial critic (No-Critic Δ , AUC ×103 , 20 seeds, T=16 , 95% bootstrap CI). Harm grows with space size on all five datasets.
Data
raw − nc
filter − nc
KL raw
KL filter
adult
−.0002
+.0015
8.96
0.15
jm1
−.0028
+.0036
8.70
0.37
kc1
−.0045
+.0018
7.32
5.17
credit-g
−.0036
−.0017
8.44
1.65
Appendix
Table 8: Detector-as-online-filter under the legacy protocol (validation-best reject branch, five features) and the family-selection KL (Llama-3.1-8B critic, 5 seeds, T=10 , LODO-trained, reject threshold τ=0.6 ). KL is between the critic’s picked model-family distribution and the oracle-optimal one.
Search policy
mean harmful Δ
n<0 /12
sequential
−0.0183
9
multi-agent (pooled)
−0.0095
9
tree-search (UCB1)
−0.0080
9
parallel k -wide
−0.0004
1
Appendix
Table 9: Harm generalizes across search backbones. Mean harmful-critic Δ (AUC vs. no-critic) over 12 datasets, 30 seeds; n<0 = datasets with Δ<−0.002 . A systematically adversarial critic degrades sequential, tree-search, and multi-agent search alike; parallel k -wide expansion is nearly immune—breadth hedges against an unreliable critic.
Figure 4: Accuracy–cost trade-off for the weak 8B critic ( adult ). Naive, chain-of-thought, and 5× -cost self-consistency all spend real LLM generations to land below the zero-cost no-critic floor. The “+VGC filter” point uses the legacy learned filter (validation-best reject branch; see above).
Family (#calls)
adult
jm1
credit-g
kc1
no-critic (0)
.000
.000
.000
.000
naive (8)
−.278
−.018
+.006
−.019
CoT (8)
−.041
−.014
+.001
−.006
self-consistency (40)
−.071
+.001
+.003
−.017
+ legacy filter (8)
+.002
+.004
−.002
+.002
Appendix
Table 10: Harm is not a naive-prompting artifact. No-Critic Δ (AUC, 8 seeds, single controlled run) by critic method family , Llama-3.1-8B. Chain-of-thought and self-consistency (#calls = LLM generations/run) reduce but do not eliminate harm (negative on 3/4; self-consistency worse than CoT on adult at 5× cost). The last row uses the legacy filter protocol.
Held-out dataset
raw (§ 4 )
Fine-tuned (LOO)
VGC
adult
−245
+0.5[−0.5,+1.4]
+1.0[0.0,+1.9]
electricity
−72
+1.6[−2.1,+7.0]
+3.5[−1.5,+8.2]
jm1
−80
−0.8[−3.4,+1.7]
+1.7[−1.7,+4.5]
kc1
−28
−3.1[−9.4,+1.2]
−1.2[−3.9,+1.9]
churn
−2
+1.7[−1.1,+4.5]
−1.8[−6.1,+1.9]
bank-marketing
−7
−0.7[−1.4,+0.0]
−0.7[−6.6,+3.6]
Appendix
Table 11: Supervised fine-tuning vs. VGC under leave-one-dataset-out (No-Critic Δ , AUC ×103 , 8 seeds, 95% bootstrap CI). For each row the critic is QLoRA-fine-tuned on oracle picks from the other seven datasets and evaluated on the held-out one. Fine-tuning averts the collapse on all eight but only returns to the floor (no CI excludes zero positively) and needs oracle labels. The VGC column uses the legacy protocol.
Dataset
ntrain
ntest
Type
credit-g
700
300
binary
kc1
1476
633
binary (defect)
mfeat-factors
1400
600
multiclass → bin
segment
1617
693
multiclass → bin
splice
2233
957
3-class → bin
churn
3500
1500
binary
Appendix
Table 12: Public OpenML suite. All splits frozen; multiclass targets binarized (minority vs. rest).
Detector
LODO mean AUC
random
0.502
majority (base rate)
0.500
best single feature
0.916
logistic (5 feat.)
0.921
gradient-boosted (ours)
0.950
Appendix
Table 13: CritFault reference baselines (leave-one-dataset-out mean AUC, 12 folds). The task is non-trivial (random/majority =0.50 ); the bad-feedback signature is strong enough that a single decision-time feature reaches 0.92 , and the full detector tops the panel.
Dataset
pworst⋆
cost 1worst
cost 1random
credit-g
0.6
0.093
0.005
adult
0.6
0.084
0.001
churn
0.6
0.078
0.004
jm1
0.9
0.061
0.004
electricity
0.9
0.051
0.002
kc1
0.9
0.048
0.004
Appendix
Table 14: Fault-injection (30 seeds). p⋆ = smallest corruption rate with >0.01 AUC drop; cost 1 = AUC lost at full corruption. Adversarial modes tolerate 60–100% bad feedback; random corruption is nearly free (cost 1≤0.005 , no threshold)—see § 5 .
Dataset
no-crit
oracle
No-Critic Δ [95% CI]
electricity
.9465
.9542
+.0053 [ .0027,.0085 ]
churn
.9412
.9463
+.0050 [ .0035,.0066 ]
credit-g
.8031
.8085
+.0049 [ .0027,.0073 ]
jm1
.7281
.7321
+.0038 [ .0025,.0053 ]
bank-marketing
.9298
.9328
+.0028 [ .0022,.0034 ]
phoneme
.9444
.9483
+.0025 [ .0016,.0034 ]
Appendix
Table 15: Four-arm ablation with a deterministic ideal critic (30 seeds). No-Critic Δ = real − no-critic, 95% bootstrap CI (paired). Small but robust: all 9 listed datasets have a CI excluding 0 (paired Wilcoxon p<0.005 ); suite-level Wilcoxon p=0.001 . The three near-zero-headroom datasets are omitted. The consequential quantity is not this upside but the downside of a bad critic (§ 4 ).
Round
Proposed family
Val AUC
Below incumbent?
1
lightgbm
0.930
— (initial)
2
lightgbm
0.927
yes
3
lightgbm
0.926
yes
4
gradient_boosting
0.922
yes
5
hist_gbdt
0.928
yes
6
extra_trees
0.902
yes
Appendix
Table 16: A real Llama-3.1-8B critic trajectory on adult (oracle: lightgbm, 0.930 ). The critic endorses the strong family in words while proposing steadily worse configurations; every post-initial pick is below the incumbent (Assumption 1 ).