BTTackler: A Diagnosis-based Framework for Efficient Deep Learning Hyperparameter Optimization
Authors: Zhongyi Pei, Zhiyao Cen, Yipeng Huang, Chen Wang, Lin Liu, Philip Yu, Mingsheng Long
Organizations: School of Software, BNRist, Tsinghua University, Beijing, China · School of Software, EIRI, Tsinghua University, Beijing, China · School of Software, Tsinghua University, Beijing, China
Hyperparameter optimization (HPO) is known to be costly in deep learning, especially when leveraging automated approaches. Most of the existing automated HPO methods are accuracy-based, i.e., accuracy metrics are used to guide the trials of different hyperparameter configurations amongst a specific search space. However, many trials may encounter severe training problems, such as vanishing gradients and insufficient convergence, which can hardly be reflected by accuracy metrics in the early stages of the training and often result in poor performance. This leads to an inefficient optimization trajectory because the bad trials occupy considerable computation resources and reduce the probability of finding excellent hyperparameter configurations within a time limitation. In this paper, we propose Bad Trial Tackler (BTTackler), a novel HPO framework that introduces training diagnosis to identify training problems automatically and hence tackles bad trials. BTTackler diagnoses each trial by calculating a set of carefully designed quantified indicators and triggers early termination if any training problems are detected. Evaluations are performed on representative HPO tasks consisting of three classical deep neural networks (DNN) and four widely used HPO methods. To better quantify the effectiveness of an automated HPO method, we propose two new measurements based on accuracy and time consumption. Results show the advantage of BTTackler on two-fold: (1) it reduces 40.33% of time consumption to achieve the same accuracy comparable to baseline methods on average and (2) it conducts 44.5% more top-10 trials than baseline methods on average within a given time budget. We also released an open-source Python library that allows users to easily apply BTTackler to automated HPO processes with minimal code changes.
Figures & tables
Figure 1. Comparing BTTackler and random search on HPO of a simple CNN on cifar10
Figure 2. The framework of BTTackler.
Figure 3. The Design of Quality Indicators.
Setup
HP- I
HP- II
HP- III
Task Type
Cifar10CNN
3
5
5
classification
Cifar10LSTM
3
1
4
classification
Ex96Trans
1
4
3
forecasting
Table 1. Number of different hyperparameters.
Task
Cifar10CNN
Cifar10LSTM
Ex96Trans
Evaluation Metric
Classification Accuracy
Classification Accuracy
MSE
Performance
Top1
Top10
Top10HR
Top1
Top 10
Top10HR
Top1
Top10
Top10HR
Random
0.7124
0.6863
-
0.5488
0.5077
-
0.1857
0.1899
-
Random-ETR
0.6635
0.6415
51%
0.5407
0.5210
56%
0.1868
0.1894
62%
Random-BTTackler
0.8200
0.7980
97%
0.5337
0.5040
35%
0.1857
0.1894
64%
GP
0.7605
0.7294
-
0.5522
0.5268
-
0.1851
0.1899
-
Table 2. The effectiveness of BTTackler-enhanced HPO methods on three tasks.
Figure 4. The effectiveness of BTTackler-enhanced HPO methods over time.
Task
Cifar10CNN
Cifar10LSTM
Ex96Trans
Time Budget
1h
3h
6h
1h
3h
6h
1h
3h
6h
Random
63
186
367
62
180
356
61
163
307
Random-ETR ( increase ratio )
105 ( 66.7% ↑ )
323 ( 73.7% ↑ )
662 ( 80.4% ↑ )
100 ( 61.3% ↑ )
321 ( 78.3% ↑ )
653 ( 83.4% ↑ )
62 ( 1.6% ↑ )
177 ( 8.6% ↑ )
343 ( 11.7% ↑ )
Random-BTTackler ( increase ratio )
110 ( 74.6% ↑ )
322 ( 73.1% ↑ )
634 ( 72.8% ↑ )
101 ( 62.9% ↑ )
305 ( 69.4% ↑ )
621 ( 74.4% ↑ )
97 ( 59.0% ↑ )
259 ( 58.9% ↑ )
486 ( 58.3% ↑ )
Table 3. The efficiency of BTTackler-enhanced HPO methods on three tasks. The numbers in the table represent the total number of hyperparameter configurations tried in each experiment.
Task
Cifar10CNN
Cifar10LSTM
Ex96Trans
SMAC
4.64h
4.41h
4.51h
SMAC-BTTackler
3.89h
2.31h
1.88h
TSBA
16%
47%
58%
Table 4. The time-saving of BTTackler in achieving best baseline accuracy on three tasks.
Figure 5. The performance distribution for both bad trials and good trials in the HPO task of Cifar10CNN.
Task
AGV
EAG
PLC
ERG
LAR
ULC
NMG
Cifar10CNN
-
-
24
317
62
-
93
Cifar10LSTM
8
20
1
296
105
19
134
Ex96Trans
1
3
-
11
3
6
35
Table 5. Number of bad trials claimed by the quality indicators of BTTackler.
Hyperparameter optimization selects model and training settings to improve predictive performance, efficiency, or reliability under limited evaluation budgets.
Hyperparameter Optimization (HPO) is essential for building high-performing ML/DL models, yet conventional optimizers often struggle in high-dimensional spaces where evaluations are costly and progress is diluted across many low-impact variables. We propose Greedy Importance First (GIF), an importance-aware scheduling strategy that uses a small-sample warm start to estimate hyperparameter importance, forms importance-based groups, allocates trials proportionally, and retains a full-space fallback. We evaluate GIF under fixed evaluation budgets on five anisotropic analytic functions, Bayesmark, and NAS-Bench-301. On the higher-dimensional benchmarks, GIF reaches better incumbents with faster convergence than TPE, BOHB, Random Search, and Sequential Grouping. On Bayesmark, where the effective dimensionality is smaller, GIF remains competitive but the margins are smaller. Ablation studies show that importance estimation, proportional allocation, and the fallback step all contribute to the gains. We also verify that the HIA component recovers the intended anisotropy on the analytic benchmarks. These results suggest that GIF is a simple and plug-compatible way to improve sample efficiency in high-dimensional HPO.
Large hyperparameter sweeps for deep neural networks spend substantial compute on configurations that are effectively doomed from the first few epochs. We study whether a single training run's own early telemetry - per-epoch loss, training accuracy, gradient signal-to-noise ratio, weight-norm growth, and an activation-saturation snapshot - together with its sampled hyperparameters, can predict that run's eventual outcome without reference to other runs. We evaluate three prediction tasks: final test accuracy, relative performance within a domain, and training-dynamics failure, including numerical divergence. Across 23,788 training runs spanning six architecture/dataset combinations, gradient-boosted trees using only the first five epochs of telemetry achieve R^2 = 0.92-0.99 for final-accuracy regression and ROC-AUC = 0.983-0.998 for relative classification on a permanently held-out set of hyperparameter configurations. Useful prediction is already available after a single epoch. A paired ablation shows that gradient- and weight-level telemetry provides a statistically consistent improvement over loss and accuracy curves alone, although the practical gain varies by domain. Transfer is strong between similar architectures, while cross-dataset transfer is limited mainly by differences in accuracy scale rather than loss of the underlying relationship. These results suggest that early-training telemetry can provide a practical decision-support signal for compute allocation while motivating human oversight for any automated intervention.
Many production systems can assess a configuration only by using it on live requests and observing noisy feedback. Modern agentic systems are a prominent example, with inference-time choices such as model selection, retrieval depth, prompting strategy, and decoding temperature, yet often with no representative validation data. We formalize this setting as Online Hyperparameter Optimization (OHPO) and cast it as an infinitely many-armed bandit over mixed and conditional search spaces. We introduce IMABO, a general framework that combines any bandit policy for choosing among already sampled configurations with any oracle for proposing new ones. We instantiate it with IMOSS, a restart-free anytime policy whose active set grows as tβ, and prove an expected cumulative quantile-regret bound of O(pρ−1/β+T(1+β)/2), where β∈(0,1) controls active-set growth and pρ lower-bounds the probability that a proposed configuration falls in the top-ρ fraction of the search space. We combine IMOSS with three practical oracles: a Tree-structured Parzen Estimator, an incumbent-mutation oracle driven by a per-coordinate bandit, and a pretrained tabular foundation model, all three improving over the uniform random oracle baseline. IMABO outperforms all baselines in terms of regret across diverse OHPO settings, from tuning classical machine-learning models to configuring LLM-based agents. Our implementation is available at https://github.com/Tiime-Software/IMABO.