Rare-event forecasting often depends on interpretable conjunctions: a single keyword, location, or temporal cue may be weak, while their joint occurrence can identify a meaningful precursor. Hierarchical-Task Numerical conjunctive feature Learning (HTNL) selects Numerical Conjunctive Features (NCFs) through a structured sparsity objective. However, the resulting optimization problem is challenging, making optimization design critical to its practical performance. We give a unified diagnosis of four HTNL optimization families: original majorization-minimization (MM), variational quadratic form (VQF), alternating direction method of multipliers (ADMM), and weighted group lasso (WGL). Our analysis distinguishes three key factors that jointly determine HTNL optimization behavior. First, VQF and MM are algebraically equivalent under matched variational updates and optimize a squared-penalty formulation. Second, ADMM optimizes the unsquared objective through consensus splitting; we replace its truncated FISTA proximal solver with a scalar bisection update (ADMM-Bisect) for more accurate treatment of near-threshold blocks. Third, WGL targets the same unsquared objective, while all methods face rapidly decaying active-set admission scores. Across two synthetic datasets and 11 civil-unrest and flu-surveillance tasks, VQF achieves the highest mean AUC with the most compact non-collapsing supports. ADMM-Bisect reduces measured runtime, while calibrated WGL avoids the empty-support failures of its original scale. Synthetic results further show how proximal accuracy, pruning, and search budget affect candidate recovery, especially for longer conjunctions. Code will be made publicly available upon publication.
Figures & tables
Figure 1: Main pipeline of HTNL optimization and NCF selection. HTNL constructs conjunction features from X and optimizes the restricted two-level group-lasso problem using MM–VQF, ADMM, or WGL. The resulting coefficients score unseen NCFs by the KKT criterion, admitted candidates expand the working set U , and the process repeats until convergence, producing event predictions and selected NCFs.
Dataset 1
Dataset 2
Method
τ=0.20
τ=0.40
τ=0.20
τ=0.40
HTNL-MM
0.580±0.201
0.592±0.253
0.581±0.188
0.590±0.246
HTNL-VQF
0.580±0.201
0.592±0.253
0.581±0.188
0.590±0.246
HTNL-ADMM (FISTA, ρ=10 )
0.468±0.133
0.674±0.187
0.457±0.147
0.674±0.211
HTNL-WGL ( λ=100 )
0.463±0.336
0.463±0.336
0.360±0.306
0.360±0.306
Best-of-class variants (post-hoc sweeps)
Table 1: NCF support-selection F1 on two controlled synthetic datasets. Results are mean ± standard deviation over 20 paired random seeds at active-set thresholds τ=0.20 and 0.40 .
Dataset
n
pos%
LogReg
MM/VQF
ADMM-Bisect
WGL( λ=1 )
Δ AUC
Flu ⋆
5,096
7.40%
0.649
0.695
0.600
0.606
+0.046
Ecuador
5,916
0.74%
0.529
0.604
0.563
0.581
+0.075
El Salvador
8,700
1.26%
0.477
0.572
0.559
0.555
+0.095
Uruguay
10,788
0.82%
0.481
0.601
0.565
0.582
+0.120
Venezuela
11,832
0.70%
0.463
0.524
0.514
0.523
+0.061
Chile
12,876
0.49%
0.459
0.527
0.501
0.503
+0.068
Table 2: Predictive AUC on 11 real-world datasets using identical train/test splits. n and pos% denote the number of training instances and positive-class fraction, respectively. Δ AUC denotes the improvement of VQF over Logistic Regression. The final row reports mean ± standard deviation across datasets.
Figure 2: Controlled optimization diagnostics. Left: Bisection improves ADMM F1, while MM and VQF are identical. Right: KKT pruning preserves F1 while reducing runtime.
Figure 3: Synthetic F1 across active-set admission thresholds τKKT . Shaded bands show uncertainty across paired random seeds.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
cu,gt=λ∑v∈AU(u)dvαv,ut,u∈U,g∈G.
(B.31)
Appendix
Algorithm 1 Two-Level Group Lasso via Weighted One-Level Group Lasso
Table 3: Sensitivity of WGL to the regularization scale on Venezuela. “Direction” indicates either high-to-low continuation from λstart to λend or a single fixed- λ run. “[email protected]” denotes the number of NCFs selected at the active-set threshold τ=0.40 .
Figure 4: Paired-seed F1 comparison between ADMM-Bisect and ADMM-FISTA at ρ=1 . Each point represents the same random seed evaluated with the two proximal solvers; points above the identity diagonal favor ADMM-Bisect. Under this diagnostic condition, ADMM-Bisect matches or improves ADMM-FISTA on every evaluated seed, showing that the aggregate improvement is consistent across paired runs rather than driven by a small number of seeds.
Method
Jac. vs Orig./VQF
∣S^∣
Δ AUC
MM/VQF
1.000
≈4 – 5
+0.074∗∗∗
ADMM-Bisect
0.277
≈10
+0.034∗
WGL( λ=1 )
0.351
≈10
+0.044∗∗
WGL-Cont (default)
0.000
0
—
Appendix
Table 4: Selected-support behavior on the LatAm civil-unrest datasets. ∣S^∣ denotes the selected set size at τ=0.40 , Jaccard similarity is measured against MM/VQF, and Δ AUC is measured against class-weighted Logistic Regression.
Method
Default ( m0=2 , add=5)
m0=3
m0=4
m0=5
MM/VQF
0.590
0.686
0.686
0.686
ADMM-Bisect
0.674
0.684
0.685
0.685
Appendix
Table 5: Active-set sensitivity on the TKDE-faithful synthetic generator (Dataset 2). m0 denotes the initial active-set size; the default configuration uses m0=2 and adds five candidates per iteration. Entries report mean F1.
Orig
VQF
ADMM
WGL-def.
WGL( λ=1 )
MM
1.000
1.000
0.277
0.045
0.351
VQF
1.000
1.000
0.277
0.045
0.351
ADMM-Bisect
0.277
0.277
1.000
0.000
0.780
WGL-Cont (def.)
0.045
0.045
0.000
1.000
0.000
WGL( λ=1 )
0.351
0.351
0.780
0.000
1.000
Appendix
Table 6: Mean Jaccard similarity between selected NCF sets across the 11 real-data benchmarks at τ=0.40 . MM and VQF have Jaccard similarity 1.000 , consistent with Proposition 3.2 .
Figure 5: Selection size and outer active-set iterations across optimization methods. Each point represents one dataset–seed pair. MM/VQF produces more compact selected sets, whereas ADMM-Bisect and WGL( λ=1 ) retain broader supports. ADMM-Bisect requires more outer iterations but remains computationally efficient in the tested implementation.
Country
ntrain
pos%
MM/VQF
ADMM-Bisect
Uruguay
10,788
0.8%
13
76
Venezuela
11,832
0.7%
15
62
Paraguay
14,964
1.5%
22
86
Colombia
19,140
0.5%
25
97
Argentina
19,836
0.5%
18
83
Mean
—
—
18.6
80.8
Appendix
Table 7: Selected-support sizes on five civil-unrest datasets. ntrain and pos% denote the number of training instances and the positive-class fraction, respectively. Support size is measured at τ=0.40 . MM and VQF are identical under matched updates and are therefore reported jointly. ADMM-Bisect uses ρ=1 .
Figure 6: Predictive AUC across the 11 real-data benchmarks. MM/VQF provides the largest mean improvement over class-weighted Logistic Regression. ADMM-Bisect and WGL( λ=1 ) also improve mean AUC but retain broader selected supports, while the default WGL continuation produces empty models on the severely imbalanced civil-unrest datasets.
Figure 7: Runtime and predictive performance across the 10 LatAm civil-unrest datasets and the flu dataset. The left panel compares wall-clock time, and the right panel compares per-dataset AUC with class-weighted Logistic Regression.
Figure 8: AUC as a function of the positive-class fraction. The 10 LatAm datasets have positive rates between 0.48% and 1.52% , whereas the flu dataset has a positive rate of 7.4% . Default WGL produces empty supports with AUC =0.500 on all 10 LatAm datasets, while this failure is not observed on the flu dataset.
Dimension
MM
VQF
ADMM-Bisect
WGL-Cont
Formulation basis
Variational quadratic formulation
Same variational quadratic formulation as MM
ADMM decomposition of the original nonsmooth objective
Reformulation of the outer ℓp hierarchy as weighted group lasso
Inner solver
Weighted ridge subproblem
Same weighted ridge subproblem as MM
L-BFGS for the smooth step plus exact bisection for the nonsmooth proximal step
FISTA for the weighted group-lasso subproblem
Convergence behavior
Linear convergence for the strongly convex ridge subproblem
Same as MM
O(1/K) ADMM residual convergence
O(1/K2) convergence within each fixed- λ FISTA stage; no recovery guarantee after support collapse
Per-iteration cost
O(∣U∣⋅ridge)
O(∣U∣⋅ridge)
O(∣U∣⋅∣D(v)∣⋅log(1/ε))
O(∣U∣⋅G⋅FISTA)
Selection regime
Sparse: ≈4 – 5 NCFs
Identical to MM
Broad: ≈10 NCFs
Broad when λ=1 ; collapse under the default continuation schedule
Class-imbalance robustness
Robust in the tested rare-event datasets
Robust in the tested rare-event datasets
Robust in the tested rare-event datasets
Scale-sensitive; the default schedule collapses on all 10 severely imbalanced civil-unrest datasets
Appendix
Table 8: Multi-dimensional comparison of the four HTNL optimization families. “Sparse” denotes the sparse-selection regime ( ≈4 – 5 selected NCFs), “broad” denotes the broader-selection regime ( ≈10 selected NCFs), and “collapse” denotes an empty or nearly empty selected set. “Ridge” refers to the weighted ridge subproblem obtained after fixing the variational auxiliary variables, and “FISTA” refers to the accelerated proximal-gradient solver used for the weighted group-lasso subproblem. The parameter ρ∈R+ is the ADMM augmented-Lagrangian penalty, while the λ schedule denotes the continuation path used by WGL.
Weighted sums of random kitchen sinks (W-RKS) is a commonly used feature extraction technique for supervised learning tasks. A major drawback of this method is the redundancy and ineffectiveness of many extracted random features. This paper investigates efficient methods for reducing such redundancy in multi-class classification problems. We specifically formulate feature selection as a variant of the sequentially nested model selection problem and introduce the Successive Empirical Excess Risk Difference Maximization (SEED-MAX) algorithm. It recursively computes empirical risks to identify the most parsimonious yet informative set of features. We derive an analytical lower bound for the probability of correct feature selection by the SEED-MAX algorithm. This lower bound on the probability of correct feature selection can be made arbitrarily close to one by increasing the training dataset size. To further enhance the rate of pruning, we evaluate the features retained by SEED-MAX using the Fisher test (F-Test). Empirical evaluations on UCR time-series classification datasets show that the sequential application of SEED-MAX with the Fisher test achieves approximately 92 percent rate of pruning with negligible degradation in classification accuracy. Implementations of the SEED-MAX and SEED-MAX-F-Test methods are available at https://github.com/OmidiAmin/seed-max-ftest.
Mohammad Amin Omidi, Babak Seyfe, Shahrokh Valaee
Information Theoretic Learning Systems Laboratory (ITLSL), Department of Electrical Engineering, Shahed University, Tehran, Iran · Department of Electrical and Computer Engineering, University of Toronto, Toronto, Canada
Learning high-quality oblique decision trees remains a significant challenge due to the discrete and non-convex nature of split optimization. We present the Hinge Regression Tree (HRT) framework, which reframes each oblique split as a nonlinear least-squares problem over two linear predictors whose max/min envelope induces ReLU-like representation capacity. We show that the resulting node-level optimization can be interpreted as a damped Newton method, and we establish the monotonic decrease of the node objective for its backtracking line-search variant. We establish, theoretically, that HRT is a universal approximator with an explicit O(δ2) approximation rate. Building upon this base learner, we propose HRT-Boost, a mathematically synergistic ensemble extension that couples node-level Newton updates with stage-wise functional gradient descent. We show that this ensemble construction admits a stage-wise empirical risk reduction guarantee under the squared loss. Empirical evaluations on synthetic and real-world benchmarks show that HRT is highly competitive with established single-tree baselines, and HRT-Boost compares favorably with strong ensemble baselines and often yields substantially more compact models. The code is publicly available at https://github.com/Hongyi-Li-sz/HRT-Boost.
Hongyi Li, Jun Xu, Hong Yan
School of Intelligence Science and Engineering, Harbin Institute of Technology, Shenzhen, 518055, China and with the Shenzhen Key Lab for Advanced Motion Control and Modern Automation Equipments, Shenzhen, 518055, China · Department of Electrical Engineering, City University of Hong Kong, Kowloon, Hong Kong
Choosing which features to collect is a deployment decision: the same limited questionnaire, test panel, or sensor set may need to serve several heterogeneous populations. Standard feature-selection methods typically optimize for one large population, while existing robust approaches tend to learn one shared model for every population. We introduce PopFS, a method for learning one shared, deployable feature set that is robust to population differences while letting each pop- ulation train its own model. PopFS uses a tunable welfare objective that lets practitioners balance overall predictive ben- efit against stronger protection of the populations that benefit least. To make this objective practical at scale, PopFS first uses multitask sparse learning to reduce the candidate pool, then searches directly over hard feature sets by ranking promising additions and swaps and fully refitting only a shortlist. Across eight population splits from six prediction tasks drawn from five tabular and public-health datasets, PopFS consistently achieves strong average and worst-population performance while scaling to thousands of candidate features. A 43-state COVID-19 nowcasting study further shows that changing the welfare objective can improve the least-served states with lit- tle change in average performance and yields an interpretable change in the selected symptom signals. Our code is available at https://github.com/Rachel-Lyu/PopFS.