Randomized neural networks enable fast and analytically tractable training by fixing the input to hidden layer parameters at random and learning the output weights in closed form; however, their performance critically depends on a single uninformed draw of hidden units. This one shot and task uninformed feature construction often leads to redundant representations and suboptimal utilization of model capacity. To address this limitation, we propose a simple and broadly applicable residual guided procedure that greedily constructs the hidden layer using a closed form residual decrease criterion. At each stage, we (i) generate a pool of random candidate units, (ii) score each candidate by the exact reduction it induces in the ridge regularized objective, (iii) select the top k units, and (iv) refit the readout in closed form using the standard design with direct input links. This procedure yields a progressive training process with a guaranteed monotonic decrease of the training objective. The method is model agnostic: only the candidate generation is architecture specific, while the scoring selection refitting loop is shared across models. Extensive experiments on 71 benchmark datasets from the UCI repository, covering both binary and multiclass classification tasks, demonstrate that the proposed residual-guided models consistently outperform their baseline counterparts in terms of accuracy, stability, and overall ranking performance.
Figures & tables
Figure 1: Architecture of a standard RVFL network.
Task
Metric
Method Comparison
RVFL
RG-RVFL †
ELM
RG-ELM †
BLS
RG-BLS †
Binary (32)
Accuracy ↑
80.92
82.92
80.82
82.76
81.65
82.78
Std. Dev. ↓
8.91
7.88
8.36
7.76
6.24
5.75
Rank ↓
1.97
1.03
1.88
1.13
1.91
1.09
Multiclass (39)
Accuracy ↑
73.33
75.94
73.34
75.35
74.96
76.20
Std. Dev. ↓
10.43
9.11
10.12
8.99
7.98
7.17
Table I : Average accuracy (%), average standard deviation, and average rank of baseline and residual-guided models on 32 binary and 39 multiclass datasets.
Dataset
RVFL
RG-RVFL †
ELM
RG-ELM †
BLS
RG-BLS †
acute_inflammation
100
100
100
100
100
100
acute_nephritis
100
100
100
100
100
100
molec_biol_promoter
71.8182
75.5411
68.9610
77.3593
83.9827
85.8874
parkinsons
81.0256
82.0513
80.5128
81.0256
81.0256
82.0256
pittsburg_bridges_T_OR_D
87.2381
89.2381
87.1429
88.1429
89.2381
90.1905
bank
89.6925
90.0022
89.5378
89.8032
89.7368
89.7589
Table S.I : Dataset-wise results on 32 binary datasets.
Dataset
RVFL
RG-RVFL †
ELM
RG-ELM †
BLS
RG-BLS †
abalone
63.4179
63.8732
63.3465
63.8729
63.4657
63.9856
annealing
89.1924
91.1962
89.6356
91.0838
89.6344
90.6400
arrhythmia
65.4823
71.0208
65.7192
68.3663
64.1587
65.2576
audiology_std
69.2949
69.8718
67.7821
69.8333
69.8333
70.8718
cardiotocography_10clases
69.6156
71.3099
69.2395
70.7451
65.5708
66.4182
cardiotocography_3clases
85.2331
86.2683
85.3743
86.2683
85.9390
86.8921
Table S.II : Dataset-wise result on 39 multi-class datasets.
Figure S.1 : Stagewise behavior of the proposed residual-guided framework on the breast_cancer dataset, illustrating the monotonic decrease of the ridge-regularized training objective and residual energy as hidden features are incrementally added.
Deep randomized models fix hidden-layer parameters through random initialization and learn only closed-form readouts, typically adding depth by stacking random trans formations without target-aware control of hidden-state evolution. We propose LAIR Net, the Leaky Alignment-Impulse Residual Network, which mixes a shallow learned anchor into each hidden state through a leaky residual transition. We derive a depth uniform bound on input-perturbation sensitivity and use controlled simulations to attribute gains over a randomized baseline to the anchor rather than recursion or added capacity. Benefits emerge when a nonlinear target structure is learnable at the available noise level and diminish for nearly linear targets or dominant noise. Across 23 benchmark datasets, LAIR-Net achieves the best average rank among eight randomized networks and twelve conventional models, with relative performance associated with the same nonlinear-structure and noise quantities identified in simulation.
Rahul Goswami, Aryan Bhambu, Bittu Karmakar
Indian Institute of Technology Guwahati · SAFIR, Sorbonne University Abu Dhabi · Indian Institute of Technology Bombay
I propose the Random Cloud method, a training-free approach to neural architecture search that discovers minimal feedforward network topologies through stochastic exploration and progressive structural reduction. Unlike post-training pruning methods that require a full train-prune-retrain cycle, this method evaluates randomly initialized networks without backpropagation, progressively reduces their topology, and only trains the best minimal candidate at the end. I evaluate on 7 classification benchmarks against magnitude pruning and random pruning baselines. The Random Cloud matches or outperforms both baselines in 6 of 7 datasets, achieving statistically significant improvements on Sonar (+4.9pp accuracy, p=0.017 vs magnitude pruning) with 87% parameter reduction. Crucially, the method is faster than both pruning baselines in 4 of 5 datasets (0.67--0.94× the cost of full training), since it avoids training the full-size network entirely.
Dropout and Random Gradient Masking (RaM) are two training techniques used to improve performance in deep learning. Both techniques inject randomness into the training dynamics, but in significantly different ways: dropout applies random masks to the activations in the forward pass, whereas RaM leaves the forward pass unchanged and instead masks the gradients. In particular, the noise induced by RaM in the parameter updates is unbiased, so standard explanations for the effectiveness of dropout, such as the penalization effect or the prevention of co-adaptation between neurons, do not apply to RaM. In this work, we show that the difference between the two methods disappears for ResNets in the large depth and width asymptotics: in the complete feature learning regime, they both converge to the same large-scale limiting dynamics. This asymptotic equivalence holds for several variants of dropout and RaM, including layerwise dropout as used in stochastic-depth ResNets, albeit at slower quantitative rates. In fact, we also show that several of these variants collapse to the same limit asymptotically.
Javier Maass, Lénaïc Chizat
École Polytechnique Fédérale de Lausanne (EPFL), Institute of Mathematics, 1015 Lausanne, Switzerland. · Department of Mathematics, National University of Singapore, Singapore.