Sparse Bayesian Learning Algorithms Revisited: From Learning Majorizers to Structured Algorithmic Learning using Neural Networks
Authors: Rushabha Balaji, Kuan-Lin Chen, Danijela Cabric, Bhaskar D. Rao
Organizations: Department of Electrical and Computer Engineering, University of California, Los Angeles, CA 90095, USA · Department of Electrical and Computer Engineering, University of California, San Diego, La Jolla, CA 92093, USA
Sparse Bayesian Learning is one of the most popular sparse signal recovery methods, and various algorithms exist under the SBL paradigm. However, given a performance metric and a sparse recovery problem, it is difficult to know a-priori the best algorithm to choose. This difficulty is in part due to a lack of a unified framework to derive SBL algorithms. We address this issue by first showing that the most popular SBL algorithms can be derived using the majorization-minimization (MM) principle, providing hitherto unknown convergence guarantees to this class of SBL methods. Moreover, we show that the two most popular SBL update rules not only fall under the MM framework but are both valid descent steps for a common majorizer, revealing a deeper analytical compatibility between these algorithms. Using this insight and properties from MM theory we expand the class of SBL algorithms, and address finding the best SBL algorithm via data within the MM framework. Second, we go beyond the MM framework by introducing the powerful modeling capabilities of deep learning to further expand the class of SBL algorithms, aiming to learn a superior SBL update rule from data. We propose a novel deep learning architecture that can outperform the classical MM based ones across different sparse recovery problems. Our architecture's complexity does not scale with the measurement matrix dimension, hence providing a unique opportunity to test generalization capability across different matrices. For parameterized dictionaries, this invariance allows us to train and test the model across different parameter ranges. We also showcase our model's ability to learn a functional mapping by its zero-shot performance on unseen measurement matrices. Finally, we test our model's performance across different numbers of snapshots, signal-to-noise ratios, and sparsity levels.
Figures & tables
Fig. 1 : Neural network architecture for the learning new SBL algorithms.
Fig. 2 : The architecture of the jth iteration. It consists of 6 layers with skip connections. The red dots represent the summation operation.
Fig. 3 : Performance curves of EM, MU-SBL ( p=1 ) and p -SBL ( p={0.25,0.5,0.75} ) update rules as a function of the number of non-zero elements for the array matrix ( N =30, M =120, SNR=30 dB).
Fig. 4 : Convergence rates of EM, MU-SBL ( p=1 ) and p -SBL ( p={0.25,0.5,0.75} ) update rules as a function of the number of non-zero elements ( N =30, M =120, SNR=40 dB).
Fig. 5 : Value of the SBL objective function as a function of the number of iterations for different SBL algorithms for the ULA matrix.
Fig. 6 : Performance curves of EM, MU-SBL ( p=1 ) and p -SBL ( p={0.25,0.5,0.75} ) update rules as a function of the number of non-zero elements for the random matrix( N =30, M =120, SNR=40 dB).
Fig. 7 : Evolution of the weights as a function of iterations. We showcase the results for different number of iterations as well as the different types of update rule generation for the array matrix ( N=30,M=120 , SNR=40 dB).
Fig. 8 : Performance of our proposed neural network architecture on different types of measurement matrices.
Objective: Sparse Bayesian learning provides an effective framework to solve high-dimensional problems in brain signal decoding. However, conventional likelihoods regarding data distributions, such as Gaussian or Bernoulli, are potentially inadequate for handling the noisy recordings of brain activity. Hence, this work aims to formulate a robust sparse Bayesian learning framework to address noisy high-dimensional brain activity decoding. Methods: Motivated by the commendable robustness of the minimum error entropy learning criterion for addressing non-Gaussian signals, this study reformulated the sparse Bayesian learning framework under a generalized Bayesian paradigm, in which the model parameter is regulated with the minimum error entropy loss rather than a conventional likelihood function. Results: Our developed SBL-MEE algorithm was evaluated with two real-world brain decoding tasks of regression and classification scenarios, respectively. Experimental results demonstrated that our approach not only realizes superior brain decoding performance than existing methods, but also presents more physiologically interpretable decoder patterns. Conclusion: Although minimum error entropy is not constructed from an arbitrary probabilistic distribution, it is effective to establish noise-robust inference in sparse Bayesian learning method. Significance: This work provides a powerful tool to improve brain activity decoding capability, particularly regarding the noisy high-dimensional setting, thus promoting biomedical engineering applications such as brain-computer interface.
Yuanhao Li, Badong Chen, Wenjun Bai +2
Center for Advanced Intelligence Project, RIKEN, Tokyo 103-0027, Japan · Department of Computational Brain Imaging, ATR Neural Information Analysis Laboratories, Kyoto 619-0237, Japan · Institute of Artificial Intelligence and Robotics (IAIR), Xi’an Jiaotong University, Xi’an 710049, China +1
Sparse recovery in linear systems underpins applications from signal processing to high-dimensional regression. Sparse Bayesian Learning, grounded in the principle of automatic relevance determination (ARD), offers a practical Bayesian mechanism for feature sparsity via marginal likelihood optimization. Yet, its reliance on a homoscedastic noise model renders it sensitive to data contaminations such as outliers or misspecified noise, harming model fit and predictions. Instead, we propose jointly learning individual feature and sample relevancies, enabling simultaneous model and data sparsification via a single Bayesian objective. This symmetric pruning of model and data offers a natural extension that preserves conjugacy, admits closed-form updates for standard optimization procedures, and aligns with perspectives from robust regression and influence functions. Empirical results across diverse regression tasks affirm that a joint ARD approach consistently yields both sparse and robust prediction models.
Alexander Timans, Thomas Möllenhoff, Christian A. Naesseth +2
1UvA-Bosch Delta Lab, University of Amsterdam · 2RIKEN Center for AI Project, Tokyo, Japan · Department of Computer Science, Johns Hopkins University.
We propose a unified fractional regularization framework for sparse signal recovery based on the ℓ1/ℓpq model. This model generalizes several widely used sparsity-promoting regularizers and provides additional flexibility through the parameters p and q. Our main theoretical contribution is the characterization of the equivalence between the first-order stationary points of the ℓ1/ℓpq formulation and the subtractive ℓ1−αℓp model, thereby offering a unified perspective on these nonconvex regularizers. In addition, we establish a new sufficient recovery condition under the Restricted Isometry Property (RIP), which shows that the proposed framework can provide relaxed recovery guarantees and improved robustness. To solve the resulting nonconvex problem, we develop a majorization--minimization (MM) algorithm and prove its convergence by using the Kurdyka--Łojasiewicz (KL) property. Numerical experiments on sparse recovery problems with different sensing matrices and MRI reconstruction demonstrate that the proposed approach outperforms existing methods in recovery accuracy.
Yinhao Zhao, Yaohua Hu, Hao Wang
School of Information Science and Technology, ShanghaiTech University, 201210