eess.SPDec 19, 2025

Alternating Direction Method of Multipliers for Nonlinear Matrix Decompositions

Authors: Atharva Awari, Nicolas Gillis, Arnaud Vandaele

Organizations: University of Mons

Abstract

We present an algorithm based on the alternating direction method of multipliers (ADMM) for solving nonlinear matrix decompositions (NMD). Given an input matrix X∈Rm×nX \in \mathbb{R}^{m \times n} and a factorization rank r≪min⁡(m,n)r \ll \min(m, n), NMD seeks matrices W∈Rm×rW \in \mathbb{R}^{m \times r} and H∈Rr×nH \in \mathbb{R}^{r \times n} such that X≈f(WH)X \approx f(WH), where ff is an element-wise nonlinear function. We evaluate our method on several representative nonlinear models: the rectified linear unit activation f(x)=max⁡(0,x)f(x) = \max(0, x), suitable for nonnegative sparse data approximation, the component-wise square f(x)=x2f(x) = x^2, applicable to probabilistic circuit representation, and the MinMax transform f(x)=min⁡(b,max⁡(a,x))f(x) = \min(b, \max(a, x)), relevant for recommender systems. The proposed framework flexibly supports diverse loss functions, including least squares, ℓ1\ell_1 norm, and the Kullback-Leibler divergence, and can be readily extended to other nonlinearities and metrics. We illustrate the applicability, efficiency, and adaptability of the approach on real-world datasets, highlighting its potential for a broad range of applications.

Explore similar work

Mar 31, 2026

Nonnegative Matrix Factorization in the Component-Wise L1 Norm for Sparse Data

Nonnegative matrix factorization (NMF) approximates a nonnegative matrix, X, by the product of two nonnegative factors, WH, where W has r columns and H has r rows. In this paper, we consider NMF using the component-wise L1 norm as the error measure (L1-NMF), which is suited for data corrupted by heavy-tailed noise, such as Laplace noise or salt and pepper noise, or in the presence of outliers. Our first contribution is an NP-hardness proof for L1-NMF, even when r=1, in contrast to the standard NMF that uses least squares. Our second contribution is to analyze, under simplified probabilistic assumptions, how the sparsity in the data enforces zero solution in the optimal scalar update in the factors of L1-NMF when all the other entries are kept fixed. This provides an intuition of the connection between the sparsity of the L1-NMF factors with the sparsity of the input. Even though sparsity favors interpretability, if the data is affected by false zeros, too sparse solutions might degrade the model. Our third contribution is a new, more general, L1-NMF model for sparse data, dubbed weighted L1-NMF (wL1-NMF), where the sparsity of the factorization is controlled by adding a penalization parameter to the entries of WH associated with zeros in the data. The fourth contribution is a new coordinate descent (CD) approach for wL1-NMF, denoted as sparse CD (sCD), where each subproblem is solved by a weighted median algorithm. Although it lacks convergence guarantees to a stationary point, sCD is, to the best of our knowledge, the first algorithm for L1-NMF whose complexity scales with the number of nonzero entries in the data, making it efficient in handling large-scale, sparse data. We perform extensive numerical experiments on synthetic and real-world data, including imaging mass spectrometry and topic modeling, to show the effectiveness of our new proposed model (wL1-NMF) and algorithm (sCD).
Aug 28, 2026

Separable Nonnegative Matrix Factorization Using Powered Ratio-of-Norms Regularization

Separable nonnegative matrix factorization (SNMF) has been widely used for low-rank representation and clustering of nonnegative data, owing to its ability to produce part-based and interpretable decompositions. In particular, SNMF is closely related to graph clustering and community detection. To enhance sparsity and identifiability of the learned factors, we propose an ℓ1p/ℓ2\ell_1^p/\ell_2-regularized SNMF model based on a powered ratio-of-norms regularizer. The resulting formulation is nonconvex and nonsmooth, which poses significant challenges for optimization. To address this, we develop efficient algorithms based on the difference-of-convex function algorithm (DCA) and the alternating direction method of multipliers (ADMM). The proposed methods decompose the original problem into tractable subproblems, leveraging closed-form proximal operators associated with the powered norm terms. We establish descent and limiting criticality properties for the DCA scheme and convergence under standard assumptions for the ADMM scheme. Extensive numerical experiments on synthetic datasets and hand gesture classification tasks demonstrate that the proposed approach achieves competitive or improved performance in anchor identification and classification accuracy compared with existing SNMF methods, while maintaining competitive computational efficiency.
Mar 15, 2024

Over-fitting Control in Deep Nonlinear Matrix Completion via Gradual Nonsmooth Regularization

Conventional matrix completion methods exploit the underlying low-rank structure of a target matrix to recover its missing entries, but may be inadequate for capturing complex nonlinear dependencies. Deep neural networks have therefore been increasingly adopted for nonlinear matrix completion (NLMC). However, their high representational capacity, particularly in deep fully connected neural networks (FCNNs), can lead to over-fitting when training is restricted to incomplete observations. In this paper, we propose a Deep Neural Network with Nonsmooth Regularization (DNN-NSR) framework to address this issue by incorporating two complementary structural priors: sparsity of intermediate representations and low-rankness of layer-wise weight matrices. Specifically, the ℓ1\ell_1-norm of hidden representations promotes sparse latent features, while the nuclear norm of weight matrices encourages low-rank layer-wise transformations. To optimize the resulting nonsmooth and nonconvex objective, we develop an extrapolated proximal gradient algorithm that employs soft-thresholding and singular-value thresholding to handle the two regularization terms, respectively. We further introduce a gradual regularization strategy that progressively increases the influence of these structural penalties during training. Under suitable assumptions, we establish convergence properties of the proposed algorithm toward a critical point of the training objective. The effectiveness of DNN-NSR is evaluated through extensive experiments on synthetic matrix recovery, single-image inpainting, and recommender systems using the MovieLens 100K and MovieLens 1M datasets. Comparisons with representative linear and nonlinear matrix completion methods demonstrate the potential of the proposed framework to improve recovery performance by jointly exploiting sparse latent representations and low-rank network transformations.