A Nonconvex Framework for Joint Sparse Sample Weighting and Parameter Estimation

Hassan Nojavan; Chul Moon; Miju Ahn

math.OCOct 25, 2026

A Nonconvex Framework for Joint Sparse Sample Weighting and Parameter Estimation

Authors: Hassan Nojavan, Chul Moon, Miju Ahn

Organizations: Department of Operations Research and Engineering Management, Southern Methodist University · Department of Statistics and Data Science, Southern Methodist University

Abstract

We propose a joint sparse sample weighting and parameter estimation (SWAPE) framework that jointly optimizes model parameters and sample weights to train a model while simultaneously assigning an individual weight to each training data point. Our framework accommodates several widely used loss functions for the model parameters, including convex, weakly convex, and difference-of-convex loss, and weakly convex sparsity functions to control the complexity of the sample weights. To solve the resulting bivariate problem, we propose an alternating minimization method in which the subproblems for updating the model parameters and weights are each approximated using majorize-minimize techniques. We provide several approximation methods for the loss functions considered in this paper, and show the overall algorithm's convergence. Numerical experiments on various learning tasks and data anomaly settings demonstrate the effectiveness of the proposed framework in minimizing the impact of undesirable data points compared to benchmark methods.

Figures & tables

Figure 1: Penalty functions: LASSO ( $\lambda=1$ ) on the left, and MCP ( $\eta=1$ , $\theta=1.5$ ) on the right.

Figure 2: Residuals versus true fitted values from the first replication. From left to right: no contamination; interior-clustered contamination with two clouds for $\alpha=0.4$ ; and vertical contamination with $\alpha=0.3$ .

Method	MSE	Signal MSE	Estimation Error	Retained
SWAPE	1.020 (0.026)	0.020 (0.014)	0.030 (0.011)	50.29
$\Theta$ -IPOD	1.014 (0.024)	0.014 (0.011)	0.025 (0.009)	94.98
OLS	1.010 (0.022)	0.010 (0.005)	0.022 (0.006)	–
LAD	1.016 (0.024)	0.016 (0.009)	0.028 (0.008)	–
Huber	1.011 (0.023)	0.011 (0.006)	0.023 (0.007)	–
LTS	1.030 (0.029)	0.030 (0.018)	0.037 (0.012)	75.00

Table 1: Results for linear regression with uncontaminated data.

Setting	Method	MSE	Signal MSE	Estimation Error	Retained	Precision	Recall	F1
$\alpha=0.2$ $\tau=5$	SWAPE	1.033 (0.038)	0.033 (0.032)	0.034 (0.013)	74.12	0.804 (0.189)	0.964 (0.064)	0.859 (0.111)
	$\Theta$ -IPOD	1.024 (0.031)	0.023 (0.023)	0.030 (0.011)	79.79	0.953 (0.060)	0.957 (0.062)	0.952 (0.036)
	OLS	2.003 (0.105)	1.001 (0.099)	0.072 (0.010)	–	–	–	–
	LAD	1.131 (0.050)	0.130 (0.045)	0.040 (0.011)	–	–	–	–
	Huber	1.193 (0.049)	0.192 (0.046)	0.041 (0.009)	–	–	–	–
	LTS	1.020 (0.023)	0.020 (0.012)	0.030 (0.010)	75.00	0.799 (0.003)	0.999 (0.003)	0.888 (0.003)

Table 2: Results for linear regression on data with vertical contamination. Parameters

\alpha

and

\tau

, indicating the percentage and severity of contamination, are annotated for each scenario.

Setting	Method	MSE	Signal MSE	Estimation Error	Retained	Precision	Recall	F1
$\alpha=0.2$ $\tau=5$ 1 cloud	SWAPE	1.084 (0.307)	0.084 (0.311)	0.044 (0.050)	70.86	0.732 (0.222)	0.967 (0.121)	0.814 (0.167)
	$\Theta$ -IPOD	1.127 (0.414)	0.126 (0.413)	0.045 (0.068)	79.08	0.862 (0.185)	0.928 (0.233)	0.887 (0.216)
	OLS	2.503 (0.180)	1.500 (0.173)	0.247 (0.040)	–	–	–	–
	LAD	1.529 (0.360)	0.527 (0.356)	0.145 (0.060)	–	–	–	–
	Huber	1.854 (0.421)	0.852 (0.416)	0.186 (0.064)	–	–	–	–
	LTS	1.100 (0.414)	0.100 (0.415)	0.041 (0.061)	75.00	0.774 (0.127)	0.967 (0.159)	0.860 (0.141)

Table 3: Results for linear regression on data with interior-clustered contamination. Parameters

\alpha

and

\tau

, indicating the percentage and severity of contamination, are annotated for each scenario.

Figure 3: Representative training and test datasets for $\alpha=0.30$ in a replication. Left: the contaminated training set, where circles and triangles denote the observed class labels. Center: the label contamination in the training set, with flipped-label observations highlighted in red and the remaining observations shown in gray. Right: the uncontaminated test set with the true class labels.

Figure 4: Estimated decision boundaries for $\alpha=0.30$ in the same replication as Figure 3 .

$\alpha$	Regularization	Accuracy		AUC		Weight Trained by SWAPE
$\alpha$	Regularization	Baseline	SWAPE	Baseline	SWAPE	Retained	Precision	Recall	F1
0.25	None	0.815 (0.025)	0.908 (0.026)	0.864 (0.023)	0.962 (0.016)	74.05	0.822 (0.046)	0.849 (0.119)	0.828 (0.086)
	Early stopping	0.892 (0.033)	0.892 (0.032)	0.960 (0.013)	0.959 (0.015)	100.00	0.000 (0.000)	0.000 (0.000)	0.000 (0.000)
	$\ell_{2}$	0.849 (0.024)	0.910 (0.027)	0.903 (0.022)	0.966 (0.017)	73.34	0.809 (0.070)	0.850 (0.160)	0.813 (0.122)
	Dropout	0.895 (0.017)	0.902 (0.025)	0.939 (0.014)	0.957 (0.018)	76.19	0.831 (0.071)	0.777 (0.224)	0.772 (0.184)
0.30	None	0.771 (0.032)	0.881 (0.044)	0.820 (0.031)	0.943 (0.036)	70.92	0.819 (0.060)	0.786 (0.194)	0.780 (0.159)

Table 4: Results for binary classification on data with flipped label contamination

$\alpha$	Method	Target HitRate			Target MRR
$\alpha$	Method	$K=5$	$K=10$	$K=15$	$K=5$	$K=10$	$K=15$
$0.05$	SWAPE	0.0000 (0.0000)	0.0000 (0.0000)	0.0000 (0.0000)	0.0000 (0.0000)	0.0000 (0.0000)	0.0000 (0.0000)
	SLIM	0.0366 (0.0029)	0.0496 (0.0029)	0.0530 (0.0045)	0.0165 (0.0021)	0.0183 (0.0021)	0.0185 (0.0022)
	SLIM-LTS	0.0004 (0.0005)	0.0016 (0.0009)	0.0032 (0.0016)	0.0002 (0.0003)	0.0003 (0.0003)	0.0004 (0.0003)
	SVD	0.0574 (0.0031)	0.0806 (0.0039)	0.0996 (0.0053)	0.0284 (0.0022)	0.0315 (0.0020)	0.0330 (0.0021)
	UserKNN	0.0515 (0.0014)	0.0598 (0.0022)	0.0677 (0.0032)	0.0265 (0.0028)	0.0276 (0.0027)	0.0282 (0.0027)

Table 8: Robustness to target promotion under bandwagon attacks. Lower Target HitRate and Target MRR indicate better robustness.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix


$\displaystyle\mathbf{x}^{t+1}$	$\displaystyle=\operatornamewithlimits{argmin}_{\mathbf{x}\in\mathbb{R}^{d}}\Bigl[\tfrac{1}{2}\\|(\mathbf{w}^{t})^{1/2}\odot(\mathbf{b}-A\mathbf{x})\\|_{2}^{2}+\tfrac{c_{\mathbf{x}}}{2}\\|\mathbf{x}-\mathbf{x}^{t}\\|_{2}^{2}\Bigr],$	(38a)
$\displaystyle\mathbf{w}^{t+1}$	$\displaystyle=\operatornamewithlimits{argmin}_{\begin{subarray}{c}\mathbf{w}\in\mathbb{R}_{+}^{n}\end{subarray}}\Bigl[\tfrac{1}{2}\\|\mathbf{w}^{1/2}\odot(\mathbf{b}-A\mathbf{x}^{t+1})\\|_{2}^{2}+\gamma\sum_{i=1}^{n}\rho^{\rm mcp}(w_{i})+\tfrac{c_{\mathbf{w}}}{2}\\|\mathbf{w}-\mathbf{w}^{t}\\|_{2}^{2}\Bigr],$	(38b)

Appendix

Algorithm 1 Proposed Method for Solving ( 37 )

Method	Training Time (s)
SWAPE	0.0025 (0.0009)
$\Theta$ -IPOD	0.0045 (0.0049)
Least Squares	0.0002 (0.0001)
LAD	0.7347 (0.0144)
Huber	0.0116 (0.0012)
LTS	0.0069 (0.0011)

Appendix

Table 9: Training wall-clock time (seconds) for the vertical-contamination setting with

\alpha=0.3

and

\tau=5

. Reported values are the mean (and standard deviation) over 200 independent replications.


$\displaystyle\mathbf{x}^{t+1}$	$\displaystyle=\operatornamewithlimits{argmin}_{\mathbf{x}\in\mathbb{R}^{d}}\Biggl[-\sum_{i=1}^{n}w_{i}^{t}\left[b_{i}\log p_{i}(\mathbf{x})+(1-b_{i})\log\!\left(1-p_{i}(\mathbf{x})\right)\right]$
	$\displaystyle\hskip 110.99927pt+\lambda R(\mathbf{x})+\frac{c_{\mathbf{x}}}{2}\\|\mathbf{x}-\mathbf{x}^{t}\\|_{2}^{2}\Biggr],$	(45a)
$\displaystyle\mathbf{w}^{t+1}$	$\displaystyle=\operatornamewithlimits{argmin}_{\mathbf{w}\in\mathbb{R}_{+}^{n}}\Biggl[-\sum_{i=1}^{n}w_{i}\left[b_{i}\log p_{i}(\mathbf{x}^{t+1})+(1-b_{i})\log\!\left(1-p_{i}(\mathbf{x}^{t+1})\right)\right]$
	$\displaystyle\hskip 110.99927pt+\gamma\sum_{i=1}^{n}\rho^{\rm lasso}(w_{i})+\frac{c_{\mathbf{w}}}{2}\\|\mathbf{w}-\mathbf{w}^{t}\\|_{2}^{2}\Biggr].$	(45b)

Appendix

Algorithm 2 Proposed Method for Solving ( 43 )

Regularization	Baseline	SWAPE
None	20.82 (0.21)	285.12 (82.02)
Early stopping	1.04 (0.46)	1.13 (0.52)
$\ell_{2}$	20.79 (0.25)	277.23 (81.64)
Dropout	22.97 (1.56)	391.26 (129.18)

Appendix

Table 10: Wall-clock training time (seconds) for the neural network experiment at

\alpha=0.35

. Entries report the mean and standard deviation over 200 replications.


$\displaystyle\mathbf{x}^{t+1}$	$\displaystyle=\operatornamewithlimits{argmin}_{\mathbf{x}\in\mathbb{R}^{d}_{+};x_{j}=0}\Biggl[\left(\mathbf{w}^{t}\odot(A\mathbf{x}^{t}-\mathbf{a})\right)^{\mathsf{T}}A\left(\mathbf{x}-\mathbf{x}^{t}\right)+\lambda\\|\mathbf{x}\\|_{1}+\frac{c_{\mathbf{x}}}{2}\left\\|\mathbf{x}-\mathbf{x}^{t}\right\\|_{2}^{2}\Biggr],$	(50a)
$\displaystyle\mathbf{w}^{t+1}$	$\displaystyle=\operatornamewithlimits{argmin}_{\mathbf{w}\in\mathbb{R}_{+}^{n}}\Biggl[\frac{1}{2}\left\\|(\mathbf{w})^{1/2}\odot(\mathbf{a}-A\mathbf{x}^{t+1})\right\\|_{2}^{2}+\gamma\sum_{i=1}^{n}\rho^{\rm mcp}(w_{i})+\frac{c_{\mathbf{w}}}{2}\left\\|\mathbf{w}-\mathbf{w}^{t}\right\\|_{2}^{2}\Biggr].$	(50b)

Appendix

Algorithm 3 Proposed Framework for Solving ( 49 )

Method	$\lambda$	$\theta$	$\gamma$	$\eta$	$c_{\mathbf{x}}$	$c_{\mathbf{w}}$	$\beta$	$h$	$K$
SWAPE	0.05	0.5	0.1	2.0	600000	0.1	–	–	–
SLIM	0.5	–	–	–	–	–	1.0	–	–
SLIM-LTS	0.5	–	–	–	–	–	1.0	0.8	–

Appendix

Table 11: Hyperparameter settings for each recommendation method

Method	Training Time
SWAPE	1.47 (0.07)
SLIM	1512.63 (45.27)
SLIM–LTS	2277.18 (67.54)
SVD	0.66 (0.02)
UserKNN	44.37 (0.88)
ItemKNN	29.23 (0.57)

Appendix

Table 12: Training wall-clock time (seconds) for the first setting. Reported values are the mean (standard deviation) over 30 independent replications.

Figure 5: Distribution of the predicted scores $\widehat{A}_{ij}$ for a replication of the first recommender system experiment. SLIM generally produces scores below the corresponding true ratings $A_{ij}$ , which is consistent with the sparsity-inducing regularization in its formulation that shrinks the estimated coefficients toward zero. SVD exhibits a similarly compressed score distribution. These results show that methods achieving relatively strong hit rate performance may still produce predicted scores whose magnitudes differ substantially from the observed ratings. For each true rating $A_{ij}\in\{1,\ldots,5\}$ , each box summarizes the predicted scores for all observed test user-item pairs having that rating.

Instance Reweighting

See all 18 papers

Meta Additive Model: Interpretable Sparse Learning With Auto Weighting
Apr 22, 2026Xuelin Zhang, Xinyue Liu, Lingjuan Wu +1Robust Regression Adaptive Loss Weighting
College of Informatics, Huazhong Agricultural University, Wuhan, China
Accelerated Dynamic Importance Weighting with Versatile Divergence-Minimizing Estimators
May 25, 2026Tongtong Fang, Nan Lu, Gang Niu +2Importance Weighting Instance Reweighting
The Institute of Statistical Mathematics, Tokyo, Japan. · University of Bristol, Bristol, U.K. · RIKEN Center for Advanced Intelligence Project, Tokyo, Japan. +1
Joint Model and Data Sparsification via the Marginal Likelihood
May 28, 2026Alexander Timans, Thomas Möllenhoff, Christian A. Naesseth +2Dataset Pruning Robust Regression
1UvA-Bosch Delta Lab, University of Amsterdam · 2RIKEN Center for AI Project, Tokyo, Japan · Department of Computer Science, Johns Hopkins University.

Apr 22, 2026J/K move · Enter open · S save

Meta Additive Model: Interpretable Sparse Learning With Auto Weighting

Xuelin Zhang, Xinyue Liu, Lingjuan Wu, Hong Chen

College of Informatics, Huazhong Agricultural University, Wuhan, China

Robust Regression Adaptive Loss Weighting Feature Selection

Open paper AskPDF arXiv

Sparse additive models have attracted much attention in high-dimensional data analysis due to their flexible representation and strong interpretability. However, most existing models are limited to single-level learning under the mean-squared error criterion, whose empirical performance can degrade significantly in the presence of complex noise, such as non-Gaussian perturbations, outliers, noisy labels, and imbalanced categories. The sample reweighting strategy is widely used to reduce the model's sensitivity to atypical data; however, it typically requires prespecifying the weighting functions and manually selecting additional hyperparameters. To address this issue, we propose a new meta additive model (MAM) based on the bilevel optimization framework, which learns data-driven weighting of individual losses by parameterizing the weighting function via an MLP trained on meta data. MAM is capable of a variety of learning tasks, including variable selection, robust regression estimation, and imbalanced classification. Theoretically, MAM provides guarantees on convergence in computation, algorithmic generalization, and variable selection consistency under mild conditions. Empirically, MAM outperforms several state-of-the-art additive models on both synthetic and real-world data under various data corruptions.

Abstract

Instance Reweighting

Explore similar work

Meta Additive Model: Interpretable Sparse Learning With Auto Weighting

Accelerated Dynamic Importance Weighting with Versatile Divergence-Minimizing Estimators

Joint Model and Data Sparsification via the Marginal Likelihood