Asymptotic Optimism for Tensor Regression Models with Applications to Neural Network Compression

Haoming Shi; Eric C. Chi; Hengrui Luo

stat.MLMar 27, 2026

Asymptotic Optimism for Tensor Regression Models with Applications to Neural Network Compression

Authors: Haoming Shi, Eric C. Chi, Hengrui Luo

Organizations: Department of Statistics, Rice University · School of Statistics, University of Minnesota

Abstract

We study rank selection for low-rank tensor regression under random covariates design. Under a Gaussian random-design model and some mild conditions, we derive population expressions for the expected training-testing discrepancy (optimism) for both CP and Tucker decomposition. We further demonstrate that the optimism is minimized at the true tensor rank for both CP and Tucker regression. This yields a prediction-oriented rank-selection rule that aligns with cross-validation and extends naturally to tensor-model averaging. We also discuss conditions under which under- or over-ranked models may appear preferable, thereby clarifying the scope of the method. Finally, we showcase its practical utility on a real-world image regression task and extend its application to tensor-based compression of neural network, highlighting its potential for model selection in deep learning.

Figures & tables

Panel of Figure 1 — Figure 1: Average optimism of the tensor KRR model over $10000$ Monte Carlo replicates. The left panel varies the target CP rank and noise level, with the training sample size held constant at $n_{\text{train}}=200$ . The right panel varies the target CP rank and the training sample size, with the noise level held constant at $5\%$ of the signal standard deviation. In all cases, the regularization parameter is $\lambda=1$ . Results are shown for the oracle case, where the CP kernel is constructed from the true tensor coefficient $\bm{\mathscr{{B}}}$ , which has a default rank of $3$ .

Figure 2: Average optimism for low-rank CP and Tucker regression with varying ranks and sample size. The left panel shows the results for CP regression over $10000$ MC replicates. The noise level is fixed at $5\%$ of the signal standard deviation, and the true CP rank is $R=3$ . The right panel shows the results for Tucker regression (using TensorGP from Yu et al. (2018) ). The noise level is fixed at $1\%$ of the signal standard deviation to maintain a similar noise magnitude as the CP case, and the true Tucker rank is $R=(3,3,3)$ . Here we reduce the MC replicates to $100$ due to its high computational complexity and our fixed computational power. In both plots, the model rank varies along the x-axis, while different colors correspond to varying training sample sizes.

Panel of Figure 3 — Figure 3: Different Selection Criteria for a low-rank CP regression on the FGNET dataset Fu et al. (2015) across different target CP ranks (x-axis). The model is evaluated using an 80% training and 20% testing split. Optimism is calculated via the hold-out algorithm ( $100$ MC replicates) in Luo and Zhu (2025) . Error bars show one standard deviation. The table on the right summarizes the optimal rank selected by each criterion (the minimum value on its curve) and the corresponding Test MSE at that rank. For optimism, the global minimum at rank 1. But we report 29 as its optimal rank, which has the second-lowest value to avoid the poor model fit (high training MSE) at rank 1 as a rank-1 model fails to learn the data structure.

Panel of Figure 4 — Figure 4: Different Selection Criteria for a three-layer CNN fitted on the FGNET dataset Fu et al. (2015) , with two convolutional layers compressed by CP decomposition of varying ranks (x-axis). The model is trained on an 80% training and 20% testing split using the Adam optimizer (fixed learning rate 0.001) with MSE loss and 500 epochs. Optimism is calculated via the hold-out algorithm ( $100$ MC replicates) in Luo and Zhu (2025) . Error bars show one standard deviation. The table on the right summarizes the optimal rank selected by each criterion (the minimum value on its curve) and the corresponding Test MSE at that rank.

Panel of Figure 5 — Figure 5: Different Selection Criteria for a two-layer MLP with a CP-decomposed input layer, fitted on the Infrared Thermography Temperature dataset Wang et al. (2021) with 1020 samples, 30-dimensional inputs, and 2-dimensional outputs. The input layer is compressed using CP decompositions with varying ranks (x-axis). The model is trained on an 80% training and 20% testing split using Adam optimizer (fixed learning rate 0.001) with MSE loss and 200 epochs. Optimism is calculated via the hold-out algorithm ( $100$ MC replicates) in Luo and Zhu (2025) . Error bars show one standard deviation. The table on the right summarizes the optimal rank selected by each criterion (the minimum value on its curve) and the corresponding Test MSE at that rank.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Average optimism of tensor KRR model over $10000$ Monte Carlo runs for varying CP ranks (rows) and regularization strength (column): low regularization $\lambda=10^{-1},\,1,\,10,\,10^{2}$ , moderate regularization $\lambda=10^{3},\,10^{4},\,10^{5}$ , and high regularization $\lambda=10^{6},\,10^{7},\,10^{8}$ . The noise level is $5\%$ of signal standard deviation and training sample size is $n_{\text{train}}=200$ . Results are shown for the oracle case, where the CP kernel is constructed from the true tensor coefficient $\bm{\mathscr{{B}}}$ , which has a rank of $3$ .

Figure 7: Average optimism of tensor KRR model over $10000$ Monte Carlo runs for varying CP ranks (row) and a zoom-in moderate regularization strength regime (column) of Figure 6 : $\lambda\in[10^{4},10^{5}]$ with $5000$ increments. The noise level is $5\%$ of signal standard deviation and training sample size is $n_{\text{train}}=200$ . Results are shown for the oracle case, where the CP kernel is constructed from the true tensor coefficient $\bm{\mathscr{{B}}}$ , which has a default rank of $3$ .

Panel of Figure 8 — Figure 8: AIC and BIC selection criteria for low-rank CP and Tucker regression with varying ranks and sample size. The top row of plots (a) shows the AIC, and the bottom row of plots (b) shows the BIC. Within each row of plots, the left plot displays the results for CP regression over $10000$ MC runs, where the noise level is $5\%$ of the signal standard deviation and the true CP rank is $R=3$ . The right plot shows the results for Tucker regression (using TensorGP from Yu et al. (2018) ) over $100$ MC runs, where the noise level is $1\%$ of the signal standard deviation and a true Tucker rank of $R=(3,3,3)$ . In all plots, the x-axis represents the model rank, while different colors correspond to varying training sample sizes. The noise level for Tucker case is reduced to $1\%$ to maintain a similar noise magnitude as the CP case. The MC replicates are limited to $100$ due to the high computational complexity and our fixed computational power.

Figure 9: Average optimism of CP ensemble regression over 10000 MC runs with varying number of ensembles (x-axis) and ensemble CP ranks. The noise level is $5\%$ of signal standard deviation and training sample size is $n_{\text{train}}=1000$ . Each ensemble member is trained on a random subsample of size $n_{k}=200$ . The true model CP rank is $R=3$ . Two horizontal dash lines represent the baseline optimism for a single (non-ensembled) rank- $3$ model trained on $200$ (black) and $1000$ (red) samples, respectively.

Figure 10: Average optimism of CP ensemble regression over $10000$ MC runs with $K=8$ ensemble members and varying ensemble CP ranks (x-axis). The noise level is $5\%$ of signal standard deviation and training sample size is $n_{\text{train}}=1000$ . Each ensemble member is trained on a random subsample of size $n_{k}=200$ . The true model CP rank is $R=3$ . Two horizontal dash lines represent the baseline optimism for a single (non-ensembled) rank- $3$ model trained on $200$ (black) and $1000$ (red) samples, respectively.

Panel of Figure 11 — Figure 11: Different Selection Criteria for a two-layer MLP with a CP-decomposed input layer, fitted on the Infrared Thermography Temperature dataset ( Wang et al., 2021 ) . Compared to Figure 5 in the main manuscript, here the input layer is reformulated as $\bm{\mathscr{{W}}}\in\mathbb{R}^{6\times 5\times 8\times 18}$ and compressed using CP decompositions with varying ranks (x-axis). The model is trained on an 80% training and 20% testing split using Adam optimizer (fixed learning rate 0.001) with MSE loss and 200 epochs. Optimism is calculated via the hold-out algorithm ( $100$ MC replicates) in Luo and Zhu (2025) . Error bars show one standard deviation. The table on the right summarizes the optimal rank selected by each criterion (the minimum value on its curve) and the corresponding Test MSE at that rank.

Model Selection

Model selection chooses models, features, or configurations that balance predictive fit, generalization, complexity, and deployment constraints.

See all 88 papers

Prototype-Rule Neurosymbolic Regularization for Rank-Constrained Tensor Neural Networks under Label Scarcity
Sep 30, 2026Eftychios Protopapadakis, Konstantinos Makantasis, Konstantinos M. GiannoutakisTensor Networks Prototype Learning
Department of Applied Informatics, University of Macedonia, 156 Egnatia Street, 54636 Thessaloniki, Greece · Department of Artificial Intelligence, Faculty of Information & Communication Technology, University of Malta, Msida, Malta
A New Tensor Network: Tubal Tensor Train and Its Applications
Mar 11, 2026Salman Ahmadi-Asl, Valentin Leplat, Anh-Huy Phan +1Tensor Decomposition TT Decomposition
Lab of Machine Learning and Knowledge Representation, Innopolis University, Innopolis, Russia · Innopolis University, Innopolis, Russia · Skolkovo Institute of Science and Technology, Center for Artificial Intelligence Technology, Moscow, Russia +1
Dual-Channel Tensor Neural Networks: Finite-Sample Theory and Conformal Structure Selection
May 18, 2026Elynn Chen, Jiayu Li, Zheshi Zheng +1Tensor Networks Uncertainty Quantification
New York University · University of Michigan · Duke University

Sep 30, 2026J/K move · Enter open · S save

Prototype-Rule Neurosymbolic Regularization for Rank-Constrained Tensor Neural Networks under Label Scarcity

Eftychios Protopapadakis, Konstantinos Makantasis, Konstantinos M. Giannoutakis

Department of Applied Informatics, University of Macedonia, 156 Egnatia Street, 54636 Thessaloniki, Greece · Department of Artificial Intelligence, Faculty of Information & Communication Technology, University of Malta, Msida, Malta

Tensor Networks Prototype Learning Few-Shot Learning Hyperspectral Image Classification Neurosymbolic Learning

Open paper AskPDF arXiv

Rank-constrained tensor neural networks reduce the parameterization of high-order inputs, but they do not explicitly constrain class geometry in the learned representation. This study investigates whether a differentiable prototype-rule can provide a complementary inductive bias for Rank-R tensor learning under limited supervision. The proposed framework augments the Rank-R objective with prototype-based regularization and optionally fuses prototype evidence with neural logits at inference. Four hyperspectral benchmarks are evaluated with four Rank-R configurations under both seven-fold stratification and spatially separated folds that mitigate leakage; a separate spatial study varies the class support budget from 2 to 20 samples. Under spatial evaluation, full neurosymbolic inference changes Macro-F1 score by +8.82 percentage points on Botswana, +5.49 on Indian Pines, +1.59 on Pavia University, and -0.62 on Salinas. Most of the benefit arises from training-time regularization, whereas inference fusion is small and dataset dependent.

Abstract

Model Selection

Explore similar work

Prototype-Rule Neurosymbolic Regularization for Rank-Constrained Tensor Neural Networks under Label Scarcity

A New Tensor Network: Tubal Tensor Train and Its Applications

Dual-Channel Tensor Neural Networks: Finite-Sample Theory and Conformal Structure Selection