Organizations: School of Mathematical Sciences/Research Center for Image and Vision Computing, University of Electronic Science and Technology of China, Chengdu, Sichuan 611731, China · School of Information Science and Technology, Southwest Jiaotong University, Chengdu 611756, China · School of Mathematics and Statistics and Ministry of Education Key Lab of Intelligent Networks and Network Security, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, China, and also with Pazhou Laboratory (Huangpu), Guangzhou, Guangdong 510335, China
Recently, tensor decompositions have attracted increasing attention. Fundamentally, different interactions among factors induce distinct tensor decomposition structures (i.e., tensor decomposition). Identifying an appropriate interaction-induced tensor decomposition structure for given data is a fundamental yet challenging problem in tensor modeling, and remains largely under-explored. Existing tensor decomposition structure search methods are typically restricted to a predefined interaction family, such as tensor contraction. To address this problem, we suggest a tensor decomposition structure search framework (I-TSS) from an interaction perspective that can identify either a single structure beyond a predefined interaction family or a mixture of structures involving heterogeneous interaction families. Specifically, we first systematically review existing tensor decomposition structures from an interaction perspective and construct a heterogeneous interaction-induced candidate structure set. Based on it, we propose a unified energy-based rank estimation scheme for heterogeneous interaction-induced candidate structure set. We then introduce the top-k gating mechanism with learnable gating scores to dynamically select a suitable single-candidate structure or combine suitable multi-candidate structures, thereby identifying the tensor decomposition structure in a data-adaptive manner. Theoretically, we derive an approximation error bound for I-TSS, thereby establishing its approximation capability. Extensive experiments on synthetic and real-world datasets demonstrate that I-TSS consistently outperforms state-of-the-art tensor decomposition methods.
Figures & tables
Characteristic
Search Capability
Data Adaptability
Method
CP
Tucker
TT
TR
FCTN
T-SVD
Mixture
Tucker
✓
✓
FCTN
✓
✓
✓
T-SVD
✓
SVDinsTN
✓
✓
✓
✓
I-TSS
✓
✓
✓
✓
✓
✓
✓
✓
TABLE I: Comparison between classic tensor decomposition structure search methods and our I-TSS in terms of search capability and data adaptability.
Fig. 1: Diagram of our I-TSS. First, we perform the energy-based rank estimation scheme for the input data. Then, the estimated ranks are fed into the heterogeneous interaction-induced candidate structure set. Finally, the top- k gating mechanism is introduced to dynamically select a suitable single-candidate structure or combine suitable multi-candidate structures, leading to either a single structure ( k=1 ) or mixture of structures ( k>1 ).
Method
Rank: 3
Rank: 5
Rank: 7
CR
RE
CR
RE
CR
RE
Synthetic Data (Tucker)
Tucker
0.3%
3∗10−5
0.7%
2∗10−5
1.1%
2∗10−5
TF
0.4%
4∗10−1
1.0%
2∗10−1
1.9%
2∗10−1
TNGreedy
0.4%
6∗10−2
1.5%
9∗10−4
3.0%
8∗10−4
SVDinsTN
0.6%
8∗10−4
2.2%
5∗10−4
4.3%
2∗10−4
TABLE II: Quantitative comparison of CR (↓) and RE (↓) of different methods on third-order synthetic data.
Method
Cloth
Beads
Jelly
CR
RE
CR
RE
CR
RE
Tucker
37.8%
1∗10−2
39.3%
1∗10−2
34.5%
1∗10−2
TF
34.4%
1∗10−2
39.0%
1∗10−2
35.4%
1∗10−2
TNGreedy
31.7%
7∗10−3
37.8%
6∗10−3
32.9%
7∗10−3
SVDinsTN
32.8%
7∗10−3
39.9%
9∗10−3
34.8%
8∗10−3
I-TSS
31.7%
6∗10−3
37.8%
5∗10−3
32.6%
6∗10−3
TABLE III: Quantitative comparison of CR (↓) and RE (↓) of different methods on real-world MSIs.
SR
0.1
0.15
0.2
0.25
0.3
Data
Method
PSNR
SSIM
RE
PSNR
SSIM
RE
PSNR
SSIM
RE
PSNR
SSIM
RE
PSNR
SSIM
RE
HaLRTC
20.54
0.672
0.592
29.89
0.830
0.202
31.73
0.864
0.163
33.50
0.888
0.133
33.86
0.895
0.127
SiLRTCTT
28.06
0.749
0.249
31.22
0.809
0.173
33.53
0.843
0.132
35.44
0.864
0.106
36.93
0.880
0.089
Flowers
TRLRF
20.32
0.552
0.607
25.34
0.568
0.340
26.52
0.656
0.297
26.81
0.670
0.287
27.09
0.685
0.278
HTNN
29.94
0.790
0.200
32.92
0.844
0.142
34.70
0.873
0.116
36.28
0.890
0.096
37.41
0.894
0.085
(256 × 256 × 31)
LTNN
29.74
0.784
0.205
32.95
0.840
0.142
34.81
0.869
0.114
36.32
0.886
0.096
37.52
0.898
0.083
TABLE IV: The quantitative results for real-world MSI completion by different methods. The best and second-best values are highlighted.
Fig. 2: The completion results of MSIs by different methods on Flowers , Beers , Balloons , and Feathers with SR = 0.1.
SR
0.1
0.15
0.2
0.25
0.3
Data
Method
PSNR
SSIM
RE
PSNR
SSIM
RE
PSNR
SSIM
RE
PSNR
SSIM
RE
PSNR
SSIM
RE
HaLRTC
18.24
0.589
0.328
22.09
0.705
0.211
24.28
0.772
0.164
25.72
0.814
0.138
27.08
0.848
0.118
SiLRTCTT
27.18
0.851
0.117
29.84
0.903
0.086
31.81
0.931
0.068
33.36
0.947
0.057
34.67
0.959
0.049
News
TRLRF
29.75
0.861
0.087
30.11
0.863
0.083
31.02
0.882
0.075
33.10
0.919
0.059
34.82
0.942
0.048
HTNN
31.70
0.916
0.069
33.53
0.939
0.056
34.94
0.953
0.048
36.13
0.962
0.041
37.35
0.970
0.036
(144 × 176 × 3 × 30)
LTNN
27.24
0.860
0.116
31.29
0.929
0.073
33.54
0.952
0.056
35.13
0.964
0.047
36.58
0.972
0.039
TABLE V: The quantitative results for real-world color video completion by different methods. The best and second-best values are highlighted.
Fig. 3: The completion results of real-world color videos by different methods on News , Claire , Grandma , and Akiyo with SR = 0.1.
SR
0.1
0.15
0.2
0.25
0.3
Data
Method
PSNR
SSIM
RE
PSNR
SSIM
RE
PSNR
SSIM
RE
PSNR
SSIM
RE
PSNR
SSIM
RE
HaLRTC
17.16
0.524
0.247
22.78
0.737
0.129
25.10
0.815
0.099
26.57
0.856
0.083
27.94
0.888
0.071
SiLRTCTT
27.53
0.880
0.074
28.18
0.887
0.069
30.07
0.921
0.055
31.67
0.943
0.046
33.15
0.957
0.039
Greek
TRLRF
24.37
0.715
0.107
28.70
0.869
0.065
29.05
0.876
0.062
29.41
0.887
0.060
29.76
0.890
0.057
HTNN
29.06
0.890
0.062
31.07
0.927
0.049
32.95
0.950
0.040
34.67
0.965
0.032
36.39
0.975
0.027
(128 × 128 × 3 × 30)
LTNN
29.37
0.909
0.060
32.28
0.952
0.043
34.86
0.972
0.032
36.85
0.982
0.025
38.62
0.987
0.020
TABLE VI: The quantitative results for real-world light field data completion by different methods. The best and second-best values are highlighted.
Fig. 4: The completion results of light field data by different methods on Greek , Museum , Medieval2 , and Vinyl with SR = 0.1.
TABLE VII: The PSNR values and corresponding interaction families of the representative search method (i.e., SVDinsTN) and the proposed I-TSS across various data with a sampling rate of 0.2. The symbol ‘+’ represents the mixture of structures.
Data
Flowers
Balloons
Feathers
Selection Strategy
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
Max Sampling
42.51
0.985
49.49
0.996
41.54
0.980
Random Sampling
37.64
0.880
40.70
0.973
37.97
0.949
TABLE VIII: The proposed I-TSS variants with different candidate structure selection strategies for MSIs Flowers , Balloons , and Feathers with a sampling rate of 0.2.
Data
Flowers
Balloons
Feathers
Updating Order
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
I
42.99
0.938
46.76
0.994
41.65
0.981
II
42.70
0.939
49.49
0.996
42.54
0.984
III
34.34
0.895
41.63
0.983
35.94
0.941
TABLE IX: The proposed I-TSS variants with different parameters updating orders for MSIs Flowers , Balloons , and Feathers with a sampling rate of 0.2.
Fig. 5: The PSNR values of the proposed I-TSS with different values of energy ratio thresholds for MSIs Flowers and Feathers with a sampling rate of 0.1.
We introduce the tubal tensor train (TTT) decomposition, a tensor-network model that combines the t-product algebra of the tensor singular value decomposition (T-SVD) with the low-order core structure of the tensor train (TT) format. For an order-(N+1) tensor with a distinguished tube mode, the proposed representation consists of two third-order boundary cores and N−2 fourth-order interior cores linked through the t-product. As a result, for bounded tubal ranks, the storage scales linearly with the number of modes, in contrast to direct high-order extensions of T-SVD. We present two computational strategies: a sequential fixed-rank construction, called TTT-SVD, and a Fourier-slice alternating scheme based on the alternating two-cores update (ATCU). We also state a TT-SVD-type error bound for TTT-SVD and illustrate the practical performance of the proposed model on image compression, video compression, tensor completion, and hyperspectral imaging.
Lab of Machine Learning and Knowledge Representation, Innopolis University, Innopolis, Russia · Innopolis University, Innopolis, Russia · Skolkovo Institute of Science and Technology, Center for Artificial Intelligence Technology, Moscow, Russia +1
Tensor networks provide efficient representations for compressing large neural networks. By carefully designing shapes and topologies, they can significantly reduce memory and computational costs. However, identifying implicit low-rank structures in large foundation models remains challenging due to their enormous scale and un-structured weight distributions. We propose an adaptive tensorization method that discovers inherent low-rank structure in a target tensor by index ordering. Experiments on weight and KV-cache compression demonstrate improved reconstruction quality compared to baselines.
Toshiaki Koike-Akino, Jing Liu, Ye Wang
Mitsubishi Electric Research Laboratories (MERL), 20 Broadway, Cambridge, MA 02139, USA.
Matricized low-rank approximation via SVD is a standard surrogate for tensor decompositions, but entry-wise reconstruction error fails to capture multiway geometric degradation. Under an orthogonal Tucker model, we characterize this degradation using two metrics: cross-mode Direction Loss, measuring geometric subspace deviation from rank truncation and noise rotation, and Interaction Loss, quantifying multilinear interaction distortion in the core tensor. We prove that squared relative reconstruction error orthogonally decomposes into interaction loss and out-of-subspace energy, and derive a Wedin-type bound establishing the stability of a plug-in Direction Loss estimator. Experiments on synthetic and hyperspectral datasets demonstrate that nearly identical reconstruction errors can yield markedly different structural-loss profiles; hyperspectral patches with comparable reconstruction errors exhibit up to a 4.6-fold difference in Direction Loss, correlating with severe visual blurring.
Hiroki Hasegawa
Graduate School of Science and Technology, University of Tsukuba, Tsukuba, 305-8573, Japan