Lightweight Representation Learning for Low Tucker Rank Tensors

Hiroki Hasegawa; Yukihiko Okada

cs.LGOct 31, 2026

Lightweight Representation Learning for Low Tucker Rank Tensors

Authors: Hiroki Hasegawa, Yukihiko Okada

Organizations: Graduate School of Science and Technology, University of Tsukuba, Tsukuba, Japan · Center for Artificial Intelligence Research, Tsukuba Institute for Advanced Research, University of Tsukuba, Tsukuba, Japan · Institute of Systems and Information Engineering, University of Tsukuba, Tsukuba, Japan

Abstract

Low Tucker rank tensors provide compact representations of multidimensional dependencies, but standard learning pipelines can incur substantial computational and memory overhead from redundant intermediate quantities. We reduce this overhead without modifying the underlying representation objective or parameterization. Our approach combines three complementary components: (i) sequential extraction of only task-required spectral components, avoiding unnecessary full decompositions; (ii) factor-wise multilinear contractions without materializing large Kronecker operators; and (iii) compact latent-state updates without storing full ambient tensor iterates. These components preserve the algebraic operations and learning trajectories of the original algorithms while reducing their computational and memory footprints. Across five low Tucker rank tensor estimators, our implementations match the original trajectories to numerical precision (relative discrepancies of $10^{-16}$ to $10^{-13}$ ), while reducing per-iteration runtime by up to $3.12\times$ and resident memory by up to $616\times$ ( $99.8\%$ ) on the main order-3 benchmark at $n=100$ , with memory reductions exceeding $900\times$ across the full evaluation suite. These results show that substantial computational redundancy can be eliminated from Tucker representation learning without changing the underlying model or learned representation.

Figures & tables

Strategy	Representative Works	Target
I. Selective computation	Rank-1 SVD ( Guo et al., 2017a ) ; truncated SVD ( Deng et al., 2024 )	FLOPs
II. Compact representation	Factor states ( Bader and Kolda, 2007 ; Nimishakavi et al., 2018 ) ; sparse+low-rank ( Yao et al., 2022 ; Cai et al., 2023 )	Memory
III. Implicit computation	ST-HOSVD ( Vannieuwenhoven et al., 2012 ) ; retraction simplification ( Zhang et al., 2026 )	FLOPs + Memory
IV. Approximate / randomized	Sketching ( Malik and Becker, 2018 ) ; randomized Tucker ( Minster et al., 2020 ) ; streaming ( Sun et al., 2020 ) ; Nyström ( Bucci and Robol, 2024 )	FLOPs + Data
V. Optimization acceleration	Preconditioning ( Kasai and Mishra, 2016 ) ; ScaledGD ( Tong et al., 2022 ) ; PRGD ( Zhang et al., 2025 ) ; RGN ( Luo and Zhang, 2023 )	Iterations

Table 1: Taxonomy of computational efficiency strategies in low-rank tensor estimation.

ID	Estimator	Paradigm	Components
E0	Subspace Norm ( Zheng and Tomioka, 2015 )	Convex Prox.	Alg. 1, 2, 3
E1	SNN / HaLRTC ( Liu et al., 2012 )	Convex ADMM	Alg. 1, 3
E2	Frank–Wolfe (FFW) ( Guo et al., 2017b )	Convex Greedy	Alg. 1, 3
E3	HOOI ( De Lathauwer et al., 2000b )	Nonconvex ALS	Alg. 1, 2, 3
E4	ScaledGD ( Tong et al., 2022 )	Nonconvex Precond.	Alg. 2, 3

Table 2: Evaluated estimators and applicable algorithmic components (ADMM: alternating direction method of multipliers; ALS: alternating least squares).

Figure 1: Trajectory exactness under lockstep execution: relative iterate discrepancies $\|D(z^{t})-\mathcal{X}^{t}\|_{F}/\|\mathcal{X}^{t}\|_{F}$ over iterations $t\leq 20$ for $\kappa\in\{1,100\}$ ; the dashed line marks the $10^{-10}$ threshold.

Figure 2: Empirical per-iteration runtime (left, bars) and resident state memory (right, markers) of the best component configuration against Raw at

n=100

; full ablation stages and

n=50

results are in Table 7 .

Estimator	Runtime (s)	Speed-up	Memory (MB)	Reduction
E0 Subspace Norm	.1366 $\to$ .0873	1.57 $\times$	66.81 $\to$ 15.32	4.36 $\times$
E1 SNN / HaLRTC	.5400 $\to$ .5849	0.92 $\times$	53.41 $\to$ 10.09	5.29 $\times$
E2 Frank–Wolfe (FFW)	.2374 $\to$ .2271	1.05 $\times$	16.41 $\to$ 1.54	10.68 $\times$
E3 HOOI	.0386 $\to$ .0124	3.12 $\times$	7.64 $\to$ .0124	616.38 $\times$
E4 ScaledGD	.0330 $\to$ .0195	1.69 $\times$	15.27 $\to$ .3933	38.83 $\times$

Table 3: Per-estimator runtime (median per iteration) and persistent memory relative to Raw at

n=100

on order-3 tensors (

p=0.05

for completion); the best lightweight configuration is selected independently for runtime and memory (more sizes in Appendix E ).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Paradigm	FLOP / Iteration	Peak Memory	Algebraic Loss	$\kappa$ -Sensitivity
Naive Ambient	$\Theta(Nn^{N+1})$	$\Theta(n^{N})$	None (Exact)	Unchanged
Randomized Sketching	$O(\|\Omega\|+Nnd^{2})$	$O(Nnd+d^{N})$	Approximate	Unchanged
Riemannian / ScaledGD	$O(\|\Omega\|r^{N}+Nnr^{2})$	$O(n^{N})$ or $O(Nnr+r^{N})$	None (Exact)	Invariant
Proposed Method	$O(q(q+r)n^{N-1}+(q+r^{N})\|\Omega\|)$	state $O(Nnr+r^{N}+\|\Omega\|)$ ; peak $+\,O(qn^{N-1})$	None (Exact)	Invariant

Appendix

Table 4: Analytical comparison of arithmetic, spatial, and optimization properties across computing paradigms.

Estimator	Config	Tol Mode	$\kappa$	$\max_{t}\mathrm{err}_{\mathrm{exact}}(t)$	$\max_{t}\|\Delta\mathcal{F}\|$	Status
Subspace Norm (E0)	All	Tight ( $10^{-14}$ )	1	$3.63\times 10^{-14}$	$9.38\times 10^{-14}$	Pass
Subspace Norm (E0)	All	Default ( $10^{-10}$ )	1	$3.62\times 10^{-14}$	$9.33\times 10^{-14}$	Pass
Subspace Norm (E0)	All	Tight ( $10^{-14}$ )	100	$3.25\times 10^{-14}$	$5.34\times 10^{-14}$	Pass
Subspace Norm (E0)	All	Default ( $10^{-10}$ )	100	$2.05\times 10^{-13}$	$5.77\times 10^{-14}$	Pass
SNN / Latent (E1)	+Alg 1+Alg 3	Tight ( $10^{-14}$ )	1	$7.65\times 10^{-15}$	$2.71\times 10^{-15}$	Pass
SNN / Latent (E1)	+Alg 1+Alg 3	Default ( $10^{-10}$ )	1	$6.86\times 10^{-15}$	$2.71\times 10^{-15}$	Pass

Appendix

Table 5: Empirical trajectory exactness and objective gaps under lockstep execution (

T=20

n=30

r=5

p=0.3

Estimator	Runtime at $n=40$ (s)		Runtime at $n=120$ (s)		Empirical Slope		Slope Change
	Raw	All	Raw	All	Raw	All	( $\Delta$ )
E0	0.0101	0.0198	0.2289	0.1539	2.829	1.858	$-0.971$
E1	0.0241	0.0429	1.1991	1.7835	3.618	3.414	$-0.204$
E2	0.0113	0.0190	0.5412	0.3220	3.567	2.584	$-0.983$
E3	0.0045	0.0060	0.0628	0.0234	2.417	1.246	$-1.171$
E4	0.0016	0.0022	0.0541	0.0314	3.215	2.401	$-0.814$

Appendix

Table 6: Empirical scaling profiles on order-3 tensors (

n\in\{40,60,80,120\}

r=5

p=0.05

). Slopes reflect least-squares fits on

\log(\text{runtime})

versus

\log(n)

. Terminal relative error

\mathrm{rel\_err\_final}

matches identically between Raw and All up to numerical precision across all dimensions.

Estimator	Config	Time $(n=50)$	Mem $(n=50)$	Time $(n=100)$	Mem $(n=100)$
Subspace Norm (E0)	Raw	$0.0155\,\mathrm{s}$	$9.09\,\mathrm{MB}$	$0.1369\,\mathrm{s}$	$66.81\,\mathrm{MB}$
	+Alg 1	$0.0284\,\mathrm{s}$	$9.09\,\mathrm{MB}$	$0.1472\,\mathrm{s}$	$66.81\,\mathrm{MB}$
	+Alg 1+Alg 2	$0.0268\,\mathrm{s}$	$7.66\,\mathrm{MB}$	$0.1387\,\mathrm{s}$	$61.09\,\mathrm{MB}$
	+Alg 1+Alg 3	$0.0260\,\mathrm{s}$	$3.37\,\mathrm{MB}$	$0.1117\,\mathrm{s}$	$21.04\,\mathrm{MB}$
	+Alg 2+Alg 3	$0.0114\,\mathrm{s}$	$1.94\,\mathrm{MB}$	$0.0874\,\mathrm{s}$	$15.32\,\mathrm{MB}$
	All (+Alg 1+Alg 2+Alg 3)	$0.0251\,\mathrm{s}$	$1.94\,\mathrm{MB}$	$0.0984\,\mathrm{s}$	$15.32\,\mathrm{MB}$

Appendix

Table 7: Ablation benchmark profiles reporting per-iteration runtime and persistent state memory (

r=5

p=0.05

\kappa=1

T=8

Primitive	Order	$n$	Config. / Parameter	Full / Ambient	Lazy / Compact	Ratio / Speedup
SVT (Alg 1, Dense)	2	50	$s=1$	$0.0072\,\mathrm{s}$	$0.0232\,\mathrm{s}$	$0.311\times$
SVT (Alg 1, Dense)	2	100	$s=5$	$0.0714\,\mathrm{s}$	$0.3297\,\mathrm{s}$	$0.217\times$
SVT (Alg 1, Operator)	2	50	$s=1$	$0.0682\,\mathrm{s}$	$0.0169\,\mathrm{s}$	$4.031\times$
SVT (Alg 1, Operator)	2	100	$s=1$	$1.5523\,\mathrm{s}$	$0.1293\,\mathrm{s}$	$12.002\times$
SVT (Alg 1, Operator)	2	100	$s=5$	$1.6554\,\mathrm{s}$	$0.1412\,\mathrm{s}$	$11.728\times$
TTM (Alg 2, Contraction)	3	100	$r=10$	$0.0194\,\mathrm{s}$ ( $7.63\,\mathrm{MB}$ )	$0.0134\,\mathrm{s}$	$1.453\times$

Appendix

Table 8: Micro-benchmark measurements evaluating individual computational primitives.

Estimator	Regime	Setting	Runtime Raw $\to$ Light. (s)	Config.	Memory Raw $\to$ Light. (MB)	Config.	Reduction
E0 Subspace Norm	Denoise	$50,5$	.0155 $\to$ .0114	A2+A3	9.09 $\to$ 1.94	A2+A3	4.69 $\times$
E0 Subspace Norm	Denoise	$100,5$	.1366 $\to$ .0873	A2+A3	66.81 $\to$ 15.32	A2+A3	4.36 $\times$
E0 Subspace Norm	Denoise ( $N=3$ )	$120,5$	.2289 $\to$ .1539	All	113.78 $\to$ 26.44	All	4.30 $\times$
E0 Subspace Norm	Denoise ( $N=4$ )	$24,3$	.0690 $\to$ .0590	All	36.72 $\to$ 5.08	All	7.23 $\times$
E1 Latent norm	$p=.05$	$50,5$	.0386 $\to$ .0698	A1	6.68 $\to$ .8262	A1+A3	8.08 $\times$
E1 Latent norm	$p=.10$	$50,5$	.0389 $\to$ .0637	A1	6.68 $\to$ .7569	A1+A3	8.82 $\times$

Appendix

Table 9: Per-estimator cost of the best lightweight configuration relative to Raw (medium preset, MATLAB R2025b, single thread). Runtime is the median time per iteration. Configurations are selected independently for runtime and memory.

Low-Rank Representation Learning

See all 16 papers

A New Tensor Network: Tubal Tensor Train and Its Applications
Mar 11, 2026Salman Ahmadi-Asl, Valentin Leplat, Anh-Huy Phan +1Tensor Decomposition TT Decomposition
Lab of Machine Learning and Knowledge Representation, Innopolis University, Innopolis, Russia · Innopolis University, Innopolis, Russia · Skolkovo Institute of Science and Technology, Center for Artificial Intelligence Technology, Moscow, Russia +1
Representing Tensors as Compositions for Efficient Computation on Sparse and Structured Data
Nov 7, 2026Valérian Rey, Praneth Paruchuri
SimplexLab
EinSort: Sorting is All We Need for Tensorizing LLM
Jun 7, 2026Toshiaki Koike-Akino, Jing Liu, Ye WangTensor Networks LLM Compression
Mitsubishi Electric Research Laboratories (MERL), 20 Broadway, Cambridge, MA 02139, USA.

Mar 11, 2026J/K move · Enter open · S save

A New Tensor Network: Tubal Tensor Train and Its Applications

Salman Ahmadi-Asl, Valentin Leplat, Anh-Huy Phan, Andrzej Cichocki

Lab of Machine Learning and Knowledge Representation, Innopolis University, Innopolis, Russia · Innopolis University, Innopolis, Russia · Skolkovo Institute of Science and Technology, Center for Artificial Intelligence Technology, Moscow, Russia+1

Tensor Decomposition TT Decomposition Low-Rank Compression

Open paper AskPDF arXiv

We introduce the tubal tensor train (TTT) decomposition, a tensor-network model that combines the t-product algebra of the tensor singular value decomposition (T-SVD) with the low-order core structure of the tensor train (TT) format. For an order-

(N+1)

tensor with a distinguished tube mode, the proposed representation consists of two third-order boundary cores and

N-2

fourth-order interior cores linked through the t-product. As a result, for bounded tubal ranks, the storage scales linearly with the number of modes, in contrast to direct high-order extensions of T-SVD. We present two computational strategies: a sequential fixed-rank construction, called TTT-SVD, and a Fourier-slice alternating scheme based on the alternating two-cores update (ATCU). We also state a TT-SVD-type error bound for TTT-SVD and illustrate the practical performance of the proposed model on image compression, video compression, tensor completion, and hyperspectral imaging.

Abstract

Low-Rank Representation Learning

Explore similar work

A New Tensor Network: Tubal Tensor Train and Its Applications

Representing Tensors as Compositions for Efficient Computation on Sparse and Structured Data

EinSort: Sorting is All We Need for Tensorizing LLM