Lightweight Representation Learning for Low Tucker Rank Tensors

Hiroki Hasegawa; Yukihiko Okada

cs.LGOct 31, 2026

Lightweight Representation Learning for Low Tucker Rank Tensors

Authors: Hiroki Hasegawa, Yukihiko Okada

Organizations: Graduate School of Science and Technology, University of Tsukuba, Tsukuba, Japan · Center for Artificial Intelligence Research, Tsukuba Institute for Advanced Research, University of Tsukuba, Tsukuba, Japan · Institute of Systems and Information Engineering, University of Tsukuba, Tsukuba, Japan

Abstract

Low Tucker rank tensors provide compact representations of multidimensional dependencies, but standard learning pipelines can incur substantial computational and memory overhead from redundant intermediate quantities. We reduce this overhead without modifying the underlying representation objective or parameterization. Our approach combines three complementary components: (i) sequential extraction of only task-required spectral components, avoiding unnecessary full decompositions; (ii) factor-wise multilinear contractions without materializing large Kronecker operators; and (iii) compact latent-state updates without storing full ambient tensor iterates. These components preserve the algebraic operations and learning trajectories of the original algorithms while reducing their computational and memory footprints. Across five low Tucker rank tensor estimators, our implementations match the original trajectories to numerical precision (relative discrepancies of $10^{-16}$ to $10^{-13}$ ), while reducing per-iteration runtime by up to $3.12\times$ and resident memory by up to $616\times$ ( $99.8\%$ ) on the main order-3 benchmark at $n=100$ , with memory reductions exceeding $900\times$ across the full evaluation suite. These results show that substantial computational redundancy can be eliminated from Tucker representation learning without changing the underlying model or learned representation.

Figures & tables

Strategy	Representative Works	Target
I. Selective computation	Rank-1 SVD ( Guo et al., 2017a ) ; truncated SVD ( Deng et al., 2024 )	FLOPs
II. Compact representation	Factor states ( Bader and Kolda, 2007 ; Nimishakavi et al., 2018 ) ; sparse+low-rank ( Yao et al., 2022 ; Cai et al., 2023 )	Memory
III. Implicit computation	ST-HOSVD ( Vannieuwenhoven et al., 2012 ) ; retraction simplification ( Zhang et al., 2026 )	FLOPs + Memory
IV. Approximate / randomized	Sketching ( Malik and Becker, 2018 ) ; randomized Tucker ( Minster et al., 2020 ) ; streaming ( Sun et al., 2020 ) ; Nyström ( Bucci and Robol, 2024 )	FLOPs + Data
V. Optimization acceleration	Preconditioning ( Kasai and Mishra, 2016 ) ; ScaledGD ( Tong et al., 2022 ) ; PRGD ( Zhang et al., 2025 ) ; RGN ( Luo and Zhang, 2023 )	Iterations

Table 1: Taxonomy of computational efficiency strategies in low-rank tensor estimation.

ID	Estimator	Paradigm	Components
E0	Subspace Norm ( Zheng and Tomioka, 2015 )	Convex Prox.	Alg. 1, 2, 3
E1	SNN / HaLRTC ( Liu et al., 2012 )	Convex ADMM	Alg. 1, 3
E2	Frank–Wolfe (FFW) ( Guo et al., 2017b )	Convex Greedy	Alg. 1, 3
E3	HOOI ( De Lathauwer et al., 2000b )	Nonconvex ALS	Alg. 1, 2, 3
E4	ScaledGD ( Tong et al., 2022 )	Nonconvex Precond.	Alg. 2, 3

Table 2: Evaluated estimators and applicable algorithmic components (ADMM: alternating direction method of multipliers; ALS: alternating least squares).

Figure 1: Trajectory exactness under lockstep execution: relative iterate discrepancies $\|D(z^{t})-\mathcal{X}^{t}\|_{F}/\|\mathcal{X}^{t}\|_{F}$ over iterations $t\leq 20$ for $\kappa\in\{1,100\}$ ; the dashed line marks the $10^{-10}$ threshold.

Figure 2: Empirical per-iteration runtime (left, bars) and resident state memory (right, markers) of the best component configuration against Raw at

n=100

; full ablation stages and

n=50

results are in Table 7 .

Estimator	Runtime (s)	Speed-up	Memory (MB)	Reduction
E0 Subspace Norm	.1366 $\to$ .0873	1.57 $\times$	66.81 $\to$ 15.32	4.36 $\times$
E1 SNN / HaLRTC	.5400 $\to$ .5849	0.92 $\times$	53.41 $\to$ 10.09	5.29 $\times$
E2 Frank–Wolfe (FFW)	.2374 $\to$ .2271	1.05 $\times$	16.41 $\to$ 1.54	10.68 $\times$
E3 HOOI	.0386 $\to$ .0124	3.12 $\times$	7.64 $\to$ .0124	616.38 $\times$
E4 ScaledGD	.0330 $\to$ .0195	1.69 $\times$	15.27 $\to$ .3933	38.83 $\times$

Table 3: Per-estimator runtime (median per iteration) and persistent memory relative to Raw at

n=100

on order-3 tensors (

p=0.05

for completion); the best lightweight configuration is selected independently for runtime and memory (more sizes in Appendix E ).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Paradigm	FLOP / Iteration	Peak Memory	Algebraic Loss	$\kappa$ -Sensitivity
Naive Ambient	$\Theta(Nn^{N+1})$	$\Theta(n^{N})$	None (Exact)	Unchanged
Randomized Sketching	$O(\|\Omega\|+Nnd^{2})$	$O(Nnd+d^{N})$	Approximate	Unchanged
Riemannian / ScaledGD	$O(\|\Omega\|r^{N}+Nnr^{2})$	$O(n^{N})$ or $O(Nnr+r^{N})$	None (Exact)	Invariant
Proposed Method	$O(q(q+r)n^{N-1}+(q+r^{N})\|\Omega\|)$	state $O(Nnr+r^{N}+\|\Omega\|)$ ; peak $+\,O(qn^{N-1})$	None (Exact)	Invariant

Appendix

Table 4: Analytical comparison of arithmetic, spatial, and optimization properties across computing paradigms.

Estimator	Config	Tol Mode	$\kappa$	$\max_{t}\mathrm{err}_{\mathrm{exact}}(t)$	$\max_{t}\|\Delta\mathcal{F}\|$	Status
Subspace Norm (E0)	All	Tight ( $10^{-14}$ )	1	$3.63\times 10^{-14}$	$9.38\times 10^{-14}$	Pass
Subspace Norm (E0)	All	Default ( $10^{-10}$ )	1	$3.62\times 10^{-14}$	$9.33\times 10^{-14}$	Pass
Subspace Norm (E0)	All	Tight ( $10^{-14}$ )	100	$3.25\times 10^{-14}$	$5.34\times 10^{-14}$	Pass
Subspace Norm (E0)	All	Default ( $10^{-10}$ )	100	$2.05\times 10^{-13}$	$5.77\times 10^{-14}$	Pass
SNN / Latent (E1)	+Alg 1+Alg 3	Tight ( $10^{-14}$ )	1	$7.65\times 10^{-15}$	$2.71\times 10^{-15}$	Pass
SNN / Latent (E1)	+Alg 1+Alg 3	Default ( $10^{-10}$ )	1	$6.86\times 10^{-15}$	$2.71\times 10^{-15}$	Pass

Appendix

Table 5: Empirical trajectory exactness and objective gaps under lockstep execution (

T=20

n=30

r=5

p=0.3

Estimator	Runtime at $n=40$ (s)		Runtime at $n=120$ (s)		Empirical Slope		Slope Change
	Raw	All	Raw	All	Raw	All	( $\Delta$ )
E0	0.0101	0.0198	0.2289	0.1539	2.829	1.858	$-0.971$
E1	0.0241	0.0429	1.1991	1.7835	3.618	3.414	$-0.204$
E2	0.0113	0.0190	0.5412	0.3220	3.567	2.584	$-0.983$
E3	0.0045	0.0060	0.0628	0.0234	2.417	1.246	$-1.171$
E4	0.0016	0.0022	0.0541	0.0314	3.215	2.401	$-0.814$

Appendix

Table 6: Empirical scaling profiles on order-3 tensors (

n\in\{40,60,80,120\}

r=5

p=0.05

). Slopes reflect least-squares fits on

\log(\text{runtime})

versus

\log(n)

. Terminal relative error

\mathrm{rel\_err\_final}

matches identically between Raw and All up to numerical precision across all dimensions.

Estimator	Config	Time $(n=50)$	Mem $(n=50)$	Time $(n=100)$	Mem $(n=100)$
Subspace Norm (E0)	Raw	$0.0155\,\mathrm{s}$	$9.09\,\mathrm{MB}$	$0.1369\,\mathrm{s}$	$66.81\,\mathrm{MB}$
	+Alg 1	$0.0284\,\mathrm{s}$	$9.09\,\mathrm{MB}$	$0.1472\,\mathrm{s}$	$66.81\,\mathrm{MB}$
	+Alg 1+Alg 2	$0.0268\,\mathrm{s}$	$7.66\,\mathrm{MB}$	$0.1387\,\mathrm{s}$	$61.09\,\mathrm{MB}$
	+Alg 1+Alg 3	$0.0260\,\mathrm{s}$	$3.37\,\mathrm{MB}$	$0.1117\,\mathrm{s}$	$21.04\,\mathrm{MB}$
	+Alg 2+Alg 3	$0.0114\,\mathrm{s}$	$1.94\,\mathrm{MB}$	$0.0874\,\mathrm{s}$	$15.32\,\mathrm{MB}$
	All (+Alg 1+Alg 2+Alg 3)	$0.0251\,\mathrm{s}$	$1.94\,\mathrm{MB}$	$0.0984\,\mathrm{s}$	$15.32\,\mathrm{MB}$

Appendix

Table 7: Ablation benchmark profiles reporting per-iteration runtime and persistent state memory (

r=5

p=0.05

\kappa=1

T=8

Primitive	Order	$n$	Config. / Parameter	Full / Ambient	Lazy / Compact	Ratio / Speedup
SVT (Alg 1, Dense)	2	50	$s=1$	$0.0072\,\mathrm{s}$	$0.0232\,\mathrm{s}$	$0.311\times$
SVT (Alg 1, Dense)	2	100	$s=5$	$0.0714\,\mathrm{s}$	$0.3297\,\mathrm{s}$	$0.217\times$
SVT (Alg 1, Operator)	2	50	$s=1$	$0.0682\,\mathrm{s}$	$0.0169\,\mathrm{s}$	$4.031\times$
SVT (Alg 1, Operator)	2	100	$s=1$	$1.5523\,\mathrm{s}$	$0.1293\,\mathrm{s}$	$12.002\times$
SVT (Alg 1, Operator)	2	100	$s=5$	$1.6554\,\mathrm{s}$	$0.1412\,\mathrm{s}$	$11.728\times$
TTM (Alg 2, Contraction)	3	100	$r=10$	$0.0194\,\mathrm{s}$ ( $7.63\,\mathrm{MB}$ )	$0.0134\,\mathrm{s}$	$1.453\times$

Appendix

Table 8: Micro-benchmark measurements evaluating individual computational primitives.

Estimator	Regime	Setting	Runtime Raw $\to$ Light. (s)	Config.	Memory Raw $\to$ Light. (MB)	Config.	Reduction
E0 Subspace Norm	Denoise	$50,5$	.0155 $\to$ .0114	A2+A3	9.09 $\to$ 1.94	A2+A3	4.69 $\times$
E0 Subspace Norm	Denoise	$100,5$	.1366 $\to$ .0873	A2+A3	66.81 $\to$ 15.32	A2+A3	4.36 $\times$
E0 Subspace Norm	Denoise ( $N=3$ )	$120,5$	.2289 $\to$ .1539	All	113.78 $\to$ 26.44	All	4.30 $\times$
E0 Subspace Norm	Denoise ( $N=4$ )	$24,3$	.0690 $\to$ .0590	All	36.72 $\to$ 5.08	All	7.23 $\times$
E1 Latent norm	$p=.05$	$50,5$	.0386 $\to$ .0698	A1	6.68 $\to$ .8262	A1+A3	8.08 $\times$
E1 Latent norm	$p=.10$	$50,5$	.0389 $\to$ .0637	A1	6.68 $\to$ .7569	A1+A3	8.82 $\times$

Appendix

Table 9: Per-estimator cost of the best lightweight configuration relative to Raw (medium preset, MATLAB R2025b, single thread). Runtime is the median time per iteration. Configurations are selected independently for runtime and memory.

Low-Rank Representation Learning

See all 16 papers

Mar 11, 2026

A New Tensor Network: Tubal Tensor Train and Its Applications

We introduce the tubal tensor train (TTT) decomposition, a tensor-network model that combines the t-product algebra of the tensor singular value decomposition (T-SVD) with the low-order core structure of the tensor train (TT) format. For an order-

(N+1)

tensor with a distinguished tube mode, the proposed representation consists of two third-order boundary cores and

N-2

fourth-order interior cores linked through the t-product. As a result, for bounded tubal ranks, the storage scales linearly with the number of modes, in contrast to direct high-order extensions of T-SVD. We present two computational strategies: a sequential fixed-rank construction, called TTT-SVD, and a Fourier-slice alternating scheme based on the alternating two-cores update (ATCU). We also state a TT-SVD-type error bound for TTT-SVD and illustrate the practical performance of the proposed model on image compression, video compression, tensor completion, and hyperspectral imaging.

Salman Ahmadi-Asl, Valentin Leplat, Anh-Huy Phan +1

Lab of Machine Learning and Knowledge Representation, Innopolis University, Innopolis, Russia · Innopolis University, Innopolis, Russia · Skolkovo Institute of Science and Technology, Center for Artificial Intelligence Technology, Moscow, Russia +1

Tensor Decomposition TT Decomposition Low-Rank Compression

Ask PDF

Nov 7, 2026

Representing Tensors as Compositions for Efficient Computation on Sparse and Structured Data

We propose a new mathematical formalism in which all kinds of tensors and of operations on those tensors are represented as simple function compositions. Tensors are functions from indices to values. An element-wise operation (e.g. an activation function) is a composition with a function applied to values; re-indexing (e.g. reshape, transposition) is a composition with a function applied to indices; and a reduction is a composition with both an index function, mapping each index to several indices, and a reduction operation (e.g. product, sum, max). Combined with tensor stacking, these compositions can represent most interesting kinds of tensors (e.g. dense, jagged, all kinds of sparsity patterns, matrix factorizations), as well as lazy versions of almost every operation available in modern tensor computation frameworks, including einsums between tensors of any kind. Identities between compositions can then be used to rewrite such expressions into equivalent ones, without evaluating them. For instance, the product of two sparse matrices can be rewritten so that only products between non-zero values are computed, turning a sparse problem into a dense one. Using this algebra as the intermediate representation of a tensor computation engine would shift the focus from defining operations on tensors (e.g. einsum between sparse tensors) to defining an algebra on index functions. This should reduce the codebase complexity from

\mathcal{O}(mn^2)

, with

m

the number of binary operations between tensors and

n

the number of tensor types, to

\mathcal{O}(m + k^2)

, with

k

the number of types of index functions. A draft implementation of a tensor computation framework based on this algebra is publicly available.

Valérian Rey, Praneth Paruchuri

SimplexLab

Topic pending

Ask PDF

Jun 7, 2026

EinSort: Sorting is All We Need for Tensorizing LLM

Tensor networks provide efficient representations for compressing large neural networks. By carefully designing shapes and topologies, they can significantly reduce memory and computational costs. However, identifying implicit low-rank structures in large foundation models remains challenging due to their enormous scale and un-structured weight distributions. We propose an adaptive tensorization method that discovers inherent low-rank structure in a target tensor by index ordering. Experiments on weight and KV-cache compression demonstrate improved reconstruction quality compared to baselines.

Toshiaki Koike-Akino, Jing Liu, Ye Wang

Mitsubishi Electric Research Laboratories (MERL), 20 Broadway, Cambridge, MA 02139, USA.

Tensor Networks LLM Compression Low-Rank Compression Low-Rank Approximation

Ask PDF

Abstract

Low-Rank Representation Learning

Explore similar work

A New Tensor Network: Tubal Tensor Train and Its Applications

Representing Tensors as Compositions for Efficient Computation on Sparse and Structured Data

EinSort: Sorting is All We Need for Tensorizing LLM