Subspace Kernel Learning on Tensor Sequences

Lei Wang; Xi Ding; Yongsheng Gao; Piotr Koniusz

cs.LGMar 20, 2026

Subspace Kernel Learning on Tensor Sequences

Authors: Lei Wang, Xi Ding, Yongsheng Gao, Piotr Koniusz

Organizations: Griffith University · Data61 CSIRO · University of New South Wales

Abstract

Learning from structured multi-way data, represented as higher-order tensors, requires capturing complex interactions across tensor modes while remaining computationally efficient. We introduce Uncertainty-driven Kernel Tensor Learning (UKTL), a novel kernel framework for $M$ -mode tensors that compares mode-wise subspaces derived from tensor unfoldings, enabling expressive and robust similarity measure. To handle large-scale tensor data, we propose a scalable Nyström kernel linearization with dynamically learned pivot tensors obtained via soft $k$ -means clustering. A key innovation of UKTL is its uncertainty-aware subspace weighting, which adaptively down-weights unreliable mode components based on estimated confidence, improving robustness and interpretability in comparisons between input and pivot tensors. Our framework is fully end-to-end trainable and naturally incorporates both multi-way and multi-mode interactions through structured kernel compositions. Extensive evaluations on action recognition benchmarks (NTU-60, NTU-120, Kinetics-Skeleton) show that UKTL achieves state-of-the-art performance, superior generalization, and meaningful mode-wise insights. This work establishes a principled, scalable, and interpretable kernel learning paradigm for structured multi-way and multi-modal tensor sequences.

Figures & tables

Figure 1: Mode-wise factor matrices from the Tucker decomposition for the action called “draw x” . Each row shows one latent factor (from 4 leading factors), and each column corresponds to one tensor mode: temporal block , body joints , 3D coordinates , and time . Structured patterns reveal interpretable, mode-specific information which motivates our approach.

Figure 2: Overview of the proposed Uncertainty-driven Kernel Tensor Learning (UKTL) pipeline for action recognition. For brevity, we use skeletons as an example. Each skeleton sequence is divided into temporal blocks

\mathbf{B}_{1},\ldots,\mathbf{B}_{\tau}

, embedded via an MLP, and processed by a Higher-order Transformer (HoT) to obtain feature tensor

\bm{\mathcal{X}}_{i}

. These tensors undergo mode-

m

matricization (

1,\ldots,M

) and SVD to extract

M

subspaces per sample. Soft

k

-means clustering yields

C

Nyström pivots, each represented by also

M

subspaces. A Multi-mode SigmaNet (MSN) estimates uncertainty vectors over all subspaces, which are used to regularize kernel computations. The Nyström-approximated KTL maps inputs to compact, uncertainty-aware representations,

\tilde{\mathbf{g}}_{i}

for final classification. The entire model is trained end-to-end.

	Method	NTU-60		NTU-120		Kinetics-Skeleton
	Method	X-Sub(%)	X-View(%)	X-Sub(%)	X-Setup(%)	Top-1(%)	Top-5(%)
Graph-based	TCN ( Kim and Reiter, 2017 )	-	-	-	-	20.3	40.0
	ST-GCN ( Yan et al., 2018 )	81.5	88.3	70.7	73.2	30.7	52.8
	AS-GCN ( Li et al., 2019 )	86.8	94.2	78.3	79.8	34.8	56.5
	2S-AGCN ( Shi et al., 2019 )	88.5	95.1	82.5	84.2	36.1	58.7
	NAS-GCN ( Peng et al., 2020 )	89.4	95.7	-	-	37.1	60.1

Table 1: Results on NTU-60, NTU-120, and Kinetics-Skeleton. UKTL outperforms graph, hypergraph, and transformer models by using uncertainty-aware tensor kernels. All tensor methods use the same MLP + HoT backbone for fair comparison.

Modality	NTU-60		NTU-120
Modality	X-Sub (%)	X-View (%)	X-Sub (%)	X-Setup (%)
Skeleton	93.1	97.3	90.0	91.4
RGB	83.2	86.1	79.0	81.0
Depth	85.0	87.5	80.5	82.1
Skeleton + RGB	94.5	97.9	91.3	92.5
Skeleton + Depth	94.8	98.0	91.5	92.7

Table 2: Evaluation of single- and multi-modal performance on NTU-60 and NTU-120. Skeleton alone provides a strong, competitive baseline, while RGB and Depth demonstrate the framework’s flexibility across modalities. Fusing multiple modalities consistently improves accuracy, showing that UKTL effectively captures complementary information and inter-modal correlations.

Panel of Figure 3 — Figure 3: Ablation study evaluating the effects of subspace order (on NTU-60/120), and of kernel choice, Nyström pivots, and kernel composition (on NTU-60) within the UKTL framework.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Visualization of Tucker decomposition in each mode of tensor representations for action draw x and action draw tick .

Model	#Params	FPS	Memory (train)
CTR-GCN ( Zhang et al., 2025a )	$1.46$ M	$520$	$2.6$ GB
ST-TR ( Plizzari et al., 2021 )	$10$ - $12$ M	$410$	$3.2$ GB
UKTL (Ours)	$1.2$ - $1.5$ M	$450$ - $480$	$2.9$ GB

Appendix

Table 3: End-to-end efficiency comparison on NTU-60.

$\mu$	$0$	$0.25$	$0.3$	$0.4$	$0.5$	$0.6$	$0.75$	$1$
Acc (%)	$91.8$	$92.6$	$92.7$	$\mathbf{93.1}$	$\mathbf{93.1}$	$92.6$	$92.4$	$81.6$

Appendix

Table 4: Sensitivity analysis of the mixture weight

\mu

between sum and product interactions.

Nyström Method

See all 15 papers

Laplace Approximation for Bayesian Tensor Network Kernel Machines
Apr 29, 2026Albert Saiapin, Kim BatselierUncertainty Quantification Bayesian Inference
1Delft Center for Systems and Control, Delft University of Technology
Tensor Decomposition Structure Search Framework from an Interaction Perspective
Mar 3, 2026Ting-Wei Zhou, Xi-Le Zhao, Sheng Liu +3Tensor Decomposition
School of Mathematical Sciences/Research Center for Image and Vision Computing, University of Electronic Science and Technology of China, Chengdu, Sichuan 611731, China · School of Information Science and Technology, Southwest Jiaotong University, Chengdu 611756, China · School of Mathematics and Statistics and Ministry of Education Key Lab of Intelligent Networks and Network Security, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, China, and also with Pazhou Laboratory (Huangpu), Guangzhou, Guangdong 510335, China
Breaking the Periodicity Assumption: Robust Tensorial Multi-View Clustering via Graph-Spectral Low-Rank Learning
Jul 28, 2026Jintian Ji, Xingsu Li, Songhe FengRepresentation Learning Multi-View Learning
Computer Science and Engineering The Hong Kong University of Science and Technology Hong Kong, China · School of Computer and Information Technology Beijing Jiaotong University, Beijing, China

Apr 29, 2026J/K move · Enter open · S save

Laplace Approximation for Bayesian Tensor Network Kernel Machines

Albert Saiapin, Kim Batselier

1Delft Center for Systems and Control, Delft University of Technology

Uncertainty Quantification Bayesian Inference Kernel Methods

Open paper AskPDF arXiv

Uncertainty estimation is essential for robust decision-making in the presence of ambiguous or out-of-distribution inputs. Gaussian Processes (GPs) are classical kernel-based models that offer principled uncertainty quantification and perform well on small- to medium-scale datasets. Alternatively, formulating the weight space learning problem under tensor network assumptions yields scalable tensor network kernel machines. However, these assumptions break Gaussianity, complicating standard probabilistic inference. This raises a fundamental question: how can tensor network kernel machines provide principled uncertainty estimates? We propose a novel Bayesian Tensor Network Kernel Machine (LA-TNKM) that employs a (linearized) Laplace approximation for Bayesian inference. A comprehensive set of numerical experiments shows that the proposed method consistently matches or surpasses Gaussian Processes and Bayesian Neural Networks (BNNs) across diverse UCI regression benchmarks, highlighting both its effectiveness and practical relevance.

Abstract

Nyström Method

Explore similar work

Laplace Approximation for Bayesian Tensor Network Kernel Machines

Tensor Decomposition Structure Search Framework from an Interaction Perspective

Breaking the Periodicity Assumption: Robust Tensorial Multi-View Clustering via Graph-Spectral Low-Rank Learning