Organizations: City University of Hong Kong, Tat Chee Avenue 83, Hong Kong, 999077, China · Columbia University, 116th St & Broadway, New York, 10027, NY, USA
High-dimensional tensors are ubiquitous in real-world applications but lead to massive storage overhead. Functional tensor decomposition addresses this by employing neural networks to map element coordinates to tensor cores, significantly reducing the parameters for data compression. However, existing functional methods typically flatten high-order tensors into 1D sequences. This folding operation causes highly overlapping local coordinate sequences and struggles to capture complex cross-mode spatial dependencies, severely limiting their expressiveness on diverse datasets. To tackle these challenges, we propose TensorSSM, a novel Implicit Neural Representation (INR) for functional decomposition method that memorizes spatial positions within a dynamic State Space Model (SSM). Specifically, TensorSSM first decomposes mode-wise coordinates into sequences spit by K-ary rule, encoding spatial tokens from coarse to fine, which effectively captures spatial variations. Furthermore, we extend the SSM's state space to explicitly map cross-mode spatial dependencies, allowing inter-mode interactions to enhance the updates of each mode. By integrating cross-mode information from previous hidden states into the current state transitions, TensorSSM effectively expands tensor representation along two dimensions: the sequence dimension for coarse-to-fine spatial tokens, and the state dimension for cross-mode memorization. This design effectively integrates spatial information into the state space and the tensor core, substantially advancing the state adaptability and enhancing the expressiveness of the data compression. Extensive experiments demonstrate that TensorSSM achieves significantly high reconstruction accuracy and storage efficiency, exhibiting robust expressiveness across diverse tensor datasets without relying on strict spatial assumptions.
Figures & tables
Figure 1: (a). Adjacent tensor entries usually have highly overlapping raw mode indices or sequences from folded mode indices, leading to the same or similar tensor core representations, which is only effective for local similar entries. (b). Entries with shared mode indices may have similar representations, but their true values may be totally different from each other.
Figure 2: (a). Framework Overview of TensorSSM via Functional Tensor Decomposition. (b). The index of each mode is decomposed via base- b rule and form a coarse-to-fine sequence as the spatial token. (c). The cross-mode information is spanned to the state transition operator, selectively influencing the spatial dependencies in the tensor core representations.
Dataset
Size
Order
Density
Smoothness
Uber
183×24×1,140
3
0.138
0.861
Action
100×570×567
3
0.393
0.484
Air
5,600×362×6
3
0.917
0.513
PEMS-SF
963×144×440
3
0.999
0.461
Activity
337×570×320
3
0.569
0.553
NYC
265×265×28×35
4
0.118
0.788
Table 1: Statistics of the datasets used in our experiments.
Dataset
Metric
CPD
TTD
NeuKron
TRD
SZ3
TensorCodec
TensorSSM
Improve (%)
Uber
CS (bytes)
156,368
153,468
155,154
157,824
646,537
153,766
152,720
2.55
Fitness
0.7289
0.7355
0.6925
0.6409
0.7382
0.7380
0.7570 ∗
Action
CS (bytes)
153,512
167,264
153,388
153,760
4,980,510
155,558
152,624
12.36
Fitness
0.7725
0.6767
0.3767
0.7716
0.7493
0.7542
0.8680 ∗
Air
CS (bytes)
143,256
148,320
149,891
158,416
1,592,014
160,999
152,720
7.34
Fitness
0.6775
0.6794
0.6580
0.5880
0.6761
0.6762
0.7293 ∗
Table 2: Comparison of compressed size and Fitness on six datasets, where CS denotes compressed size. For CPD, TTD, TRD, NeuKron, and TensorSSM, we report the available configuration whose compressed size is closest to that of TensorCodec. For SZ3, we report the operating point whose Fitness is closest to that of TensorCodec. The best results are in bold and the second best are underlined. Improve denotes the relative Fitness gain of TensorSSM over the strongest competing method on each dataset. “ ∗ ” indicates that the improvements over baselines are statistically significant.
Figure 3: Training Fitness curves of TensorSSM, TensorCodec, and NeuKron on six datasets. For Uber, Air Quality, PEMS-SF, and NYC, the NeuKron curves are displayed up to the larger final epoch of TensorSSM and TensorCodec.
Figure 4: Effect of the numeral base on the Fitness of TensorSSM on the Air Quality and Action datasets.
Figure 5: Fitness of TensorSSM and TensorCodec with different reconstruction heads on the Action dataset.
Variant
Fitness
Full TensorSSM
0.8680
Self-only FiLM
0.8304
Self-only Fusion
0.8390
w/o Pairwise Interaction
0.8530
w/o SSM
0.7778
w/o Coarse-to-Fine
0.6819
Table 3: Ablation results of TensorSSM on the Action dataset. Self-only FiLM and self-only fusion remove information from the other tensor modes in the corresponding modules.
Figure 6: Fitness comparison between TensorSSM and TensorCodec under matched hidden sizes. The value of H below each dataset denotes the hidden size shared by both models.
Figure 7: Intensity of state transitions from the perspectives of entries and states. (a). Spatial modulation of the first-layer dynamic transition matrices on Action with i1=50 fixed. The axes correspond to sampled mode-2 and mode-3 coordinates. (b). Entry-wise deviation of the first-layer dynamic transition matrices over the coordinate grid in (a). Rows and columns correspond to the two state dimensions, respectively.
Figure 8: Raw transition coefficients at the hundreds-digit scale in the first SSM layer, with i1=50 . Panel (r,c) corresponds to matrix row r and column c . Each gray curve fixes one of 64 sampled mode-3 coordinates and varies over 64 sampled mode-2 coordinates. The blue line shows the slice mean, and the shaded region spans the slice minimum and maximum.
Figure 9: Raw transition coefficients at the tens-digit scale. Coordinates, layer, channel, curve definitions, and entry-wise vertical limits follow Figure 8 . Slices with identical same-scale digits at the same panel can produce overlapping curves.
Figure 10: Raw transition coefficients at the units-digit scale. Coordinates, layer, channel, curve definitions, and entry-wise vertical limits follow Figure 8 . Slices with identical same-scale digits at the same panel can produce overlapping curves.
Figure 11: Cross-mode sensitivity of the SSM outputs at the sampled entry (94,279,330) of the Action dataset. Each cell reports the Frobenius norm Sa,b of the channel Jacobian block between the input embedding at position b (columns) and the final scan output at position a (rows), where each position is indexed by mode (M1–M3) and digit scale (100s, 10s, 1s). Measurements are taken before the final channel mixing and FiLM, and both panels share the same color scale. (a). The full TensorSSM exhibits nonzero off-diagonal mode blocks, indicating that each SSM stream responds to the other modes. (b). In the self-only fusion variant, all off-diagonal mode blocks are exactly zero.
We propose a scalable tensorization framework for neural network compression based on slice-wise feature distillation. Unlike conventional tensor decomposition methods that rely on costly global finetuning, our approach decomposes the network into slices consisting of either individual layers or blocks (e.g., convolutional layers or MLPs), or small groups of consecutive layers, and tensorizes each slice independently to reproduce the intermediate representations of the original pretrained model. This modular strategy improves accuracy recovery, reduces data requirements, and enables efficient parallel optimization. Experiments on ResNet-34 show significant gains over conventional global tensorization, achieving near-lossless compression at moderate compression rates with faster optimization. Results on GPT-2 XL further demonstrate the scalability of the method and its applicability to large-scale models, particularly in distributed settings.
Safa Hamreras, Sukhbinder Singh, Román Orús
Donostia International Physics Center, Paseo Manuel de Lardizabal 4, E-20018 San Sebastián, Spain · Multiverse Computing, Spadina Ave., Toronto, ON M5T 2C2, Canada · Multiverse Computing, Paseo de Miramón 170, E-20014 San Sebastián, Spain +1
We introduce the tubal tensor train (TTT) decomposition, a tensor-network model that combines the t-product algebra of the tensor singular value decomposition (T-SVD) with the low-order core structure of the tensor train (TT) format. For an order-(N+1) tensor with a distinguished tube mode, the proposed representation consists of two third-order boundary cores and N−2 fourth-order interior cores linked through the t-product. As a result, for bounded tubal ranks, the storage scales linearly with the number of modes, in contrast to direct high-order extensions of T-SVD. We present two computational strategies: a sequential fixed-rank construction, called TTT-SVD, and a Fourier-slice alternating scheme based on the alternating two-cores update (ATCU). We also state a TT-SVD-type error bound for TTT-SVD and illustrate the practical performance of the proposed model on image compression, video compression, tensor completion, and hyperspectral imaging.
Lab of Machine Learning and Knowledge Representation, Innopolis University, Innopolis, Russia · Innopolis University, Innopolis, Russia · Skolkovo Institute of Science and Technology, Center for Artificial Intelligence Technology, Moscow, Russia +1
Selective state space models (SSMs) have rapidly become a compelling backbone for large language models, especially for long-context workloads. Yet in deployment, their inference performance is often bounded by the memory capacity, bandwidth, and latency limits of a single GPU, making multi-GPU execution increasingly necessary. Although tensor parallelism (TP) is widely used to scale Transformer inference, applying it to selective SSM blocks is non-trivial because the SSM mixer couples large projections with a sequence-wise recurrent state update and local mixing whose efficiency depends on preserving locality and avoiding synchronization in the critical path. This paper presents a communication-efficient TP design for selective SSM inference that addresses three practical engineering challenges: enabling TTFT improvements via an SSM state cache across prefill and decode, partitioning the mixer's packed parameter tensor so that recurrent updates remain local while minimizing communication, and reducing TP aggregation overhead with quantized AllReduce. We evaluate on three representative SSM-based LLMs spanning pure-SSM and hybrid architectures - Mamba, Falcon-Mamba, and Zamba - on NVIDIA A6000 and A100 clusters. Our experiments show substantial throughput gains from tensor-parallel SSM inference, improving batch-request throughput by ~1.6-2.1x on 2 GPUs and ~2.6-4.0x on 4 GPUs for Mamba, with the largest benefits at long context lengths, and achieving a further ~10-18% throughput improvement from quantized all-reduce by lowering synchronization bandwidth overhead.