cs.LGNov 18, 2023

Deep Tensor Network

Authors: Yifan Zhang

Abstract

The Deep Tensor Network is a family of sequence operators built from contractions of queries, keys, and values. Tensor Attention mixes tokens through the Gram operators Q(K⊤K)Q⊤Q(K^\top K)Q^\top and K(Q⊤Q)K⊤K(Q^\top Q)K^\top; Tensor Interaction mixes channels through operators derived from C=Q⊤KC=Q^\top K. Associativity lets us evaluate these products exactly without forming n×nn\times n token matrices. For query/key width dd and value width dvd_v, the Gram constructions use O(d2+ddv)O(d^2+dd_v) work and summary state per causal token, or O(d2)O(d^2) when dv=O(d)d_v=O(d). We derive prefix recurrences and distinguish trace, diagonal, and row normalization, including the conditions under which the weights are nonnegative and row-stochastic. The unmasked token operators are positive semidefinite with rank at most dd, but their entries can be negative. We also give exact evaluations and costs for Hadamard and matrix-exponential variants.

Explore similar work

May 28, 2026

Automatically Differentiable Nonlinear Tensor Networks (ADNTNs) for Exponential Parameter Compression of Deep Neural Networks

Large deep neural networks are costly to store and deploy because inference must move and evaluate many parameters. This paper studies Automatically Differentiable Nonlinear Tensor Networks (ADNTNs), compact differentiable weight generators for replacing selected dense, convolutional, and attention layers. An ADNTN maps a small set of trainable tensor cores to a full weight tensor through hierarchical contractions and learnable nonlinearities; the generated layer is then used as an ordinary linear or convolutional operator. We investigate three multilayered topologies: Tree Tensor Networks, augmented Tree Tensor Networks with boundary disentanglers, and MERA-style multi-scale decoders. Compared with flat brick-wall automatically differentiable tensor networks, these hierarchies provide logarithmic-depth communication between tensorised modes and optional lateral mixing, which can improve long-range structure without large increases in stored parameters. We give a unified forward--adjoint formulation showing how reverse-mode automatic differentiation computes pre-activation adjoints and contracted-environment gradients for all trainable cores. The formulation supports task losses, reconstruction losses, distillation, quantisation-aware terms, batching, and modern optimisers. Proof-of-concept experiments on selected AlexNet and VGG-16 layers on CIFAR-10 datasets achieve per-layer parameter-compression ratios from about 2,000×2{,}000\times to 430,000×430{,}000\times. Several VGG-16 compressed models match or slightly exceed the dense baseline, whereas AlexNet shows moderate degradation under more restrictive redundancy. These results indicate that nonlinear tensor-network generators are a promising structured route to compact pattern-recognition models, while also showing that contraction schedules and hardware-aware implementations remain essential for practical speedups.
May 29, 2026

Graphical einops: bridging tensor networks and computation graphs

Architecture diagrams are ubiquitous in deep learning, but they are usually only representational: the tensor-program identities they suggest are still proved by prose and tensor-axis manipulation. We introduce a formal graphical calculus for the structural fragment of tensor programming underlying einops, making such diagrams proof-enabling. Our calculus represents tensor axes as nested graded tubes around a base type. The tube boundary recovers the undirected tensor-network view of axes, while the directed interior retains the operational reading of computation graphs. The key rewrite is grade-naturality: sliding spectacles over tubes. Standard equivariance proofs become short diagrammatic derivations. We additionally demonstrate how our rewrite system may be applied to convert attention masks into pre-processing operations, recovering efficient implementations of sparse attention blocks.
Sep 11, 2026

RunningTensor: Generalizing Linear Attention to Higher-Order Recurrent States

Linear attention and state-space models provide linear-time sequence modeling, but their recurrent memory remains a second-order tensor (a matrix), limiting the order of interactions that can be represented in the state. We introduce the RunningTensor, which generalizes this memory to an order-oo tensor, updated by a rank-1 outer product and read by contracting against o−1o-1 vector queries. Order 22 recovers linear attention; we study order 33 as a proof of concept, retaining both recurrent and parallel forms while remaining linear in sequence length TT and improving working memory capacity from O(W2)\mathcal{O}(W^2) to O(Wo)\mathcal{O}(W^o). On synthetic multi-query associative recall, RunningTensor outperforms linear-attention and SSM baselines. After pretraining, it also improves performance on language-understanding and non-synthetic retrieval tasks, suggesting that higher-order recurrent state can provide useful additional memory capacity beyond matrix-valued state.