cs.LGNov 18, 2023

Deep Tensor Network

Authors: Yifan Zhang

Abstract

The Deep Tensor Network is a family of sequence operators built from contractions of queries, keys, and values. Tensor Attention mixes tokens through the Gram operators Q(K⊤K)Q⊤Q(K^\top K)Q^\top and K(Q⊤Q)K⊤K(Q^\top Q)K^\top; Tensor Interaction mixes channels through operators derived from C=Q⊤KC=Q^\top K. Associativity lets us evaluate these products exactly without forming n×nn\times n token matrices. For query/key width dd and value width dvd_v, the Gram constructions use O(d2+ddv)O(d^2+dd_v) work and summary state per causal token, or O(d2)O(d^2) when dv=O(d)d_v=O(d). We derive prefix recurrences and distinguish trace, diagonal, and row normalization, including the conditions under which the weights are nonnegative and row-stochastic. The unmasked token operators are positive semidefinite with rank at most dd, but their entries can be negative. We also give exact evaluations and costs for Hadamard and matrix-exponential variants.

Explore similar work

CardsList
  1. Automatically Differentiable Nonlinear Tensor Networks (ADNTNs) for Exponential Parameter Compression of Deep Neural Networks

    May 28, 2026Andrzej Cichocki, Michal WietczakTensor NetworksNeural Network Compression

  2. Graphical einops: bridging tensor networks and computation graphs

    May 29, 2026Vincent Wang-Maścianica, Nikhil KhatriTensor Networks