Deep Tensor Network
Abstract
The Deep Tensor Network is a family of sequence operators built from contractions of queries, keys, and values. Tensor Attention mixes tokens through the Gram operators and ; Tensor Interaction mixes channels through operators derived from . Associativity lets us evaluate these products exactly without forming token matrices. For query/key width and value width , the Gram constructions use work and summary state per causal token, or when . We derive prefix recurrences and distinguish trace, diagonal, and row normalization, including the conditions under which the weights are nonnegative and row-stochastic. The unmasked token operators are positive semidefinite with rank at most , but their entries can be negative. We also give exact evaluations and costs for Hadamard and matrix-exponential variants.
Transformer Attention
Transformer attention lets models relate tokens by weighting information from other positions, enabling context-dependent sequence and multimodal processing.
See all 343 papers