On The Application of Linear Attention in Multimodal Transformers

Armin Gerami; Seyedehanita Madani; Ramani Duraiswami

cs.CVApr 11, 2026

On The Application of Linear Attention in Multimodal Transformers

Authors: Armin Gerami, Seyedehanita Madani, Ramani Duraiswami

Organizations: University of Maryland, Department of Computer Science and UMIACS · Johns Hopkins University, Department of Electrical and Computer Engineering

Abstract

Multimodal Transformers serve as the backbone for state-of-the-art vision-language models, yet their quadratic attention complexity remains a critical barrier to scalability. In this work, we investigate the viability of Linear Attention (LA) as a high-efficiency alternative within multimodal frameworks. By integrating LA, we reduce the computational overhead from quadratic to linear relative to sequence length while preserving competitive performance. We evaluate our approach across ViT-S/16, ViT-B/16, and ViT-L/16 architectures trained on the LAION-400M dataset, with validation focused on ImageNet-21K zero-shot accuracy. Our systematic evaluation demonstrates that Linear Attention not only yields significant computational savings but also adheres to the same scaling laws as standard softmax attention. These findings position Linear Attention as a robust, scalable solution for next-generation multimodal Transformers tasked with processing increasingly large and complex datasets.

Figures & tables

Figure 1: Heatmap visualization of attention scores $a_{ij}$ of Eq. 4 and 7 . As $i$ increases, the attention scores of Eq. 4 smooth-out, making it difficult for the model to distinguish relevant query and key pairs. The query and key are random matrices

Figure 2: Forward and backward pass time for a single attention layer with 8 heads, 64 dimensions per head, and batch size of 4 on an H200 GPU.

Figure 3: Training metrics for ViT-S/16, ViT-B/16, and ViT-L/16 models trained on LAION-400M using four NVIDIA A5500 GPUs. Linear attention with our adjustments achieves performance comparable to standard attention.

Model	Param	Layers	Width	Heads
ViT-S-16	22M	12	384	6
ViT-B-16	86M	12	512	8
ViT-L-16	304M	24	768	12

Table 1: Specifications of the models used in our study.

Figure 4: Training loss scaling with model size for ViTs on LAION-400M. Our adjusted LA follows a trend similar to standard attention.

Linear Attention

Linear attention reduces sequence-length costs by replacing pairwise token interactions with kernel features, recurrent states, or compact memory.

See all 129 papers

Exact Linear Attention
May 13, 2026Weinuo OuEfficient Transformer Inference Self-Attention
Wuyi University
Scaling Native Multimodal Pre-Training From Scratch
Jul 24, 2026Haoyuan Wu, Aoqi Wu, Hai Wang +3Cross-Modal Learning Multimodal Pretraining
The Chinese University of Hong Kong · LLM Department, Tencent
LT2: Linear-Time Looped Transformers
May 20, 2026Chunyuan Deng, Yizhe Zhang, Rui-Jie Zhu +4Efficient Transformer Inference Recurrent Transformers
1Rice University · 2Apple · 3UC Santa Cruz +1

May 13, 2026J/K move · Enter open · S save

Exact Linear Attention

Weinuo Ou

Wuyi University

Efficient Transformer Inference Self-Attention Linear Attention

Open paper AskPDF arXiv

This paper introduces Exact Linear Attention (ELA), a mechanism that achieves linear computational complexity for Transformer attention by exploiting the exact decomposition property of kernel functions, thereby eliminating approximation error. We identify and address two key limitations of prior linear attention -- gradient explosion and token attention dilution -- by imposing kernel constraints that ensure non-negativity, discriminability, and geometric interpretability. Several kernel functions are proposed, including the Hadamard Exp Kernel, Summation Squared Euclidean Distance Kernel, and Subtraction Squared Euclidean Distance Kernel, each tailored for specific attention behaviors. Beyond the core attention formulation, the paper presents three engineering innovations: (1) a Hyper-Link structure that replaces traditional residual connections to mitigate gradient degradation; (2) a Memory Lobe module based on bidirectional linear attention, which captures "transformation flow" across layers to implement qualitative memory and an implicit reinforcement learning paradigm; and (3) a routing-score-based bias mechanism for Mixture-of-Experts (MoE) to improve interpretability and semantic alignment. Experimental results demonstrate that ELA achieves up to 6x faster decoding speed and 75% reduction in KV cache memory usage compared to full attention, while maintaining comparable or superior training performance. The proposed memory module accelerates convergence and enhances generalization. Furthermore, we extend the linear attention principle to vision models, yielding YOLO-LAT, which attains up to 4.3x GPU inference speedup and 7.9x parameter reduction with competitive detection accuracy. These results underline the broad applicability of exact linear attention for scaling Transformer models to ultra-long sequences and efficient visual tasks.

Abstract

Linear Attention

Explore similar work

Exact Linear Attention

Scaling Native Multimodal Pre-Training From Scratch

LT2: Linear-Time Looped Transformers