HSMLA: Hierarchical Softmax Multi-scale Linear Attention for Efficient Vision Transformers

Dong Liu; Yanxuan Yu; Renata Borovica-Gajic; Tong Geng; Ying Nian Wu

cs.CVAug 7, 2026

HSMLA: Hierarchical Softmax Multi-scale Linear Attention for Efficient Vision Transformers

Authors: Dong Liu, Yanxuan Yu, Renata Borovica-Gajic, Tong Geng, Ying Nian Wu

Organizations: University of California, Los Angeles, USA · Columbia University, USA · The University of Melbourne, Australia · Rice University, USA

Abstract

Vision transformers face significant computational overheads in high-resolution dense prediction due to the quadratic complexity of self-attention. Linear attention offers efficiency but sacrifices local context modeling. We propose \textbf{HSMLA (Hierarchical Softmax Multi-scale Linear Attention)}, which combines ReLU-based linear attention for global context, selective softmax refinement for critical local features, and multi-scale token representations via depthwise convolutions. HSMLA achieves superior accuracy-efficiency trade-offs: up to $4.2\times$ inference-time speedup across dense prediction tasks, $87.3%$ Dice with $3.2\times$ speedup on CT organ segmentation, and $94.2%$ AUC with $4.1\times$ speedup on pathology WSI.

Explore similar work

Full-resolution MLPs Empower Medical Dense Prediction
Nov 28, 2023Mingyuan Meng, Yuxin Xue, Dagan Feng +2Dense Prediction Hybrid Transformer
Exact Linear Attention
May 13, 2026Weinuo OuQuadratic Attention Transformer Attention
Elastic Attention Cores for Scalable Vision Transformers
May 12, 2026Alan Z. Song, Yinjie Chen, Mu Nan +8Self-Supervised Vision Transformers Vision Transformer

Nov 28, 2023 · eess.IVJ/K move · Enter open · S save

Full-resolution MLPs Empower Medical Dense Prediction

Mingyuan Meng, Yuxin Xue, Dagan Feng, Lei Bi+1

School of Computer Science, The University of Sydney, Australia. · Institute of Translational Medicine, Shanghai Jiao Tong University, China.

Dense prediction is a fundamental requirement for many medical vision tasks such as medical image restoration, registration, and segmentation. The most popular vision model, Convolutional Neural Networks (CNNs), has reached bottlenecks due to the intrinsic locality of convolution operations. Recently, transformers have been widely adopted for dense prediction for their capability to capture long-range visual dependence. However, due to the high computational complexity and large memory consumption of self-attention operations, transformers are usually used at downsampled feature resolutions. Such usage cannot effectively leverage the tissue-level textural information available only at the full image resolution. This textural information is crucial for medical dense prediction as it can differentiate the subtle human anatomy in medical images. In this study, we hypothesize that Multi-layer Perceptrons (MLPs) are superior alternatives to transformers in medical dense prediction where tissue-level details dominate the performance, as MLPs enable long-range dependence at the full image resolution. To validate our hypothesis, we develop a full-resolution hierarchical MLP framework that uses MLPs beginning from the full image resolution. We evaluate this framework with various MLP blocks on a wide range of medical dense prediction tasks including restoration, registration, and segmentation. Extensive experiments on six public well-benchmarked datasets show that, by simply using MLPs at full resolution, our framework outperforms its CNN and transformer counterparts and achieves state-of-the-art performance on various medical dense prediction tasks.

Dense Prediction Hybrid Transformer Convolutional Multilayer Perceptrons Tissue Architecture Long-Range Temporal Dependencies

Open paper Ask about it PDF

Abstract

Explore similar work

Full-resolution MLPs Empower Medical Dense Prediction

Exact Linear Attention

Elastic Attention Cores for Scalable Vision Transformers