Saccade Attention Networks: Using Transfer Learning of Attention to Reduce Network Sizes

Marc Estafanous

cs.CVApr 13, 2026

Saccade Attention Networks: Using Transfer Learning of Attention to Reduce Network Sizes

Authors: Marc Estafanous

Organizations: Johns Hopkins University · Neurobaby Corporation

Abstract

One of the limitations of transformer networks is the sequence length due to the quadratic nature of the attention matrix. Classical self attention uses the entire sequence length, however, the actual attention being used is sparse. Humans use a form of sparse attention when analyzing an image or scene called saccades. Focusing on key features greatly reduces computation time. By using a network (Saccade Attention Network) to learn where to attend from a large pre-trained model, we can use it to pre-process images and greatly reduce network size by reducing the input sequence length to just the key features being attended to. Our results indicate that you can reduce calculations by close to 80% and produce similar results.

Explore similar work

Scaling Attention via Feature Sparsity
Mar 17, 2026Yan Xie, Tiansheng Wen, Tangda Huang +4Transformer Attention Efficient Attention
School of Electronic Engineering, Xidian University · Stony Brook University · TUM +2
A Controlled Study of Attention-Only Transformers
Jul 20, 2026Henry Ndubuaku, Karen Mosoyan, Jakub Mroz +5Transformer FFNs Transformer
Cactus Compute, Inc.
Why Attend to Everything? Focus is the Key
Mar 12, 2026Hengshuai Yao, Xing Chen, Ahmed Murtadha +8Attention Mechanisms Efficient Attention
Sapient Intelligence · University of Alberta · Tsinghua University

Mar 17, 2026J/K move · Enter open · S save

Scaling Attention via Feature Sparsity

Yan Xie, Tiansheng Wen, Tangda Huang, Bo Chen+3

School of Electronic Engineering, Xidian University · Stony Brook University · TUM+2

Transformer Attention Efficient Attention Sparse Attention Long-Context Modeling

Open paper AskPDF arXiv

Scaling Transformers to ultra-long contexts is bottlenecked by the

O(n^2 d)

cost of self-attention. Existing methods reduce this cost along the sequence axis through local windows, kernel approximations, or token-level sparsity, but these approaches consistently degrade accuracy. In this paper, we instead explore an orthogonal axis: feature sparsity. We propose Sparse Feature Attention (SFA), where queries and keys are represented as

k

-sparse codes that preserve high-dimensional expressivity while reducing the cost of attention from

Θ(n^2 d)

Θ(n^2 k^2/d)

. To make this efficient at scale, we introduce FlashSFA, an IO-aware kernel that extends FlashAttention to operate directly on sparse overlaps without materializing dense score matrices. Across GPT-2 and Qwen3 pretraining, SFA matches dense baselines while improving speed by up to

2.5\times

and reducing FLOPs and KV-cache by nearly 50%. On synthetic and downstream benchmarks, SFA preserves retrieval accuracy and robustness at long contexts, outperforming short-embedding baselines that collapse feature diversity. These results establish feature-level sparsity as a complementary and underexplored axis for efficient attention, enabling Transformers to scale to orders-of-magnitude longer contexts with minimal quality loss. Code is available at https://github.com/YannX1e/Sparse-Feature-Attention.

Abstract

Explore similar work

Scaling Attention via Feature Sparsity

A Controlled Study of Attention-Only Transformers

Why Attend to Everything? Focus is the Key