Softmax Attention

Follow this topic Ask about this topic

Recent momentum

-50%

4 papers in the last 28 days · 0.1% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-21

1 new paper

A weekly snapshot of new work published in Softmax Attention.

Period ending 2026-09-07

2 new papers

A weekly snapshot of new work published in Softmax Attention.

74 papers

Latest in Softmax Attention

Browse all topics →

VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention
Sep 14, 2026Xingyang Li, Dongyun Zou, Shining Zhang +8Dynamic Sparse Attention Pre-Trained Video Diffusion Models
Hierarchical and Permutation-Invariant Feature Transformation Learning via Policy-Guided Embedding Search
Sep 9, 2026Rui Liu, Tao Zhe, Yanyong Huang +5Tabular Learning Softmax Attention
ProbPlug: A Plugin Uncertainty Network for Reliable Confidence in LLM Binary Classification
Sep 9, 2026Jianzong Wang, Chuhang Liu, Botao Zhao +6Confidence Calibration Binary Classification
Why shared attention vectors fail: a case for outcome-indexed tuning
Sep 8, 2026Lenard DomeSoftmax Attention Gradient Descent
Lapis: Laplacian Spiking Attention via First-Spike Timing and Membrane Leakage
Aug 12, 2026Kaiwen Tang, Jiaqi Zheng, Zixuan Zhu +3Binary Spikes Softmax Attention
A Quantum Roadmap for Softmax Attention: Exact Born-Rule Analogs for Softmax Attention on the Probability Simplex
Aug 11, 2026Eric A. F. Reinhardt, Adam J. HauserSoftmax Attention Qubit
Training-Free Universal Approximation by Prompting Random Transformers
Aug 10, 2026Alexander Hsu, Rongjie LaiSoftmax Attention Approximation
Faster Query-Key Learning Sharpens Attention in Self-Attention Models
Aug 7, 2026Rahul Vashisht, Harish G. RamaswamySoftmax Attention Model Interpretability
Riemannian Attention Mechanisms for Transformers: A Theoretical Framework and Architecture Design
Aug 2, 2026Sen SongTransformer Attention Softmax Attention
ResKV: Reconstructing Omitted Attention Contributions for Fixed-Budget KV Cache Compression
Jul 31, 2026Yuhang Zhan, Lisi Chen, Shuo ShangKey-Value Cache Compression Cache
Causal Discovery with Inverted Self-attention for Multivariate Time Series
Jul 30, 2026Yusen Liu, Yong Wang, Yifan Yin +3Causal Foundation Models Time Series
Self-Attention Dynamics with Rotary Position Embeddings: Twisted States and Explicit Consensus Rates on the Sphere
Jul 27, 2026Hao YeRotary Position Embedding Softmax Attention
On the Diverse Dynamical Behaviors Arising in Deep Linear Transformers
Jul 20, 2026Sixu Li, Thomas Jacob Maranzatto, Jan Peszek +5Transformer Architectures Softmax Attention
L1 Augmented Attention as an Improved Vector Similarity Metric
Jul 20, 2026Kurt GoddenSoftmax Attention Cosine Similarity
Brain-Aligned Multi-Stream Video Transformers with Sparse Self-Selection
Jul 20, 2026Amir Hosein Fadaei, Mahyar Maleki, Mohammad-Reza A. DehaqaniVideo Understanding Transformer Architectures
In-context learning of closed form solution to simple linear regression task using transformer with linear self-attention
Jul 17, 2026Katsuyuki HagiwaraIn-Context Learning Transformer Architectures
Screening Is Effective for Visual Recognition
Jul 15, 2026Shunya Shimomura, Kazuhiro HottaVision Transformer Softmax Attention
AVQ-Attention: Adaptive Vector-Quantized Attention
Jul 14, 2026Winfried van den dool, Patrick Forré, Amir Habibian +2Softmax Attention Codebooks
Forgetful Attention: A Trainable Support-Vector Memory with Certified Selection and Exact Unlearning
Jul 13, 2026Vishwajith RameshSoftmax Attention Forgetting
Is Energy Guidance All You Need? Training-Free Norm Injection for Driving World Models
Jul 12, 2026Xiyan Su, Frank Diermeyer, Markus LienkampVideo World Models World Models
TSR-Ego: Temporally Guided Stereo Refinement Framework for Egocentric 3D Human Pose Estimation
Jul 10, 2026Md Mushfiqur Azam, John Quarles, Kevin Desai3D Pose Estimator Stereo Matching
Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing
Jul 8, 2026Tommaso Cerruti, Tim Rieder, George Rowlands +2Quadratic Attention Global Attention Mechanism
HyperVAttention: Efficient Sparse Attention with Spatio-Temporal Clustering for Video Diffusion
Jul 3, 2026Dongyeun Lee, Amir Zandieh, Vahab Mirrokni +2Dynamic Sparse Attention High-Fidelity Video Generation
Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling
Jul 3, 2026Xiang Hu, Xinyu Wei, Hao Gu +10Efficient Long-Context Inference Dynamic Sparse Attention
X-LogSMask: Expand Transformer for Graph-Structured Data
Jul 2, 2026Leyan Li, Rennong Yang, Zhenxing Zhang +1Heterogeneous Graph Transformers Transformer Attention
The risk of KV cache compression
Jul 1, 2026Lukas Haverbeck, Carmen Amo Alonso, Andres Felipe Posada-Moreno +2Key-Value Cache Compression Context Compression
Transformer Architectures as Complete Bayes Processes: A Formal Proof in the Measure-Theoretic Kernel Framework
Jun 29, 2026Haobo YangTransformer Architectures Bayesian
Depth-Staggered Fibonacci Spacing for Sparse Attention: Static Schedules Beat Learned Dilation and Extrapolate Where Dense Attention Fails
Jun 26, 2026Chad A. CappsDynamic Sparse Attention Softmax Attention
Why Do Accumulated Transformations Extrapolate?
Jun 23, 2026Mahesh GodavartiRotary Position Embedding Softmax Attention
Parallel Manifold Steering: Efficient Adaptation of Large Associative Memories via Residual Energy Shaping
Jun 23, 2026Kanishk AwadhiyaTransformer Residual Streams Dense Associative Memory
Asymptotic Signal Subspace Recovery in Softmax Attention Models
Jun 21, 2026Lan V. TruongSoftmax Attention Learning Dynamics
All Routes Lead to Collapse
Jun 21, 2026K. R. BalasubramanianSoftmax Attention Representation Collapse
FiLM-Coordinated Dual-Branch Transformer for Global-Local Dependency Modeling in Language Modeling
Jun 19, 2026Zhiqiang Zhou, Xu Ling, Junliang DaiTransformer Layers Softmax Attention
ATT-CR: Adaptive Triangular Transformer for Cloud Removal
Jun 4, 2026Yang Wu, Ye Deng, Pengna Li +4Hybrid Transformer Removal
UtVAA: Ultra-tiny Vision Transformer with Affix Attention for Mobile Image Classification
Jun 2, 2026Romiyal George, Sathiyamohan Nishankar, Selvarajah Thuseethan +1Self-Supervised Vision Transformers Mobilenetv2
Forget Attention: Importance-Aware Attention Is All You Need
Jun 1, 2026Suhyeong Shin, Yeongwook YangSoftmax Attention Transformer Architectures
Interpolation between Convolution and Attention via K-Nearest Neighbors
May 31, 2026Mingi KangConvolutional Softmax Attention
Blurry Window Attention
May 31, 2026Axel Laborieux, Christos Sourmpis, Juan Gabriel Kostelec +1Softmax Attention Kernel Method
Improving Adversarial Robustness of Attribution via Implicit Regularization
May 28, 2026Amir Mehrpanah, Matteo Gamba, Hossein AzizpourTraining Data Attribution Explainability
Multi-Mixer Models: Flexible Sequence Modeling with Shared Representations
May 27, 2026Kevin Y. Li, Asher Trockman, Ananda Theertha Suresh +1Sequence Modeling Softmax Attention
PLS in the Mirror of Self-Attention
May 27, 2026Jiangsheng, YouSoftmax Attention Least Squares
Meta-Attention: Bayesian Per-Token Routing for Efficient Transformer Inference
May 27, 2026Alan FerrariSoftmax Attention Posterior
Balancing Fidelity and Diversity in Diffusion Models via Symmetric Attention Decomposition: Hopfield Perspective
May 26, 2026Hyunmin Cho, Woo Kyoung Han, Kyong Hwan JinSoftmax Attention Hopfield Networks
Preisach Attention: A Hysteretic Model of Sequential Memory
May 22, 2026Piotr FrydrychTransformer Layers Softmax Attention
Optimising CSRNet with parameter-free attention mechanisms for crowd counting in public transport
May 18, 2026Aida Rostamza, Enrico Del Re, Joshua Cherian Varughese +1Softmax Attention Density Ratio Estimation
InfoFlow: A Framework for Multi-Layer Transformer Analysis
May 18, 2026Penghao Yu, Haotian Jiang, Zeyu Bao +1Transformer Layers Softmax Attention
Attend Locally, Remember Linearly: Linear Attention as Cross-Frame Memory for Autoregressive Video Diffusion
May 15, 2026Kunyang Li, Mubarak Shah, Yuzhang ShangAutoregressive Video Diffusion Softmax Attention
WorldParticle: Unified World Simulation of Lagrangian Particle Dynamics via Transformer
May 14, 2026Caoliwen Wang, Minghao Guo, Siyuan Chen +13Physical Dynamics Deformable Objects
CoReDiT: Spatial Coherence-Guided Token Pruning and Reconstruction for Efficient Diffusion Transformers
May 13, 2026Zhuojin Li, Hsin-Pai Cheng, Hong Cai +2Multimodal Diffusion Transformers Visual Token Pruning
OSDN: Improving Delta Rule with Provable Online Preconditioning in Linear Attention
May 13, 2026Chenyu Zhou, Hongpei Li, Yuerou Liu +3Softmax Attention Spectral Preconditioning
A Unified Framework for Critical Scaling of Inverse Temperature in Self-Attention
May 12, 2026Tomohiro Hayase, Ryo KarakidaSoftmax Attention Gumbel-Softmax Relaxation
The Routing and Filtering Structure of Attention
May 12, 2026Shafayeth Jamil, Rehan KapadiaSoftmax Attention Quadratic Attention
Breaking Global Self-Attention Bottlenecks in Transformer-based Spiking Neural Networks with Local Structure-Aware Self-Attention
May 12, 2026Lingdong Li, Hangming Zhang, Qiang YuSpiking Neural Networks Transformer Attention
Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime
May 11, 2026Albert Alcalde, Leon Bungert, Konstantin Riedl +1Transformer Encoder Transformer Architectures
Self-Attention as a Covariance Readout: A Unified View of In-Context Learning and Repetition
May 11, 2026Haoren Xu, Guanhua FangSoftmax Attention In-Context Learning
Complex-Valued Phase-Coherent Transformer
May 11, 2026Leona HiokiTransformer Attention Unified Transformer
Rethinking Random Transformers as Adaptive Sequence Smoothers for Sleep Staging
May 11, 2026Guisong Liu, Xin Gao, Martin Dresler +2Sleep Stage Classification Sleep
Nectar: Neural Estimation of Cached-Token Attention via Regression
May 10, 2026João Monteiro, Michal Klein, Pierre Ablin +1Softmax Attention Efficient Long-Context Inference
Scaling Limits of Long-Context Transformers
May 8, 2026Giuseppe Bruno, Shi Chen, Zhengjiang Lin +2Softmax Attention Context Length
Long Context Pre-Training with Lighthouse Attention
May 7, 2026Bowen Peng, Subho Ghosh, Jeffrey QuesnelleSoftmax Attention Causal Reasoning

Sep 14, 2026 · cs.CVJ/K move · Enter open · S save

VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention

Xingyang Li, Dongyun Zou, Shining Zhang, Jiacheng Chen+7

2MIT · 1Nunchux AI · 3CMU · 4UC Berkeley · 5Stanford

Diffusion Transformers deliver state-of-the-art video generation, but their long spatiotemporal sequences make attention the dominant deployment cost, and a deployable low-bit kernel must be accurate and fast. Accuracy is limited by outliers: a block's quantization scale is set by its largest entries, leaving typical entries confined to a narrow range of representable values. Prior work smooths queries and keys, but value outliers follow no fixed channel or spatiotemporal structure and remain the dominant source of output error. Speed is limited by softmax: low-bit Tensor Cores accelerate only the two matrix multiplications, so the high-precision exponential between them becomes the longest pipeline stage on datacenter GPUs. We propose VC-Attention, a training-free low-bit attention framework that addresses both by pairing Value smoothing with a fused probability Cast. V-Smooth reorders value tokens by lightweight online clustering, so the tokens in a hardware block quantize well together. It quantizes only the residual after subtracting the block mean, and restores that mean from the row sum the online softmax already maintains. ExpCast-FP8 maps log-domain scores directly to E4M3 probability codes with one fused multiply-add, eliminating the FP32 exponential and the format conversion. We implement VC-Attention for B200, B300, H200, RTX PRO 6000, and RTX 5090. Across Wan2.2, LongCat-Video, HunyuanVideo-1.5, and MiniMax-H3, VC-Attention improves fidelity over low-bit baselines, speeds up the attention kernel over BF16 FlashAttention-4 by 1.46-1.59x on datacenter Blackwell and Hopper and by 2.3-3.6x on workstation cards, and generates a clip 1.13-1.19x and 1.36-1.70x faster end to end.

Dynamic Sparse Attention Pre-Trained Video Diffusion Models Softmax Attention Tensor Programs Smoothing Activation Functions

Open paper Ask about it PDF