Attention Mechanisms

Follow this topic Ask about this topic

Momentum

42 papers in the last four weeks, up 100% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 378

All topics

VGGT-Occ: Geometry-Grounded and Density-Aware Gated Fusion for 3D Occupancy Prediction
May 16, 2026Xun Chen, Tianchen Deng, Rui Wang +53D Semantic Segmentation 3D Occupancy Prediction
Nanyang Technological University · Shanghai Jiao Tong University · ETH Zurich
How Few-Shot Examples Add Up: A Causal Decomposition of Function Vectors in In-Context Learning
May 15, 2026Entang Wang, Yiwei Wang, Aleksandra Bakalova +1Transformer Interpretability Task Vectors
Saarland Informatics Campus, Saarland Univesity, Saarbrücken, Germany.
Attention Dispersion in Dynamic Graph Transformers: Diagnosis and a Transferable Fix
May 15, 2026Jinhao Zhang, Kangfei Zhao, Qiuhao Zeng +1Distribution Shift Robustness Temporal GNNs
Beijing Institute of Technology · University of Toronto · Hong Kong Baptist University
AdaEraser: Training-Free Object Removal via Adaptive Attention Suppression
May 15, 2026Dingming LiuImage Inpainting Attention Mechanisms
School of Computer Science, Peking University.
Grokking as Structural Inference: Transformers Need Bayesian Lottery Tickets
May 15, 2026Kai Hidajat, Solden Stoll, Joseph AnNeural Network Generalization Transformer
Department of Computer Science University of Washington Seattle, WA 98195
Spectral Priors vs. Attention: Investigating the Utility of Attention Mechanisms in EEG-Based Diagnosis
May 14, 2026Tawsik Jawad, Gowtham Atluri, Vikram RavindraElectroencephalography Time Series Classification
University of Cincinnati, Cincinnati, OH, USA
DualKV: Shared-Prompt Flash Attention for Efficient RL Training with Large Rollouts and Long Contexts
May 14, 2026Jiading Gai, Shuai Zhang, Xiang Song +2GPU Acceleration Self-Attention
1Amazon Web Services · 2Google · University of Minnesota
Multi-Block Attention for Efficient Channel Estimation in IRS-Assisted mmWave MIMO
May 14, 2026Mehrdad Momen-Tayefeh, Mehrshad Momen-Tayefeh, Maryam SabbaghianWireless Communications Channel Estimation
School of Electrical and Computer Engineering, University of Tehran, Tehran, Iran · Department of Computer Engineering, Sharif University of Technology, Tehran, Iran
GQLA: Group-Query Latent Attention for Hardware-Adaptive Large Language Model Decoding
May 14, 2026Fanxu MengGrouped-Query Attention GPU Acceleration
Institute for Artificial Intelligence, Peking University
AttnGen: Attention-Guided Saliency Learning for Interpretable Genomic Sequence Classification
May 13, 2026Rayhaneh Shabani Nia, Ali KarkehabadiNeural Network Interpretability Attention Mechanisms
University of California, Davis California, USA
Delta Attention Residuals
May 13, 2026Cheng Luo, Zefan Cai, Junjie HuAttention Residual Connections Residual Learning
Independent Researcher · University of Wisconsin–Madison
When Attention Closes: How LLMs Lose the Thread in Multi-Turn Interaction
May 13, 2026Vardhan Dongre, Joseph Hsieh, Viet Dac Lai +3Self-Attention LLM Interpretability
University of Illinois Urbana-Champaign · Adobe Research
Precision Tracked Transformer via Kalman Filtering, Kriging and Process Noise
May 12, 2026Bo Long, Deepak Agarwal, Jelena Markovic-Voronov +2Bayesian Filtering Transformer
LinkedIn Core AI
The Routing and Filtering Structure of Attention
May 12, 2026Shafayeth Jamil, Rehan KapadiaAttention Head Analysis Self-Attention
Ming Hsieh Department of Electrical and Computer Engineering University of Southern California Los Angeles, CA 90089
Phylogenetic Tree Inference with Tropical Axial Attention
May 12, 2026Chris Teska, Kurt Pasque, Ruriko Yoshida +1Phylogenetic Inference Attention Mechanisms
In this work, we introduce a Tropical Axial Attention neural reasoning architecture...
Dual-Temporal LSTM with Hybrid Attention for Airline Passenger Load Factor Forecasting: Integrating Intra-Flight and Inter-Flight Booking Dynamics
May 12, 2026ASM Nazrul Islam, Md. Hasanul Kabir, Md. Liakot Ali +1Time Series Forecasting Long Short-Term Memory Networks
Institute of Information and Communication Technology, Bangladesh University of Engineering and Technology, Dhaka, Bangladesh · Islamic University of Technology, Dhaka, Bangladesh
Breaking Global Self-Attention Bottlenecks in Transformer-based Spiking Neural Networks with Local Structure-Aware Self-Attention
May 12, 2026Lingdong Li, Hangming Zhang, Qiang YuSpiking Transformers Efficient ViTs
School of Future Technology, Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, Tianjin 300354, China · School of Artificial Intelligence, Tianjin Key Laboratory of Cognitive Computing and Application, College of Intelligence and Computing, Tianjin University, Tianjin 300354, China
Transformer Interpretability from Perspective of Attention and Gradient
May 12, 2026Yongjin Cui, Xiaohui Fan, Huajun ChenGradient-Based Attribution Transformer Interpretability
Zhejiang University
Block-Based Double Decoders
May 11, 2026Asher Labovich, Benjamin Bradley, Vanessa Alexander +1Efficient Transformer Inference LLM Inference Acceleration
Department of Computer Science Brown University Providence, RI 02912
SLASH the Sink: Sharpening Structural Attention Inside LLMs
May 11, 2026Yiming Liu, Bin Lu, Xinbing Wang +2LLM Reasoning with Graphs LLM Interpretability
Shanghai Jiao Tong University, Shanghai, China · Institute of Geographical Science and Natural Resources Research, Chinese Academy of Sciences, Beijing, China
Polyphonia: Zero-Shot Timbre Transfer in Polyphonic Music with Acoustic-Informed Attention Calibration
May 11, 2026Haowen Li, Tianxiang Li, Yi Yang +2Audio Editing Controllable Music Generation
School of Future Technology, South China University of Technology, Guangzhou, China
Complex-Valued Phase-Coherent Transformer
May 11, 2026Leona HiokiSoftmax Attention Complex-Valued Neural Networks
HYPERPOSE: Hyperbolic Kinematic Phase-Space Attention for 3D Human Pose Estimation
May 11, 2026Vinduja Thekkath, Ashish Musale, Ajay Waghumbare +13D Human Pose Estimation Attention Mechanisms
Key-Value Means: Transformers with Expandable Block-Recurrent Compressed Memory
May 11, 2026Daniel Goldstein, Eugene CheahTransformer Inference Self-Attention
Featherless AI · Eleuther AI
Positional LSH: Binary Block Matrix Approximation for Attention with Linear Biases
May 10, 2026Daniel Wolfson, Tal WagnerSelf-Attention Long-Context Language Modeling
Blavatnik School of Computer Science and AI, Tel Aviv University
The Transformer as a Polar State Estimator
May 10, 2026Peter RacioppoTransformer Layer Normalization
Independent Researcher Los Angeles, CA
Sparsity Moves Computation: How FFN Architecture Reshapes Attention in Small Transformers
May 10, 2026Gabriel Smithline, Chris MascioliTransformer FFNs Transformer
University of Michigan
CATO: Charted Attention for Neural PDE Operators
May 9, 2026Chun-Wun Cheng, Sifan Wang, Carola-Bibiane Schönlieb +1PDE Operator Learning Attention Mechanisms
DAMTP University of Cambridge · Institute for Foundations of Data Science Yale University · Yau Mathematical Sciences Center Tsinghua University
Curvature-Aware Captioning:Leveraging Geodesic Attention for 3D Scene Understanding
May 9, 2026Ziyao He, Yingjie Liu, ZhangYangRui +3Attention Mechanisms Hyperbolic Representation Learning
East China Normal University
A Single Layer to Explain Them All:Understanding Massive Activations in Large Language Models
May 8, 2026Zeru Shi, Zhenting Wang, Fan Yang +2LLM Interpretability Attention Sinks
Rutgers University · Wake Forest University · Meta AI

May 16, 2026, cs.CVJ/K move · Enter open · S save

VGGT-Occ: Geometry-Grounded and Density-Aware Gated Fusion for 3D Occupancy Prediction

Xun Chen, Tianchen Deng, Rui Wang, Fangjinhua Wang+4

Nanyang Technological University · Shanghai Jiao Tong University · ETH Zurich

3D Semantic Segmentation 3D Occupancy Prediction Transformer Attention Deformable Attention Attention Mechanisms

Open paper AskPDF arXiv

3D semantic occupancy prediction requires accurate 2D-to-3D feature lifting, yet current methods restrict camera geometry to initial projections. Subsequent operations like offset learning, attention weighting, and cross-camera aggregation remain geometry-agnostic, ignoring essential physical constraints. We propose VGGT-Occ, a framework that embeds geometric tokens throughout the entire pipeline. We introduce Projection-Aware Deformable Attention (PA-DA) to inject geometry into all attention stages. PA-DA projects 3D offsets back to image planes and leverages the projection Jacobian as an additive bias to suppress unreliable observations. Features are then integrated through a view-quality semantic gate for cross-view consistency. To optimize both efficiency and performance, we employ a sequential coarse-to-fine decoder with gated fusion, where low-resolution features are refined into higher resolutions, allocating computation by information density while substantially reducing decoder cost. Extensive evaluations demonstrate the effectiveness and accuracy of our approach. On SurroundOcc-nuScenes, VGGT-Occ achieves 33.00% IoU and 21.08% mIoU (

T{=}1

), and 33.64% IoU and 21.43% mIoU with

T{=}2

inference, outperforming existing methods, with only

{\sim}41

M trainable parameters in the occupancy head. Code will be released publicly.

Momentum

Related topics

Latest papers 378