Attention Mechanisms

Follow this topic Ask about this topic

Momentum

42 papers in the last four weeks, up 100% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 378

All topics

Cross-Modal Attention Analysis and Optimization in Vision-Language Models: A Study on Visual Reliability
Apr 19, 2026Lijie ZhouCross-Modal Learning VLM Robustness
School of Computer Science University of Nottingham Ningbo China Ningbo, China
Open-TQ-Metal: Fused Compressed-Domain Attention for Long-Context LLM Inference on Apple Silicon
Apr 18, 2026Sai VegasenaAI Accelerator Inference Self-Attention
Ensue
SinkRouter: Sink-Aware Routing for Efficient Long-Context Decoding in Large Language and Multimodal Models
Apr 18, 2026Junnan Liu, Xinyan Liu, Peifeng Gao +4GPU Acceleration Self-Attention
Harbin Institute of Technology (Weihai) · University of Hong Kong, Hong Kong, China · City University of Hong Kong, Hong Kong, China
AtManRL: Towards Faithful Reasoning via Differentiable Attention Saliency
Apr 17, 2026Max Henning Höth, Kristian Kersting, Björn Deiseroth +1CoT Faithfulness Feature Attribution
Aleph Alpha Research Lab1141 · TU Darmstadt Hessian.AI
Improving Reasoning Capabilities in Small Models through Mixture-of-Layers Distillation with Stepwise Attention on Key Information
Apr 17, 2026Yao Chen, Jiawei Sheng, Wenyuan Zhang +1CoT Distillation Small Language Models
Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences
LACE: Lattice Attention for Cross-thread Exploration
Apr 16, 2026Yang Li, Zirui Zhang, Yang Liu +1LLM Inference Attention Mechanisms
Department of Computer Science Rutgers University · Amazon AGI Labs
AdaSplash-2: Faster Differentiable Sparse Attention
Apr 16, 2026Nuno Gonçalves, Hugo Pitorro, Vlad Niculae +4GPU Kernel Optimization Self-Attention
Instituto Superior Técnico, Universidade de Lisboa · Instituto de Telecomunicações · Carnegie Mellon University +4
Dispatch-Aware Ragged Attention for Pruned Vision Transformers
Apr 16, 2026Seifeldin Abdellatif, Ahmad AlmasriEfficient Transformer Inference Visual Attention
College of Engineering Al Ain University Abu Dhabi, United Arab Emirates
What Is the Minimum Architecture for Prolepsis? Early Irrevocable Commitment Across Tasks in Small Transformers
Apr 16, 2026Éric JacopinTransformer Interpretability Attention Head Analysis
Cosmic AI
Expressivity of Transformers: A Tropical Geometry Perspective
Apr 16, 2026Ye Su, Yong LiuTransformer Expressivity Transformer
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China · University of Chinese Academy of Sciences, Beijing 101407, China · Gaoling School of Artificial Intelligence, Renmin University of China, Beijing 100872, China
A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigation
Apr 16, 2026Yuval Ran-Milo, Hila Ofek, Shahar MendelSelf-Attention Decoder-Only Language Models
Tel Aviv University
Gating Enables Curvature: A Geometric Expressivity Gap in Attention
Apr 16, 2026Satwik Bathula, Anand A. JoshiInformation Geometry Neural Representation Geometry
Department of Electrical and Computer Engineering University of Southern California
What Drives Representation Steering? A Mechanistic Case Study on Steering Refusal
Apr 9, 2026Stephen Cheng, Sarah Wiegreffe, Dinesh ManochaLanguage Model Steering LLM Interpretability
University of Maryland, College Park
Sinkhorn doubly stochastic attention rank decay analysis
Apr 9, 2026Michela Lapenna, Rita Fioresi, Bahman GharesifardSoftmax Attention Self-Attention
Department of Physics and Astronomy, University of Bologna, Bologna, Italy · Department of Pharmacy and Biotechnologies, University of Bologna Bologna, Italy · Department of Mathematics and Statistics, Queen’s University, Kingston, ON, Canada
Flux Attention: Context-Aware Hybrid Attention for Efficient LLMs Inference
Apr 8, 2026Quantong Qiu, Zhiyi Hong, Yi Yang +5Self-Attention Long-Context Language Model Inference
School of Computer Science and Technology, Soochow University · Baidu Inc, China
Invertible Query-Key Coupling Composes with Attention Mechanisms
Apr 2, 2026Barak Gahtan, Alex M. BronsteinInvertible Neural Networks Self-Attention
Technion – Israel Institute of Technology · ISTA – Institute of Science and Technology Austria
Screening Is Enough
Apr 1, 2026Ken M. NakanishiSelf-Attention Long-Context Language Modeling
Center for Emergent Matter Science (CEMS), RIKEN · Graduate School of Science, The University of Tokyo
IWP: Token Pruning as Implicit Weight Pruning in Large Vision Language Models
Apr 1, 2026Dong-Jae Lee, Sunghyun Baek, Junmo KimEfficient VLM Inference Large Vision-Language Models
KAIST, South Korea
When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models
Apr 1, 2026Jiho Choi, Jaemin Kim, Sanghwan Kim +2Large Vision-Language Models VLM Interpretability
KAIST, South Korea · Chung-Ang University, South Korea · Technical University of Munich, Germany
From Dispersion to Attraction: Spectral Dynamics of Hallucination Across Whisper Model Scales
Mar 31, 2026Ivan Viakhirev, Kirill Borodin, Grach MkrtchianTransformer Interpretability Automatic Speech Recognition
Information Technologies, Mechanics and Optics University, Russia · lab260, Moscow Technical University of Communications and Informatics, Russia · BitmanagerAI, UAE
HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention
Mar 30, 2026Yufei Xu, Fanxu Meng, Fan Jiang +11Self-Attention LLM Inference Acceleration
Xiaohongshu
Pure and physics-guided deep learning approaches for spatio-temporal groundwater level prediction
Mar 26, 2026Matteo Salis, Gabriele Sartor, Rosa Meo +2Physics-Informed ML Attention Mechanisms
Computer Science Department, University of Turin, Turin, Italy · LISTIC Laboratory, Université Savoie Mont Blanc, Annecy-le-vieux, France · Interuniversity Department of Regional and Urban Studies and Planning, Politecnico di Torino and University of Turin, Turin, Italy
Learning When to Attend: Conditional Memory Access for Long-Context LLMs
Mar 18, 2026Sakshi Choudhary, Aditya Chattopadhyay, Luca Zancato +4Self-Attention Long-Context Language Modeling
Department of Electrical and Computer Engineering, Purdue University, USA · AWS Agentic AI, USA
WAND: Windowed Attention and Knowledge Distillation for Efficient Autoregressive Text-to-Speech Models
Mar 17, 2026Hanna Lee, Tan Dat Nguyen, Jaehoon Kang +1TTS Synthesis Self-Attention
Korea Advanced Institute of Science and Technology, Republic of Korea · Sungkyunkwan University, Republic of Korea
Not All Retrievals are Useful: Cross-Attention for Input-Aware RAG in Time Series Forecasting
Mar 16, 2026Seunghan Lee, Jaehoon Lee, Jun Seo +7Zero-Shot Time Series Forecasting Cross-Attention
LG AI Research Seoul, South Korea
Beyond Pairwise Attention: Higher-Order Modular Attention for Efficient Sequence Learning
Mar 11, 2026Shirin Amiraslani, Xin GaoSelf-Attention Transformer Attention
Department of Mathematics and Statistics York University Toronto, ON M3J 1P3, Canada
StructSAM: Structure- and Spectrum-Preserving Token Merging for Segment Anything Models
Mar 7, 2026Duy M. H. Nguyen, Tuan A. Tran, Duong Nguyen +17Token Merging Image Segmentation
University of Stuttgart · DFKI · Max Planck Research School +10
SHINE: Sequential Hierarchical Integration Network for EEG and MEG
Feb 27, 2026Xiran Xu, Yujie Yan, Songyi Li +4Electroencephalography Neural Processes
Speech and Hearing Research Center, School of Intelligence Science and Technology, Peking University, China · National Key Laboratory of General Artificial Intelligence, China · Center for BioMed-X Research, Academy for Advanced Interdisciplinary Studies, Peking University, China
Bridging Latent Reasoning and Target-Language Generation via Retrieval-Transition Heads
Feb 25, 2026Shaswat Patel, Vishvesh Trivedi, Yue Han +2Multilingual Language Models Cross-Lingual Reasoning in Language Models
New York University
Incremental Learning of Sparse Attention Patterns in Transformers
Feb 22, 2026Oğuz Kaan Yüksel, Rodrigo Alvarez Lucendo, Nicolas FlammarionSelf-Attention Transformer Attention
TML Lab, EPFL

Apr 19, 2026, cs.CVJ/K move · Enter open · S save

Cross-Modal Attention Analysis and Optimization in Vision-Language Models: A Study on Visual Reliability

Lijie Zhou

School of Computer Science University of Nottingham Ningbo China Ningbo, China

Cross-Modal Learning VLM Robustness Adversarial Attacks on VLMs Vision-Language Models VLM Adaptation Attention Mechanisms+2

Open paper AskPDF arXiv

Vision-Language Models (VLMs) achieve strong cross-modal performance, yet recent evidence suggests they over-rely on textual descriptions while under-utilizing visual evidence -- a phenomenon termed ``text shortcut learning.'' We propose an adversarial evaluation framework that quantifies this cross-modal dependency by measuring accuracy degradation (Drop) when semantically conflicting text is paired with unchanged images. Four adversarial strategies -- shape_swap, color_swap, position_swap, and random_text -- are applied to a controlled geometric-shapes dataset (

n{=}1{,}000

). We compare three configurations: Baseline CLIP (ViT-B/32), LoRA fine-tuning, and LoRA Optimized (integrating Hard Negative Mining, Label Smoothing, layer-wise learning rates, Cosine Restarts, curriculum learning, and data augmentation). The optimized model reduces average Drop from 27.5% to 9.8% (64.4% relative improvement,

p{<}0.001

) while maintaining 97% normal accuracy. Attention visualization and embedding-space analysis confirm that the optimized model attends more to visual features and achieves tighter cross-modal alignment.

Momentum

Related topics

Latest papers 378