Attention Mechanisms

Follow this topic Ask about this topic

Momentum

42 papers in the last four weeks, up 100% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 378

All topics

TEEP-RCNN: Texture-Enhanced Edge-aware Perception for Steel Surface Defect Detection via Improved Convolutional Block Attention in Faster R-CNN
Sep 23, 2026Kirtan RajeshObject Detection Convolutional Neural Networks
Independent Researcher
A Systematic Benchmark of Explainable Methods for Temporal Attribution in Sequential Recommendation Systems
Sep 23, 2026Akash Pandey, Kanisha Shah, Addrish Roy +5Gradient-Based Attribution Sequential Recommendation
Capital One, AI Foundations San Jose, CA, USA · Capital One, AI Foundations New York, NY, USA · Capital One, AI Foundations McLean, VA, USA +1
Rethinking Pairwise Token Interaction in Spiking Transformers
Sep 22, 2026Sicheng Shen, Dongcheng Zhao, Zhiyuan Li +5Spiking Transformers Transformer Attention
Institute of Automatoin, CAS · Center for Excellence in Brain Science and Intelligence Technology, State Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology,Institute of Neuroscience, CAS · Zhongguancun Academy
Attention as a Routing Graph: Live Circuit Extraction from a Single Forward Pass
Sep 21, 2026Ash Manvi, Samreena TajreenCircuit Discovery Attention Mechanisms
Aquin Labs
TACIT: Tactile Contact Supervision for Spatial Attention in Dexterous Manipulation
Sep 21, 2026Yanhou Lai, Fucai Zhu, Ruiqiang Wang +1Contact-Rich Robotic Manipulation Robot Imitation Learning
Graduate School of Information Sciences, Tohoku University, Sendai 980-8579, Japan
On-Demand Attention: Language Models Know When to Recall
Sep 17, 2026Haibo Feng, Ruiqi Liang, Hanyang Peng +1Language Model Decoding Long-Context Language Model Inference
Southern University of Science and Technology · Peking University · Peng Cheng Laboratory
Relational Attention for Data-Efficient Language Modeling
Sep 17, 2026Adrian Brasoveanu, Ece Takmaz, Jakub DotlačilLanguage Model Pretraining Decoder-Only Language Models
SETTer: Sparse-Encoder Transformer for Long-term Multivariate Time Series Forecasting
Sep 17, 2026Abraham Ezema, Chijioke Eze, Ferdinanda Ponci +1Multivariate Time Series Forecasting Long-Term Time Series Forecasting
Institute for Automation of Complex Power Systems, RWTH Aachen University, 52064 Aachen, Germany · RWTH Aachen University, 52064 Aachen, Germany · institute for Monitoring and Distributed Control for Power Systems Teaching and Research Area, RWTH Aachen University, 52064 Aachen, Germany +1
Beyond Flattened Tokens: Structure-Preserving EEG Decoding with Reusable TriDim Blocks
Sep 17, 2026Shiyue Su, Song Wang, Zekai Zhan +7Cross-Subject EEG Decoding EEG Decoding
Department of Biomedical Engineering, Southern University of Science and Technology, Shenzhen, 518055, China. · Omni-Intelligence, Shenzhen, China. · Department of Computer Science, The Chinese University of Hong Kong, Shenzhen, 518172, China. +2
Directions That Don't Drift: Stiefel Manifold Routing for Transformer Attention
Sep 16, 2026Rubén Darío GuerreroStiefel Manifold Optimization Riemannian Optimization
NeuroTechNet S.A.S., 1108831, Bogotá, Colombia
Event-based Selective Attention for Multi-resolution Fast Region of Interest (ROI) Detection
Sep 15, 2026Luca Peres, Giulia D'Angelo, Chiara Bartolozzi +1Event-Based Vision Visual Attention
Department of Cybernetics Faculty of Electrical Engineering Czech Technical University in Prague Prague, Czech Republic · Istituto Italiano di Tecnologia Genoa, Italy
Training Trajectories Determine Circuit Removability in Annealable Soft-Prior Transformers
Sep 9, 2026Zonglin Yang, Ziming Zhao, Wei Tang +5Transformer Interpretability Attention Mechanisms
Criminal Science and Technology, Guangdong Police College, China · Cybersecurity and Law Enforcement, Guangdong Police College, China · Traffic Management Engineering, Guangdong Police College, China
It's Not RoPE that Creates Sinks: The Role of Self-Concentration and Value-Non-Mixing in Attention
Sep 8, 2026Raito Kiya, Satoki Ohashi, Kosuke Sato +6Self-Attention Attention Sinks
1Tohoku University · 2SOKENDAI · 3NINJAL +3
Adaptive Anisotropic Attention for Axis-Structured Signals
Sep 8, 2026Mahir Jain, Parshva Runwal, Aditya Ray Mishra +4EEG Decoding Attention Mechanisms
Mannas AI · IIT Mandi
Why shared attention vectors fail: a case for outcome-indexed tuning
Sep 8, 2026Lenard DomeAttention Mechanisms
Department of Psychiatry and Psychotherapy, Faculty of Medicine University of Tübingen, Tübingen, Germany · German Center for Mental Health (DZPG), Tübingen, Germany
Do New Attention Mechanisms Actually Fix Attention Sinks at Million-Token Context?
Sep 8, 2026Sara Rizwan, Samaanah Abdus SalamLong-Context Language Model Evaluation Long-Context Language Modeling
Shadan Women’s College of Engineering and Technology Hyderabad, Telangana, India
Nyström Attention Matches Full Attention for Cross-Sectional Stock Prediction
Sep 8, 2026Kunhan GuoLow-Rank Attention Financial Forecasting
The University of Hong Kong
DeepTable: Structural Attention Biases and Tree Path Encoding for Hierarchical Table Understanding
Sep 7, 2026Jyun-Ying Yen, Cheng-Kuan Lin, Yu-Chee TsengTable QA Table Structure Recognition
Department of Computer Science National Yang Ming Chiao Tung University, Taiwan
RedKnot-MLA: Multi-Head Offline-Online Reuse for DeepSeek-V4 Long-Context Serving
Sep 7, 2026Yang Liu, Zhaokai Luo, Huayi Jin +10KV Caching LLM Inference Acceleration
Xiaohongshu Inc., China · Huawei Cloud · Shanghai Jiao Tong University +2
RoLA: Rotary-Positioned Low-Rank Linear Attention for Efficient Diffusion Transformers
Sep 6, 2026Zekun Zhang, Yixiang Cai, Yuxi Liu +9Rotary Positional Embeddings Low-Rank Attention
Peking University, Melon Group · Alibaba Group · University of Electronic Science and Technology of China +1
Hardware-Aware FP4 FlashAttention-4
Sep 3, 2026Robert HuMixed-Precision Quantization GPU Kernel Optimization
GRAPHCORE RESEARCH
The Head Complexity of Boolean Functions in Single-Layer Attention
Sep 3, 2026Rajmohan Rajaraman, Ravi Sundaram, Amanuel TesfayeSelf-Attention Attention Mechanisms
Northeastern University.
VestigeKV: The NoPE-MLA KV Cache Carries Its Own Sparse-Attention Signal in a Vestigial Branch
Sep 3, 2026WenJie FanKV Caching KV-Cache Management
Yotta Labs
High-Dimensional Learning Dynamics of Attention-Indexed Models
Sep 3, 2026Yizhou Xu, Margarita Sagitova, Lenka Zdeborová +1Implicit Bias Attention Mechanisms
1Information, Learning and Physics Laboratory, École Polytechnique Fédérale de Lausanne (EPFL) · 2Statistical Physics of Computation Laboratory, École Polytechnique Fédérale de Lausanne (EPFL)
Scaled Idempotence in Transformer Attention: Paired OV Geometry and Shared-Value Algebras
Sep 1, 2026Jiming Feng, Junliang LiAttention Head Analysis Transformer Attention
College of Computer Science, Beijing University of Technology Beijing, China · School of Mathematics, Statistics and Mechanics, Beijing University of Technology Beijing, China
Toppling the Hierarchy in Byte-level Language Modeling
Aug 31, 2026Lukas Edman, Alexander FraserEfficient Language Model Training Byte-Level Language Model
School of Computation, Information and Technology, TU Munich · Munich Center for Machine Learning · Munich Data Science Institute
SELECT: SELEctive Context Transfer for Class-Incremental Semantic Segmentation
Aug 31, 2026Avi Gupta, Saurabh Yadav, Koteswar Rao Jerripothula +1Class-Incremental Learning Attention Mechanisms
IIIT Delhi · Microsoft, India · IIT Kanpur +1
TPR-Attention for Combinatorial Generalization
Aug 31, 2026Melisa Civelekoğlu, Isabeau Prémont-SchwarzAttention Mechanisms Compositional Representation Learning
Universit´e Laval · Institut intelligence et donn´ees · Mila – Qu´ebec Artificial Intelligence Institute
A Lightweight Phenology-Aware YOLOv5 Framework for Tomato Growth Stage Detection in Resource-Constrained Bhutanese Greenhouse Environments
Aug 30, 2026Sherab Gocha, Sou NobukawaPlant Phenotyping Object Detection
Graduate School of Information and Computer Science · Chiba Institute of Technology · Chiba, Japan +1
Intrinsic Interaction Geometry Controls the Low-Rank Complexity of Softmax Attention
Aug 28, 2026Yuhe Sui, Jianing Zhang, Yingzhi TangSoftmax Attention Low-Rank Attention
Nanyang Technological University, Singapore · Carnegie Mellon University

Sep 23, 2026, cs.CVJ/K move · Enter open · S save

TEEP-RCNN: Texture-Enhanced Edge-aware Perception for Steel Surface Defect Detection via Improved Convolutional Block Attention in Faster R-CNN

Kirtan Rajesh

Independent Researcher

Object Detection Convolutional Neural Networks Attention Mechanisms Anomaly Detection

Open paper AskPDF arXiv

Steel surface defect detection is critical for automated industrial quality control but remains challenging due to subtle inter-class texture differences and pronounced class imbalance. We introduce TEEP-RCNN (Texture-Enhanced Edge-aware Perception Region-based CNN), a two-stage detector built on Faster R-CNN with a Feature Pyramid Network backbone and an improved Convolutional Block Attention Module (CBAM). Our CBAM adds dropout regularization in the channel attention MLP and batch normalization on the spatial attention branch, reducing co-adaptation and stabilizing gating logits. Training uses a differential learning rate protocol with cosine annealing warm-up, separating update rates for the pre-trained ResNet-101 backbone and the detection head. At inference, predictions are refined via Test-Time Augmentation fused with Weighted Box Fusion (WBF), improving localization stability on elongated and boundary-adjacent defects. On the NEU-DET benchmark across six defect categories, TEEP-RCNN achieves 73.3% mAP@50 and 37.9% mAP@50-95 in only 10 training epochs on a single GPU, competitive with YOLOv11m (76.2% mAP@50, 100 epochs) while outperforming it on the rolled-in-scale category under the COCO metric. Per-class analysis shows the spatial attention branch is most effective on elongated texture defects such as patches and scratches, while crazing remains an open challenge across both paradigms due to its distributed non-local texture structure.

Momentum

Related topics

Latest papers 378