Transformers

Follow this field Ask about this field

Recent momentum

emerging

0 papers in the last 28 days · 0.0% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this field, kept on the site without email delivery.

Period ending 2026-09-21

42 new papers

A weekly snapshot of new work published in Transformers.

Period ending 2026-09-14

3 new papers

A weekly snapshot of new work published in Transformers.

Inside this field

Focused directions

Focused topic

Transformer Architectures

Within Transformers

684 papersExplore

Focused topic

Quadratic Attention

Within Transformers

144 papersExplore

Focused topic

Dynamic Sparse Attention

Within Transformers

110 papersExplore

Focused topic

Transformer Attention

Within Transformers

89 papersExplore

Focused topic

Transformer Encoder

Within Transformers

80 papersExplore

Focused topic

Transformer Layers

Within Transformers

80 papersExplore

Focused topic

Softmax Attention

Within Transformers

74 papersExplore

Focused topic

Unified Transformer

Within Transformers

40 papersExplore

Focused topic

Multi-Head Self-Attention

Within Transformers

35 papersExplore

Focused topic

Autoregressive Transformers

Within Transformers

33 papersExplore

Focused topic

Global Attention Mechanism

Within Transformers

20 papersExplore

1,209 papers

Latest in Transformers

Browse all topics →

Benchmarking Convolutional, Transformer, Hybrid, and Vision Language Models for Multi Disease Retinal Screening
May 25, 2026Durjoy Dey, Aymane Ajbar, Yuhong YanHybrid Transformer Medical Vision-Language Models
Looped Diffusion Language Models
May 25, 2026Sanghyun Lee, Chunsan Hong, Seungryong Kim +3Masked Diffusion Language Models Diffusion Language Models
Generic Interpretation Approach for Transformer Models Incorporating Heterogenous Attention Structures
May 25, 2026Yongjin Cui, Xiaohui Fan, Huajun ChenTransformer Attention Multimodal Transformers
ParkourFormer: Integrating Predictive Supervision and Sequence Modeling into Parkour Locomotion
May 25, 2026Yanheng Mai, Wenhao Xu, Zirui Huang +6Parkour Locomotion Human Motion Prediction
Phantom transitions in language model fine-tuning
May 25, 2026Vaibhav Prakash, Jayasri DontabhaktuniLarge Language Model Fine-Tuning Finetuning
Profiling-Driven Adaptive Distributed Transformer Inference on Embedded Edge Deployment
May 25, 2026Muhammad Azlan Qazi, Alexandros Iosifidis, Qi ZhangNvidia Jetson Orin Nano Transformer Architectures
StrTransformer: Source-Wise Structured Transformers for Unsupervised Blind Source Recovery
May 25, 2026Yuan-Hao WeiTransformer Architectures
Analogies between Transformer Layers and Power Method
May 25, 2026Chenglong Li, Claudio AltafiniTransformer Layers Transformer Architectures
Max-Window Scale Estimation for Near-Lossless HiF8 W8A8 Quantization-Aware Training
May 25, 2026Yingying Cheng, Jinquan Shi, Li Zhou +4Large Models Quadratic Attention
Towards the Connection between Activation Sparsity and Flat Minima
May 25, 2026Ze Peng, Jian Zhang, Lei Qi +2Sparse Expert Activation Modern Deep Networks
Transformer Field Theory: A Response-Theoretic Approach to Mechanistic Interpretability
May 24, 2026David N. Olivieri, Antonio F. Pérez RodríguezAutoregressive Transformers Mechanistic Interpretability
TGFormer: Towards Temporal Graph Transformer with Auto-Correlation Mechanism
May 24, 2026Hongjiang Chen, Pengfei Jiao, Ming Du +4Temporal Graph Neural Networks Temporal Graphs
H $^{2}$ MT: Semantic Hierarchy-Aware Hierarchical Memory Transformer
May 24, 2026Maryam Haghifam, Zifan He, Jason Cong +1LLM Inference Efficiency Transformer Architectures
BandVQ: Band-Wise Vector-Quantized EEG Foundation Model
May 24, 2026Jamiyan Sukhbaatar, Satoshi Imamura, Toshihisa TanakaElectroencephalography Foundation Models Electrocorticography
The Concept Allocation Zone: Tracking How Concepts Form Across Transformer Depth
May 24, 2026James HenryTransformer Architectures Model Activations
CONF-KV: Confidence-Aware KV Cache Eviction with Mixed-Precision Storage for Long-Horizon LLM
May 24, 2026Yubo Li, Yidi MiaoKv-Cache Blocks Key-Value Cache Eviction
Muon in Vision Transformers: Optimizer-Recipe Interactions and Gradient Spectra
May 23, 2026Ben S. Southworth, Shuai Jiang, Daniel McBride +2Vision Transformer Muon
Reasoning to Align: Implicit Reasoning in Diffusion Transformers for Video Editing
May 23, 2026Yan Li, Lin Liu, Xiaopeng Zhang +1Video Editing Pre-Trained Video Diffusion Models
CSP-Atlas: Concept-Specific Neural Circuits in a Sparse Python Transformer
May 23, 2026Piotr WilamCircuits Discrete Tokens
Polymorphism Is Rotation: Operational Mechanistic Interpretability from a Two-Layer Transformer to Pythia-70m
May 23, 2026Jordan F. McCannTransformer Architectures Transformer Layers
Emission-Aware Reinforcement Learning for Sustainable Electric Vehicle Charging and Carbon Dioxide Reduction Under Varying Renewable Penetration
May 23, 2026Ninglin Ou, Mohammad A. Razzaque, Iftekher Islam Shovon +5Electric Vehicles Carbon Emissions
Grammatically-Guided Sparse Attention for Efficient and Interpretable Transformers
May 23, 2026Spandan PratyushTransformer Attention Dynamic Sparse Attention
Momentum Streams for Optimizer-Inspired Transformers
May 23, 2026Jingchu Gai, Nai-Chieh Huang, Jiayun WuTransformer Architectures Batch Normalization
Treatment Effect Estimation with Differentiated Networked Effect on Graph Data
May 23, 2026Xiaofeng Lin, Han Bao, Hisashi KashimaHeterogeneous Treatment Effects Complex Networks
Inference Time Context Sparsity: Illusion or Opportunity?
May 22, 2026Sahil Joshi, Prithvi Dixit, Agniva Chowdhury +5LLM Inference Efficiency Dynamic Sparse Attention
Good Token Hunting: A Hitchhiker's Guide to Token Selection for Visual Geometry Transformers
May 22, 2026Shuhong Zheng, Michael Oechsle, Erik Sandström +3Self-Supervised Vision Transformers 3D Scene Understanding
Training-Free Looped Transformers
May 22, 2026Lizhang Chen, Jonathan Li, Chen Liang +2Transformer Architectures Training-Free Inference Framework
Vision Transformers Need Better Token Interaction
May 22, 2026Linxiang SuVision Transformer Dense Prediction
Approaching I/O-optimality for Approximate Attention
May 22, 2026Pál András Papp, Aleksandros Sobczyk, Anastasios ZouziasQuadratic Attention Matrix Multiplication
DAStatFormer: A Hybrid Multibranch Transformer with Statistical Feature Integration for DAS-Based Pattern Recognitions
May 22, 2026Michel Dione, Jerry Lonlac, Hélène Louis +2Hybrid Transformer Passive Acoustic Monitoring
Preisach Attention: A Hysteretic Model of Sequential Memory
May 22, 2026Piotr FrydrychTransformer Layers Softmax Attention
DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation
May 22, 2026Jie Hu, Zixiang Gao, Yutong He +1Dynamic Sparse Attention Pre-Trained Video Diffusion Models
Every Component Is a Lookup: One Linear Graph for Interaction, Composition and Attribution
May 22, 2026Po-Kai Chen, Aske Plaat, Niki van SteinTransformer Attention Transformer Architectures
Adaptive Mass-Segmented KV Compression for Long-Context Reasoning
May 22, 2026Junzhe Yang, Xiaoyu ShenKey-Value Cache Compression Key-Value Cache
Spectral Probe-Circuits: A Three-Step Recipe for Identifying Attention-Head Circuits in Pretrained Transformers
May 22, 2026Yongzhong XuTransformer Attention Transformer Architectures
The Cognitive Categorical Transformer: Category-Theoretic Inductive Biases for Language Modeling
May 22, 2026Al KariGenerative Pretrained Transformers Language Model Perplexity
Uncovering the Latent Potential of Deep Intermediate Representations
May 21, 2026Arnesh Batra, Arush Gumber, Aniket Khandelwal +2Geometric Deep Learning Transformer Layers
World Machine: Towards Generative World Modeling for Time-Series
May 21, 2026Elton Cardoso do Nascimento, Alexandre da Silva Simões, Esther Luna Colombini +2Generative Models Unified Transformer
Certification from Examples is Hard for Circuits and Transformers under Minimal Overparametrization
May 21, 2026Artur Back de Luca, Kimon FountoulakisCertification Over-Parameterized Neural Networks
Represented Is Not Computed: A Causal Test of Candidate Algorithmic Intermediates in a Transformer
May 21, 2026Ishita Darade, Sushrut ThoratArithmetic Intermediate Representations
Structured-Sparse Attention for Entity Tracking with Subquadratic Sequence Complexity
May 21, 2026Hangyue Zhao, Paul Caillon, Erwan Fagnou +1Dynamic Sparse Attention Sequence
FastTab: A Fast Table Recognizer with a Tiny Recursive Module and 1D Transformers
May 21, 2026Laziz Hamdi, Amine Tamasna, Pascal Boisson +1Transformer Encoder Structure-Aware
One LR Doesn't Fit All: Heavy-Tail Guided Layerwise Learning Rates for LLMs
May 21, 2026Di He, Songjun Tu, Keyu Wang +2Learning Rates Transformer Layers
Direct content-based retrieval from music scores images
May 21, 2026Noelia Luna-Barahona, Antonio Ríos-Vila, Félix Fuentes-Hurtado +2Composed Image Retrieval Retrieval Task
How Many Different Outputs Can a Transformer Generate?
May 21, 2026Maxime Meyer, Mario Michelessa, Caroline Chaux +1Transformer Architectures Theoretical Foundations
Can Transformers Learn to Verify During Backtracking Search?
May 21, 2026Yin Jun Phua, Tony Ribeiro, Tuan Nguyen +1Transformer Architectures Backward Induction
Towards Verifiable Transformers: Solver-Checkable Circuit Explanations
May 21, 2026Neel SomaniTransformer Architectures Model Interpretability
From TF-IDF to Transformers: A Comparative and Ensemble Approach to Sentiment Classification
May 21, 2026Dip Biswas Shanto, Mitali Yadav, Prajwal Panth +1Sentiment Analysis Term Frequency-Inverse Document Frequency
Tensor Cache: Eviction-conditioned Associative Memory for Transformers
May 21, 2026Kabir Swain, Sijie Han, Daniel Karl I. Weidele +2Tensor Programs Sliding-Window Attention
Energy-Gated Attention: Spectral Salience as an Inductive Bias for Transformer Attention
May 21, 2026Athanasios ZerisTransformer Attention Inductive Bias
Same Architecture, Different Capacity: Optimizer-Induced Spectral Scaling Laws
May 20, 2026Nandan Kumar Jha, Brandon ReagenOptimizer Design Scaling Laws
Patch Hierarchical Attention Transformer for Efficient Particle Jet Tagging
May 20, 2026Aaron Wang, Zihan Zhao, Alan Xia +5High Energy Physics Transformer Attention
EntmaxKV: Support-Aware Decoding for Entmax Attention
May 20, 2026Gonçalo Duarte, Miguel Couceiro, Marcos V. TrevisoEfficient Long-Context Inference Dynamic Sparse Attention
Post-Hoc Understanding of Metaphor Processing in Decoder-Only Language Models via Conditional Scale Entropy
May 20, 2026Lawhori Chakrabarti, Jennifer Johnson-Leung, Bert Baumgaertner +3Transformer Encoder Metaphors
Fast and Stable Triangular Inversion for Delta-Rule Linear Transformers
May 20, 2026Aleksandros Sobczyk, Gioele Gottardo, Christos K. Matzoros +4Quadratic Attention Full-Precision Performance
Large-Step Training Dynamics of a Two-Factor Linear Transformer Model
May 20, 2026Krishnakumar BalasubramanianTransformer Architectures Gradient Descent
OCTOPUS: Optimized KV Cache for Transformers via Octahedral Parametrization Under optimal Squared error quantization
May 20, 2026Mark Boss, Vikram Voleti, Simon Donné +1Key-Value Cache Quantization Key-Value Cache Compression
FTerViT: Fully Ternary Vision Transformer
May 20, 2026Szymon Ruciński, Pietro Bonazzi, Engin Türetken +3Self-Supervised Vision Transformers Vision Transformer
UniT: Unified Geometry Learning with Group Autoregressive Transformer
May 20, 2026Haotian Wang, Yusong Huang, Zhaonian Kuang +4Cross-Modal Attention Autoregressive Transformers
HORST: Composing Optimizer Geometries for Sparse Transformer Training
May 20, 2026Tom Jacobs, Rohan Jain, Rebekka BurkholzSparsity Optimizer Design

May 25, 2026 · cs.CVJ/K move · Enter open · S save

Benchmarking Convolutional, Transformer, Hybrid, and Vision Language Models for Multi Disease Retinal Screening

Durjoy Dey, Aymane Ajbar, Yuhong Yan

Department of Computer Science and Software Engineering Concordia University Montreal, QC, Canada · Ebovir Biotechnologie Inc. Montreal, QC, Canada

Modern deep learning offers powerful tools for automated retinal screening, but it remains unclear how different visual model families compare in realistic multi-disease settings and under domain shift. In this work, we benchmark twelve architectures across four model families: convolutional neural networks, vision transformers, hybrid CNN-transformer backbones, and vision-language models, using the Retinal Fundus Multi-disease Image Dataset (RFMiD). We evaluate two tasks: binary screening for any retinal disease and multi-label classification across 28 disease classes. Using standardized training, calibration, and evaluation protocols, we report AUC, F1, precision, recall, and sensitivity at a clinically relevant operating point with specificity near 80%. On RFMiD, all architectures perform well on binary screening, with AUC above 84%, but attention-based models perform best. SwinTiny and the hybrid CoAtNet0 and MaxViTTiny models achieve the strongest binary screening results and improve macro and micro F1 in the multi-label setting. Vision-language models, including CLIP ViT-B/16 and SigLIP-Base384, are competitive with CNN baselines but do not surpass the best transformer and hybrid backbones. In external validation on Messidor-2 for referable diabetic retinopathy, AUC ranges from 66.8% to 84.7%, with hybrid and transformer models again showing strong performance. These results provide a reproducible reference for model selection in multi-disease retinal screening and guide future automated screening tools for clinical deployment.

Hybrid Transformer Medical Vision-Language Models Vision Foundation Models Screening Transformer Architectures

Open paper Ask about it PDF