Transformers

Follow this field Ask about this field

Recent momentum

emerging

0 papers in the last 28 days · 0.0% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this field, kept on the site without email delivery.

Period ending 2026-09-21

42 new papers

A weekly snapshot of new work published in Transformers.

Period ending 2026-09-14

3 new papers

A weekly snapshot of new work published in Transformers.

Inside this field

Focused directions

Focused topic

Transformer Architectures

Within Transformers

684 papersExplore

Focused topic

Quadratic Attention

Within Transformers

144 papersExplore

Focused topic

Dynamic Sparse Attention

Within Transformers

109 papersExplore

Focused topic

Transformer Attention

Within Transformers

89 papersExplore

Focused topic

Transformer Encoder

Within Transformers

80 papersExplore

Focused topic

Transformer Layers

Within Transformers

80 papersExplore

Focused topic

Softmax Attention

Within Transformers

74 papersExplore

Focused topic

Unified Transformer

Within Transformers

40 papersExplore

Focused topic

Multi-Head Self-Attention

Within Transformers

35 papersExplore

Focused topic

Autoregressive Transformers

Within Transformers

33 papersExplore

Focused topic

Global Attention Mechanism

Within Transformers

20 papersExplore

1,208 papers

Latest in Transformers

Browse all topics →

Towards Understanding Self-Pretraining for Sequence Classification
May 20, 2026Omar Coser, Loredana Zollo, Paolo Soda +1Self-Supervised Transformer Architectures
A Sharper Picture of Generalization in Transformers
May 20, 2026Paul Lintilhac, Sair ShaikhGeneralization Bounds Sharpness-Aware Minimization
Runtime-Certified Bounded-Error Quantized Attention
May 20, 2026Dean CalverKey-Value Cache Quantization Systematic Fp16 Divergence
Markovian Circuit Tracing for Transformer State Dynamic
May 20, 2026Abdullah XTransformer Architectures Hidden Markov Models
TERDNet: Transformer Encoder-Recurrent Decoder Network for Scene Change Detection
May 20, 2026Jiae Yoon, Ue-Hwan KimChange Detection Rt-Detr
PulseCol: Periodically Refreshed Column-Sparse Attention for Accelerating Diffusion Language Models
May 20, 2026Yanyi Lyu, Letian Chen, Futing Sun +3Dynamic Sparse Attention Diffusion Language Models
The Devil is in the Condition Numbers: Why is GLU Better than non-GLU Structure?
May 20, 2026Xingyu Lyu, Qianqian Xu, Zhiyong Yang +2Neural Tangent Kernel Quadratic Attention
Rethinking Cross-Layer Information Routing in Diffusion Transformers
May 20, 2026Chao Xu, Maohua Li, Qirui Li +9Multimodal Diffusion Transformers Transformer Residual Streams
LT2: Linear-Time Looped Transformers
May 20, 2026Chunyuan Deng, Yizhe Zhang, Rui-Jie Zhu +4Quadratic Attention Transformer Architectures
RoPeSLR: 3D RoPE-driven Sparse-LowRank Attention for Efficient Diffusion Transformers
May 20, 2026Yuxi Liu, Zekun Zhang, Yixiang Cai +3Dynamic Sparse Attention Rotary Position Embedding
Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnostics
May 19, 2026Lucky VermaModel Weights Transformer Attention
Efficient Long-Context Modeling in Diffusion Language Models via Block Approximate Sparse Attention
May 19, 2026Wenhu Zhang, Yiming Wu, Huanyu Wang +6Dynamic Sparse Attention Diffusion Language Models
Position: The Turing-Completeness of Autoregressive Transformers Relies Heavily on Context Management
May 19, 2026Guanyu Cui, Zhewei Wei, Kun HeAutoregressive Transformers Transformer Architectures
CODA: Rewriting Transformer Blocks as GEMM-Epilogue Programs
May 19, 2026Han Guo, Jack Zhang, Arjun Menon +4Graphics Processing Unit Kernels Transformer Architectures
Rotation-Aligned Key Channel Pruning for Efficient Vision-Language Model Inference
May 19, 2026Beomseok Kang, Dongwon Jo, Jiwon Song +2Visual Token Pruning Quadratic Attention
Multi-Head Attention as Ensemble Nadaraya-Watson Estimation: Variance Reduction, Decorrelation, and Optimal Head Diversity
May 18, 2026Ernest FokouéMulti-Head Self-Attention Variance Reduction
Chronicle: A Multimodal Foundation Model for Joint Language and Time Series Understanding
May 18, 2026Paul Quinlan, Jeremy Levasseur, Qingguo Li +1Time Series Foundation Models Time Series
Performance Monitoring of Proton Exchange Membrane Water Electrolyzer by Transformers-Based Machine Learning Model
May 18, 2026Bingqing Chen, Ivan Batalov, Qiu Chen +2Hydrogen Multi-Energy Systems Transformer Encoder
Chessformer: A Unified Architecture for Chess Modeling
May 18, 2026Daniel Monroe, George Eilender, Philip Chalmers +2Chess Transformer Encoder
KVBuffer: IO-aware Serving for Linear Attention
May 18, 2026Longwei Zou, Lin ZhongQuadratic Attention Speculative Decoding
SAGA: A Sequence-Adaptive Generative Architecture for Multi-Horizon Probabilistic Forecasting with Adaptive Temporal Conformal Prediction
May 18, 2026Gustav Olaf Yunus Laitinen-Fredriksson Lundström-Imanov, Hafize Gonca CömertConformal Prediction Conditional Distribution
DashAttention: Differentiable and Adaptive Sparse Hierarchical Attention
May 18, 2026Yuxiang Huang, Nuno M. T. Gonçalves, Federico Alvetreti +5Dynamic Sparse Attention Efficient Long-Context Inference
Optimising CSRNet with parameter-free attention mechanisms for crowd counting in public transport
May 18, 2026Aida Rostamza, Enrico Del Re, Joshua Cherian Varughese +1Softmax Attention Density Ratio Estimation
From Division to Decision: Leveraging Temporal Cell-Stage Segmentation for Embryo Transferability Prediction
May 18, 2026Yasmine Hachani, Patrick Bouthemy, Elisa Fromont +3Reproduction Early Layers
The Expressive Power of Low Precision Softmax Transformers with (Summarized) Chain-of-Thought
May 18, 2026Moritz Brösamle, Stephan EcksteinTransformer Layers Gumbel-Softmax Relaxation
Protection Is (Nearly) All You Need: Structural Protection Dominates Scoring in Globally Capped KV Eviction
May 18, 2026Gabriel GarciaKey-Value Cache Eviction Key-Value
Function graph transformers universally approximate operators between function spaces
May 18, 2026Takashi Furuya, David Mis, Ivan Dokmanić +2Probabilistic Operator Learning Heterogeneous Graph Transformers
InfoFlow: A Framework for Multi-Layer Transformer Analysis
May 18, 2026Penghao Yu, Haotian Jiang, Zeyu Bao +1Transformer Layers Softmax Attention
SNLP: Layer-Parallel Inference via Structured Newton Corrections
May 18, 2026Ligong Han, Kai Xu, Hao Wang +1Transformer Layers Transformer Residual Streams
OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization
May 18, 2026Zhongzhu Zhou, Donglin Zhuang, Jisen Li +4Key-Value Cache Quantization Quantization
Training Infinitely Deep and Wide Transformers
May 17, 2026Raphaël Barboni, Maarten V. de Hoop, Takashi Furuya +1Transformer Architectures Mean-Field Limit
LLMForge: Multi-Backend Hardware-Aware Neural Architecture Search with Infinite-Head Attention for Edge Language Models
May 17, 2026Xinting Jiang, Junyi Luo, Ruichen Qi +4Neural Architecture Search LLM Inference Efficiency
Descriptive versus Regulatory Uncertainty in Bounded Predictive Systems
May 17, 2026Ahmed Gamal EldinEpistemic Uncertainty Large Language Model Uncertainty
Hamiltonian-Inspired Attention Mechanism for Scalable RF Transmitter Fingerprinting
May 17, 2026Chitraksh Singh, Monisha Dhanraj, Akram SheriffRadio Frequency Transformer Attention
Mamba-VGGT: Persistent Long-Sequence Video Geometry Grounded Transformer via External Sliding Window Mamba Memory
May 17, 2026Tianchen Deng, Zhenxiang Xiong, Nailin Wang +4Spatio-Temporal Transformers Feed-Forward 3D Reconstruction
A Two-Parameter Weibull Framework for Diagnosing Transformer Weight Distributions
May 17, 2026Tiexin DingModel Weights Transformer Architectures
Cross-Subject Intracranial EEG Reconstruction from Scalp Recordings Using Multi-Scale Cross-Attention Transformers
May 17, 2026Tien-Dat Pham, Xuan-The TranElectroencephalography Spatial Super-Resolution Neural Recordings
VISTA: Triplet-Supervised Video Style Transfer with Diffusion Transformers
May 17, 2026Yiren Song, Wangzi Yao, Haofan Wang +1Pre-Trained Video Diffusion Models Stylization
FishBack: Pullback Fisher Geometry for Optimal Activation Steering in Transformers
May 17, 2026Sihan Wang, Jiayi ZhaoActivation Steering Steering
OPTNet: Ordering Point Transformer Network for Post-disaster 3D Semantic Segmentation
May 16, 2026Nhut Le, Ehsan Karimi, Maryam Rahnemoonfar3D Semantic Segmentation Post-Disaster Damage Assessment
The Geometry of Projection Heads: Conditioning, Invariance, and Collapse
May 16, 2026Faris ChaudhryMulti-Head Self-Attention Riemannian Manifolds
TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks
May 16, 2026Hanzhang Shen, Haoran Wu, Yiren Zhao +1Key-Value Cache Quantization LLM Inference Efficiency
Parallel Recursive LSTM
May 16, 2026Tristan Gaudreault, Yongyi MaoParallel Recurrent Neural Networks
Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps
May 16, 2026Yanke Zhou, Yiduo Li, Hanlin Tang +6Dynamic Sparse Attention Efficient Long-Context Inference
Propagation of Chaos in Contextual Flow Maps
May 16, 2026Shi Chen, Zhengjiang Lin, Kaizhao Liu +1Transformer Architectures Flow Map
DiRotQ: Rotation-Aware Quantization for 4-bit Diffusion Transformers
May 16, 2026Sayeh Sharify, Mahsa Salmani, Hesham MostafaMultimodal Diffusion Transformers Low-Bit
Attend Locally, Remember Linearly: Linear Attention as Cross-Frame Memory for Autoregressive Video Diffusion
May 15, 2026Kunyang Li, Mubarak Shah, Yuzhang ShangAutoregressive Video Diffusion Softmax Attention
No Free Swap: Protocol-Dependent Layer Redundancy in Transformers
May 15, 2026Gabriel GarciaTransformer Layers Transformer Architectures
Attention Dispersion in Dynamic Graph Transformers: Diagnosis and a Transferable Fix
May 15, 2026Jinhao Zhang, Kangfei Zhao, Qiuhao Zeng +1Transformer Attention Transformer Architectures
Multi-Headed Transformer Architectures as Time-dependent Wasserstein Gradient Flows
May 15, 2026Alex Massucco, Leonardo Del Grande, Marcello Carioni +2Transformer Architectures Wasserstein Gradient Flows
Towards Generalization of Block Attention via Automatic Segmentation and Block Distillation
May 15, 2026Shuaiyi Li, Zhisong Zhang, Yan Wang +5Global Attention Mechanism Efficient Long-Context Inference
Grokking as Structural Inference: Transformers Need Bayesian Lottery Tickets
May 15, 2026Kai Hidajat, Solden Stoll, Joseph AnAttention-Based Models Transformer Architectures
HyperDiT: Hyper-Connected Transformers for High-Fidelity Pixel-Space Diffusion
May 15, 2026Yu He, Lichen Ma, Zipeng Guo +5Multimodal Diffusion Transformers High-Fidelity
From Sparsity to Simplicity: Enabling Simpler Sequential Replacements via Sparse Attention Distillation
May 15, 2026Yuxin Ren, Maxwell D Collins, Miao Hu +1Dynamic Sparse Attention Sparsity
DreamSR: Towards Ultra-High-Resolution Image Super-Resolution via a Receptive-Field Enhanced Diffusion Transformer
May 15, 2026Qingji Dong, Hang Dong, Mingqin Chen +2Generative Super-Resolution Receptive Fields
Transformer-like Inference from Optimal Control
May 15, 2026Aditya Kudre, Heng-Sheng Chang, Prashant G. MehtaTransformer Encoder Optimal Control
Gaussian Relational Graph Transformer
May 15, 2026Zezhong Ding, Jin Li, Xugang Wang +1Heterogeneous Graph Transformers Spatio-Temporal Transformers
GiLT: Augmenting Transformer Language Models with Dependency Graphs
May 15, 2026Tianyu Huang, Yida Zhao, Chuyan Zhou +1Graph Language Models Transformer Architectures
3DTMDet: A Dual-Path Synergy Network of Transformer and SSM for 3D Object Detection in Point Clouds
May 15, 2026Bingwen Qiu, Yuan Liu, Junqi Bai +53D Object Detection Feature Pyramid Networks
STS: Efficient Sparse Attention with Speculative Token Sparsity
May 15, 2026Ceyu Xu, Jiangnan Yu, Yongji Wu +1Dynamic Sparse Attention LLM Inference Efficiency

May 20, 2026 · cs.LGJ/K move · Enter open · S save

Towards Understanding Self-Pretraining for Sequence Classification

Omar Coser, Loredana Zollo, Paolo Soda, Antonio Orvieto

Amos et al. (2024) showed that the accuracy of Transformer models in sequence classification can be significantly improved by first pretraining with a masked token prediction objective without external data or augmentation, a procedure referred to as self-pretraining (SPT). While the primary objective of Amos et al. (2024) was to showcase that Transformers can achieve strong performance on the Long-Range Arena (LRA), their pipeline raises more fundamental questions: How does SPT drive optimization to better solutions? Why can standard supervised training fail in Transformers? To better understand this, we replicate and systematically ablate the findings of Amos et al. (2024). Our ablations suggest that a central bottleneck in the studied settings is not depth or generalization alone, but the ability of label supervision to learn useful query-key Attention patterns from random initialization. With a minimal setup, we identify learning proximity interactions - turning absolute positional encodings into proximity-biased Attention scores - as a key source of the improvements brought by SPT. Finally, in a simplified theoretical setup, we show that label supervision can be locally blind to certain Attention-score directions that are instead detectable through masked reconstruction.

Self-Supervised Transformer Architectures Token-Level Supervision

Open paper Ask about it PDF