Transformers

Ask about this field

Recent momentum

emerging

0 papers in the last 28 days · 0.0% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this field, kept on the site without email delivery.

Period ending 2026-09-21

42 new papers

A weekly snapshot of new work published in Transformers.

Period ending 2026-09-14

3 new papers

A weekly snapshot of new work published in Transformers.

Inside this field

Focused directions

1,208 papers

Latest in Transformers

Browse all topics →
CardsList

  1. MetaOthello: A Controlled Study of Multiple World Models in Transformers

    Feb 26, 2026Aviral Chawla, Galen Hall, Juniper LovatoTransformer ArchitecturesMultimodal Transformers

  2. CeRA: Breaking the Linear Ceiling of Low-Rank Adaptation with Non-linearity Retained at Inference

    Feb 26, 2026Hung-Hsuan ChenLow-Rank AdaptationParameter-Efficient Fine-Tuning

  3. Transformers converge to invariant algorithmic cores

    Feb 26, 2026Joshua S. SchiffmanTransformer ArchitecturesMemorization

  4. Bridging Latent Reasoning and Target-Language Generation via Retrieval-Transition Heads

    Feb 25, 2026Shaswat Patel, Vishvesh Trivedi, Yue Han +2Multilingual Language ModelsEfficient Latent Reasoning

  5. How Transformers Reject Wrong Answers: Rotational Dynamics of Factual Constraint Processing

    Feb 25, 2026Javier MarínTransformer EncoderTransformer Architectures

  6. Incremental Learning of Sparse Attention Patterns in Transformers

    Feb 22, 2026Oğuz Kaan Yüksel, Rodrigo Alvarez Lucendo, Nicolas FlammarionTransformer AttentionTransformer Architectures

  7. FGAA-FPN: Foreground-Guided Angle-Aware Feature Pyramid Network for Oriented Object Detection

    Feb 11, 2026Jialin MaFeature Pyramid NetworksObject Detection

  8. Central Dogma Transformer II: An AI Microscope for Understanding Cellular Regulatory Mechanisms

    Feb 9, 2026Nobuyuki OtaGene-Regulatory NetworksRegulation

  9. Language-Guided Transformer Tokenizer for Human Motion Generation

    Feb 9, 2026Sheng Yan, Yong Wang, Xin Du +2Human Motion GenerationTokenization

  10. Entropy-Generated Attention Beyond Softmax and Entmax: Kaniadakis and Reciprocal-Symmetric Abe Operators

    Feb 9, 2026Gunn KimEntropy-Regularized Equilibrium SelectionGumbel-Softmax Relaxation

  11. Revisiting the Shape Convention of Transformer Language Models

    Feb 6, 2026Feng-Ting Liao, Guan-Ting Yi, Tzu-Quan Lin +2Transformer ArchitecturesNeural Architectures

  12. FlashBlock: Attention Caching for Efficient Long-Context Block Diffusion

    Feb 5, 2026Zhuokun Chen, Jianfei Cai, Bohan ZhuangDynamic Sparse AttentionDiffusion Language Models

  13. A Constrained Optimization Perspective of Unrolled Transformers

    Jan 24, 2026Javier Porras-Valenzuela, Samar Hadou, Alejandro RibeiroTransformer ArchitecturesConstrained Optimization

  14. Threshold Differential Attention for Sink-Free, Ultra-Sparse, and Non-Dispersive Language Modeling

    Jan 17, 2026Xingyue Huang, Xueying Ding, Mingxuan Ju +3Dynamic Sparse AttentionGlobal Attention Mechanism

  15. Mixture of Distributions Matters: Dynamic Sparse Attention for Efficient Video Diffusion Transformers

    Jan 14, 2026Yuxi Liu, Yipeng Hu, Zekun Zhang +2Pre-Trained Video Diffusion ModelsDynamic Sparse Attention

  16. From Local Windows to Adaptive Candidates via Individualized Exploratory: Rethinking Attention for Image Super-Resolution

    Jan 13, 2026Chunyu Meng, Wei Long, Shuhang GuGenerative Super-ResolutionSelf-Supervised Vision Transformers

  17. SnapGen++: Unleashing Diffusion Transformers for Efficient High-Fidelity Image Generation on Edge Devices

    Jan 13, 2026Dongting Hu, Aarush Gupta, Magzhan Gabidolla +12Multimodal Diffusion TransformersReal-Time Inference

  18. The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models

    Jan 9, 2026Nora Graichen, Iria de-Dios-Flores, Gemma BoledaSyntactic StructureBert-Based Models

  19. TexTailor: Inference-Time Textual Guidance Tailoring for Multimodal Diffusion Transformers

    Jan 5, 2026Binglei Li, Mengping Yang, Zhiyu Tan +2Multimodal Diffusion TransformersText-To-Image

  20. SwinIFS: Landmark Guided Swin Transformer For Identity Preserving Face Super Resolution

    Jan 4, 2026Habiba Kausar, Saeed Anwar, Omar Jamal Hammad +1Diffusion-Based Super-Resolution FrameworkBlind Face Restoration

  21. GeoTransolver: Learning Physics on Irregular Domains Using Multi-scale Geometry Aware Physics Attention Transformer

    Dec 23, 2025Corey Adams, Rishikesh Ranade, Ram Cherukuri +1Representational GeometryFinite Element Method

  22. Probing Spectrum-Like Organization of States of Mind in Transformer Representation Spaces

    Dec 23, 2025Sophie ZhaoTransformer ArchitecturesRepresentation Quality

  23. Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference

    Dec 18, 2025Dhruv Deshmukh, Saurabh Goyal, Nipun Kwatra +1Dynamic Sparse AttentionLLM Inference Efficiency

  24. Sparse-LaViDa: Sparse Multimodal Discrete Diffusion Language Models

    Dec 16, 2025Shufan Li, Jiuxiang Gu, Kangning Liu +4Masked Diffusion Language ModelsDiffusion Language Models

  25. Flexible Gravitational-Wave Parameter Estimation with Transformers

    Dec 2, 2025Annalena Kofler, Maximilian Dax, Stephen R. Green +6Gravitational WavesDiscrete Wavelet Transform

  26. Understanding the Staged Dynamics of Transformers in Learning Latent Structure

    Nov 24, 2025Rohan Saha, Farzane Aminmansour, Alona FysheTransformer ArchitecturesLatent Structure

  27. Equivalence of Context and Parameter Updates in Modern Transformer Blocks

    Nov 22, 2025Adrian Goldwaser, Michael Munn, Javier Gonzalvo +1Transformer LayersModel Weights

  28. Unifying Convolution and Attention via Convolutional Nearest Neighbors

    Nov 18, 2025Mingi Kang, Jeová Farias Sales Rocha NetoConvolutional Neural NetworksConvolutional

  29. A Space-Time Transformer for Precipitation Nowcasting

    Nov 14, 2025Levi Harris, Tianlong ChenPrecipitationAdverse Weather

  30. Emergence of Minimal Circuits for Indirect Object Identification in Attention-Only Transformers

    Oct 28, 2025Rabin AdhikariTransformer AttentionTransformer Architectures

  31. A Unified Geometric Space for Topological Alignment Between Transformer-Based Models and Human Brain Networks

    Oct 28, 2025Silin Chen, Yuzhong Chen, Caiwei Wang +9Human Brain NetworksTransformer Attention

  32. Higher Embedding Dimension Creates a Stronger World Model for a Simple Sorting Task

    Oct 21, 2025Brady Bhalla, Honglu Fan, Nancy Chen +1Transformer ArchitecturesEmbedding Dimension

  33. Performance-Efficiency Tradeoffs in Transformers: An Approximation Theory Perspective

    Oct 4, 2025Ruoxi Yu, Haotian Jiang, Jingpu Cheng +3Transformer ArchitecturesActivation Functions

  34. Uncovering the Computational Ingredients of Human-Like Representations in LLMs

    Oct 1, 2025Zach Studdiford, Timothy T. Rogers, Kushin Mukherjee +1Large Language Model AlignmentMulti-Head Self-Attention

  35. Test time training enhances in-context learning of nonlinear functions

    Sep 30, 2025Kento Kuwataka, Taiji SuzukiIn-Context LearningTransformer Layers

  36. Unveiling the Mechanisms of Multi-Hop Reasoning in Transformers via Identity Bridge

    Sep 29, 2025Pengxiao Lin, Zheng-An Chen, Zhi-Qin John XuTransformer ArchitecturesMultimodal Transformers

  37. Rotary Position Encodings for Graphs

    Sep 26, 2025Isaac Reid, Arijit Sehanobish, Cederik Höfs +7Positional EncodingTransformer Architectures

  38. Multipole Semantic Attention: A Fast Approximation of Softmax Attention for Pretraining

    Sep 12, 2025Rupert Mitchell, Kristian KerstingSoftmax AttentionModel Pretraining

  39. FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernel

    Aug 25, 2025Ran Yan, Youhe Jiang, Zhuoming Chen +3Dynamic Sparse AttentionAlternative

  40. Vivid-VR: Distilling Concepts from Text-to-Video Diffusion Transformer for Photorealistic Video Restoration

    Aug 20, 2025Haoran Bai, Xiaoxu Chen, Canqian Yang +3Pre-Trained Video Diffusion ModelsPhotorealistic Video Restoration

  41. Symmetry-Aware Transformer Training for Automated Planning

    Aug 11, 2025Markus Fritzsche, Elliot Gestrin, Jendrik SeippClassical PlanningTransformer Architectures

  42. A Systematic Analysis of Hybrid Linear Attention

    Jul 8, 2025Dustin Wang, Rui-Jie Zhu, Steven Abreu +9Global Attention MechanismSystematic Study

  43. DETRPose: Real-Time End-to-End Multi-Person Pose Estimation via Modified Transformer Decoder and Novel Denoising Keypoints

    Jun 16, 2025Sebastian Janampa, Marios Pattichis3D Pose EstimatorRt-Detr

  44. FPTQuant: Function-Preserving Transforms for LLM Quantization

    Jun 5, 2025Boris van Breugel, Yelysei Bondarenko, Paul Whatmough +1Large Language Model QuantizationQuantization

  45. Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism

    Jun 2, 2025Sameera Ramasinghe, Thalaiyasingam Ajanthan, Gil Avraham +2Large Language Model CompressionBillion-Parameter Models

  46. Normalizing Flows are Capable Models for Continuous Control

    May 29, 2025Raj Ghugare, Benjamin EysenbachNormalizing FlowsFlow-Based Policy

  47. Freqformer: Image-Demoiréing Transformer via Effective Frequency Decomposition

    May 25, 2025Xiaoyang Liu, Bolin Qiu, Zheng Chen +5Image RestorationImage Denoising

  48. Fast and Accurate Image Restoration and Generation with Rank Enhanced Linear Attention

    May 22, 2025Yuang AiImage RestorationSelf-Supervised Vision Transformers

  49. Quick ViTs: Speeding up Vision Transformers through Equivariance

    May 21, 2025David Nordström, Johan Edstedt, Fredrik Kahl +1Self-Supervised Vision TransformersVision Transformer

  50. The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs

    Apr 24, 2025Piotr Nawrot, Robert Li, Renjie Huang +3Dynamic Sparse AttentionImportance

  51. Fewer yet critical: Reducing Redundant Token Dependencies for Transformer-based Time Series Forecasting

    Mar 10, 2025Jianqi Zhang, Yuchan Liu, Zeen Song +2Transformer ArchitecturesReward-Prediction Error

  52. LLM-Microscope: Uncovering the Hidden Role of Punctuation in Context Memory of Transformers

    Feb 20, 2025Anton Razzhigaev, Matvey Mikhalchuk, Temurbek Rahmatullaev +4Efficient Long-Context InferenceToken-Level Uncertainty

  53. You Do Not Fully Utilize Transformer's Representation Capacity

    Feb 13, 2025Gleb Gerasimov, Yaroslav Aksenov, Nikita Balagansky +2Transformer LayersTransformer Architectures

  54. A Mechanistic Study of Transformers Training Dynamics

    Oct 31, 2024Ambroise Odonnat, Wassim Bouaziz, Vivien CabannesTransformer ArchitecturesModel Pretraining

  55. Tuning Language Models by Mixture-of-Depths Ensemble

    Oct 16, 2024Haoyan Luo, Lucia SpeciaLarge Language Model Fine-TuningTransformer Layers

  56. On Generalisation Error Bounds for Transformers

    Oct 15, 2024Lan V. TruongGeneralization BoundsTransformer Layers

  57. Efficient Concertormer for Image Deblurring and Beyond

    Apr 9, 2024Pin-Hung Kuo, Jinshan Pan, Shao-Yi Chien +1Self-Supervised Vision TransformersMotion Blur

  58. Introduction to Transformers: an NLP Perspective

    Nov 29, 2023Tong Xiao, Jingbo ZhuTransformer ArchitecturesNatural Language Processing

  59. Generalizing Adam to Manifolds for Efficiently Training Transformers

    May 26, 2023Benedikt BrantnerAdamManifolds

  60. SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers

    Date pendingShaowen Wang, Ge Zhang, Kairong Luo +6Transformer LayersTransformer Architectures