Transformer Architectures

Ask about this topic

Momentum

148 papers in the last four weeks, up 164% on the four weeks before. 1.5% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 1,120

All topics
CardsList

  1. Applications of the Transformer Architecture in AI-Assisted English Reading Comprehension

    Apr 26, 2026Ping LiTransformer Architectures

    a Shandong Management University, Jinan, Shandong Province, 250357, China

  2. Hybrid JIT-CUDA Graph Optimization for Low-Latency Large Language Model Inference

    Apr 25, 2026Divakar Kumar Yadav, Tian ZhaoLLM Inference OptimizationInference Latency

    Department of Computer Science University of Wisconsin-Milwaukee Milwaukee, WI, USA

  3. TEMPO: Transformers for Temporal Disease Progression from Cross-Sectional Data

    Apr 25, 2026Hongtao Hao, Joseph L. AusterweilBiomarkerTransformer Architectures

    University of Wisconsin—Madison, USA · Chiba Institute of Technology, Japan & University of Wisconsin—Madison, USA

  4. Robust Audio-Text Retrieval via Cross-Modal Attention and Hybrid Loss

    Apr 25, 2026Meizhu Liu, Matthew Rowe, Amit Agarwal +8Text-To-AudioCross-Modal Attention

  5. Toeplitz MLP Mixers are Low Complexity, Information-Rich Sequence Models

    Apr 24, 2026Benjamin L. Badger, Ethan RolandAsymmetric Grouped-Attention Variate MixerTransformer Architectures

    IBM · AE Studio

  6. Adaptive Head Budgeting for Efficient Multi-Head Attention

    Apr 24, 2026Bilal Faye, Abdoulaye Mbaye, Hanane Azzag +1Multi-Head AttentionTransformer Architectures

  7. SpikingBrain2.0: Brain-Inspired Foundation Models for Efficient Long-Context and Cross-Platform Inference

    Apr 24, 2026Yuqi Pan, Jinghao Zhuang, Yupeng Feng +16Dynamic Sparse AttentionEfficient Long-Context Inference

    Institute of Automation, Chinese Academy of Sciences · The Hong Kong Polytechnic University · Beijing Key Laboratory of Brain-Inspired General Intelligence Large Model +1

  8. Learning Reactive Human Motion Generation from Paired Interaction Data Using Transformer-Based Models

    Apr 24, 2026Masato Soga, Ryuki TakebayashiHuman Motion GenerationHuman Motion

    Faculty of Systems Engineering Graduate School of Systems Engineering Wakayama University

  9. Dissociating Decodability and Causal Use in Bracket-Sequence Transformers

    Apr 24, 2026Aryan Sharma, Cutter Dawes, Shivam RavalTransformer EncoderTransformer Architectures

    1Yale University · 2Independent · 3Harvard University

  10. LayerBoost: Layer-Aware Attention Reduction for Efficient LLMs

    Apr 23, 2026Mohamed Ali Souibgui, Jan Fostier, Rodrigo Abadía-Heredia +3Transformer ArchitecturesEnsemble

    Openchip & Software Technologies, S.L.

  11. Universal Transformers Need Memory: Depth-State Trade-offs in Adaptive Recursive Reasoning

    Apr 23, 2026Grigory SapunovBounded-MemoryTransformer Architectures

    Intento

  12. To See the Unseen: on the Generalization Ability of Transformers in Symbolic Reasoning

    Apr 23, 2026Nevena Lazić, Liam Fowl, András György +1Transformer EncoderTransformer Architectures

  13. Sculpt4D: Generating 4D Shapes via Sparse-Attention Diffusion Transformers

    Apr 23, 2026Minghao Yin, Wenbo Hu, Jiale Xu +24D GenerationFour-Dimensional

    1The University of Hong Kong · 2ARC Lab, Tencent PCG

  14. Reasoning Primitives in Hybrid and Non-Hybrid LLMs: Do Architectural Differences Yield Advantages in State-Tracking and Recall?

    Apr 23, 2026Shivam Rawat, Lucie Flek, Florian Mai +1LLM Reasoning StrategiesSearch-Augmented Reasoning

    Lamarr Institute for Machine Learning and Artificial Intelligence · Rheinische Friedrich-Wilhelms-Universität Bonn

  15. EdgeFormer: local patch-based edge detection transformer on point clouds

    Apr 23, 2026Yifei Xie, Zhikun Tu, Tong Yang +2Edge-AwarePoint Clouds

    School of Information Science and Technology, Northwest University, 2026 Xuefu Street, Xi’an, 710127, Shannxi, China.

  16. Sub-Token Routing for KV Cache Compression

    Apr 23, 2026Wei Jiang, Wei WangKey-Value Cache CompressionToken Compression

    Futurewei Technologies San Jose, CA 95131, USA

  17. an interpretable vision transformer framework for automated brain tumor classification

    Apr 23, 2026Chinedu Emmanuel Mbonu, Tochukwu Sunday Belonwu, Okwuchukwu Ejike Chukwuogo +1Brain Tumor ClassificationSelf-Supervised Vision Transformers

    Department of Computer Science, Nazarbayev University, Astana · Department of Computer Science, Nnamdi Azikiwe University, Nigeria

  18. Hyperloop Transformers

    Apr 23, 2026Abbas Zeitoun, Lucas Torroba-Hennigen, Yoon KimTransformer ArchitecturesEnergy-Based Transformer

    Massachusetts Institute of Technology

  19. The Recurrent Transformer: Greater Effective Depth and Efficient Decoding

    Apr 23, 2026Costin-Andrei Oncescu, Depen Morwani, Samy Jelassi +3Transformer ArchitecturesKv-Cache Management

    Harvard University

  20. climt-paraformer: Stable Emulation of Convective Parameterization using a Temporal Memory-aware Transformer

    Apr 22, 2026Shuochen Wang, Nishant Yadav, Joy Merwin Monteiro +1Global Climate ModelsEmulators

    Sustainability and Data Sciences Laboratory, Northeastern University · Microsoft · Indian Institute of Science Education and Research Pune +2

  21. PanGuide3D: Cohort-Robust Pancreas Tumor Segmentation via Probabilistic Pancreas Conditioning and a Transformer Bottleneck

    Apr 22, 2026Sunny Joy Ma, Xiang MaTumor SegmentationPanoptic Segmentation

    Trailridge School, Waukee, IA, 50263, USA · Department of Biochemistry, Grand View University, Des Moines, IA 50316, USA · Department of Computer Science, Iowa State University, Ames, IA 50011, USA

  22. Working Memory Constraints Scaffold Learning in Transformers under Data Scarcity

    Apr 22, 2026Pranava Madhyastha, Dagmar AdamcovaTransformer ArchitecturesWorking Memory

    △City, University of London · ∞The Alan Turing Institute · ⊕Grounded Machines

  23. Explicit Dropout: Deterministic Regularization for Transformer Architectures

    Apr 22, 2026Vidhi Agrawal, Illia Oleksiienko, Alexandros IosifidisDropoutTransformer Attention

    Faculty of Information Technology and Communications,Tampere University, Tampere, 33720, Finland · Department of Electrical and Computer Engineering, Aarhus University, Aarhus, 8000, Denmark

  24. MAE-Based Self-Supervised Pretraining for Data-Efficient Medical Image Segmentation Using nnFormer

    Apr 22, 2026R. M. Krishna Sureddi, T. Satyanarayana Murthy, Nomula Varsha Reddy +2Semi-Supervised Medical Image SegmentationSelf-Supervised Learning

    Information Technology, Chaitanya Bharathi Institute of Technology, Hyderabad, India

  25. LaplacianFormer:Rethinking Linear Attention with Laplacian Kernel

    Apr 22, 2026Zhe Feng, Sen Lian, Changwei Wang +5Linear AttentionTransformer Architectures

    1MAIS,Institute of Automation, Chinese Academy of Sciences · usr 3China Electronics Data Corporation · 5The Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Shandong Computer Science Center, Qilu University of Technology +4

  26. PASTA: A Patch-Agnostic Twofold-Stealthy Backdoor Attack on Vision Transformers

    Apr 21, 2026Dazhuang Liu, Yanqi Qiao, Rui Wang +2Clean Label Backdoor AttackVision Transformer

    Delft University of Technology, Delft, the Netherlands. · University of Turku, Turku, Finland

  27. Benign Overfitting in Adversarial Training for Vision Transformers

    Apr 21, 2026Jiaming Zhang, Meng Ding, Shaopeng Fu +2Adversarial TrainingSelf-Supervised Vision Transformers

    Division of CEMSE, King Abdullah University of Science and Technology · School of Statistics, Renmin University of China · Department of Computer Science and Engineering, State University of New York at Buffalo +1

  28. GRAFT: Geometric Refinement and Fitting Transformer for Human Scene Reconstruction

    Apr 21, 2026Pradyumna YM, Yuxuan Xue, Yue Chen +3Scene ReconstructionArticulated Objects

    University of Tübingen · Tübingen AI Center · Westlake University +1

  29. Volume Transformer: Revisiting Vanilla Transformers for 3D Scene Understanding

    Apr 21, 2026Kadir Yilmaz, Adrian Kruse, Tristan Höfer +23D Scene UnderstandingVisual Geometry Grounded Transformer

    RWTH Aachen University · Eindhoven University of Technology

  30. EX-FIQA: Leveraging Intermediate Early eXit Representations from Vision Transformers for Face Image Quality Assessment

    Apr 21, 2026Guray Ozgur, Tahar Chettaoui, Eduarda Caldeira +4Self-Supervised Vision TransformersImage Quality Assessment

    Fraunhofer Institute for Computer Graphics Research IGD, Darmstadt, Germany · Department of Computer Science, TU Darmstadt, Germany

  31. Multimodal Transformer for Sample-Aware Prediction of Metal-Organic Framework Properties

    Apr 21, 2026Seunghee Han, Jaewoong Lee, Jihan KimMolecular Property PredictionPowder X-Ray Diffraction

    Department of Chemical and Biomolecular Engineering, Korea Advanced Institute of Science and Technology, Daejeon 34141, Republic of Korea. · Department of Materials Science and Engineering, Korea Advanced Institute of Science and Technology, Daejeon 34141, Republic of Korea.

  32. Multi-view Crowd Tracking Transformer with View-Ground Interactions Under Large Real-World Scenes

    Apr 21, 2026Qi Zhang, Jixuan Chen, Kaiyi Zhang +3Multi-Object TrackingMulti-View

    College of Computer Science and Software Engineering, Shenzhen University, China · Department of Computer Science, City University of Hong Kong, China

  33. Multimodal embodiment-aware navigation transformer

    Apr 21, 2026Louis Dezons, Quentin Picard, Rémi Marsal +2Object Goal NavigationMultimodal Fusion

    AMIAD Pˆole Recherche, Palaiseau, France · U2IS, ENSTA, Institut Polytechnique de Paris, Palaiseau, France

  34. One-Block Transformer (1BT) for EEG-Based Cognitive Workload Assessment

    Apr 21, 2026Stefanos Gkikas, Christian Arzate Cruz, Thomas Kassiotis +3Inference WorkloadsCognitive Load

    Honda Research Institute Japan · Wako City, Japan · Department of Electronic Engineering +6

  35. ConvVitMamba: Efficient Multiscale Convolution, Transformer, and Mamba-Based Sequence modelling for Hyperspectral Image Classification

    Apr 20, 2026Mohammed Q. AlkhatibHyperspectral ImageMulti-Scale Convolution

    College of Engineering and IT, University of Dubai, Dubai 14143, United Arab Emirates

  36. One Step Forward and K Steps Back: Better Reasoning with Denoising Recursion Models

    Apr 20, 2026Chris Cameron, Wangzheng Wang, Nikita Ivanov +3Denoising Diffusion Probabilistic ModelRecursive Models

    Huawei Technologies · University of Waterloo · University of Toronto

  37. Towards Understanding the Robustness of Sparse Autoencoders

    Apr 20, 2026Ahson Saiyed, Sabrina Sadiekh, Chirag AgarwalLarge Language Model JailbreaksTransformer Architectures

    University of Virginia · Independent Researcher

  38. URoPE: Universal Relative Position Embedding across Geometric Spaces

    Apr 20, 2026Yichen Xie, Depu Meng, Chensheng Peng +4Visual Geometry Grounded TransformerRotary Position Embedding

    Applied Intuition · University of California, Berkeley

  39. Linear-Time and Constant-Memory Text Embeddings Based on Recurrent Language Models

    Apr 20, 2026Tobias Grantner, Emanuel Sallinger, Martin FlechlHybrid Mamba-Transformer ModelTransformer Architectures

    Dynatrace Research · TU Wien · University of Oxford

  40. Towards Real-Time ECG and EMG Modeling on μμNPUs

    Apr 20, 2026Josh Millar, Ashok Samraj Thangarajan, Soumyajit Chatterjee +1Physiological SignalsNeural Processing Units

    Imperial College London London, United Kingdom · Nokia Bell Labs London, United Kingdom · Brave Software & University of Cambridge London, United Kingdom

  41. Chatting about Upper-Body Expressive Human Pose and Shape Estimation

    Apr 20, 2026Yuxiang Zhao, Wei Huang, Yujie Song +2Statistical Shape ModelTransformer Architectures

    Sun Yat-sen University · Alibaba Group

  42. How Much Cache Does Reasoning Need? Depth-Cache Tradeoffs in KV-Compressed Transformers

    Apr 20, 2026Xiao WangKey-Value Cache CompressionKey-Value Cache

  43. Attention Is not Everything: Efficient Alternatives for Vision

    Apr 19, 2026Nur Mohammad Kazi, Ibteshum Khaled, Md. Luthful Hasan Galib +2Self-Supervised Vision TransformersComputer Vision

    Ahsanullah University of Science and Technology Dhaka, Bangladesh

  44. Towards Generalizable Deepfake Image Detection with Vision Transformers

    Apr 19, 2026Kaliki V Srinanda, M Manvith Prabhu, Hemanth K Mogilipalem +4Deepfake DetectionSelf-Supervised Vision Transformers

    Department of Electronics and Communication Engineering National Institute of Technology Karnataka (NITK), Surathkal - 575025, India · Department of Information Technology National Institute of Technology Karnataka (NITK), Surathkal - 575025, India · Department of Electrical and Electronics Engineering National Institute of Technology Karnataka (NITK), Surathkal - 575025, India

  45. The Topological Trouble With Transformers

    Apr 18, 2026Michael C. Mozer, Shoaib Ahmed Siddiqui, Rosanne LiuTransformer ArchitecturesFeed-Forward

    Google DeepMind

  46. One Mask to Rule Them All: On Hidden Facts after Editing and How to Find Them

    Apr 18, 2026Ali Holmov, Paul Youssef, Nandi Schoots +1Multimodal Knowledge EditingModifications

    Technical University of Munich · Marburg University · University of Oxford

  47. NaviFormer: A Deep Reinforcement Learning Transformer-like Model to Holistically Solve the Navigation Problem

    Apr 18, 2026Daniel Fuertes, Andrea Cavallaro, Carlos R. del-Blanco +2Path PlanningOffline Reinforcement Learning

    Grupo de Tratamiento de Im´agenes, Information Processing and Telecommunications Center, ETSI Telecomunicaci´on, Universidad Polit´ecnica de Madrid, 28040 Madrid, Spain · Idiap Research Institute, Martigny, Switzerland

  48. Adaptive Computation Depth via Learned Token Routing in Transformers

    Apr 18, 2026Ahmed Abdelmuniem Abdalla MohammedTransformer ArchitecturesMultilayer Perceptrons

    Independent Researcher

  49. Closing the Theory-Practice Gap in Spiking Transformers via Effective Dimension

    Apr 17, 2026Dongxin Guo, Jikun Wu, Siu Ming YiuNeuromorphic ComputingTransformer Architectures

    *The University of Hong Kong, Hong Kong, China · †Brain Investing Limited, Hong Kong, China

  50. Stability and Generalization in Looped Transformers

    Apr 16, 2026Asher LabovichTransformer ArchitecturesFixed-Point Iteration

  51. What Is the Minimum Architecture for Prolepsis? Early Irrevocable Commitment Across Tasks in Small Transformers

    Apr 16, 2026Éric JacopinTransformer ArchitecturesCommitment

    Cosmic AI

  52. Continued Pretraining of FinBERT on Finnish Histopathological Reports: Train-Time Signals and Proxy Downstream Correlations

    Apr 16, 2026Rami Luisto, Liisa Petäinen, Tommi Grönholm +5Bert-Based ModelsLarge Language Model Classification

    Faculty of Information Technology, University of Jyväskylä, Jyväskylä, Finland. · Digital Workforce Services, Helsinki, Finland. · Heart and Lung Center, Helsinki University Hospital, Helsinki, Finland. +2

  53. Personalized and Context-Aware Transformer Models for Predicting Post-Intervention Physiological Responses from Wearable Sensor Data

    Apr 16, 2026Esther Brown, Victoria Dean, Finale Doshi-VelezWearable PhysiologyPhysiological Data

    Harvard University, Cambridge, MA · Princeton University, Princeton, NJ

  54. Expressivity of Transformers: A Tropical Geometry Perspective

    Apr 16, 2026Ye Su, Yong LiuTransformer AttentionMulti-Head Attention

    Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China · University of Chinese Academy of Sciences, Beijing 101407, China · Gaoling School of Artificial Intelligence, Renmin University of China, Beijing 100872, China

  55. FreqTrack: Frequency Learning based Vision Transformer for RGB-Event Object Tracking

    Apr 16, 2026Jinlin You, Muyu Li, Xudong ZhaoRgb-Event Object TrackingRgbt

    Dalian University of Technology

  56. THEIA: Learning Complete Kleene Three-Valued Logic in a Pure-Neural Modular Architecture

    Apr 13, 2026Augustus Haoyang LiLogicMultilayer Perceptrons

    Irvine Valley College Irvine, CA, USA

  57. Relational Preference Encoding in Looped Transformer Internal States

    Apr 10, 2026Jan KirinPreference Alignment LearningTransformer Architectures

  58. LSRM: High-Fidelity Object-Centric Reconstruction via Scaled Context Windows

    Apr 6, 2026Zhengqin Li, Cheng Zhang, Jakob Engel +1Feed-Forward 3D ReconstructionNovel View Synthesis

    Meta Reality Labs Research

  59. FusionBERT: Multi-View Image--3D Retrieval via Cross-Attention Visual Fusion and Normal-Aware 3D Encoder

    Apr 2, 2026Wei Li, Yufan Ren, Hanqing Jiang +6Cross-View FusionProgressive Attention Fusion

    IROOTECH TECHNOLOGY, Hangzhou, Zhejiang, China · Wolf 1069 b Lab, Sany Group, Hangzhou, Zhejiang, China · Zhejiang University, Hangzhou, Zhejiang, China +4

  60. Crystalite: A Lightweight Transformer for Efficient Crystal Modeling

    Apr 2, 2026Tin Hadži Veljković, Joshua Rosenthal, Ivor Lončarić +1Crystal StructureCrystal

    1UvA-Bosch Delta Lab · University of Amsterdam · 3Ruđer Bošković Institute