Transformer Architectures

Follow this topic Ask about this topic

Momentum

148 papers in the last four weeks, up 164% on the four weeks before. 1.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,120

All topics

GenGait: A Transformer-Based Model for Human Gait Anomaly Detection and Normative Twin Generation
Apr 2, 2026Elisa Motta, Marta Lorenzini, Clara Mouawad +3Gait Recognition Gait Analysis
HRI2 Laboratory, Istituto Italiano di Tecnologia (IIT), Genoa, Italy · Department of Occupational and Environmental Medicine, Epidemiology and Hygiene, INAIL, Rome, Italy · Department of Medical and Surgical Sciences and Biotechnologies, Sapienza University of Rome, Rome, Italy
Sampling at intermediate temperatures is optimal for training large language models in protein structure prediction
Mar 31, 2026L. Ghiringhelli, A. Zambon, G. TianaProtein Structure Prediction Transformer Architectures
Department of Physics, via Celoria 16, 20133 Milano, Italy · Department of Physics and INFN, via Celoria 16, 20133 Milano, Italy
ExFusion: Efficient Transformer Training via Multi-Experts Fusion
Mar 30, 2026Jiacheng Ruan, Daize Dong, Xiaoye Qu +5Transformer Architectures Progressive Attention Fusion
School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, No 800, Dongchuan Road, Minhang District, Shanghai, China, 200240 · Shanghai Artificial Intelligence Laboratory, Shanghai, China, 200240 · Huazhong University of Science and Technology, Hubei, China, 430074 +3
When Perplexity Lies: Generation-Focused Distillation of Hybrid Sequence Models
Mar 27, 2026Juan Gabriel Kostelec, Qinghai GuoEffective Distillation Symbolic Distillation
Huawei Switzerland · ACS Lab, Huawei Technologies
OptiSAR-Net++: A Large-Scale Benchmark and Transformer-Free Framework for Cross-Domain Remote Sensing Visual Grounding
Mar 25, 2026Xiaoyu Tang, Jun Dong, Jintao Cheng +1Remote Sensing Sequential Visual Localization
School of Electronics and Information Engineering, and Xingzhi College, South China Normal University, Foshan 528225, China · School of Data Science and Engineering, and Xingzhi College, South China Normal University, Shanwei, 516600, China · Department of Electronic and Computer Engineering, Hong Kong University of Science and Technology, Hong Kong SAR, China +1
Thinking Deeper, Not Longer: Memory-Efficient Test-Time Reasoning with Depth-Recurrent Transformers for Compositional Generalization
Mar 23, 2026Hung-Hsuan ChenTransformer Architectures Chain-of-Thought Reasoning
Computer Science and Information Engineering National Central University Taoyuan, Taiwan
A Two-stage Transformer Framework for Temporal Localization of Distracted Driver Behaviors
Mar 22, 2026Gia-Bao Doan, Nam-Khoa Huynh, Minh-Nhat-Huy Ho +3Temporal Action Localization Human Driving
Department of Information Technology FPT University Ho Chi Minh City 700000, Vietnam · Faculty of Electronics Technology Industrial University of Ho Chi Minh City Ho Chi Minh City 700000, Vietnam · School of Computer Science and Engineering The Saigon International University Ho Chi Minh City 700000, Vietnam
On the Ability of Transformers to Verify Plans
Mar 20, 2026Yash Sarrof, Yupei Du, Katharina Stein +3Classical Planning Transformer Architectures
Saarland University · Australian National University · University of Toulouse / LAAS-CNRS
Dual Path Attribution: Efficient Attribution for SwiGLU-Transformers through Layer-Wise Target Propagation
Mar 20, 2026Lasse Marten Jantsch, Dong-Jae Koh, Seonghyeon Lee +1Transformer Attention Transformer Architectures
Department of Computer Science and Engineering, Kyungpook National University, Daegu, Korea.
Effective Distillation to Hybrid xLSTM Architectures
Mar 16, 2026Lukas Hauzenberger, Niklas Schmidinger, Thomas Schmied +7Probe-Logit Distillation Effective Distillation
1NXAI · 2Johannes Kepler University Linz
MoE-ACT: Scaling Multi-Task Bimanual Manipulation with Sparse Task-Conditioned Mixture-of-Experts Transformers
Mar 16, 2026Kangjun Guo, Haichao Liu, Yanji Sun +3Bimanual Manipulation Robotic Manipulation
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, China · The Hong Kong University of Science and Technology, Hong Kong SAR, China
Human-like Object Grouping in Self-supervised Vision Transformers
Mar 14, 2026Hossein Adeli, Seoyoung Ahn, Andrew Luo +3Self-Supervised Vision Transformers Self-Supervised Learning
Zuckerman Mind Brain Behavior Institute, Columbia University, New York · Department of Social Science and AI, Hankuk University of Foreign Studies, Seoul · University of Hong Kong, Hong Kong +2
AccelAes: Accelerating Diffusion Transformers for Training-Free Aesthetic-Enhanced Image Generation
Mar 13, 2026Xuanhua Yin, Chuanzhi Xu, Haoxian Zhou +2Diffusion Transformers Image Generation
School of Computer Science, The University of Sydney, NSW 2006, Australia
HiAP: A Multi-Granular Stochastic Auto-Pruning Framework for Vision Transformers
Mar 12, 2026Andy Li, Aiden Durrant, Milan Markovic +1Self-Supervised Vision Transformers Vision Transformer
School of Natural and Computing Sciences, University of Aberdeen · School of Computing Sciences, University of East Anglia · Department of Physics and Technology, UiT The Arctic University of Norway
Feature-level Interaction Explanations in Multimodal Transformers
Mar 4, 2026Yeji Kim, Housam Khalifa Bashier Babiker, Mi-Young Kim +1Cross-Modal Detection Transformer
University of Alberta, Edmonton, AB, Canada · University of Waikato, Hamilton, New Zealand · University of Alberta, Camrose, AB, Canada
A novel network for classification of cuneiform tablet metadata
Mar 4, 2026Frederik HagelskjærPoint Clouds Historical Manuscripts
SDU Robotics · The Mærsk Mc-Kinney Møller Institute · University of Southern Denmark +1
Retrievit: In-context Retrieval Capabilities of Transformers, State Space Models, and Hybrid Architectures
Mar 3, 2026Georgios Pantazopoulos, Malvina Nikandrou, Ioannis Konstas +1Transformer Architectures Positional Encoding
University of Edinburgh · Heriot-Watt University
MetaOthello: A Controlled Study of Multiple World Models in Transformers
Feb 26, 2026Aviral Chawla, Galen Hall, Juniper LovatoTransformer Architectures World Models
Vermont Complex Systems Institute, University of Vermont, Burlington, VT · University of Michigan, Ann Arbor, MI
Transformers converge to invariant algorithmic cores
Feb 26, 2026Joshua S. SchiffmanTransformer Architectures Algorithmic Cores
New York Genome Center, New York, NY, USA.
Bridging Latent Reasoning and Target-Language Generation via Retrieval-Transition Heads
Feb 25, 2026Shaswat Patel, Vishvesh Trivedi, Yue Han +2Multilingual Language Models Efficient Latent Reasoning
New York University
How Transformers Reject Wrong Answers: Rotational Dynamics of Factual Constraint Processing
Feb 25, 2026Javier MarínTransformer Encoder Transformer Architectures
Groundlens-dev
Incremental Learning of Sparse Attention Patterns in Transformers
Feb 22, 2026Oğuz Kaan Yüksel, Rodrigo Alvarez Lucendo, Nicolas FlammarionTransformer Attention Transformer Architectures
TML Lab, EPFL
Context-aware Skin Cancer Epithelial Cell Classification with Scalable Graph Transformers
Feb 17, 2026Lucas Sancéré, Noémie Moreau, Katarzyna BozekMelanoma Whole-Slide Images
Faculty of Mathematics and Natural Sciences, University of Cologne, Cologne, North Rhine-Westphalia, Germany · Institute for Biomedical Informatics, Faculty of Medicine and University Hospital Cologne, University of Cologne, North Rhine-Westphalia, Germany · Center for Molecular Medicine Cologne (CMMC), Faculty of Medicine and University Hospital Cologne, University of Cologne, North Rhine-Westphalia, Germany +1
Universal priors: solving empirical Bayes via Bayesian inference and pretraining
Feb 16, 2026Nick Cannella, Anzo Teh, Yanjun Han +1Empirical Bayes Bayesian Inference
Courant Institute of Mathematical Sciences, NYU, New York City, NY · Department of EECS, MIT, Cambridge, MA · Courant Institute of Mathematical Sciences and Center for Data Science, NYU, New York City, NY
Compressed Sensing for Capability Localization in Large Language Models
Feb 11, 2026Anna Bair, Yixuan Even Xu, Mingjie Sun +1Large Language Model Compression Transformer Architectures
Carnegie Mellon University
Central Dogma Transformer II: An AI Microscope for Understanding Cellular Regulatory Mechanisms
Feb 9, 2026Nobuyuki OtaRegulatory Networks Regulation
Independent Researcher, Burlingame, CA, USA
Language-Guided Transformer Tokenizer for Human Motion Generation
Feb 9, 2026Sheng Yan, Yong Wang, Xin Du +2Human Motion Generation Tokenizer
1Transsion Ltd. · 2Chongqing University of Technology · 3Afari Intelligence Drive +2
Revisiting the Shape Convention of Transformer Language Models
Feb 6, 2026Feng-Ting Liao, Guan-Ting Yi, Tzu-Quan Lin +2Transformer Architectures Deep Learning Architectures
MediaTek Research · National Taiwan University
Poly-attention: a general scheme for higher-order self-attention
Feb 2, 2026Sayak Chakrabarti, Toniann Pitassi, Josh AlmanLinear Attention Transformer Architectures
Computer Science, Columbia University, New York, NY 10027.
Plain Transformers are Surprisingly Powerful Link Predictors
Feb 2, 2026Quang Truong, Yu Song, Donald Loveland +4Zero-Shot Link Prediction Graph Neural Networks
Department of Computer Science and Engineering, Michigan State University, East Lansing, MI, USA. · Snap Inc., Bellevue, WA, USA.
Semi-supervised CAPP Transformer Learning via Pseudo-labeling
Feb 1, 2026Dennis Gross, Helge Spieker, Arnaud Gotlieb +3Semi-Supervised Learning Transformer Architectures
1- Simula Research Laboratory, Oslo Norway · 2- National Technical University of Athens, School of Mechanical Engineering, Manufacturing Technology Laboratory Greece
A Constrained Optimization Perspective of Unrolled Transformers
Jan 24, 2026Javier Porras-Valenzuela, Samar Hadou, Alejandro RibeiroTransformer Architectures Constrained Optimization
Department of Electrical and Systems Engineering, University of Pennsylvania, Philadelphia, PA, United States of America.
Human Values in a Single Sentence: Moral Presence, Hierarchies, and Transformer Ensembles on the Schwartz Continuum
Jan 20, 2026Víctor Yeste, Paolo RossoHuman Values Moral Reasoning
PRHLT Research Center, Universitat Politècnica de València, Valencia, 46022, Spain · School of Science, Engineering and Design, Universidad Europea de Valencia, Valencia, 46010, Spain · Valencian Graduate School and Research Network of Artificial Intelligence (ValgrAI)
From Local Windows to Adaptive Candidates via Individualized Exploratory: Rethinking Attention for Image Super-Resolution
Jan 13, 2026Chunyu Meng, Wei Long, Shuhang GuReal-World Image Super-Resolution Transformer Architectures
University of Electronic Science and Technology of China
The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models
Jan 9, 2026Nora Graichen, Iria de-Dios-Flores, Gemma BoledaSyntactic Structure Transformer Architectures
Universitat Pompeu Fabra1 · ICREA2
TexTailor: Inference-Time Textual Guidance Tailoring for Multimodal Diffusion Transformers
Jan 5, 2026Binglei Li, Mengping Yang, Zhiyu Tan +2Diffusion Transformers Transformer Architectures
Fudan University, Shanghai 200433, China · Shanghai Academy of AI for Science, Shanghai 200030, China · Shanghai Innovation Institute, Shanghai 200030, China
Hidden State Poisoning Attacks against Mamba-based Language Models
Jan 5, 2026Alexandre Le Mercier, Chris Develder, Thomas DemeesterHybrid Mamba-Transformer Model Attacker Large Language Model
IDLab–T2K, Ghent University–imec
SwinIFS: Landmark Guided Swin Transformer For Identity Preserving Face Super Resolution
Jan 4, 2026Habiba Kausar, Saeed Anwar, Omar Jamal Hammad +1Generative Super-Resolution Blind Face Restoration
King Fahd University of Petroleum and Minerals, KSA · The University of Western Australia, Australia · The University of Regina, Canada
GeoTransolver: Learning Physics on Irregular Domains Using Multi-scale Geometry Aware Physics Attention Transformer
Dec 23, 2025Corey Adams, Rishikesh Ranade, Ram Cherukuri +1Geometric Constraints Spatio-Temporal Transformers
NVIDIA Corporation, Santa Clara, US
Probing Spectrum-Like Organization of States of Mind in Transformer Representation Spaces
Dec 23, 2025Sophie ZhaoTransformer Architectures Representation Quality
School of Computer Science Georgia Institute of Technology
Flexible Gravitational-Wave Parameter Estimation with Transformers
Dec 2, 2025Annalena Kofler, Maximilian Dax, Stephen R. Green +6Gravitational Waves Canada-France-Hawaii Telescope Data
Max Planck Institute for Intelligent Systems, Max-Planck-Ring 4, 72076 Tübingen, Germany · ELLIS Institute Tübingen, Maria-von-Linden-Straße 2, 72076 Tübingen, Germany · Tübingen AI Center, Maria-von-Linden-Straße 1, 72076 Tübingen, Germany +4
Harmonic-Percussive Disentangled Neural Audio Codec for Bandwidth Extension
Nov 26, 2025Benoît Giniès, Xiaoyu Bie, Olivier Fercoq +1Neural Audio Codecs Audio Tokenizers
LTCI, Télécom Paris, Institut Polytechnique de Paris Palaiseau, France
Understanding the Staged Dynamics of Transformers in Learning Latent Structure
Nov 24, 2025Rohan Saha, Farzane Aminmansour, Alona FysheTransformer Architectures Language Modeling
Department of Computing Science University of Alberta Edmonton, Canada · Department of Computing Science and Psychology University of Alberta Edmonton, Canada
Equivalence of Context and Parameter Updates in Modern Transformer Blocks
Nov 22, 2025Adrian Goldwaser, Michael Munn, Javier Gonzalvo +1Transformer Architectures Model Weights
University of Cambridge, UK · Google Research
A Space-Time Transformer for Precipitation Nowcasting
Nov 14, 2025Levi Harris, Tianlong ChenPrecipitation Nowcasting Artificial Intelligence Weather Models
UNC Chapel Hill
You Had One Job: Per-Task Quantization Using LLMs' Hidden Representations
Nov 9, 2025Amit LeVi, Raz Lapid, Rom Himelstein +3Post-Training Quantization Transformer Architectures
1Technion—Israel Institute of Technology · 2Intuit · 3Ben-Gurion University of the Negev +1
Controllably Efficient Language Models
Nov 7, 2025Jatin Prakash, Aahlad Puli, Rajesh RanganathTransformer Architectures Linear Attention
New York University
Emergence of Minimal Circuits for Indirect Object Identification in Attention-Only Transformers
Oct 28, 2025Rabin AdhikariTransformer Attention Transformer Architectures
Saarland University 66123 Saarbrücken
A Unified Geometric Space for Topological Alignment Between Transformer-Based Models and Human Brain Networks
Oct 28, 2025Silin Chen, Yuzhong Chen, Caiwei Wang +9Human Brain Networks Transformer Attention
The Clinical Hospital of Chengdu Brain Science Institute, MOE-K Lab for NeuroInformation, Brain-Apparatus Communication Institute, School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu. 611731, China. · School of Automation, Northwestern Polytechnical University, Xi’an 710072, China. · Department of Biomedical Engineering, New Jersey Institute of Technology, Newark, NJ 07102, USA. +1
Beyond Semantics: How Temporal Biases Shape Retrieval in Transformer and State-Space Models
Oct 26, 2025Anooshka Bajaj, Deven Mahesh Mistry, Sahaj Singh Maini +2In-Context Learning Large Language Model Memory
Department of Computer Science Indiana University Bloomington
Higher Embedding Dimension Creates a Stronger World Model for a Simple Sorting Task
Oct 21, 2025Brady Bhalla, Honglu Fan, Nancy Chen +1Transformer Architectures Behavioral Embeddings
California Institute of Technology · Pasadena, CA 91125 · Google DeepMind +3
Scaling Vision Transformers for Functional MRI with Flat Maps
Oct 15, 2025Connor Lane, Mihir Tripathy, Leema Krishna Murali +15Functional Magnetic Resonance Imaging Vision Transformer
MedARC · Sophont · Baylor College of Medicine +3
Performance-Efficiency Tradeoffs in Transformers: An Approximation Theory Perspective
Oct 4, 2025Ruoxi Yu, Haotian Jiang, Jingpu Cheng +3Transformer Attention Transformer Architectures
Center for Data Science, Peking University · Institute for Functional Intelligent Materials, National University of Singapore · Department of Mathematics, National University of Singapore +1
How Can Mamba Learn In Context with Outliers and Generalize Provably?
Oct 1, 2025Hongkang Li, Songtao Lu, Xiaodong Cui +2Mamba-Based Models In-Context Learning
Department of Electrical and Systems Engineering, University of Pennsylvania, Philadelphia, PA, USA · Department of Computer Science and Engineering, The Chinese University of Hong Kong, Sha Tin, New Territories, Hong Kong · IBM Thomas J. Watson Research Center, Yorktown Heights, NY, USA +1
Test time training enhances in-context learning of nonlinear functions
Sep 30, 2025Kento Kuwataka, Taiji SuzukiIn-Context Learning Test-Time Training
Department of Mathematical Engineering and Information Physics The University of Tokyo Hongo 7-3-1, Bunkyo-ku, Tokyo
Unveiling the Mechanisms of Multi-Hop Reasoning in Transformers via Identity Bridge
Sep 29, 2025Pengxiao Lin, Zheng-An Chen, Zhi-Qin John XuTransformer Architectures Multi-Hop Reasoning
School of Mathematical Sciences, Shanghai Jiao Tong University · Institute of Natural Sciences, MOE-LSC, Shanghai Jiao Tong University · Shanghai Seres Information Technology Co., Ltd, Shanghai 200040, China.
HyMaTE: A Hybrid Mamba and Transformer Model for EHR Representation Learning
Sep 28, 2025Md Mozaharul Mottalib, Thao-Ly T. Phan, Rahmatollah BeheshtiMedical World Model Hybrid Transformer
University of Delaware Newark, Delaware, USA · Thomas Jefferson University & Nemours Children’s Health Wilmington, Delaware, USA
Rotary Position Encodings for Graphs
Sep 26, 2025Isaac Reid, Arijit Sehanobish, Cederik Höfs +7Positional Encoding Transformer Architectures
University of Cambridge · Google DeepMind · Independent Researcher +3
Geometrically Constrained and Token-Based Probabilistic Spatial Transformers
Sep 14, 2025Johann Schmidt, Tom Siegl, Martin Becker +1Spatio-Temporal Transformers Transformer Architectures
Artificial Intelligence Lab, Otto-von-Guericke University Magdeburg, Germany · University of Rostock, Germany · Becker Lab, University of Marburg, Germany
Vivid-VR: Distilling Concepts from Text-to-Video Diffusion Transformer for Photorealistic Video Restoration
Aug 20, 2025Haoran Bai, Xiaoxu Chen, Canqian Yang +3Text-To-Video Diffusion Models Video Restoration
Alibaba Group - Taobao & Tmall Group

Apr 2, 2026, cs.AIJ/K move · Enter open · S save

GenGait: A Transformer-Based Model for Human Gait Anomaly Detection and Normative Twin Generation

Elisa Motta, Marta Lorenzini, Clara Mouawad, Alberto Ranavolo+2

HRI2 Laboratory, Istituto Italiano di Tecnologia (IIT), Genoa, Italy · Department of Occupational and Environmental Medicine, Epidemiology and Hygiene, INAIL, Rome, Italy · Department of Medical and Surgical Sciences and Biotechnologies, Sapienza University of Rome, Rome, Italy

Gait Recognition Gait Analysis Gait Dynamics Graph Anomaly Detection Autoencoder Architectures Transformer Architectures+1

Open paper AskPDF arXiv

Gait analysis provides an objective characterization of locomotor function and is widely used to support diagnosis and rehabilitation monitoring across neurological and orthopedic disorders. Deep learning has been increasingly applied to this domain, yet most approaches rely on supervised classifiers trained on disease-labeled data, limiting generalization to heterogeneous pathological presentations. The methodological objective of this work is to develop a label-free framework for joint-level anomaly detection and kinematic correction based on a Transformer masked autoencoder trained exclusively on normative gait sequences from 150 adults, acquired with a markerless multi-camera motion-capture system. At inference, a two-pass procedure is applied to potentially pathological input sequences: first, it estimates joint inconsistency scores by occluding individual joints and measuring deviations from the learned normative prior. Then, it withholds the flagged joints from the encoder input and reconstructs the full skeleton from the remaining spatiotemporal context, yielding corrected kinematic trajectories at the flagged positions. The validation objective is to assess whether the framework preserves unseen normative gait and reduces angular deviation in simulated abnormal gait patterns. In this proof-of-concept evaluation, data from 10 held-out normative participants, who performed seven simulated abnormal gait patterns, showed a significant reduction in angular deviation across all analyzed joints with large effect sizes, and preservation of normative kinematics. The proposed approach enables interpretable, subject-specific localization of joints that are inconsistent with learned normative gait patterns and generation of an individualized normative reconstruction without requiring disease labels. Video is available at https://youtu.be/Rcm3jqR5pN4.

Momentum

Related topics

Latest papers 1,120