Transformer Encoder

Ask about this topic

Momentum

11 papers in the last four weeks, up 83% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 80

All topics
CardsList

  1. Decoding Looped Transformers Better for (Almost) Free

    Oct 1, 2026Weihao Liu, Huangjie Zheng, Tianrong Chen +5Transformer EncoderTransformer Architectures

    Apple

  2. DeBERTa-ConPara: Attack-Aware and Deployment-Realistic Detection of AI-Generated Text

    Oct 1, 2026Mohamed Mady, Yupei Li, Johannes Reschke +1Machine-Generated Text DetectionBert-Based Models

    CHI, Chair of Health Informatics, Technical University of Munich, Germany · Smart Embedded Systems Lab, OTH Regensburg, Germany · GLAM, Group on Language, Audio & Music, Imperial College London, UK

  3. Predictive Geometry of Hidden Trajectories in Transformers

    Sep 29, 2026Timur Mudarisov, Mikhail Burtsev, Tatiana Petrova +1Transformer EncoderTransformer Architectures

    University of Luxembourg, Luxembourg

  4. Relational Attention for Data-Efficient Language Modeling

    Sep 17, 2026Adrian Brasoveanu, Ece Takmaz, Jakub DotlačilTransformer AttentionLanguage Modeling

  5. MeshKV: A Network-on-Chip KV Cache Fabric for Scalable Transformer Decoding Accelerators

    Sep 16, 2026Dong Liu, Yanxuan YuKey-Value CacheAutoregressive Transformers

    Columbia University New York, NY, USA

  6. Self-Supervised Learning for Robust Resonance Mass Regression in Cascade Decays

    Sep 15, 2026Ho Fung Tsoi, Alex Yang, Luis Felipe Gutierrez Zagazeta +2Self-Supervised LearningTransformer Encoder

    Department of Physics and Astronomy University of Pennsylvania, USA · Department of Physics Kyoto University, Japan

  7. Persistent Recurrent Memory Between Transformer Layers - Improves Language Model Generalization

    Sep 15, 2026Eduardo Novaes HeringTransformer EncoderTransformer Architectures

    FITec Labs / Ericsson São Paulo

  8. Same path, different: a mechanistic comparison of looped and stacked transformer encoders on 12-lead ECG

    Sep 14, 2026Pawel Olszowiec, Michal Byra, Grzegorz Gruszczynski +2Transformer EncoderAdult-To-Pediatric Electrocardiogram Transfer

    Samsung AI Center, Warsaw, Poland · IFTR, Polish Academy of Sciences, Warsaw, Poland

  9. Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models

    Sep 14, 2026Kalyani Marathe, Artidoro Pagnoni, Tomasz Limisiewicz +4Small Large Language ModelsTransformer Encoder

    University of Washington, Seattle · Work done at Meta FAIR · Meta FAIR

  10. Towards a knowledge-enhanced single-cell foundation model

    Sep 14, 2026Hanqing Zhang, Jie Bao, Mei Ma +7Regulatory NetworksPretraining

    College of Animal Science and Technology, China Agricultural University, Beijing 100193, China · College of Information and Electrical Engineering, China Agricultural University, Beijing 100193, China

  11. Sparse Weight and Edge Circuit Discovery in Transformer-based Acoustic Models

    Sep 11, 2026Jiankun Wei, Ewan Dunbar, Gerald PennTransformer EncoderNeural Audio

    Department of Computer Science University of Toronto Toronto, Canada

  12. Stride-k Subsampling: Train-Free Audio Token Reduction for Whisper

    Aug 31, 2026Chanhee Cho, Junhyuk Choi, Bugeun KimAudio TokenizersSubsampling

    Department of Artificial Intelligence Chung-Ang University, Seoul, Republic of Korea

  13. AQ3D: Adaptive Query Transformer for 3D Instance Segmentation

    Aug 31, 2026Keno Moenck, Thorsten Schüppstuhl3D GenerationTransformer Encoder

    Hamburg University of Technology

  14. LabelFusion-TS: Fusing Large Language Models, Transformer Encoders, and Financial Time Series for Monetary-Policy Stance Classification

    Aug 12, 2026Michael Schlee, Fabian Lukassen, Christoph WeisserFinancial Sentiment AnalysisCnn-Lstm

    Centre for Statistics, Georg-August-Universität Göttingen, Germany · Hochschule Bielefeld (HSBI) - University of Applied Sciences and Arts, Bielefeld, Germany

  15. G0.5: One Autoregressive Stream for Robot Reasoning and Action

    Aug 12, 2026Yicheng Liu, Zibin Dong, Baijun Ye +24Diffusion-Based Vision-Language-ActionsVision Encoders

    Galaxea Team

  16. ODE-Based Transformer Decoders for Iterative Sign Language Translation

    Aug 11, 2026Tuğçe Kızıltepe, Hacer Yalim KelesGloss-Free Sign Language TranslationSign Language Translation

    ASELSAN, Ankara, Türkiye · Department of Computer Engineering, Hacettepe University, Ankara, Türkiye

  17. Why Post-Norm Transformers Collapse: Attention Amplification and Gradient Repair Failure

    Aug 10, 2026Xingjian Wang, Qingyu Han, Xiaodong Luo +1Batch NormalizationTransformer Encoder

    The Chinese University of Hong Kong, Shenzhen · Shenzhen Research Institute of Big Data

  18. ZIPBrain: Can EEG Foundation Models Be Faster, Locally Deployable, but Accurate?

    Aug 7, 2026Lingwei Li, Yirong Kan, Peng Chen +3Electroencephalography Foundation ModelsElectroencephalography

    Nara Institute of Science and Technology, Nara, Japan · RIKEN Center for Computational Science, Hyogo, Japan · University of Illinois Urbana-Champaign, USA +1

  19. Separating quantum circuits from classical LLMs

    Aug 4, 2026Srinivasan Arunachalam, Arkopal Dutt, Hari Krovi +1Quantum CircuitsLarge Language Model Quantization

    IBM Research

  20. Divergent large language model predictions from convergent representations in ambiguous word pairs

    Aug 3, 2026K. Jack Scott, Narun Pat, Veronica LiesaputraTransformer EncoderLayer-Wise

    Department of Psychology, University of Otago, Dunedin, New Zealand · School of Computing, University of Otago, Dunedin, New Zealand

  21. HealthCAT: An Interpretable Encoder-only Transformer Framework for Health Indicator Prediction and Temporal Interpretation of Wearable Sensor Data

    Jul 30, 2026Xiaotong Yu, Joshua Y. Kim, HaeJin Lee +1Wearable PhysiologyReadmission Prediction

  22. Penelope: Localized Latent Recurrence for Efficient Structured Reasoning

    Jul 28, 2026Yutong Chen, Shouqian Shi, Xinran Liu +5Efficient Latent ReasoningTransformer Encoder

  23. Grounding latent algorithm routing in transformer reasoning

    Jul 27, 2026Xiangbo Zhang, Xiaoxu MaTransformer ArchitecturesLarge Language Model Routing

    Georgia Institute of Technology

  24. Sensitivity Analysis of GRU, LSTM and Transformer Encoder in Classification of Automated Driving Systems

    Jul 25, 2026Bidhya Shrestha, Christos PapadopoulosAutomated VehiclesRobustness Verification

    University of Memphis

  25. Enhancing Transformer-based Routing by Encoding Distance via Relative Positional Encoding

    Jul 21, 2026Leyre Encío, Daniel Fuertes, Carlos R. del-Blanco +1Multiplex Graph TransformersPositional Encoding

    Grupo de Tratamiento de Imágenes, Information Processing and Telecomunications Center, ETSI Telecomunicación, Universidad Politécnica de Madrid, 28040, Madrid, Spain

  26. A Controlled Study of Attention-Only Transformers

    Jul 20, 2026Henry Ndubuaku, Karen Mosoyan, Jakub Mroz +5Transformer AttentionTransformer Encoder

    Cactus Compute, Inc.

  27. An Explainable FFT-Based Spatial-Frequency Fusion Framework for Deepfake Detection

    Jul 19, 2026Pamela Kirui, Cho Hyuk, Qingzhong Liu +1Deepfake DetectionTwo-Level Spatial-Frequency Fusion Strategy

    Department of Computer Science Sam Houston State University Huntsville, Texas, USA

  28. Do Transformer Temporal Heads and Post-Pooling Motion Gates Help CorrNet-based CSLR? An Empirical Study

    Jul 10, 2026Lisi Wang, Zhidong Xiao, Jianjun PengTransformer EncoderSign Language Translation

    University of New South Wales, Sydney, Australia · National Centre for Computer Animation, Faculty of Media, Science and Technology, Bournemouth University, BH12 5BB, United Kingdom · School of Information Science and Engineering, Dalian Polytechnic University, Dalian 116034, China

  29. IG-Lens: Exact Additive Probability Attribution Across Transformer Layers via Telescoping Integrated Gradients

    Jun 29, 2026Duc Anh NguyenTransformer EncoderTransformer Architectures

    Hanoi University of Science and Technology

  30. DDStereo: Efficient Dual Decoder Transformers for Stereo 3D Road Anomaly Detection

    Jun 23, 2026Shiyi Mu, Zichong Gu, Zhiqi Ai +23D Object DetectionAutonomous Driving Perception

    Shanghai University, Shanghai, China · Xi’an Jiaotong-Liverpool University, Suzhou, China

  31. moBERTo: A Modern Encoder for Portuguese via Continued Pretraining of ModernBERT

    Jun 21, 2026Thiago Laitz, Thales Sales Almeida, João Guilherme Alves Santos +1Bert-Based ModelsTransformer Encoder

    UNICAMP, Campinas, Brazil · Tropic AI · Maritaca AI

  32. Short-Term Electricity Demand Forecasting for New England Using a Hybrid Transformer-XGBoost Framework with Weather, Calendar, and COVID-19 Indicators

    Jun 18, 2026Reza Ghanavati, Behrooz MosallaeiAccurate Short-Term Electricity Load ForecastingForecasting Backbone

    Department of Chemical and Materials Engineering, New Mexico State University · Department of Electrical and Communications Engineering, New Jersey Institute of Technology

  33. Dual Dimensionality for Local and Global Attention

    Jun 17, 2026Zhiyuan Wang, Xuan Luo, Sirui Zeng +1Transformer EncoderRepresentational Capacity

    UC Santa Barbara

  34. Transformers Learn the Mestre-Nagao Heuristic

    Jun 13, 2026Pranav Venkata KondaTransformer EncoderTransformer Architectures

  35. Binary Black Hole Parameter Estimation with Hybrid CNN-Transformer Neural Networks

    Jun 11, 2026Panagiotis N. Sakellariou, Spiros V. Georgakopoulos, Sotiris Tasoulis +1Gravitational WavesGeneral Relativity

    Department of Mathematics, University of Thessaly, 3rd km. Old National Road Lamia-Athens, Lamia, 35100, Greece · Department of Computer Science and Biomedical Informatics, University of Thessaly, 35100, Greece

  36. SpikeDecoder: Realizing the GPT Architecture with Spiking Neural Networks

    Jun 10, 2026Claas Beger, Florian Walter, Alois KnollSpiking Neural NetworksTransformer Encoder

    Chair of Robotics, Artificial Intelligence and Real-time Systems Technical University of Munich, Germany

  37. GeoMetric: Injecting Metric Geographic Structure into Worldwide Image Geo-Localization

    Jun 8, 2026Junchao Cui, Xuanzi Ma, Wenqi Shi +3Cross-View Geo-LocalizationGeogs-Slam

    Information Engineering University Zhengzhou, China

  38. Where does Absolute Position come from in decoder-only Transformers?

    Jun 4, 2026Valeria Ruscio, Umberto Nanni, Fabrizio SilvestriTransformer EncoderRotary Position Embedding

    Intuition Machines · Sapienza University of Rome

  39. An Empirical Audit of Input Encoders for Multi-Channel Signal Transformers

    Jun 3, 2026Ossi LehtinenTransformer EncoderEncoders

    Ocon Oy

  40. ChessMimic: Per-Rating Transformer Models for Human Move, Clock, and Outcome Prediction in Online Blitz Chess

    Jun 3, 2026Thomas JohnsonChessClocks

    Nascent

  41. A Training-Efficient Transformer-Based Anti-Spoofing Network for Logical Access in ASVspoof 5

    Jun 2, 2026Sidan Yin, Bo ZhaoSpoofingAutomatic Speaker Verification

    San Domenico School, San Anselmo, CA, United States · University of Washington, Seattle, WA, United States

  42. Evaluating Transformer and LSTM Frameworks for Prediction in Ungauged Basins

    Jun 1, 2026Taye Akinrele, James Halgren, Noorbakhsh Amiri Golilarz +2Forecasting BackboneSen1Floods11

    Dept. of Computer Science, The University of Alabama, Tuscaloosa, USA · Alabama Water Institute, USA

  43. GIRL-DETR: Gradient-Isolated Reinforcement Learning for Video Moment Retrieval

    May 30, 2026Shihang Zhang, Mingjin Kuai, Ye Wei +2Detection TransformerTransformer Encoder

    Shihang Zhang and Ye Wei are with the College of Electronics and Information Engineering, Sichuan University, Chengdu 610065, China · Mingjin Kuai, Zhen Zhang, and Wei Ji are with the School of Intelligence Science and Technology, Nanjing University, Suzhou 215163, China

  44. Giving Sensors a Voice: Multimodal JEPA for Semantic Time-Series Embeddings

    May 29, 2026Utsav Dutta, Gerardo Pastrana, Sina Khoshfetrat Pakazad +1Timestep EmbeddingsJoint-Embedding Predictive Architectures

    C3 AI, Redwood City, CA, USA

  45. Reliable Multilingual Orthopedic Decision Support from Clinical Narratives: Language-Aware Adaptation and Verification-Guided Deferral

    May 29, 2026Danish Ali, Li Xiaojian, Sundas Iqbal +1Multilingual Healthcare Q&ALanguage Pairs

    School of Computer Science, Wuhan University, China · School of Software, Nanjing University of Information Science and Technology (NUIST), China · Department of Orthopedic, Bahawal Victoria Hospital, Pakistan

  46. Give it Space! Explicit Disentangling of Positional and Semantic Representations in Encoders

    May 28, 2026Pierre-Antoine Lequeu, Camille Barboule, Benjamin PiwowarskiPositional EncodingTransformer Encoder

    Sorbonne Université, CNRS, ISIR, Paris, France · Orange Innovation

  47. Benchmarking Positional Encoding Strategies for Transformer-Based EEG Foundation Models

    May 28, 2026Ayse Betul Yuce, Sebastian StoberElectroencephalography Foundation ModelsElectroencephalography

    Department of Computer Science, Otto von Guericke University, Magdeburg, Germany

  48. BandVQ: Band-Wise Vector-Quantized EEG Foundation Model

    May 24, 2026Jamiyan Sukhbaatar, Satoshi Imamura, Toshihisa TanakaElectroencephalography Foundation ModelsElectroencephalography

    National University of Mongolia, Ulaanbaatar, Mongolia · Tokyo University of Agriculture and Technology, Tokyo, Japan

  49. FastTab: A Fast Table Recognizer with a Tiny Recursive Module and 1D Transformers

    May 21, 2026Laziz Hamdi, Amine Tamasna, Pascal Boisson +1Transformer EncoderRecursive Models

    LITIS, Rouen Normandy, France · Malakoff Humanis, Paris, France

  50. Post-Hoc Understanding of Metaphor Processing in Decoder-Only Language Models via Conditional Scale Entropy

    May 20, 2026Lawhori Chakrabarti, Jennifer Johnson-Leung, Bert Baumgaertner +3Transformer EncoderTransformer Architectures

    Department of Computer Science, University of Idaho, Moscow ID 83844, USA · Department of Mathematics and Statistical Science, University of Idaho, Moscow ID 83844, USA · Department of Politics and Philosophy, University of Idaho, Moscow ID 83844, USA +2

  51. TERDNet: Transformer Encoder-Recurrent Decoder Network for Scene Change Detection

    May 20, 2026Jiae Yoon, Ue-Hwan KimDecoder LayersTransformer Encoder

    Department of AI Convergence, Gwangju Institute of Science and Technology (GIST), Gwangju 61005, Republic of Korea

  52. Chronicle: A Multimodal Foundation Model for Joint Language and Time Series Understanding

    May 18, 2026Paul Quinlan, Jeremy Levasseur, Qingguo Li +1Time Series Foundation ModelsTime Series

    1InertialAI · Department of Electrical and Computer Engineering, Queen’s University · Department of Mechanical and Materials Engineering, Queen’s University

  53. Performance Monitoring of Proton Exchange Membrane Water Electrolyzer by Transformers-Based Machine Learning Model

    May 18, 2026Bingqing Chen, Ivan Batalov, Qiu Chen +2Hydrogen System ComponentsTransformer Encoder

    Bosch Research & Technology Center

  54. Chessformer: A Unified Architecture for Chess Modeling

    May 18, 2026Daniel Monroe, George Eilender, Philip Chalmers +2ChessTransformer Encoder

    University of Toronto · Williams College

  55. SAGA: A Sequence-Adaptive Generative Architecture for Multi-Horizon Probabilistic Forecasting with Adaptive Temporal Conformal Prediction

    May 18, 2026Gustav Olaf Yunus Laitinen-Fredriksson Lundström-Imanov, Hafize Gonca CömertRetrieval-Augmented ForecastingOnline Conformal Prediction

    Department of Economics, Stockholm University, SE-106 91 Stockholm, Sweden

  56. Transformer-like Inference from Optimal Control

    May 15, 2026Aditya Kudre, Heng-Sheng Chang, Prashant G. MehtaTransformer EncoderTransformer Architectures

    Coordinated Science Laboratory · Electrical and Computer Engineering · University of Illinois Urbana-Champaign +1

  57. Ghosted Layers: Unconstrained Activation Alignment for Recovering Layer-Pruned LLMs

    May 15, 2026Vincent-Daniel Yun, Junhyuk Jo, Sai Praneeth Karimireddy +1Model ActivationsTransformer Encoder

    University of Southern California, USA · Inha University, Republic of Korea

  58. Exploiting Pre-trained Encoder-Decoder Transformers for Sequence-to-Sequence Constituent Parsing

    May 13, 2026Daniel Fernández-González, Cristina Outeiriño CidTransformer EncoderSequence Modeling

    Universidade de Vigo, Departamento de Informática, Campus As Lagoas s/n, Ourense, 32004,, Spain

  59. HEPA: A Self-Supervised Horizon-Conditioned Event Predictive Architecture for Time Series

    May 11, 2026Jonas Petersen, Gian-Alessandro Lombardi, Riccardo Maggioni +3Forecasting BackboneMultivariate Time-Series

    1ETH Zurich · 2Forgis · University of Vienna

  60. Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime

    May 11, 2026Albert Alcalde, Leon Bungert, Konstantin Riedl +1Transformer EncoderTransformer Architectures

    Department of Mathematics Friedrich-Alexander-University Erlangen-Nürnberg · Institute of Mathematics CAIDAS University of Würzburg · Mathematical Institute University of Oxford Reuben College +1