Transformer Encoder

Follow this topic Ask about this topic

Recent momentum

-67%

4 papers in the last 28 days · 0.1% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-14

2 new papers

A weekly snapshot of new work published in Transformer Encoder.

Period ending 2026-09-07

3 new papers

A weekly snapshot of new work published in Transformer Encoder.

66 papers

Latest in Transformer Encoder

Open your feed →

Sparse Weight and Edge Circuit Discovery in Transformer-based Acoustic Models
Sep 11, 2026Jiankun Wei, Ewan Dunbar, Gerald PennTransformer Encoder Self-Supervised Audio Representations
A Transformer-Based Delta Expression Encoder for Psilocybin Transcriptional Response: Architecture, Representations, and Biological Validation
Sep 8, 2026Sai JayakumarTranscriptomics Transformer Encoder
Stride-k Subsampling: Train-Free Audio Token Reduction for Whisper
Aug 31, 2026Chanhee Cho, Junhyuk Choi, Bugeun KimAudio Tokens Subsampling
AQ3D: Adaptive Query Transformer for 3D Instance Segmentation
Aug 31, 2026Keno Moenck, Thorsten Schüppstuhl3D Instance Segmentation 3D
LabelFusion-TS: Fusing Large Language Models, Transformer Encoders, and Financial Time Series for Monetary-Policy Stance Classification
Aug 12, 2026Michael Schlee, Fabian Lukassen, Christoph WeisserFinancial Sentiment Analysis Time Series Foundation Models
G0.5: One Autoregressive Stream for Robot Reasoning and Action
Aug 12, 2026Yicheng Liu, Zibin Dong, Baijun Ye +24Diffusion-Based Vision-Language-Actions Vision-Language-Action
ODE-Based Transformer Decoders for Iterative Sign Language Translation
Aug 11, 2026Tuğçe Kızıltepe, Hacer Yalim KelesSign Language Translation Transformer Encoder
Why Post-Norm Transformers Collapse: Attention Amplification and Gradient Repair Failure
Aug 10, 2026Xingjian Wang, Qingyu Han, Xiaodong Luo +1Batch Normalization Transformer Encoder
ZIPBrain: Can EEG Foundation Models Be Faster, Locally Deployable, but Accurate?
Aug 7, 2026Lingwei Li, Yirong Kan, Peng Chen +3Electroencephalography Foundation Models Electroencephalogram
Separating quantum circuits from classical LLMs
Aug 4, 2026Srinivasan Arunachalam, Arkopal Dutt, Hari Krovi +1Quantum Circuits Large Language Model Quantization
Divergent large language model predictions from convergent representations in ambiguous word pairs
Aug 3, 2026K. Jack Scott, Narun Pat, Veronica LiesaputraTransformer Encoder Protein Language Model
HealthCAT: An Interpretable Encoder-only Transformer Framework for Health Indicator Prediction and Temporal Interpretation of Wearable Sensor Data
Jul 30, 2026Xiaotong Yu, Joshua Y. Kim, HaeJin Lee +1Wearable Devices Transformer Encoder
Penelope: Localized Latent Recurrence for Efficient Structured Reasoning
Jul 28, 2026Yutong Chen, Shouqian Shi, Xinran Liu +5Latent Reasoning Transformer Encoder
Grounding latent algorithm routing in transformer reasoning
Jul 27, 2026Xiangbo Zhang, Xiaoxu MaTransformer Encoder Dynamic Routing
Sensitivity Analysis of GRU, LSTM and Transformer Encoder in Classification of Automated Driving Systems
Jul 25, 2026Bidhya Shrestha, Christos PapadopoulosAutonomous Driving Automatic Identification System
Enhancing Transformer-based Routing by Encoding Distance via Relative Positional Encoding
Jul 21, 2026Leyre Encío, Daniel Fuertes, Carlos R. del-Blanco +1Multiplex Graph Transformers Positional Encoding
An Explainable FFT-Based Spatial-Frequency Fusion Framework for Deepfake Detection
Jul 19, 2026Pamela Kirui, Cho Hyuk, Qingzhong Liu +1Deepfake Detection Spatio-Temporal Fusion
Do Transformer Temporal Heads and Post-Pooling Motion Gates Help CorrNet-based CSLR? An Empirical Study
Jul 10, 2026Lisi Wang, Zhidong Xiao, Jianjun PengCnn-Lstm Transformer Encoder
IG-Lens: Exact Additive Probability Attribution Across Transformer Layers via Telescoping Integrated Gradients
Jun 29, 2026Duc Anh NguyenMulti-Layer Transformers Transformer Encoder
DDStereo: Efficient Dual Decoder Transformers for Stereo 3D Road Anomaly Detection
Jun 23, 2026Shiyi Mu, Zichong Gu, Zhiqi Ai +23D Object Detection Autonomous Driving
moBERTo: A Modern Encoder for Portuguese via Continued Pretraining of ModernBERT
Jun 21, 2026Thiago Laitz, Thales Sales Almeida, João Guilherme Alves Santos +1Modernbert Transformer Encoder
Short-Term Electricity Demand Forecasting for New England Using a Hybrid Transformer-XGBoost Framework with Weather, Calendar, and COVID-19 Indicators
Jun 18, 2026Reza Ghanavati, Behrooz MosallaeiEnergy Forecasting Coronavirus Disease
Dual Dimensionality for Local and Global Attention
Jun 17, 2026Zhiyuan Wang, Xuan Luo, Sirui Zeng +1Transformer Encoder Representational Capacity
Binary Black Hole Parameter Estimation with Hybrid CNN-Transformer Neural Networks
Jun 11, 2026Panagiotis N. Sakellariou, Spiros V. Georgakopoulos, Sotiris Tasoulis +1Gravitational Waves Hybrid Transformer
A systematic investigation of molecular encoding methods for drug property predictions across neural network and Transformer encoder-based model
Jun 8, 2026Sheng-Ya Chen, Shan-Ju YehMolecular Property Prediction Transformer Encoder
GeoMetric: Injecting Metric Geographic Structure into Worldwide Image Geo-Localization
Jun 8, 2026Junchao Cui, Xuanzi Ma, Wenqi Shi +3Geo-Localization Geoquery
Where does Absolute Position come from in decoder-only Transformers?
Jun 4, 2026Valeria Ruscio, Umberto Nanni, Fabrizio SilvestriRotary Position Embedding Transformer Encoder
TS-ICL: A Flexible Time-Indexed Foundation Model for Time Series via In-Context Learning
Jun 4, 2026Etienne Le Naour, Tahar Nabil, Adrien PetraliaTime Series Foundation Models Time Series
An Empirical Audit of Input Encoders for Multi-Channel Signal Transformers
Jun 3, 2026Ossi LehtinenTransformer Encoder Multilayer Perceptrons
ChessMimic: Per-Rating Transformer Models for Human Move, Clock, and Outcome Prediction in Online Blitz Chess
Jun 3, 2026Thomas JohnsonChess Clocks
A Training-Efficient Transformer-Based Anti-Spoofing Network for Logical Access in ASVspoof 5
Jun 2, 2026Sidan Yin, Bo ZhaoSpoofed Speech Transformer Encoder
Evaluating Transformer and LSTM Frameworks for Prediction in Ungauged Basins
Jun 1, 2026Taye Akinrele, James Halgren, Noorbakhsh Amiri Golilarz +2Fundamental Hydrological Principles Transformer Encoder
Rethinking the Role of Positional Encoding: Sliding-Window Transformers without PE Remain Turing Complete
Jun 1, 2026Qian Li, Xinyu Mao, Shang-Hua TengPositional Encoding Transformer Encoder
GIRL-DETR: Gradient-Isolated Reinforcement Learning for Video Moment Retrieval
May 30, 2026Shihang Zhang, Mingjin Kuai, Ye Wei +2Transformer Encoder Surrogate Gradient
Giving Sensors a Voice: Multimodal JEPA for Semantic Time-Series Embeddings
May 29, 2026Utsav Dutta, Gerardo Pastrana, Sina Khoshfetrat Pakazad +1Joint-Embedding Predictive Architectures Representation Learning
Reliable Multilingual Orthopedic Decision Support from Clinical Narratives: Language-Aware Adaptation and Verification-Guided Deferral
May 29, 2026Danish Ali, Li Xiaojian, Sundas Iqbal +1Indic Languages Clinical Notes
Give it Space! Explicit Disentangling of Positional and Semantic Representations in Encoders
May 28, 2026Pierre-Antoine Lequeu, Camille Barboule, Benjamin PiwowarskiPositional Encoding Transformer Encoder
Benchmarking Positional Encoding Strategies for Transformer-Based EEG Foundation Models
May 28, 2026Ayse Betul Yuce, Sebastian StoberElectroencephalography Foundation Models Positional Encoding
BandVQ: Band-Wise Vector-Quantized EEG Foundation Model
May 24, 2026Jamiyan Sukhbaatar, Satoshi Imamura, Toshihisa TanakaElectroencephalography Foundation Models Calibration-Free Low-Bit Vector Quantization
FastTab: A Fast Table Recognizer with a Tiny Recursive Module and 1D Transformers
May 21, 2026Laziz Hamdi, Amine Tamasna, Pascal Boisson +1Transformer Encoder Structure-Aware
Chronicle: A Multimodal Foundation Model for Joint Language and Time Series Understanding
May 18, 2026Paul Quinlan, Jeremy Levasseur, Qingguo Li +1Time Series Foundation Models Time Series
Performance Monitoring of Proton Exchange Membrane Water Electrolyzer by Transformers-Based Machine Learning Model
May 18, 2026Bingqing Chen, Ivan Batalov, Qiu Chen +2Perovskite Electrochemical Models
Chessformer: A Unified Architecture for Chess Modeling
May 18, 2026Daniel Monroe, George Eilender, Philip Chalmers +2Chess Transformer Encoder
SAGA: A Sequence-Adaptive Generative Architecture for Multi-Horizon Probabilistic Forecasting with Adaptive Temporal Conformal Prediction
May 18, 2026Gustav Olaf Yunus Laitinen-Fredriksson Lundström-Imanov, Hafize Gonca CömertConformal Prediction Observational Data
Transformer-like Inference from Optimal Control
May 15, 2026Aditya Kudre, Heng-Sheng Chang, Prashant G. MehtaTransformer Encoder Inference-Time Control
Exploiting Pre-trained Encoder-Decoder Transformers for Sequence-to-Sequence Constituent Parsing
May 13, 2026Daniel Fernández-González, Cristina Outeiriño CidSequence-To-Sequence Models Transformer Encoder
HEPA: A Self-Supervised Horizon-Conditioned Event Predictive Architecture for Time Series
May 11, 2026Jonas Petersen, Gian-Alessandro Lombardi, Riccardo Maggioni +3Joint-Embedding Predictive Architectures Time Series
Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime
May 11, 2026Albert Alcalde, Leon Bungert, Konstantin Riedl +1Transformer Encoder Concentration
Kinetic theory for Transformers and the lost-in-the-middle phenomenon
May 9, 2026Mitia Duerinckx, Borjan Geshkovski, Stefano RossiMean-Field Limit Transformer Encoder
Emergent Semantic Role Understanding in Language Models
May 9, 2026Carla Griffiths, Mirco MusolesiPre-Trained Vision-Language Models Finetuning
Cross-Attention and Encoder-Decoder Transformers: A Logical Characterization
May 8, 2026Veeti Ahvonen, Damian Heiman, Antti Kuusisto +2Transformer Encoder Cross-Attention
Average Attention Transformers and Arithmetic Circuits
May 6, 2026Lena Ehrmuth, Laura StriekerTransformer Encoder Transformer Attention
Self-Improvement for Fast, High-Quality Plan Generation
May 5, 2026Robert Gieselmann, Henrike von Huelsen, Mihai Samson +9Tool Planning Planner
Trust, but Verify: Peeling Low-Bit Transformer Networks for Training Monitoring
May 4, 2026Arian Eamaz, Farhang Yeganegi, Mojtaba SoltanalianNeural Network Optimization Low-Precision Training
PINN-Cast: Exploring the Role of Continuous-Depth NODE in Transformers and Physics Informed Loss as Soft Physical Constraints in Short-term Weather Forecasting
Apr 30, 2026Hira Saleem, Flora Salim, Cormac PurcellArtificial Intelligence Weather Models Conventional Physics-Informed Neural Networks
SG-UniBuc-NLP at SemEval-2026 Task 6: Multi-Head RoBERTa with Chunking for Long-Context Evasion Detection
Apr 29, 2026Gabriel Stefan, Sergiu NisioiNatural Language Processing Evasion
Graph Memory Transformer (GMT)
Apr 26, 2026Nicola Zanarini, Niccolò Ferrari, Evelina LammaTransformer Encoder Feed-Forward
Scaling Point-in-Time Language Models
Apr 24, 2026Bryan Kelly, Semyon Malamud, Johannes Schwab +1Transformer Encoder Lookahead
To See the Unseen: on the Generalization Ability of Transformers in Symbolic Reasoning
Apr 23, 2026Nevena Lazić, Liam Fowl, András György +1Transformer Encoder Vector Embeddings
Volume Transformer: Revisiting Vanilla Transformers for 3D Scene Understanding
Apr 21, 2026Kadir Yilmaz, Adrian Kruse, Tristan Höfer +23D Scene Understanding 3D

Sep 11, 2026 · cs.SDJ/K move · Enter open · S save

Sparse Weight and Edge Circuit Discovery in Transformer-based Acoustic Models

Jiankun Wei, Ewan Dunbar, Gerald Penn

Transformer-based foundation models are powerful but opaque, motivating Mechanistic Interpretation methods to uncover the black-box by identifying small computation subgraphs responsible for a task. DiscoGP is a joint weight-and-edge circuit discovery framework originally developed for text decoders. We extend DiscoGP to speech encoders and present, to our knowledge, the first circuit discovery study for modern speech foundation models. Across HuBERT and Wav2Vec 2.0 on several speech classification tasks, we find that the discovered circuits are extremely compact, yet often match or even exceed the performance of the full pretrained encoder with the same downstream head. Through ablations, we show that these circuits reflect pretrained computation rather than random structure or task-head artifacts. We also introduce a memory-efficient DiscoGP variant that reduces the GPU memory cost of edge-circuit discovery at runtime from quartic to cubic. Overall, our results broaden Mechanistic Interpretation beyond text decoders and show that circuit-level analysis can reveal both explanatory structure and unexpected functional behavior in speech encoders.

Transformer Encoder Self-Supervised Audio Representations Circuits

Open paper Ask about it PDF