Hybrid Transformer Architectures

Follow this topic Ask about this topic

Latest papers

24 papers6 papers in the last four weeks, up 100% on the four weeks before. 0.0% of all new papers.

Anchor: Augmented Cross-Layer Recurrent State for Long-Context Retrieval in Test-Time Training
Nov 3, 2026Vimal William, Yuanzhe Li, Danella Zhao +1Memory-Augmented Language Models Long-Context Retrieval
School of Electrical, Computer & Software Engineering, University of Arizona, Tucson, AZ, USA.
Inner-Speech Decoding from EEG Across Three Corpora Under a Fixed Architecture and Protocol
Nov 1, 2026Md Wahiduzzaman Suva, Md Imtiaj Alam Sajin, Esm E Moula Chowdhury Abha +5EEG Decoding EEG-to-Text Decoding
Computer Science Department, American International University-Bangladesh, Dhaka, Bangladesh · Department of Software Engineering, N.I. Lobachevsky National Research Nizhny Novgorod State University, Nizhny Novgorod, Russia · School of Computing, RMIT University, Melbourne, Australia
SASFormer: Stage-Aware Selective Attention Transformer for All-in-One Blind Image Restoration
Oct 31, 2026Xiangpeng Tian, Mingyang Li, Fajing Liu +3All-in-One Image Restoration Image Restoration
College of Electronics and Information Engineering, Sichuan University, Chengdu, 610065, China · College of Intelligent Systems Science and Engineering, Hubei Minzu University, Enshi 445000, China
Learning from Points and Patches: A Pre-trained Hybrid Attention Transformer for Time Series Analysis
Oct 27, 2026Peiwang Tang, Cheng HE, Yupeng Ye +3Self-Supervised Time Series Representation Learning Time Series Transformers
University of Science and Technology of China, Hefei, Anhui, China · State Key Laboratory of Cognitive Intelligence, Hefei, Anhui, China · Shanghai Black Wing Asset Management, Shanghai, China +1
DGCM-SSFA: Dual-Granularity Cross-Modulation and Sparse Strip Frequency Attention for Efficient 3T-to-7T MRI Synthesis
Oct 21, 2026Zhichun Fan, Zhiqiang Chen, Lanling Liu +2Medical Image Synthesis Magnetic Resonance Imaging
School of Computer Science and Engineering, Huizhou University, Huizhou, China
A Hybrid TFT-NBEATS Framework for Robust Multi-Horizon Photovoltaic Power Forecasting
Oct 19, 2026Shafi Bin Sultan, Sabik Bin Sultan, Mohammad Tajrian Arafat Alvi +2Solar Power Forecasting Time Series Forecasting
St. Joseph Higher Secondary School, Dhaka, Bangladesh · Bangladesh Air Force Shaheen College, Kurmitolla, Dhaka, Bangladesh · Computer Science & Engineering, Jahangirnagar University, Savar, Bangladesh +2
The Effect of Layer Ordering in Hybrid Architectures
Oct 13, 2026John Cooper, Mingchen Ma, Jiaqi Mo +4Hybrid Transformer Architectures Hybrid Attention
Department of Computer Sciences, University of Wisconsin-Madison · Department of Computer Sciences, John Hopkins University
Learning Rate Transfer for Hybrid Transformer-SSM Architectures
Oct 1, 2026Jimin Seo, Gyubok Lee, Yeonsik Jo +11Hyperparameter Transfer Hybrid Transformer-State Space Model Architectures
Seoul National University · SB Intuitions · LG AI Research +1
On-Policy Attention Linearization
Sep 25, 2026Arian Raje, Anupam Nayak, Anthony Fei +3Linear Attention Hybrid Transformer Architectures
Carnegie Mellon University · Cornell University
Benchmarking Hybrid Deep Learning Architectures for Predictive Maintenance in Industry 4.0
Sep 18, 2026Zhengyang, Gu, Joseph E. Hernandez +4Hybrid Transformer Architectures Predictive Maintenance
Cissy · Data Science, Liveline Technologies Livonia, MI, USA · Engineering, Liveline Technologies Livonia, MI, USA
Modern Transformers Are Implicit Hybrids: From Functional Differentiation to Principled Hybrid Architecture Design
Sep 2, 2026Runlin Shi, Bojian Yin, Guoqi LiRotary Positional Embeddings Linear Attention
Institute of Automation, Chinese Academy of Sciences, Beijing, China
Morphing into Hybrid Attention Models
Jun 29, 2026Disen Lan, Jianbin Zheng, Yuxi Ren +5Efficient Attention Neural Architecture Search
1Fudan University · 2ByteDance Seed · Work done at ByteDance Seed +1
Comparing Transformers and Hybrid Models at the Token Level
Jun 18, 2026Yanhong Li, William MerrillLLM Evaluation Transformer
Allen Institute for AI
A Hybrid LSTM--Vision Transformer Architecture for Predicting HRRR Forecast Errors
Jun 17, 2026David Aaron Evans, Jay C. Rothenberger, Kara J. Sulia +2Multivariate Time Series Forecasting Recurrent Transformers
Atmospheric Sciences Research Center, University at Albany, SUNY, Albany, NY, USA · University of Oklahoma, Norman, OK, USA · State Weather Risk Communication Center, University at Albany, SUNY, Albany, NY, USA
Priming: Hybrid State Space Models From Pre-trained Transformers
May 8, 2026Aditya Chattopadhyay, Elvis Nunez, Prannay Kaul +6State Space Models Long-Context Language Model Inference
AWS Agentic AI
Domain-Adaptive Arrhythmia Classification Using a Hybrid Transformer on Wearable Heart Signals
May 5, 2026Maedeh H. Toosi, Siamak MohammadiTransformer Domain Adaptation
School of Electrical and Computer Engineering, University of Tehran, Tehran, Iran
SE-Enhanced ViT and BiLSTM-Based Intrusion Detection for Secure IIoT and IoMT Environments
Apr 6, 2026Afrah Gueriani, Hamza Kheddar, Ahmed Cherif Mazari +2Hybrid Transformer Architectures IoT Intrusion Detection
LSEA Lab., Faculty of Technology, University of MEDEA, Medea 26000, Algeria · LSEA Lab, Faculty of Science, University of MEDEA, Medea 26000, Algeria · Department of Computer Engineering, Gazi University, Ankara, Turkey +1
Olmo Hybrid: From Theory to Practice and Back
Apr 3, 2026William Merrill, Yanhong Li, Tyler Romero +19Hybrid Language Models Hybrid Transformer-State Space Model Architectures
Allen Institute for AI · ETH Zürich · Lambda ML Team +2
BiTA: Bidirectional Gated Recurrent Unit-Transformer Aggregator in a Temporal Graph Network Framework for Alert Prediction in Computer Networks
Apr 3, 2026Zahra Makki Nayeri, Mohsen RezvaniHybrid Transformer Architectures Temporal GNNs
Faculty of Computer Engineering, Shahrood University of Technology, Shahrood, Iran
Switch Attention: Towards Dynamic and Fine-grained Hybrid Transformers
Mar 27, 2026Yusheng Zhao, Hourun Li, Bohan Wu +5Hybrid Transformer Architectures Hybrid Attention
State Key Laboratory for Multimedia Information Processing, School of Computer Science, PKU-Anker LLM Lab, Peking University, Beijing, China · Contribution during internship at Huawei Technologies Co., Ltd. · Huawei Technologies Co., Ltd.
Fusing Memory and Attention: A study on LSTM, Transformer and Hybrid Architectures for Symbolic Music Generation
Mar 22, 2026Soudeep Ghoshal, Sandipan Chakraborty, Pradipto Chowdhury +1Hybrid Transformer Architectures Symbolic Music Generation
Kalinga Institute of Industrial Technology (KIIT), Bhubaneswar, India · IMC University of Applied Sciences Krems, Krems, Austria
A Fusion of context-aware based BanglaBERT and Two-Layer Stacked LSTM Framework for Multi-Label Cyberbullying Detection
Feb 25, 2026Mirza Raquib, Asif Pervez Polok, Kedar Nath Biswas +3Bengali NLP Hybrid Transformer Architectures
Department of Computer and Communication Engineering, International Islamic University Chittagong, Chattogram, Bangladesh · Department of Information and Communication Engineering, Noakhali Science and Technology University, Noakhali, Bangladesh · mPower Social Enterprise +2
From Fuzzy to Exact: The Halo Architecture for Infinite-Depth Reasoning via Rational Arithmetic
Jan 26, 2026Hansheng RenNeural Network Stability Hybrid Transformer Architectures
Integrating AI for Human-Centric Breast Cancer Diagnostics: A Multi-Scale and Multi-View Swin Transformer Framework
Mar 17, 2025Farnoush Bayatmakou, Reza Taleei, Milad Amir Toutounchian +1Mammography Image Classification Multi-View Feature Fusion
Concordia Institute for Information Systems Engineering (CIISE), Concordia University, Montreal, Canada · Thomas Jefferson University Hospital, Philadelphia, Pennsylvania, USA · College of Computing & Informatics, Drexel University, Philadelphia, Pennsylvania, USA

Nov 3, 2026J/K move · Enter open · S save

Anchor: Augmented Cross-Layer Recurrent State for Long-Context Retrieval in Test-Time Training

Vimal William, Yuanzhe Li, Danella Zhao, Huanrui Yang

School of Electrical, Computer & Software Engineering, University of Arizona, Tucson, AZ, USA.

Memory-Augmented Language Models Long-Context Retrieval Hybrid Transformer Architectures Test-Time Training

Open paper AskPDF arXiv

Parameter-efficient linearization turns a pretrained autoregressive Transformer into an efficient model by hybridizing sliding-window attention (SWA) with a constant-size recurrent memory. But does every layer's recurrent state contribute to long-context retrieval? Investigating an SWA and Test-Time Training (TTT) hybrid, we find that retrieval is sparse across depth, concentrated in a few anchor layers, and that the remaining states hold similar information expressed through layer-specific transformations. We exploit both observations in Anchor: TTT memory placed only at the anchor layers, and a cross-layer recurrent state written once and read by many layers, each through its own learned read map that realizes a layer-specific stretch and rotation, along with demonstrated generalization to other hybrid attentions. On Llama-3.1-8B and Mistral-7B, Anchor delivers 2.3-3.1x higher recall and 6.7-8.9 points higher MMLU accuracy than LoLCATs with a 10x smaller recurrent state, retains 91-95% of the teacher's MMLU accuracy, and cuts end-to-end latency by up to 4.3x relative to softmax attention.