Sequence Modeling

Ask about this topic

Overview

Sequence modeling asks how information in an ordered stream—such as words, sensor readings, user actions, or DNA—can be used to predict what comes next or interpret the sequence as a whole. Common approaches include recurrent networks such as LSTMs, Transformer attention, state-space models such as Mamba, and Markov or hidden Markov models; hybrids and diffusion methods are also used for long or structured sequences. Applications range from language and time-series forecasting to recommendation, biological analysis, and action recognition, with evaluation often testing prediction quality, long-range recall, and generalization to new lengths or domains. A central tension is balancing expressive memory and reliable long-context behavior against computation, latency, and memory costs, while ensuring benchmarks measure genuine sequence understanding rather than leakage or repetition.

Latest papers

CardsList

250 papers22 papers in the last four weeks, up 38% on the four weeks before. 0.1% of all new papers.

  1. EMA Is Not All You Need: Mapping the Boundary Between Structure and Content in Recurrent Context

    Mar 17, 2026Arth SinghSequence ModelingAutoregressive Language Modeling

    AIM Intelligence National Institute of Technology Agartala

  2. Mamba-3: Improved Sequence Modeling using State Space Principles

    Mar 16, 2026Aakash Lahoti, Kevin Y. Li, Berlin Chen +5State Space Language ModelsSequence Modeling

    Carnegie Mellon University · Princeton University · Together AI +1

  3. Embedding-Aware Feature Discovery: Bridging Latent Representations and Interpretable Features in Event Sequences

    Mar 16, 2026Artem Sakhno, Ivan Sergeev, Alexey Shestov +5Large Language Model-Based Feature EngineeringAutomated Feature Engineering

    Sber AI Lab · ISP RAS Research Center for Trusted AI

  4. Training-Free Generation of Protein Sequences from Small Family Alignments via Stochastic Attention

    Mar 16, 2026Jeffrey D. VarnerHopfield NetworksMolecular Generation

    R.F. Smith School of Chemical and Biomolecular Engineering · Cornell University, Ithaca, NY 14850

  5. Probing Length Generalization in Mamba via Image Reconstruction

    Mar 12, 2026Jan Rathjens, Robin Schiewer, Laurenz Wiskott +1MambaLength Generalization

    Institute for Neural Computation, Faculty of Computer Science, Ruhr University Bochum, Bochum, Germany · CITEC, Bielefeld University, Bielefeld, Germany · Department of Computer Science, Royal Holloway, University of London, Egham, UK

  6. EvoFlows: Evolutionary Edit-Based Flow-Matching for Protein Engineering

    Mar 12, 2026Nicolas Deutschmann, Constance Ferragu, Jonathan D. Ziegler +2Protein DesignSequence Modeling

    Cradle

  7. Single molecule localization microscopy challenge: a biologically inspired benchmark for long-sequence modeling

    Mar 11, 2026Fatemeh Valeh, Monika Farsang, Radu Grosu +1Sequence ModelingState Space Models

    Department of Biophysics, Technische Universität Wien, Vienna, Austria · Department of Informatics, Technische Universität Wien, Vienna, Austria

  8. Beyond Pairwise Attention: Higher-Order Modular Attention for Efficient Sequence Learning

    Mar 11, 2026Shirin Amiraslani, Xin GaoSelf-AttentionTransformer Attention

    Department of Mathematics and Statistics York University Toronto, ON M3J 1P3, Canada

  9. Decoding the decoder: Contextual sequence-to-sequence modeling for intracortical speech decoding

    Mar 10, 2026Michal Olak, Tommaso Boccato, Matteo FerranteSequence-to-Sequence ModelsNeural Decoding

    Tether Evo

  10. Expressivity-Efficiency Tradeoffs for Hybrid Sequence Models

    Mar 9, 2026John Cooper, Ilias Diakonikolas, Mingchen Ma +1Hybrid Transformer-State Space Model ArchitecturesSequence Modeling

    Department of Computer Sciences, University of Wisconsin-Madison

  11. Structure from rank: Rank-order coding as a bridge from sequence to structure

    Mar 9, 2026Xiaodan Chen, Alexandre Pitti, Mathias Quoy +1Speech ProductionSequence Modeling

    ETIS, CY Cergy-Paris Université - ENSEA – CNRS, UMR 8051, 2 Av. Adolphe Chauvin, Pontoise, 95300, Ile de France, France · A*Star, 1 Fusionopolis Way, #20-10 Connexis North Tower, Singapore, 138632, Singapore · IPAL (International Research Laboratory on Artificial Intelligence), 1 Fusionopolis Way, #21-01 Connexis (South Tower), Singapore, 138632, Singapore

  12. InfoMamba: An Attention-Free Hybrid Mamba-Transformer Model

    Mar 8, 2026Youjin Wang, Jiaqiao Zhao, Rong Fu +5Hybrid Mamba-Transformer ArchitecturesHybrid Transformer-State Space Model Architectures

    Renmin University of China, Beijing, China · University of Macau, Macau SAR, China · Central South University, Changsha, China +1

  13. OrthoFormer: Instrumental Variable Estimation in Transformer Hidden States via Neural Control Functions

    Mar 8, 2026Charles LuoTransformerInstrumental Variable Estimation

    Metro State University

  14. Why Depth Matters in Parallelizable Sequence Models: A Lie Algebraic View

    Mar 5, 2026Gyuryang Heo, Timothy Ngotiaoco, Kazuki Irie +2Sequence ModelingTransformer Expressivity

    Howard Hughes Medical Institute, Department of Neurobiology, Harvard Medical School, Boston, MA, USA · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University, Cambridge, MA, USA · Department of Psychology and Center for Brain Science, Harvard University, Cambridge, MA, USA

  15. IntSeqBERT: Learning Arithmetic Structure in OEIS via Modulo-Spectrum Embeddings

    Mar 5, 2026Kazuhisa NakashoSequence ModelingTransformer

    Iwate Prefectural University, Takizawa, Iwate, Japan

  16. Timer-S1: A Billion-Scale Time Series Foundation Model with Serial Scaling

    Mar 5, 2026Yong Liu, Xingjian Su, Shiyu Wang +7Time Series ForecastingTime-Series Foundation Models

    Tsinghua University · ByteDance

  17. The Volterra signature

    Mar 4, 2026Paul P. Hager, Fabian N. Harang, Luca Pelizzari +1Path SignaturesTime Series Representation Learning

  18. Coupling Local Context and Global Semantic Prototypes via a Hierarchical Architecture for Rhetorical Roles Labeling

    Mar 4, 2026Anas Belfathi, Nicolas Hernandez, Laura Monceaux +4Prototype LearningLegal NLP

    Nantes Université, École Centrale Nantes, CNRS, LS2N, UMR 6004, F-44000 Nantes, France · University of Lorraine, France

  19. MaBERT:A Padding Safe Interleaved Transformer Mamba Hybrid Encoder for Efficient Extended Context Masked Language Modeling

    Mar 3, 2026Jinwoong Kim, Sangjin ParkHybrid Transformer-State Space Model ArchitecturesLong-Context Language Modeling

    Graduate School of Industrial Data Engineering, Hanyang University, Seoul, Republic of Korea

  20. Efficient Extractive Summarization with MAMBA-Transformer Hybrids for Low-Resource Scenarios

    Mar 1, 2026Nisrine Ait KhayiHybrid Transformer-State Space Model ArchitecturesText Summarization

    Independent Researcher · University of Memphis

  21. Probabilistic Learning and Generation in Deep Sequence Models

    Mar 1, 2026Wenlong ChenBayesian InferenceSequence Modeling

    Imperial College London Department of Computing

  22. Wave-Attractor-Tree: A Hierarchical Binary Tree Reduction Architecture for Efficient Sequence Modeling

    Feb 28, 2026Igor BerezkinLong-Range Dependency ModelingSequence Modeling

    Independent Researcher

  23. Efficient Long-Sequence Diffusion Modeling for Symbolic Music Generation

    Feb 28, 2026Jinhan Xu, Xing Tang, Houpeng Yang +7Symbolic Music GenerationDiffusion Models

    School of Computer Science and Artificial Intelligence, Wuhan University of Technology, Wuhan, China 430070 · School of Computer Science, Hubei University of Technology, Wuhan, China 430068 · School of Information Engineering, Hubei University of Economics, Wuhan, China 430205; Hubei Key Laboratory of Digital Finance Innovation, Wuhan, China 430205

  24. MEDNA-DFM: A Dual-View FiLM-MoE Model for Explainable DNA Methylation Prediction

    Feb 26, 2026Yi He, Yina Cao, Jixiu Zhai +3GenomicsSequence Modeling

    Cuiying Honors College, Lanzhou University, Lanzhou, Gansu, China · School of Mathematics and Statistics, Lanzhou University, Lanzhou, Gansu, China · Cuiying Honors College, Lanzhou University, Lanzhou, Gansu, China. +7

  25. Support Tokens, Stability Margins, and a New Foundation for Robust LLMs

    Feb 25, 2026Deepak Agarwal, Dhyey Dharmendrakumar Mavani, Suyash Gupta +2Self-AttentionLanguage Model Robustness

    LinkedIn, Mountain View, CA

  26. Deep Sequence Modeling with Quantum Dynamics: Language as a Wave Function

    Feb 24, 2026Ahmed Nebli, Hadi Saadatdoorabi, Kevin YamSequence ModelingQuantum-Inspired ML

    cAI Technology GmbH

  27. HiPPO Zoo: Explicit Memory Mechanisms for Interpretable State Space Models

    Feb 24, 2026Jack Goffinet, Casey Hanks, David E. CarlsonSequence ModelingState Space Models

    Department of Computer Science, Duke University, Durham NC, USA · Departments of Civil and Environmental Engineering; Biostatistics and Bioinformatics; and Electrical and Computer Engineering, Duke University, Durham NC, USA

  28. Test-Time Training with KV Binding Is Secretly Linear Attention

    Feb 24, 2026Junchen Liu, Sven Elflein, Or Litany +2Test-Time TrainingLinear Attention

    NVIDIA, Toronto, Ontario, Canada · University of Toronto, Toronto, Ontario, Canada · Vector Institute, Toronto, Ontario, Canada +1

  29. Online Prediction of Stochastic Sequences with High Probability Regret Bounds

    Feb 18, 2026Matthias Frey, Jonathan H. Manton, Jingge ZhuSequence PredictionSequence Modeling

    Department of Electrical and Electronic Engineering, The University of Melbourne

  30. Learning State-Tracking from Code Using Linear RNNs

    Feb 16, 2026Julien Siems, Riccardo Grazzi, Korbinian Pöppel +3Recurrent Neural NetworksState Tracking

    University of Freiburg · Prior Labs · Microsoft Research +2