Sequence Modeling

Follow this topic Ask about this topic

Sequence modeling asks how information in an ordered stream—such as words, sensor readings, user actions, or DNA—can be used to predict what comes next or interpret the sequence as a whole. Common approaches include recurrent networks such as LSTMs, Transformer attention, state-space models such as Mamba, and Markov or hidden Markov models; hybrids and diffusion methods are also used for long or structured sequences. Applications range from language and time-series forecasting to recommendation, biological analysis, and action recognition, with evaluation often testing prediction quality, long-range recall, and generalization to new lengths or domains. A central tension is balancing expressive memory and reliable long-context behavior against computation, latency, and memory costs, while ensuring benchmarks measure genuine sequence understanding rather than leakage or repetition.

Latest papers

250 papers22 papers in the last four weeks, up 38% on the four weeks before. 0.1% of all new papers.

Rethinking Point Clouds as Sequences: A Causal Next-Token Predictive Learning Framework
May 17, 2026Yumeng Yao, Jingzhi Dong, Haowen Gu +4Point Cloud Learning 3D ViTs
Nanjing University of Science and Technology · Hangzhou City University · East China Normal University +1
Multi-Level Contextual Token Relation Modeling for Machine-Generated Text Detection
May 15, 2026Chenwang Wu, Yiuming Cheung, Bo Han +2AI-Generated Text Detection Markov Models
Department of Computer Science, Hong Kong Baptist University, Hong Kong, China · School of Software Engineering, South China University of Technology, Guangzhou, Guangdong 510000, China · School of Computer Science and Technology, University of Science and Technology of China, Hefei, Anhui 230000, China +1
LifeSentence: Language models can encode human life course trajectories from longitudinal panel data
May 14, 2026Samuel Liu, Muchen Xi, William Yeoh +1Computational Social Science Human Behavior Prediction
DRL-STAF: A Deep Reinforcement Learning Framework for State-Aware Forecasting of Complex Multivariate Hidden Markov Processes
May 14, 2026Manrui Jiang, Jingru Huang, Yong Chen +1Multivariate Time Series Forecasting Reinforcement Learning
Department of Industrial Engineering, Tsinghua University, Beijing 100084, China · Department of Industrial and Systems Engineering, University of Iowa, Iowa City, IA 52242, USA.
AttnGen: Attention-Guided Saliency Learning for Interpretable Genomic Sequence Classification
May 13, 2026Rayhaneh Shabani Nia, Ali KarkehabadiNeural Network Interpretability Attention Mechanisms
University of California, Davis California, USA
QLAM: A Quantum Long-Attention Memory Approach to Long-Sequence Token Modeling
May 13, 2026Hoang-Quan Nguyen, Sankalp Pandey, Khoa LuuMemory-Augmented Neural Networks Hybrid Quantum-Classical ML
Department of Electrical Engineering and Computer Science, University of Arkansas, AR · Quantum AI Lab, University of Arkansas, AR
Multimodal Hidden Markov Models for Persistent Emotional State Tracking
May 13, 2026Anamika Ragu, Aneesh JonelagaddaEmotion Recognition in Conversations Valence-Arousal Modeling
Kaliber AI, San Mateo, California, USA.
Discrete MeanFlow: One-Step Generation via Conditional Transition Kernels
May 12, 2026Fairoz Nower Khan, Nabuat Zaman Nahim, Md Sajid Ahmed +2Markov Models MeanFlow
Department of Computer Science, University of Kentucky, Lexington, Kentucky 40506, USA
From Message-Passing to Linearized Graph Sequence Models
May 12, 2026Joël Mathys, Basil Rohner, Saku Peltonen +1Long-Range Graph Learning Graph Representation Learning
1ETH Zurich, Switzerland
Complex-Valued Phase-Coherent Transformer
May 11, 2026Leona HiokiSoftmax Attention Complex-Valued Neural Networks
MambaNetBurst: Direct Byte-level Network Traffic Classification without Tokenization or Pretraining
May 11, 2026Gayan K. Kulatilleke, Siamak Layeghy, Mahsa Baktashmotlagh +1Network Intrusion Detection Encrypted Traffic Classification
University of Queensland, Brisbane, Australia
UxSID: Semantic-Aware User Interests Modeling for Ultra-Long Sequence
May 9, 2026Hongwei Zhang, Qiqiang Zhong, Junfeng Shu +8Semantic ID Recommendation Sequential Recommendation
Kuaishou Technology, Beijing, China
A Geometric Perspective on Next-Token Prediction in Large Language Models: Three Emerging Phases
May 9, 2026Gianfranco Lombardo, Giuseppe Trimigno, Stefano CagnoniNeural Representation Geometry LLM Interpretability
Department of Engineering and Architecture University of Parma Parma, IT 43125
Continuity Laws for Sequential Models
May 8, 2026Annan Yu, Dongwei Lyu, N. Benjamin ErichsonSelective SSMs Temporal Consistency
Center for Applied Mathematics, Cornell University · University of California, Berkeley · Lawrence Berkeley National Laboratory +1
RubiConv -- Efficient Boundary-Respecting Convolutions
May 8, 2026Linda Friso, Annie Marsden, Xinyi Chen +4Convolutional Neural Networks Sequence Modeling
†Google Deepmind · ‡UC Berkeley · §Princeton University
The Power of Second Order Methods for Sequence Preconditioning
May 8, 2026Annie Marsden, Elad HazanDynamical Systems Sequence Modeling
Google DeepMind · Princeton University
Exact Regular-Constrained Variable-Order Markov Generation via Sparse Context-State Belief Propagation
May 8, 2026François PachetConstrained Generative Modeling Belief Propagation
LIP6, Sorbonne Université · Ynosound
Better Protein Function Prediction by Modeling Survivorship Bias
May 7, 2026Zhongmou Chao, Poompol Buathong, Ekaterina Selivanovitch +2Protein Function Prediction Weakly Supervised Learning
Smith School of Chemical and Biomolecular Engineering, Cornell University, USA · Center for Applied Mathematics, Cornell University, USA · School of Operations Research and Information Engineering, Cornell University, USA
Wisteria: A Unified Multi-Scale Feature Learning Framework for DNA Language Model
May 7, 2026Weihua Wang, Haoji Li, Feilong Bao +2Representation Learning Language Modeling
College of Computer Science, Inner Mongolia University, Hohhot, Inner Mongolia, 010021, China · National and Local Joint Engineering Research Center of Intelligent Information Processing Technology for Mongolian, Hohhot, Inner Mongolia, 010021, China · Inner Mongolia Key Laboratory of Multilingual Artificial Intelligence Technology, Hohhot, Inner Mongolia, 010021, China +1
Understanding Cross-Language Transfer Improvements in Low-Resource HTR: The Role of Sequence Modeling
May 7, 2026Sana Al-azzawi, Chang Liu, Nudrat Habib +2Convolutional RNNs Recurrent Neural Networks
Luleå University of Technology, Department of Computer Science, Electrical and Space Engineering , Luleå, Sweden · Luleá University of Technology, Department of Computer Science, Electrical and Space Engineering , Luleå, Sweden
Spherical Flows for Sampling Categorical Data
May 7, 2026Jannis Chemseddine, Gregor Kornhardt, Gabriele SteidlFlow Matching Synthetic Data Generation
Technische Universität Berlin
The Impossibility Triangle of Long-Context Modeling
May 6, 2026Yan ZhouLong-Context Modeling Associative Memory
School of Mathematics and Statistics Changsha University of Science and Technology Changsha, Hunan 410114, China
Tree-Conditioned Edit Flows for Ancestral Sequence Reconstruction
May 5, 2026Emil Sharafutdinov, Ingemar AndréFlow Matching Phylogenetic Inference
Center for Molecular Protein Science Lund University Box 124, 221 00 Lund
TemPose-TF-ASF: Two-Stage Bidirectional Stroke Context Fusion for Badminton Stroke Classification
May 4, 2026Tzu-Yu Liu, Duan-Shin LeeSports Video Analysis Video Action Recognition
Institute of Communications Engineering National Tsing Hua University, Hsinchu City, Taiwan · Department of Computer Science National Tsing Hua University, Hsinchu City, Taiwan
Spiking Sequence Machines and Transformers
May 1, 2026Joy BoseTransformer Attention Spiking Neural Networks
Independent Researcher Bengaluru, India
Preserving Temporal Dynamics in Time Series Generation
Apr 29, 2026Ci Lin, Futong Li, Tet Yeap +1Time Series Generation Time Series Data Augmentation
Action-Aware Generative Sequence Modeling for Short Video Recommendation
Apr 28, 2026Wenhao Li, Zihan Lin, Zhengxiao Guo +7Sequential Recommendation User Preference Modeling
Kuaishou Inc. · Beijing, China · Unaffiliated +2
Mining Negative Sequential Patterns to Improve Viral Genomic Feature Representation and Classification
Apr 28, 2026Wenxi Zhu, Wensheng Gan, Zhenlian QiGenomics Sequence Modeling
College of Information Science and Technology, Jinan University, Guangzhou 510632, China · College of Cyber Security, Jinan University, Guangzhou 510632, China · Guangdong Eco-Engineering Polytechnic, Guangzhou 510520, China
Long-Context Aware Upcycling: A New Frontier for Hybrid LLM Scaling
Apr 27, 2026Parsa Ashrafi Fashi, Utkarsh Saxena, Mehdi Rezagholizadeh +7Long-Context Language Model Inference Long-Context Language Modeling
1AMD
An event-based sequence modeling approach to recognizing non-triad chords with oversegmentation minimization
Apr 27, 2026Leekyung Kim, Jonghun ParkMusic Information Retrieval Sequence Modeling
Department of Industrial Engineering and Institute of Industrial Systems Innovation, Seoul National University, Seoul, Republic of Korea

May 17, 2026J/K move · Enter open · S save

Rethinking Point Clouds as Sequences: A Causal Next-Token Predictive Learning Framework

Yumeng Yao, Jingzhi Dong, Haowen Gu, Tao Chen+3

Nanjing University of Science and Technology · Hangzhou City University · East China Normal University+1

Point Cloud Learning 3D ViTs Self-Supervised Pre-Training Sequence Modeling Next-Token Prediction Predictive Representation Learning+1

Open paper AskPDF arXiv

With the rapid progress of multimodal foundation models and predictive pre-training, an important open question is how to equip 3D point clouds with a pre-training paradigm that is better aligned with next-token and next-embedding learning. Existing point-cloud self-supervised methods are largely built on masked reconstruction or explicit geometric generation, and thus remain tied to input recovery rather than predictive dependency modeling. In this paper, we introduce PointNTP, which reformulates point cloud pre-training as a fully causal, decoder-free latent Next-Token Prediction problem. Specifically, each point cloud is first partitioned into local patches and serialized into a structured 3D token sequence according to patch-center geometry. The resulting sequence is then modeled by a causal Transformer under prefix-only conditioning, and trained with a shift-based prediction objective stabilized by stop-gradient targets. This design enables the model to learn structural dependencies directly in latent space, without reconstruction decoders or explicit geometric recovery. Extensive experiments demonstrate that the proposed PointNTP is highly competitive across multiple downstream tasks: it achieves 93.8%(+0.5%), 92.6%(+0.3%), and 89.3%(+1.1%) on OBJ_BG, OBJ_ONLY, and PB_T50_RS of ScanObjectNN, respectively; obtains 85.0%(+0.1%) in Cls.mIoU on ShapeNetPart; and reaches 71.1% mAcc on S3DIS Area 5. Overall, decoder-free causal latent prediction provides a simple, scalable, and potentially modality-agnostic paradigm for point-cloud self-supervised learning, offering a new 3D perspective on foundation-style predictive learning for 3D data.