Sequence Prediction

Follow this topic Ask about this topic

Latest papers

31 papers8 papers in the last four weeks, up 100% on the four weeks before. 0.0% of all new papers.

Oct 16, 2026

Next Token or Vector? Learning How Far to Decode

Standard autoregressive decoding advances one token at a time, even when several upcoming tokens can be predicted together. We propose CADENCE (Context-Adaptive DEcoding), a generation method that learns both what to draft and how far to draft. Gaussian prediction heads represent future spans as continuous latent vectors, and a shared decoder converts the selected vector into token predictions in parallel. A learned router selects the prediction length from the model's own hidden states and predicted Gaussian parameters, adapting generation granularity to the current context with single-token generation as a fallback. The backbone verifies each proposed continuation and retains the prefix that matches its greedy predictions. Experiments with Llama-3.1-8B-Instruct on Alpaca, GSM8K, and HumanEval show that router-selected positions yield higher draft acceptance than applying the corresponding prediction length everywhere. CADENCE achieves up to 1.3x higher generation throughput than standard greedy decoding.

Sai Sreenivas Chintha, Chidera Biringa, Rishab Sharma +6

Microsoft · University of Massachusetts Amherst

LLM Inference Acceleration Multi-Token Prediction Sequence Prediction Speculative Decoding

Oct 8, 2026

SteerCast: Retrieval-Based Latent Steering for Decoder-Only Time Series Forecasting

Time series forecasting aims to predict future values from historical observations and auxiliary features. We propose SteerCast, a retrieval-based latent steering method that improves decoder-only forecaster at inference time, without updating its parameters. SteerCast constructs a database from the training set by storing a representation of each history window together with a steering vector computed in the forecaster's latent space, defined as the difference between representations induced by the ground-truth continuation and by the model's own prediction. At test time, SteerCast retrieves nearest neighbors for a query history, aggregates their steering vectors, and injects the resulting signal into the forecaster's hidden states at every step of autoregressive generation, guiding predictions toward trajectories consistent with similar training cases. Experiments across diverse multivariate benchmarks and multiple horizons show that SteerCast consistently improves forecasting accuracy over the fine-tuned backbone and retrieval-based baselines, while requiring no additional training beyond the original fine-tuning and using only the training set as a retrieval corpus.

Van Dai Do, Huu Hiep Nguyen, Minh Hoang Nguyen +1

Deakin’s Applied Artificial Intelligence Initiative Deakin University, Geelong, Australia

Multivariate Time Series Forecasting Sequence Prediction Retrieval-Augmented Time Series Forecasting Inference-Time Guidance

Oct 5, 2026

LightMTP: Lightweight Latent Multi-Token Prediction

Next-token prediction (NTP) is the standard pretraining objective for large language models, yet it provides an explicit training signal only for the immediate next token, which can lead models to exploit local patterns instead of capturing longer-range structure and ideas. Multi-token prediction (MTP) addresses this by training models to predict several future tokens. However, existing MTP methods often introduce a large number of new parameters with limited improvements in downstream performance. Latent MTP approaches address this efficiency issue by encoding future tokens into a vector representation. However, these approaches usually rely on external helper models for future token encoding. We propose LightMTP, a lightweight, i.e., parameter-efficient, latent MTP approach that bootstraps the future token representations from the model's own hidden states. Our two LightMTP variants extend supervision to more future tokens without requiring the additional computational overhead of conventional MTP nor the external supervision latent MTP normally relies on. LightMTP adds at most 1% extra parameters, retains better performance on general language modeling benchmarks, and achieves similar gains in planning, coding, and reasoning.

Tamara Czinczoll, Julie Kallini, Gerard de Melo +1

Hasso Plattner Institute / University of Potsdam · Stanford University · Tel-Aviv University

Language Model Pretraining Multi-Token Prediction Sequence Prediction

Oct 2, 2026

How Far Back Should a Transformer Look? Repetition and Copying in Music Sequence Models

We investigate how predictive performance depends on the maximum context available to an autoregressive model of symbolic music, and what information long-context models exploit. A small causal Transformer is trained separately at each context length T in {6, 18, 48, 96, 192, 336} and evaluated on the same target positions, both over raw tokens and over non-overlapping binary latent codes. On Nottingham folk tunes, the token predictor's test NLL decreases by 71% (0.986 bits per token) between 6 and 336 tokens, and by 71% on the O'Neill's tunes long enough for the same sweep (129 of 302 test tunes). Most of the long-context gain is explained by exact copying: the improvement appears when an earlier occurrence of the target's 16-token history enters the available context; overwriting that occurrence removes the gain, whereas equally large unrelated corruption does not; and a simple copy baseline recovers 94% of the reduction. A trained 336-token model likewise loses most of this gain when its history is restricted to recent tokens at test time. Because many repeats arise from written repeat signs expanded in the score, this result pertains specifically to these rendered score representations. By contrast, on MAESTRO performances and MusicNet scores, where exact repeats are substantially less frequent, the reduction is smaller (9% and 18%) and is largely attained by 96 tokens. Finally, an audit of an earlier draft that reported saturation at 16 tokens identified split leakage, overlapping latent receptive fields, an averaging predictor, and a per-file tempo grid; we present these as methodological checks for context-length measurements.

Amir Fathi

Department of Computing Science University of Alberta Edmonton, AB, Canada

Long-Context Modeling Sequence Modeling Sequence Prediction Transformer Symbolic Music Generation

Sep 30, 2026

OpenTSLM TeeMoE: A Unified Time-Series Language Model for Forecasting, Contextual Prediction, and Reasoning

Real-world time-series applications increasingly require models that can handle time series forecasting, context-conditioned prediction, and language-based temporal reasoning. Yet current time-series foundation models remain fragmented across these capabilities: numerical specialists often provide the strongest forecasts, while language-based models offer broader contextual understanding and analysis. A central challenge is to unify these heterogeneous capabilities without reducing their individual performance. We introduce OpenTSLM TeeMoE, a generalist time-series language model that can forecast directly from observed time series, reason over textual context and temporal patterns, and synthesize and refine predictions from external numerical forecasting specialists. We independently train three low-rank experts for forecast aggregation, native forecasting, and temporal analysis over a shared backbone. A learned LoRA mixture-of-experts controller then weights their frozen parameter updates for each request. Our proposed model achieves strong performance on widely used benchmarks for time series forecasting, context-conditioned prediction, and language-based temporal reasoning, ranking among the top three on GIFT-Eval by mean MASE rank, Context is Key by RCRPS, and TimeSeriesExam by accuracy.

Tony Chen, Timo Stoffregen, Maxwell Xu +8

Columbia University, USA · Stanford University, USA · Aionic Labs, Switzerland +4

Temporal Reasoning in Language Models LLM-Based Time Series Forecasting Mixture-of-Experts Models Time-Series Foundation Models Time Series Forecasting Time Series Modeling+2

Sep 28, 2026

Information-Theoretic Analysis of Next-Token Prediction under Markovian Data

We develop an information-theoretic framework for generalization in next-token prediction under temporally dependent data. We consider independent trajectories generated by finite-memory Markov processes and distinguish algorithmic dependence, quantified by mutual information, from temporal dependence, characterized by mixing. For cross-entropy loss, we derive an expected generalization bound using the Donsker--Varadhan variational representation and a McDiarmid-type concentration inequality for Markov chains. A refinement captures the joint effect of context length and temporal mixing through the mixing properties of the history-state process. We then extend the bound through a rate--distortion formulation, replacing mutual information with the minimum information rate required to represent the learned model within a prescribed distortion in the generalization gap, yielding informative guarantees for deterministic algorithms over continuous hypothesis spaces. For margin-based prediction, we derive explicit bounds for linear and self-attention next-token predictors via noisy low-dimensional compression, revealing the roles of context length, model complexity, sample size, margin, and temporal mixing. Experiments on TinyStories and ETTh2 show that longer contexts can reduce both training and test losses, but typically reduce training loss more, enlarging the generalization gap. A complementary ETTh2 analysis identifies an effective predictive-memory scale near 24 hours, with no statistically supported improvement beyond this scale, offering a plausible explanation for test-performance saturation at larger contexts.

Masoud Kavian, Abdellatif Zaidi, Milad Sefidgaran

Mathematical and Algorithmic Science Laboratory, Huawei Paris Research Center, 92100 Boulogne-Billancourt, France · Laboratoire d’Informatique Gaspard Monge, Université Gustave Eiffel, 77420 Champs-sur-Marne, France

Information-Theoretic Generalization Bounds Next-Token Prediction Generalization Bounds Sequence Prediction

Sep 27, 2026

Decoupling Token Roles in Autoregressive Pretraining

Autoregressive pretraining increasingly draws on heterogeneous data, making it important to understand how a model learns from an individual token. The next-token prediction objective naturally identifies a token's contribution with its own loss. However, each token is not only a prediction target but also context for what follows. Using controlled corruption, we decouple these two roles and find a reversal: making a noisy token easier to predict reduces its damage as a target but increases it as context. The same decoupling helps explain text generated by language models: generation selects each token by its fit to the prefix, while its role as context is never tested against an independently determined continuation, because that continuation is generated to fit it. At known corrupted positions, acting through the context can reduce damage that removing the token's own loss does not. Understanding and controlling what a model learns from a token therefore requires decoupling its roles.

Suqin Yuan, Runqi Lin, Kevin Qinghong Lin +4

University of Sydney · University of Oxford · Southeast University

Language Model Pretraining Autoregressive Language Modeling Causal Interventions in Language Models Language Model Robustness Autoregressive Transformer Sequence Prediction

Sep 24, 2026

The Cost of Long Memory: State, Context, and Stability Complexity in Sequence Models

Long-range temporal dependence poses a resource question for sequence models: for a specified predictive-memory law, how much state, context, or dynamical criticality is required in order to forecast accurately? We study this question directly in forecasting risk. For algebraically decaying predictive memory, we prove matching upper and lower approximation bounds for exponential and finite-state modes. The best

r

-mode forecast error decays as

e^{-Θ(\sqrt r)}

, so reaching forecast error

τ

needs

r=Θ(\log^2(1/τ))

states or modes. Earlier curse-of-memory results establish broad limitations of stable recurrent models under different approximation notions; here both sides match for one canonical predictive target in forecast risk, which fixes the optimal resource exponent for that target. We then show that genuine fractional long memory changes the geometry itself. In particular, forecast error is measured after fractional integration, prediction from a finite context of length

L

has an exact

1/L

leading order, and a fixed fractional strength

d

keeps the square-log state-complexity law. Near the short-memory boundary, we identify the relevant

d^2

and

d^4

scales and give a uniform constructive law in the intermediate regime. For nonlinear contextual recurrences with uniformly contractive state dynamics, we derive an exponential first-chaos envelope and an explicit necessary condition that relates forecast accuracy to the contraction margin. Vanishing forecasting error on an algebraic target forces the recurrence quantitatively toward criticality, a condition that is necessary and not by itself sufficient. Finite-sample Kullback--Leibler calculations further connect the predictive geometry to statistical information. Theorem-matched experiments with contractive state-space, gated recurrent, and attention models reproduce the state and stability predictions.

Yuheng Song

School of Management, Huazhong University of Science and Technology

Sequence Prediction Time Series Forecasting Recurrent Neural Networks State Space Models

Sep 17, 2026

Stringological sequence prediction III: layered ziplines and a tradeoff between efficiency and expressivity

In previous papers, we began the study of sequence prediction algorithms adapted to stringological word complexity measures. In particular, we defined a complexity measure called Arithmetic Repetition Complexity (ARC) which admits a polynomial-time prediction algorithm with a mistake bound quasilinear in the complexity. Here, we show a weaker complexity measure related to ARC that admits an especially efficient prediction algorithm: an algorithm that runs in quasilinear time and polylog space for appropriate highly-structured sequences. The complexity measure is defined via a restricted class of "zipline programs" (a variant of straight-line programs), which we call layered. We thus get a less expressive measure with a more efficient algorithm (compared to our results for ARC), demonstrating a possible tradeoff.

Vanessa Kosoy

Faculty of Mathematics, Technion, Haifa, Israel · Computational Rational Agents Laboratory, Delaware, USA

Sequence Modeling Sequence Prediction

Sep 5, 2026

SIDE: Sensor Impersonation Detection at the Edge via Sequence Prediction

Some low-cost Internet of Things (IoT) sensor deployments lack device-level source authentication, leaving them vulnerable to impersonation or injected sensor readings. We present a lightweight approach to sensor impersonation detection in a small proof-of-concept study. We formulate detection as a sequence-prediction problem. A model with three LSTM layers and two fully connected layers is trained only on univariate temperature readings from a genuine sensor, and a window of readings is flagged when its mean absolute prediction error exceeds the mean genuine error by more than six standard deviations. The model is converted to TensorFlow Lite in three variants, non-quantized (554 KB), 16-bit weight quantized (298.5 KB), and 8-bit weight quantized (185 KB), targeting an Arduino Nano 33 BLE aggregator. The variants were evaluated with the TensorFlow Lite interpreter on the local server, since on-device execution of LSTM models was not yet supported by TensorFlow Lite for Microcontrollers at the time of the study. On a controlled testbed with the impostor sensor placed in a hotter outdoor location, the three variants reached detection accuracies of 99.980%, 99.972%, and 98.206%, and each flagged the change point when a test sequence switched from genuine to impostor data. Quantization made the model smaller but slower in our measurements. The genuine and impostor distributions were well separated, so these results show detection of a controlled distribution shift and should not be read as evidence of general device authentication.

Nahom Birhan

Department of Electrical and Computer Engineering Old Dominion University Virginia, USA

Internet of Things Time-Series Anomaly Detection Sequence Prediction

Aug 27, 2026

Fast Weight Attention for Continual Learning

Recurrent fast-weight memories and selective state-space models compress an expanding context into a fixed-size recurrent state, making the state transition an online learning rule. We study this rule under read-after-write autoregressive semantics. For the prefix-prediction objective considered here, the local fast-memory example revealed at step

t

is the prefix-aligned pair

(\mathbf{x}_t,\mathbf{y}_t)=(φ(\mathbf{k}_{t-1}),\mathbf{v}_t)

. The common same-step association

(φ(\mathbf{k}_t),\mathbf{v}_t)

remains causal, but optimizes a different internal objective. We derive normalized first-order updates for squared-error regression and negative inner-product objectives. The regression family comprises Falcon-1 (a scalar NLMS update), Falcon-2 (its per-column extension), and Falcon-3 (a sliding-window mini-batch update); Falcon-1A/Falcon-2A/Falcon-3A are the corresponding inner-product variants. We provide recurrent, masked-parallel, and chunk-parallel forms, together with numerically stable positive-decay renormalization. Representative variants remain competitive in language modeling and improve length extrapolation on variable-digit addition. This framework separates temporal alignment, plasticity, forgetting, and bounded rehearsal in recurrent sequence models.

Yifan Zhang, Steve Ta, Jasper Zhang +8

Bytedance Seed · Princeton University · Hyperbolic Labs +1

Recurrent Language Models Memory-Augmented Language Models Sequence Prediction Attention Mechanisms Autoregressive Language Modeling Online Continual Learning

Aug 21, 2026

Truthful Calibration Measures for Sequential Prediction

Calibration requires probabilistic reports to be conditionally unbiased and reliably interpretable as probabilities. A calibration measure assigns numerical error to miscalibrated reports. Haghtalab et al. (2024) proposed an approximately truthful calibration measure for online prediction, leaving open whether exact truthfulness is compatible with completeness and soundness. We resolve this question negatively for sequential binary prediction: exact truthfulness is incompatible with completeness and soundness, even for independent outcomes. We then show that this impossibility is specific to exact truthfulness. We give two general reductions from a base calibration measure, producing additively and multiplicatively approximately truthful calibration measures, respectively. Applying the multiplicative reduction, for every

0 < \varepsilon < 1

we construct a sound and complete calibration measure that is

(1+\exp(-T^{(1-\varepsilon)/2}/2))

-multiplicatively truthful. This improves the approximate-truthfulness guarantee of Haghtalab et al. (2024).

Anagha Gokul, Jason Hartline, Lunjia Hu +2

Khoury College of Computer Sciences · Department of Computer Science · Microsoft Research New England

Online Calibration Sequence Prediction Probability Calibration Model Calibration

Aug 18, 2026

How Far Should Tokenization Go? Predictive Effectiveness and Relational Losslessness

GPT-style models have achieved remarkable success with finite vocabularies of reusable tokens, making the token interface a central component of modern sequence modeling. Symbolic music appears naturally compatible with this paradigm: it consists of discrete note events and recurring structures such as chords, motifs, and phrases. However, when tokenization moves beyond language, the interface must be specified for each domain. Existing work offers many effective designs, but no unified criterion for deciding what tokenization should represent and how far it should go. Using predictive codelength as a common criterion, we formulate the Effectiveness--Losslessness Framework to define where tokenization should begin and where it should end. The Fact--Token Boundary marks where observation-determined structure should enter the token interface, through operations such as coordinate construction. Within this interface, the resulting carrier may be reversibly recoded without changing the represented facts. The Token--State Boundary marks where tokenization should stop: relations that depend on context should remain for model-state computation rather than being fixed in advance by the tokenizer. We validate the framework through controlled multi-seed symbolic-music experiments, with an independent-corpus replication of the temporal intervention. Making musical time explicit consistently reduces predictive code and also improves pitch and duration prediction, while tonal-frame canonicalization and pitch factorization provide further gains. Fixed circle-of-fifths pitch coordinates instead increase predictive code, suggesting that imposing a fixed pitch relation before context can burden prediction. Reversible BPE substantially shortens the carrier but increases predictive codelength in every seed, showing that carrier compaction alone does not guarantee predictive gain.

Yi Wang

Department of Electronic Engineering, Tsinghua University

Tokenization Token Efficiency Sequence Modeling Sequence Prediction Symbolic Music Generation

Aug 14, 2026

Sequence prediction under a lying oracle

We consider the problem of sequential prediction of an

m

-ary sequence, where at each epoch, (i) the environment selects an outcome from an

m

-ary alphabet, (ii) the learner selects a probability distribution over the same alphabet (unaware of the outcome generated by the environment), and finally, (iii) the learner incurs a cost that depends on the probability assigned to the outcome. The cost function we consider captures the complexity of predicting the outcome generated by the environment, in a scenario where the aforementioned prediction is performed via comparative queries to a lying oracle. We consider both stochastic and adversarial environments, propose algorithms for both settings, and establish logarithmic upper bounds on their regret.

Puspabeethi Samanta, Nikhil Karamchandani, Jayakrishnan Nair

Department of Electrical Engineering IIT Bombay

Regret Minimization Sequence Prediction Sequence Modeling

Aug 6, 2026

Hierarchical Latent Prediction for Language Models

While standard Next-Token Prediction (NTP) lays the foundation of language model pre- training, its teacher-forced training paradigm may not be optimal for long-horizon reasoning and planning. Recent works such as Multi-Token Prediction (MTP) and Next-Latent prediction (NextLat) try to mitigate the problem through predicting multiple future tokens and self-supervised prediction in the latent space. However, those auxiliary objectives either have a limited horizon or suffer from compounding error from multi-step rollout. We introduce Hierarchical Latent Prediction (HiLP), which introduces an auxiliary higher-level abstract latent to help reduce the error accumulation effect in latent-space rollouts. Experiments show that HiLP can lead to longer-horizon coherent belief state representation and demonstrate the effectiveness of our method across coding and multi-step reasoning benchmarks, and offers more speculative decoding efficiency.

Chang Shi, Tim Pearce, Manan Tomar +2

University of Texas at Austin · Microsoft Research

Language Model Pretraining Multi-Token Prediction Language Modeling Large Reasoning Models Sequence Prediction

Aug 2, 2026

Hierarchical Solomonoff Induction: An Unbounded Machine Learning Model

Solomonoff Induction, or SolInd, provides an ideal unbounded model of a priori sequence prediction but cannot naturally describe extrapolation from a given training dataset, as performed by Large Language Models. We apply de Finetti's theorem on exchangeable distributions to SolInd to produce what we call Hierarchical Solomonoff Induction, or HSI, which maintains a hyperprior over all Solomonoff priors that can be conditioned on previously observed sequences. We extend Wood et al.'s proof that universal mixtures of semimeasures are equivalent to SolInd to show that universal mixtures of these mixtures are also equivalent, proving that HSI=SolInd. We also prove that HSI's excess error on any distribution, compared to its true generator, is bounded by that generator's complexity in the hyperprior. This result is directly comparable to SolInd's prediction error being bounded by the Kolmogorov complexity of the sequence being predicted, and forces HSI's average excess error to converge to 0 as a dataset grows, leading to optimal prediction in the limit. We claim that HSI is an ideal unbounded model of sequence prediction given a dataset in the same way that SolInd is ideal over individual sequences.

Nathan Young

Strong AI Lab, University of Auckland

Hierarchical Bayesian Modeling Statistical Learning Theory Sequence Modeling Sequence Prediction Inductive Inference

Jul 19, 2026

Stringological sequence prediction II: Right-to-left automaticity and related complexity measures

In a previous paper, we began the study of sequence prediction algorithms adapted to stringological word complexity measures. One measure we considered was left-to-right (most-significant-digit-first) automaticity. Here, we show a statistically and computationally efficient algorithm adapted to the

dual'' right-to-left (least-significant-digit-first) automaticity, which turns out to be substantially different for our purpose. We also demonstrate a prediction algorithm for a more expressive measure that we call

arithmetic repetition complexity''. In particular, the latter can be used for predicting the so-called mix-automatic sequences.

Vanessa Kosoy

Faculty of Mathematics, Technion, Haifa, Israel · Computational Rational Agents Laboratory, Delaware, USA

Sequence Prediction Sequence Modeling

Jul 14, 2026

ReDiTT: Retrieval Augmented Conditional Diffusion Transformers for Asynchronous Time Series

We present a diffusion based model for asynchronous time series prediction, where the goal is to predict the next inter event time and event type. To address the inherent uncertainty of future events, we introduce ReDiTT, a retrieval augmented conditional diffusion transformer that operates in latent space. ReDiTT retrieves structurally similar latent sequences from a memory bank during both training and inference and incorporates them as reference conditions through cross attention. This retrieval based conditioning allows the model to attend to relevant temporal dynamics and provides global structural guidance for generation. As a result, ReDiTT stabilizes long horizon forecasting and improves sample diversity. Experiments on seven real world datasets demonstrate state of the art performance on next event prediction and long horizon forecasting. Our code is available at https://github.com/BorealisAI/ReDiTT.

Saiyue Lyu, Zhitian Zhang, Ruizhi Deng +1

Department of Computer Science, University of British Columbia · Work done during an internship at RBC Borealis. · RBC Borealis

Irregular Time-Series Modeling Event Time Prediction Temporal Point Processes Diffusion Transformer Retrieval-Augmented Time Series Forecasting Conditional Diffusion Models+2

Jun 4, 2026

Pretraining Recurrent Networks without Recurrence

Training recurrent neural networks (RNNs) requires assigning credit across long sequences of computations. Standard backpropagation through time (BPTT) addresses this problem poorly: it is sequential in time, limiting parallelism, and suffers from vanishing or exploding gradients, making long-range associations difficult to learn. We propose Supervised Memory Training (SMT), a method for training nonlinear RNNs that sidesteps recurrent credit propagation entirely by reducing RNN training to supervised learning on one-step memory transition labels

(m_t, x_{t+1}) \rightarrow m_{t+1}

. SMT acquires these memory labels by training a Transformer-based encoder on a predictive state objective--retaining only information from the past necessary to predict the future. By decoupling what to remember from how to update memory, SMT enables time-parallel RNN training with a stable

O(1)

length gradient path between any two tokens--without ever unrolling the RNN. We find that SMT outperforms BPTT when pretraining various RNN architectures on tasks like language modeling and pixel sequence modeling. SMT enables nonlinear RNNs to better capture long-range dependencies and train in parallel, potentially unlocking the scaling of models that build temporal abstractions of past experience.

Akarsh Kumar, Phillip Isola

MIT

Recurrent Neural Networks Recurrent Language Models Long-Range Dependency Modeling Sequence Prediction

Jun 3, 2026

Prediction Under Imperfect Compression: A Theory of Approximate MDL

Minimum Description Length (MDL) formalizes the principle of Occam's razor by optimizing the total description length:

L(\mathrm{model})+L(\mathrm{data} \ | \ \mathrm{model})

. For sequential prediction, the MDL method repeatedly selects a model with a minimum objective score of the observed prefix for the next step prediction. Classical MDL prediction theory shows that exact optimization of the MDL objective indeed provides a strong compression guarantee that supports reliable prediction. However, practical machine learning usually can only find models by approximately optimizing the objective function. To bridge this gap, this paper addresses the following fundamental question: Under what forms of approximation and regularization does approximate MDL still guarantee reliable sequential prediction? This work offers a principled characterization. We prove that for any approximation with additive slack

C

of the more general form of the balanced MDL objective:

λ\cdot L(\mathrm{model})+L(\mathrm{data} \ | \ \mathrm{model})

, the cumulative expected squared prediction error is finite for all

λ\ge1

. The case

λ>1

is proved by an affinity-telescoping argument, while the boundary case

λ=1

is proved by a likelihood-ratio stopping argument based on exact static MDL bounds. Our results establish that classical MDL regularization remains robust to any fixed additive optimization error. Furthermore, we establish that our characterization of the approximate MDL framework is sharp: When

0<λ<1

, overfits can happen to incur infinite cumulative expected error in the universal class of estimable measures, and hence a strong form of model-complexity regularization is necessary. In addition, model selection may fail in every regularized regime

λ>0

, under multiplicative approximation, and thus, additive approximation is both sufficient and essential.

Qian Li, Xinyu Mao, Shang-Hua Teng +1

Shenzhen Research Institute of Big Data · University of Southern California

Minimum Description Length Approximation Algorithms Statistical Learning Theory Sequence Prediction Sequence Modeling

May 31, 2026

Time Series as Language: A Universal Tokenizer for General-Purpose Time Series Foundation Models

While Next-Token Prediction (NTP) has unified LLM pretraining, its adaptation to unbounded, continuous time series (TS) remains open. To bridge the gap, we introduce UniTok, a universal tokenizer that transforms TS into discrete tokens, and UniTok-FM, a foundation model pretrained via NTP on these tokens. UniTok-FM is a general-purpose foundation model that supports zero-shot and prompt-boosted forecasting, as well as few-shot generation and classification via training-free in-context inference--a capability not achieved by prior works. Technically, UniTok is a vector-quantized autoencoder incorporating prefix normalization for scale stabilization, a progressive-resolution causal architecture for encoding and decoding, and a structure-preserving reconstruction loss for training. UniTok-FM adopts an off-the-shelf LLM architecture without TS-specific modifications. Instead of pretraining on isolated TS, it performs NTP on context windows formed by multiple series with similar patterns, aiming to capture their shared dynamics. Experiments on forecasting, generation, and classification show that a single unified UniTok-FM consistently outperforms statistical and supervised baselines, achieves competitive performance with task-specific foundation models, and uniquely enables training-free in-context inference across tasks.

Yunhao Zhang, Ruiying Qi, Jiale Zheng +3

Shanghai Jiao Tong University · Huawei Noah’s Ark Lab

Time Series Tokenization Time Series Generation Zero-Shot Time Series Forecasting Time-Series Foundation Models Time Series Modeling Time Series Language Modeling+1

May 19, 2026

EviTrack: Selection over Sampling for Delayed Disambiguation

Sequential prediction is challenging in regimes of delayed disambiguation, where early observations are ambiguous and multiple latent explanations remain plausible until sufficient evidence accumulates. Standard approaches based on marginal inference struggle in this setting, either collapsing uncertainty prematurely or failing to recover once informative evidence arrives. We introduce EviTrack, a test-time inference framework that operates over latent trajectories rather than marginal states. EviTrack maintains a set of competing trajectory hypotheses and applies evidence- and likelihood-ratio-based selection to delay commitment until supported by data, drawing inspiration from hypothesis management in multiple hypothesis tracking and track-before-detect. To evaluate this setting, we construct a controlled synthetic benchmark with known latent ground truth that explicitly exhibits delayed disambiguation. At matched inference budget, EviTrack substantially outperforms sampling-based baselines, achieving faster post-disambiguation recovery. These results show that, in delayed disambiguation regimes, moderate trajectory-level selection is more effective than increasing sampling coverage, highlighting selection over sampling as a key principle for reliable sequential inference.

Omer Haq

Independent Researcher · Baltimore, MD, USA

Sequence Prediction Sequential Hypothesis Testing Sequence Modeling

May 13, 2026

Conditional Attribute Estimation with Autoregressive Sequence Models

Generative models are often trained with a next-token prediction objective, yet many downstream applications require the ability to estimate or control sequence-level properties. Next-token prediction can lead to overfitting of local patterns during training, underfitting of global structure, and requires significant downstream modifications or expensive sampling to guide or predict the global attributes of generated samples at inference time. Here, we introduce Conditional Attribute Transformers, a novel method for jointly estimating the next-token probability and the value of an attribute conditional on each potential next token selection. This framework enables three critical capabilities within a single forward pass, without modification of the input sequence: (1) per-token credit assignment across an entire sequence, by identifying how each token in a sequence is associated with an attribute's value; (2) counterfactual analysis, by quantifying attribute differences conditional on alternative next token choices; (3) steerable generation, by decoding sequences based on a combination of next-token and attribute likelihoods. Our approach achieves state of the art performance on sparse reward tasks, improves next-token prediction at sufficient model sizes, estimates attribute probabilities orders of magnitude faster than sampling, and can guide decoding of autoregressive sequence models on a range of language tasks.

Erica Stutz, Giacomo Marino, Daniella Meeker +2

Department of Biomedical Informatics and Data Science Yale University New Haven, CT 06510 · Department of Biostatistics Yale University New Haven, CT 06510 · Department of Biomedical Informatics and Data Science, Department of Pediatrics Yale University New Haven, CT 06510

Next-Token Prediction Controllable Text Generation Autoregressive Language Modeling Counterfactual Reasoning in Language Models Autoregressive Generation Conditional Generative Modeling+1

May 8, 2026

The Power of Second Order Methods for Sequence Preconditioning

Sequence prediction methods for linear dynamical systems with long memory, i.e. marginally stable systems, typically achieve regret that grows linearly with the hidden dimension of the underlying generative model. While many methods have been developed to address this regime with varying success, we show that simply using the second-order Vovk-Azoury-Warmuth (VAW) algorithm to learn a short autoregressive-with-inputs (ARX) model achieves astoundingly strong results: for bounded sequential data from a marginally-stable linear dynamical system with spectra in the complex disk except for angular wedge of width

δ

around the negative real axis, this algorithm achieves dimension-free regret

O\left( δ^{-4} \log^2 T \right)

. These bounds are state-of-the-art to our knowledge. The key components for our result come from 1) using the theory of ``Universal Sequence Preconditioning'' (USP) \cite{marsdenuniversal} to prove the existence of an optimal setting of autoregressive coefficients, 2) the application of VAW which takes better advantage of the memory compression provided by USP, and 3) the analysis of Faber polynomials on circular sectors to extend these results to systems with complex spectra.

Annie Marsden, Elad Hazan

Google DeepMind · Princeton University

Dynamical Systems Sequence Modeling Autoregressive Modeling Sequence Prediction

Apr 20, 2026

Sessa: Selective State Space Attention

Modern sequence modeling is dominated by two families: Transformers, whose self-attention can access arbitrary elements of the visible sequence, and structured state-space models, which propagate information through an explicit recurrent state. These mechanisms face different limitations on long contexts: when attention is diffuse, the influence of individual tokens is diluted across the effective support, while recurrent state propagation can lose long-range sensitivity unless information is actively preserved. As a result, both mechanisms face challenges in preserving and selectively retrieving information over long contexts. We propose Sessa, a decoder that places attention inside a recurrent feedback path. This creates many attention-based paths through which past tokens can influence future states, rather than relying on a single attention read or a single recurrent chain. We prove that, under explicit assumptions and matched regimes, Sessa admits power-law memory tails

O(\ell^{-β})

for

0 < β< 1

, with slower decay than in the corresponding Transformer and Mamba-style baselines. We further give an explicit construction that achieves this power-law rate. Under the same assumptions, Sessa is the only model class among those considered that realizes flexible selective retrieval, including profiles whose influence does not decay with distance. Consistent with this theoretical advantage, across matched experiments, Sessa achieves the strongest performance on long-context benchmarks while remaining competitive with Transformer and Mamba-style baselines on short-context language modeling.

Liubomyr Horbatko

State Space Models Long-Context Language Modeling Persistent Memory for Language Models Attention Mechanisms Self-Attention Recurrent Transformers+2

Mar 27, 2026

Stringological sequence prediction I: efficient algorithms for predicting highly repetitive sequences

We propose novel algorithms for sequence prediction based on ideas from stringology. These algorithms are time and space efficient and satisfy mistake bounds related to particular stringological complexity measures of the sequence. In this work (the first in a series) we focus on two such measures: (i) the size of the smallest straight-line program that produces the sequence, and (ii) the number of states in the minimal automaton that can compute any symbol in the sequence when given its position in base k as input. These measures are interesting because multiple rich classes of sequences studied in combinatorics of words (automatic sequences, morphic sequences, characteristic words) have low complexity and hence high predictability in this sense.

Vanessa Kosoy

Faculty of Mathematics, Technion – Israel Institute of Technology, Haifa, Israel 3200003. · Computational Rational Agents Laboratory (coral-research.org), 2810 N Church St PMB 49028, Wilmington, DE 19802.

Sequence Prediction Sequence Modeling

Mar 3, 2026

Thinking into the Future: Latent Lookahead Training for Transformers

Autoregressive language models trained with next-token prediction generate text by sampling one discrete token at a time. Although very scalable, this objective forces the model to commit at every step, preventing it from exploring or reflecting upon multiple plausible continuations. Furthermore, the compute allocation across tokens is uniform; every token is formed based on a single forward-pass, potentially limiting the model's expressiveness in cases where difficult tokens require inherently more compute. Towards addressing these limitations, we introduce latent lookahead, a training strategy that enables models to "think" before generating: at selected positions in the sequence, before committing to the next token, the model performs a multi-step lookahead in latent space. More precisely, instead of sampling future tokens, we leverage the network's latent space by recursively feeding its hidden states back into the context for

τ

steps, investing more compute on predicting that token. This produces

τ

latent predictions that are supervised against the next

τ

ground-truth tokens, encouraging the model to "lookahead" and refine its prediction. We show that latent lookahead substantially outperforms both autoregressive and non-autoregressive baselines on planning tasks such as maze solving, Sudoku, and ProsQA, where foresight is essential.

Lorenzo Noci, Gregor Bachmann, Seyed-Mohsen Moosavi-Dezfooli +1

Apple

Autoregressive Language Modeling Sequence Prediction Autoregressive Transformer LLM Planning Continuous Latent Reasoning

Feb 18, 2026

Online Prediction of Stochastic Sequences with High Probability Regret Bounds

We revisit the classical problem of universal prediction of stochastic sequences with a finite time horizon

T

known to the learner. The question we investigate is whether it is possible to derive vanishing regret bounds that hold with high probability, complementing existing bounds from the literature that hold in expectation. We propose such high-probability bounds which have a very similar form as the prior expectation bounds. For the case of universal prediction of a stochastic process over a countable alphabet, our bound states a convergence rate of

\mathcal{O}(T^{-1/2} δ^{-1/2})

with probability as least

1-δ

compared to prior known in-expectation bounds of the order

\mathcal{O}(T^{-1/2})

. We also propose an impossibility result which proves that it is not possible to improve the exponent of

δ

in a bound of the same form without making additional assumptions.

Matthias Frey, Jonathan H. Manton, Jingge Zhu

Department of Electrical and Electronic Engineering, The University of Melbourne

Sequence Prediction Sequence Modeling

Feb 6, 2026

Quantum Attention by Overlap Interference: Predicting Classical and Many-Body Quantum Sequences

We propose a variational quantum implementation of self-attention (QSA)-the core operation in transformers and large language models-which predicts future elements of a sequence by forming overlap-weighted combinations of past data. At variance with previous approaches, our QSA realizes the required nonlinearity through interference of state overlaps and a degree-

k

polynomial kernel, and estimates a loss based on Rényi-

1/2

entropic functionals via two observables' expectation values, avoiding the decoding of amplitude-encoded predictions into classical probabilities. QSA also accommodates a constrained, trainable data-embedding tying state overlaps to data-level similarities. Its dominant end-to-end training complexity scales as

O\left(μ^{-1}k^2Td\right)

, versus

O\left(T d^{k+1}\right)

of the fairest classical comparison, with

μ

a training signal; we show numerically that this allows a complexity advantage in the regime where sequence length

T

dominates the embedding size

d

. In simulations, our QSA-based quantum transformer learns sequence prediction on classical data and on many-body transverse-field Ising trajectories-establishing trainable attention as a practical primitive for quantum dynamical modeling.

Alessio Pecilli, Matteo Rosati

Dipartimento di Ingegneria Civile, Informatica e delle Tecnologie Aeronautiche, Universit`a degli Studi Roma Tre, Via della Vasca Navale 79, 00146 Rome, Italy

Self-Attention Transformer Attention Variational Quantum Circuits Attention Mechanisms Sequence Prediction Sequence Modeling+1

Oct 31, 2025

Training LLMs Beyond Next Token Prediction - Filling the Mutual Information Gap

Autoregressive language models are usually trained with next-token prediction under a fixed left-to-right target order. This paper studies target-token organization as a training-time design choice across arithmetic reasoning, multi-label text classification, and text generation. Controlled order studies show that the default NTP order can be far from the best-performing tested organization, while fixed hand-crafted orders such as reverse order do not generalize across tasks. To address this, Max(MI(S;t)) is introduced as a deterministic mutual-information-based heuristic that prioritizes information-rich target tokens. For fixed-length outputs, it constructs task-specific prediction orders; for open-vocabulary text generation, the same principle is applied through lightweight prefix augmentation. Experiments with GPT-style models and pretrained Llama/Qwen variants show that Max(MI(S;t)) often improves over standard NTP and simple baselines such as reverse order, TF-IDF, and conditional-probability selection, without changing model architecture or autoregressive decoding.

Chun-Hao Yang, Bo-Han Feng, Tzu-Yuan Lai +3

National Taiwan University

Autoregressive Language Modeling Sequence Prediction Autoregressive Transformer