stat.MLMar 20, 2026

A two-step sequential approach for hyperparameter selection in finite context models

Authors: José Contente, Ana Martins, Armando J. Pinho, Sónia Gouveia

Organizations: Institute of Electronics and Informatics Engineering of Aveiro (IEETA), Department of Electronics, Telecommunications and Informatics (DETI), University of Aveiro (UA), Aveiro, Portugal · Intelligent Systems Associate Laboratory (LASI), Portugal

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Finite-context models (FCMs) are widely used for compressing symbolic sequences such as DNA, where predictive performance depends critically on the context length k and smoothing parameter α. In practice, these hyperparameters are typically selected through exhaustive search, which is computationally expensive and scales poorly with model complexity. This paper proposes a statistically grounded two-step sequential approach for efficient hyperparameter selection in FCMs. The key idea is to decompose the joint optimization problem into two independent stages. First, the context length k is estimated using categorical serial dependence measures, including Cramér's ν, Cohen's \k{appa} and partial mutual information (pami). Second, the smoothing parameter α is estimated via maximum likelihood conditional on the selected context length k. Simulation experiments were conducted on synthetic symbolic sequences generated by FCMs across multiple (k, α) configurations, considering a four-letter alphabet and different sample sizes. Results show that the dependence measures are substantially more sensitive to variations in k than in α, supporting the sequential estimation strategy. As expected, the accuracy of the hyperparameter estimation improves with increasing sample size. Furthermore, the proposed method achieves compression performance comparable to exhaustive grid search in terms of average bitrate (bits per symbol), while substantially reducing computational cost. Overall, the results on simulated data show that the proposed sequential approach is a practical and computationally efficient alternative to exhaustive hyperparameter tuning in FCMs.

Figures & tables

Figure 1: Illustration of the usage of a finite context model in a compression task, showing how the probability of the next outcome, yt+1y_{t+1} , is conditioned by the last kk outcomes ( k=5k=5 , in this example). Adapted from [ 10 ] .
Figure 2: Outline of the two-step sequential strategy for hyperparameter context ( k∗k^{*} ) and smoothing factor ( α∗\alpha^{*} ) selection.
Figure 3: Boxplots of the distribution of pami for synthetic time series of length 100,000, generated with k∈{3,8}k\in\{3,8\} and α∈{0,0.1,0.5,0.8,1}\alpha\in\{0,0.1,0.5,0.8,1\} .
Figure 4: Boxplots of the distribution of Cramér’s ν\nu (blue) and Cohen’s κ\kappa (red) for the synthetic time series of length 100,000, with k=3k=3 and α∈{0.1,0.5,0.8}\alpha\in\{0.1,0.5,0.8\} .
Figure 5: Partial auto mutual information for two data sequences with T=100,000T=100,000 . Maximum value highlighted in red.
Figure 6: Confusion matrix comparing the context kk used to generate the synthetic sequences and the optimal context k∗k^{*} obtained using the pami-based selection rule for different values of TT .
Figure 7: Dispersion plots of the estimated smoothing factor conditioned to the optimal context (α∗∣k∗)(\alpha^{*}|k^{*}) , and conditioned to the data-generating context (α∗∣k)(\alpha^{*}|k) , against the smoothing factor (α)(\alpha) for different sample sizes TT . Red line is r=1r=1 .
StatisticT=100,000T=100,000T=10,000T=10,000T=1,000T=1,000
r(α∗∣k∗,α)r(\alpha^{*}|k^{*},\alpha)0.320.080.07
r(α∗∣k,α)r(\alpha^{*}|k,\alpha)0.930.370.07
Bias α∗∣k∗\alpha^{*}|k^{*}1.067.6×1097.6\times 10^{9}1.44×10111.44\times 10^{11}
Bias α∗∣k\alpha^{*}|k0.010.062.1×10102.1\times 10^{10}
% (α∗∣k∗)>1(\alpha^{*}|k^{*})>122.541.552.3
% (α∗∣k)>1(\alpha^{*}|k)>14.38.426.1
Table 1: Summary statistics for the estimation procedure of the smoothing factor conditioned to the optimal context (α∗∣k∗)(\alpha^{*}|k^{*}) , and conditioned to the data-generating context (α∗∣k)(\alpha^{*}|k) for different sample sizes TT .
Figure 8: Dispersion diagrams of bpsbps versus the estimated bpsbps . (a) bps∗bps^{*} obtained when compressing with optimal (k∗,α∗)(k^{*},\alpha^{*}) using the proposed approach, distinguishing k∗=kk^{*}=k (blue) or k∗≠kk^{*}\neq k (black). (b) bpsgsbps_{gs} obtained when compressing with the optimal (k,α)(k,\alpha) found via grid search. Red line is r=1r=1 .
More in this topic

Hyperparameter Optimization

Hyperparameter optimization selects model and training settings to improve predictive performance, efficiency, or reliability under limited evaluation budgets.

See all 161 papers

Explore similar work

CardsList
Sep 10, 2026

FlexComp: One Model for Every Ratio in Context Compression

Soft context compression condenses a context into a few memory tokens that a frozen LLM consumes in place of the raw text, but existing compressors fix the compression ratio at training and inference: each deployed ratio requires a separately trained model, and the chosen ratio is applied uniformly to all inputs, whose actual needs vary drastically. We propose FlexComp, a method-agnostic framework that decouples the ratio from both training and deployment: Matryoshka-style training samples the memory budget KK per instance, turning one model into an any-ratio compressor, and the budget is then chosen per input by: (1) confidence-based cascade routing or (2) a lightweight learned KK predictor. Across ICAE, 500xCompressor, and SAC on MRQA, a single FlexComp model matches separately trained fixed-ratio specialists with minimal degradation. Cascade routing preserves over 98% of the mildest ratio's accuracy at up to 266x average compression; the KK predictor, in a single compression-decoding pass, reaches 158-236x within 0.7 F1 of the mildest ratio. At serving-scale batch sizes, the KK predictor cuts context KV cache by 50% and improves decoding throughput by 47%.

Kaiyan Zhao, Zhongtao Miao, Akiko Aizawa +1

The University of Tokyo · National Institute of Informatics

LLM Inference AccelerationContext CompressionLLM Context CompressionToken Compression
AskPDF
Mar 26, 2026

Density-aware Soft Context Compression with Semi-Dynamic Compression Ratio

Soft context compression reduces the computational workload of processing long contexts in LLMs by encoding long context into a smaller number of latent tokens. However, existing frameworks apply uniform compression ratios, failing to account for the extreme variance in natural language information density. While adopting a density-aware dynamic compression ratio seems intuitive, empirical investigations reveal that models struggle intrinsically with operations parameterized by input dependent, continuous structural hyperparameters. To resolve this pitfall, we introduce Semi-Dynamic Context Compression framework. Our approach features a Discrete Ratio Selector, which predicts a compression target based on intrinsic information density and quantizes it to a predefined set of discrete compression ratios. It is efficiently jointly trained with the compressor on synthetic data, with the summary lengths as a proxy to create labels for compression ratio prediction. Extensive evaluations confirm that our density-aware framework, utilizing mean pooling as the backbone, consistently outperforms static baselines, establishing a robust Pareto frontier for context compression techniques. Our code, data and model weights are available at https://github.com/yuyijiong/semi-dynamic-context-compress

Yijiong Yu, Shuai Yuan, Jie Zheng +2

Oregon State University · DeepSolution

LLM Context CompressionContext CompressionLLM CompressionLong-Context Language Modeling
AskPDF
Sep 28, 2026

Fractional State Space Transition for Long Sequence Modeling

State Space Models (SSMs) compress sequence history into a bounded recurrent state, making the resulting memory law a central architectural choice for long-context performance. Most modern SSMs rely on ODE-based dynamics that lead to exponential forgetting, limiting their ability to retain information over broad temporal ranges. We introduce FRAC, a selective SSM architecture derived from fractional dynamics that replaces this exponential decay with power-law long memory. To make fractional dynamics practical, FRAC approximates the heavy-tailed target kernel with a finite-state, log-spaced sum of exponential modes. This construction turns fractional memory into an efficient recurrent module with parallel training and prefill, while retaining bounded-state autoregressive decoding. Extensive experiments, including 1.3B-parameter language modeling, demonstrate that FRAC consistently improves long-context performance over state-of-the-art SSM baselines while staying competitive on short-context. These results show that fractional dynamics provide a practical and effective prior for long-context SSMs.

Ivan Kobyzev, Abbas Ghaddar, Ali Nasiri-Sarvi +2

Huawei Noah’s Ark Lab, Montreal Research Center, Canada

State Space Language ModelsState Space ModelsLong-Context Language ModelingSelective SSMs
AskPDF