cs.LGOct 12, 2026

Compression Through Time: An Interplay of Separability, Stationarity and Capacity

Authors: Nikolaus Kopp, Franz Pernkopf, Bernhard Geiger

Organizations: Graz University of Technology, Graz, Austria · Graz University of Technology & Know Center Research GmbH, Graz, Austria

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Information-theoretic and geometric compression have been widely studied as possible explanations for generalization. Yet their temporal dynamics in recurrent networks remain largely unexplored. Using discretized LSTMs, whose finite embedding space enables direct estimates of information quantities, we observe pronounced compression across the final timesteps of many datasets. We explain this effect through three competing mechanisms: (i) class separability permits stronger compression at the final timestep; (ii) shared recurrent dynamics propagate this pressure across timesteps with similar activation states, making its extent depend on temporal stationarity; and (iii) limited representational capacity opposes prolonged compression because it restricts information available for subsequent computation. We systematically test these mechanisms through controlled synthetic interventions and real-world sequence tasks. Our results identify class separability, temporal stationarity, and network capacity as key determinants of temporal compression, and suggest that compression acts as a regularizer-potentially beneficial when excess capacity is available, but harmful to generalization when it consumes memory needed for the task.

Figures & tables

Fig. 1 : Embedding entropy and target information of last recurrent layer of an LSTM as it evolves over time. Each trajectory corresponds to a training trial. Left: Dataset MNIST stroke . Right: Dataset Tiselac
Fig. 2 : Empirically estimated embedding entropy I^(X;H)\hat{I}(\mathbf{X};\mathbf{H}) after directly optimizing pre-embeddings to minimize softmax loss. Entropy is plotted over the separation cost λsep\lambda_{sep} , which penalizes separating pre-embeddings which are initially close. The resulting classifier accuracies are shown for λsep∈{0,0.1,3.0}\lambda_{sep}\in\{0,0.1,3.0\} .
Fig. 3 : Embedding entropy I^(X;Ht)\hat{I}(\mathbf{X};\mathbf{H}^{t}) and target information I^(Ht;Y)\hat{I}(\mathbf{H}^{t};Y) over timesteps tt . LSTM networks are trained and evaluated on the synthetic AR dataset. Each experiment comprises five trials, shown in different colors. Test set accuracies are displayed above the subplots. Left: No EOS token ( tEOS=0t_{\mathrm{EOS}}=0 ). Middle: EOS token only at final timestep ( tEOS=1t_{\mathrm{EOS}}=1 ). Right: EOS token for final 10 timesteps ( tEOS=10t_{\mathrm{EOS}}=10 ).
tEOSt_{\mathrm{EOS}}I^(X;H140)\hat{I}(\mathbf{X};\mathbf{H}^{140})I^(X;H150)\hat{I}(\mathbf{X};\mathbf{H}^{150})ΔI^(X;Ht)\Delta\hat{I}(\mathbf{X};\mathbf{H}^{t})Convergence epoch
003.5±0.53.5\pm 0.53.5±0.53.5\pm 0.5−0.06±0.01-0.06\pm 0.0197±4997\pm 49
115.2±0.75.2\pm 0.73.4±0.43.4\pm 0.4−1.81±0.65-1.81\pm 0.6553±1453\pm 14
10105.3±0.55.3\pm 0.52.5±0.42.5\pm 0.4−2.87±0.40-2.87\pm 0.4059±2259\pm 22
TABLE I : Embedding entropy I^(X;Ht)\hat{I}(\mathbf{X};\mathbf{H}^{t}) at t∈[140,150]t\in[140,150] and their difference. The epoch at which training accuracy surpassed 9898 % of it’s maximum value, is also given. Mean and standard deviation are computed over 5 trials.
Fig. 4 : Information content behavior of a network trained and tested on AR dataset with alternating information. Left Embedding entropy I^(X;Ht)\hat{I}(\mathbf{X};\mathbf{H}^{t}) and target information I^(Ht;Y)\hat{I}(\mathbf{H}^{t};Y) over timesteps tt when training on informative end and no EOS signal. Shown are the first 5 trials, with their average displayed in black. Middle: Same, but trained on uninformative end . Right: Change in information content between the beginning and end of each phase, excluding the first two and final phases of each trial. Error bars indicate the mean and standard deviation for each color–symbol combination.
DatasetEOS1 Layer2 Layers
MNIST stroke×\times90.69±0.5890.69\pm 0.5893.99±0.4293.99\pm 0.42
✓\checkmark91.12±0.6491.12\pm 0.6494.07±0.3094.07\pm 0.30
Tiselac×\times82.27±0.8682.27\pm 0.8682.11±0.8782.11\pm 0.87
✓\checkmark81.11±1.2981.11\pm 1.2982.11±0.8982.11\pm 0.89
ElectricDevices×\times60.83±2.8760.83\pm 2.8765.38±2.1265.38\pm 2.12
✓\checkmark61.43±3.1561.43\pm 3.1566.99±2.4566.99\pm 2.45
TABLE II : Generalization performance of LSTM networks across four experimental setups for each dataset, combining the presence or absence of an EOS signal with either one or two LSTM layers. Shown are the mean and standard deviation of the accuracy over the best-performing 90% of trials (27 of 30).
Dataset1 Layer2 Layers
 rrpp -valuerrpp -value
MNIST stroke0.447.8×10−4\mathbf{7.8\times 10^{-4}}0.152.6×10−12.6\times 10^{-1}
Tiselac0.456.9×10−4\mathbf{6.9\times 10^{-4}}0.455.1×10−4\mathbf{5.1\times 10^{-4}}
ElectricDevices0.038.3×10−18.3\times 10^{-1}0.312.3×10−2\mathbf{2.3\times 10^{-2}}
AR short0.393.8×10−3\mathbf{3.8\times 10^{-3}}0.075.9×10−15.9\times 10^{-1}
TABLE III : Pearson correlations between test accuracy and time-averaged embedding entropy 1/T∑t=1TI^(X,Ht)1/T\sum_{t=1}^{T}\hat{I}(\mathbf{X},\mathbf{H}^{t}) , computed across all 54 of 60 trials, irrespective of whether an EOS signal was present. pp -values <0.05<0.05 are shown in bold.
Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 5 : Illustration of the straight-through estimator. The elliptical contour plot shows the level sets of the loss, and the arrows display the negative gradient field. (a) shows the updates when no quantizer is used, i.e., Equation ( 2 ). (b) displays the STE updates given by Equation 5 , using the derivative of the identity function as the surrogate ρ(⋅)\rho(\cdot) . Note that the update is fully determined by the gradient at the quantized values, namely the vertices of the square. Even though the minimum lies in the first quadrant, updates in the second quadrant may flip h2=sign⁡(h~2)h_{2}=\operatorname{sign}(\tilde{h}_{2}) , while updates in the fourth quadrant may flip h1=sign⁡(h~1)h_{1}=\operatorname{sign}(\tilde{h}_{1}) . In either case, such flips can increase distance to the loss-minimizing quadrant. Thus, STE updates are not guaranteed to eventually reach the loss-minimizing vertex h=(1,1)\mathbf{h}=(1,1) . (c) shows the STE updates when using the derivative of tanh⁡\operatorname{tanh} as the surrogate ρ(⋅)\rho(\cdot) . Note that vectors near the quantization boundaries are pulled towards them, as the derivative of tanh⁡\operatorname{tanh} is largest near 00 . The vector field can be transformed into that of (b) by applying a strictly monotonic transformation to each coordinate. (d) shows an example where even though the loss is convex and STE moves us locally closer to the loss-minimizing orthant, we are not guaranteed to get closer with discrete steps.
Fig. 6 : Visualizations from the MNIST stroke dataset. The digit plots are generated by drawing the strokes as instructed by the sequence of pen moves. (a) shows the histogram of stroke counts in the training set. (b) shows randomly selected digits from sequences containing five strokes. (c) shows randomly selected digits from sequences containing three or fewer strokes, while (d) shows the corresponding digits generated from the resampled sequences.
Fig. 7 : Illustration of the stroke resampling procedure. The blue curve shows the stroke after rescaling and smoothing. Target points are placed equidistantly with respect to the Euclidean arc length of the blue curve and are generally located between its elements by linear interpolation. The orange curve is the resulting discrete stroke, where each movement is chosen to approach the next target point as closely as possible subject to the discrete movement constraints. The original digit had a longer sequence length, and therefore the associated blue curve is downscaled by the factor rsr_{s} before resampling, resulting in the shorter steps taken by blue compared to orange trajectory.
Fig. 8 : Sequences λι[t]\lambda_{\iota}[t] used to generate the AR datasets, controlling the informativeness of the input sequences x\mathbf{x} . For visualization, a vertical offset was added to each sequence; in reality, λι[t]∈0,1\lambda_{\iota}[t]\in{0,1} . Left: Examples of λι[t]\lambda_{\iota}[t] used to generate training sequences in the informative end mode. Right: The λι[t]\lambda_{\iota}[t] sequence used for all test sequences, together with the λι[t]\lambda_{\iota}[t] sequence used during the early epochs of curriculum learning.
Fig. 9 : Information content behavior of networks trained and tested on AR dataset with alternating measurement noise . Shown is embedding entropy I^(X;Ht)\hat{I}(\mathbf{X};\mathbf{H}^{t}) and target information I^(Ht;Y)\hat{I}(\mathbf{H}^{t};Y) over timesteps tt for all ten trials. The average embedding entropy and target information, aggregated over trials for each timestep tt , is shown in black. Time intervals with λι(t)=0\lambda_{\iota}(t)=0 in the test sequences are shown in red, while those with λι(t)=1\lambda_{\iota}(t)=1 are shown in green. Top: No EOS information present during training and testing ( tEOS=0t_{\mathrm{EOS}}=0 ). Bottom: EOS information is present during training and testing with tEOSt_{\mathrm{EOS}} . Left: Networks are trained on informative end . Right: Networks are trained on uninformative end .
Fig. 10 : Information content behavior of networks trained and tested on AR dataset with alternating information noise . Shown is embedding entropy I^(X;Ht)\hat{I}(\mathbf{X};\mathbf{H}^{t}) and target information I^(Ht;Y)\hat{I}(\mathbf{H}^{t};Y) over timesteps tt for all ten trials. The average embedding entropy and target information, aggregated over trials for each timestep tt , is shown in black. Time intervals with λι(t)=0\lambda_{\iota}(t)=0 in the test sequences are shown in red, while those with λι(t)=1\lambda_{\iota}(t)=1 are shown in green. Top: No EOS information present during training and testing ( tEOS=0t_{\mathrm{EOS}}=0 ). Bottom: EOS information is present during training and testing with tEOSt_{\mathrm{EOS}} . Left: Networks are trained on informative end . Right: Networks are trained on uninformative end .
Fig. 11 : Effect of training on informative end compared to uninformative end on the information content dynamics, with and without an EOS token. Shown is the change in information content between the beginning and end of each phase, excluding the first two and final phases of each trial. Error bars indicate the mean and standard deviation for each color–symbol combination. Left: Uninformative phases during training and testing were generated using measurement noise . Right: Same, except that information noise was used for uninformative phases.
DatasetEOS1 Layer2 Layers
MNIST stroke×\times6.00±0.246.00\pm 0.246.08±0.246.08\pm 0.24
✓\checkmark6.21±0.316.21\pm 0.316.39±0.206.39\pm 0.20
Tiselac×\times6.45±0.276.45\pm 0.275.81±0.295.81\pm 0.29
✓\checkmark6.35±0.326.35\pm 0.325.97±0.415.97\pm 0.41
ElectricDevices×\times5.15±0.445.15\pm 0.445.22±0.275.22\pm 0.27
✓\checkmark4.93±0.484.93\pm 0.485.38±0.355.38\pm 0.35
Appendix
TABLE IV : Average embedding entropy, aggregated over timesteps, of LSTM networks across four experimental setups for each dataset, combining the presence or absence of an EOS signal with either one or two LSTM layers. Shown are the mean and standard deviation of the entropy in bits, over the best-performing 90% of trials (27 of 30).
Dataset1 Layer2 Layers
 tt -statisticpp -valuett -statisticpp -value
MNIST stroke2.71.1×10−2\mathbf{1.1\times 10^{-2}}5.23.5×10−6\mathbf{3.5\times 10^{-6}}
Tiselac-1.22.2×10−12.2\times 10^{-1}1.71.1×10−11.1\times 10^{-1}
ElectricDevices-1.88.0×10−28.0\times 10^{-2}1.96.5×10−26.5\times 10^{-2}
AR short4.01.8×10−4\mathbf{1.8\times 10^{-4}}9.75.3×10−13\mathbf{5.3\times 10^{-13}}
Appendix
TABLE V : Statistical results of Welch’s tt -test assessing the effect of EOS signal presence on the average embedding entropy reported in Table IV . A positive tt -statistic means the presence of EOS leads to higher average embedding entropy.
Dataset1 Layer2 Layers
 tt -statisticpp -valuett -statisticpp -value
MNIST stroke2.51.4×10−2\mathbf{1.4\times 10^{-2}}0.74.7×10−14.7\times 10^{-1}
Tiselac-3.84.3×10−4\mathbf{4.3\times 10^{-4}}0.09.9×10−19.9\times 10^{-1}
ElectricDevices0.74.8×10−14.8\times 10^{-1}2.51.4×10−2\mathbf{1.4\times 10^{-2}}
AR short5.24.1×10−6\mathbf{4.1\times 10^{-6}}2.05.1×10−25.1\times 10^{-2}
Appendix
TABLE VI : Statistical results of Welch’s tt -test assessing the effect of EOS signal presence on the accuracies reported in Table II . pp -values <0.05<0.05 are shown in bold. A positive tt -statistic means the presence of EOS leads to higher accuracy.
Fig. 12 : Embedding entropy I^(X;Ht)\hat{I}(\mathbf{X};\mathbf{H}^{t}) and target information I^(Ht;Y)\hat{I}(\mathbf{H}^{t};Y) over timesteps tt for different datasets, showing all 30 independently trained LSTMs per dataset, each with a single recurrent layer. The black lines show the respective averages across trials at each timestep. The dataset is indicated by the label on the left. Left: Without EOS signal, Right: With EOS signal ( tEOS=1t_{\mathrm{EOS}}=1 ).
Fig. 13 : Embedding entropy I^(X;Ht)\hat{I}(\mathbf{X};\mathbf{H}^{t}) and target information I^(Ht;Y)\hat{I}(\mathbf{H}^{t};Y) over timesteps tt for different datasets, showing all 30 independently trained LSTMs per dataset, each with two recurrent layers. The black lines show the respective averages across trials at each timestep. The dataset is indicated by the label on the left. Left: Without EOS signal, Right: With EOS signal ( tEOS=1t_{\mathrm{EOS}}=1 ).
Fig. 14 : Accuracies of all 30 independently trained LSTMs per dataset with a single recurrent layer, using post-training early stopping. Top left: MNIST stroke . Top right: Tiselac . Bottom left: ElectricDevices . Bottom right: AR short .
Fig. 15 : Accuracies of all 30 independently trained LSTMs per dataset with two recurrent layers, using post-training early stopping. Top left: MNIST stroke . Top right: Tiselac . Bottom left: ElectricDevices . Bottom right: AR short .
More in this topic

Neural Network Generalization

Neural network generalization studies why trained models perform well on unseen data and how architecture, data, and training shape that ability.

See all 302 papers

Explore similar work

CardsList

  1. Dynamic Compression in Recurrent Networks

    Aug 18, 2026Jyothish Pari, Ryan Bahlous-Boldi, Pulkit AgrawalRecurrent Neural NetworksModel Compression

    Improbable AI Lab, Massachusetts Institute of Technology

  2. What to Preserve in Recursive Computation: A Local Predictive Sufficiency Principle

    Oct 3, 2026Peilin Wang, Feng Shiyang, Hongfu Gao +4Model Compression

    Shandong University · National University of Singapore · Tsinghua University

  3. Learning is Forgetting: LLM Training As Lossy Compression

    Apr 8, 2026Henry C. Conklin, Tom Hosking, Tan Yi-Chern +5Language Model PretrainingLLM Compression

    Princeton University · Cohere