Tracking Adaptation Time: Metrics for Temporal Distribution Shift

Lorenzo Iovine; Giacomo Ziffer; Emanuele Della Valle

cs.LGApr 8, 2026

Tracking Adaptation Time: Metrics for Temporal Distribution Shift

Authors: Lorenzo Iovine, Giacomo Ziffer, Emanuele Della Valle

Organizations: Politecnico di Milano, DEIB, Via Giuseppe Ponzio, 34, 20133 Milan, Italy

Abstract

Evaluating robustness under temporal distribution shift remains an open challenge. Existing metrics quantify the average decline in performance, but fail to capture how models adapt to evolving data. As a result, temporal degradation is often misinterpreted: when accuracy declines, it is unclear whether the model is failing to adapt or whether the data itself has become inherently more challenging to learn. In this work, we propose three complementary metrics to distinguish adaptation from intrinsic difficulty in the data. Together, these metrics provide a dynamic and interpretable view of model behavior under temporal distribution shift. Results show that our metrics uncover adaptation patterns hidden by existing analysis, offering a richer understanding of temporal robustness in evolving environments.

Figures & tables

Panel of Figure 1 — Figure 1: UMAP 2D projections of Yearbook feature embeddings across consecutive decades. Blue and red points correspond to different classes (e.g., male vs. female). Over time, the class clusters — initially well separated — become increasingly overlapping, reflecting higher intrinsic data difficulty and complex temporal changes. This visual evidence supports our hypothesis that performance degradation is not solely due to limited model generalization, but also to evolving data complexity.

Panel of Figure 2 — Figure 2: Temporal performance of the Fine-Tuning model on FMoW. Each row corresponds to a model trained at year $t$ and evaluated on future years $\tau>t$ . (a) $A(t,\tau)$ : darker cells indicate higher accuracy ; grey cells mark non-evaluated pairs. (b) $g(t,\tau)$ : the colormap is centered at the tolerance threshold $\delta=0.6$ ; white denotes $g(t,\tau)=0.6$ exactly, blue values $>\!0.6$ indicate smaller deviation from the oracle ( $g\!\approx\!1$ ), and red values $<\!0.6$ indicate larger deviation. Crossing below $\delta$ marks the end of the Stability Horizon for the model trained at $t$ .

Model	ID	OOD	TAS	SH	DH
MoCo	32.8%	21.3%	63.4%	2.9 y	6.1 y
SI	30.8%	26.8%	80.1%	4.8 y	6.0 y
CORAL	30.9%	24.8%	74.2%	4.9 y	6.5 y
ERM	31.9%	25.7%	74.6%	4.5 y	6.7 y
A-GEM	16.2%	6.9%	42.9%	1.7 y	6.3 y
FT	32.5%	27.1%	75.9%	5.1 y	6.4 y

Table 1: Comparison of models on the FMoW dataset. While ID and OOD accuracies reflect overall performance, TAS, SH, and DH reveal complementary aspects of temporal adaptation and stability.

Model	ID Avg	OOD Avg	OOD Min	TAS Avg	TAS Min
MoCo	95.0%	92.8%	76.8%	97.8%	89.3%
SI	93.8%	90.9%	73.3%	96.2%	75.8%
CORAL	91.0%	87.5%	61.3%	96.3%	82.5%
EWC	95.2%	92.6%	70.3%	96.4%	72.9%
ERM	94.7%	91.8%	70.3%	96.8%	73.1%
A-GEM	90.4%	87.2%	56.5%	96.2%	61.0%

Table 2: Comparison of models on the Yearbook dataset (1930–2013). While ID and OOD averages indicate overall robustness, TAS highlights differences in temporal adaptability, particularly under distributional changes.

Figure 3: ID, OOD, and TAS trends for the MoCo+SML model on Yearbook (1930–2013). Around 1970, a large apparent ID–OOD gap (99% vs. 77%) corresponds to a TAS of about 90%, indicating that the performance drop is largely due to intrinsic data difficulty rather than failure to adapt.

Distribution Shift Robustness

Distribution shift robustness studies how to keep machine-learning predictions reliable when deployment data differ from training data.

See all 280 papers

Drift Happens: An Empirical Study of Neural Architecture Robustness to Temporal Distribution Shift
Jul 7, 2026Robin Holzinger, Riccardo CollettiDistribution Shift Robustness Neural Network Generalization
Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, USA
Adversarial Vulnerability Under Temporal Concept Drift: A Longitudinal Study of Android Malware Detection
May 22, 2026Ahmed Sabbah, Mohammed Kharma, Radi Jarrar +2Distribution Shift Robustness Concept Drift
aDepartment of Computer Science, Birzeit University, P.O. Box 14, Birzeit, P627, West2026 Bank, Palestine · bDepartment of Computer Science, University of Central Florida, 4328 Scorpius St., Building 116, Room 246, Orlando, 32816, FL, USAMay 22
When Test-Time Adaptation Helps, Harms, or Becomes Inactive: A Condition-Level Study on CIFAR-10-C
Aug 23, 2026Sreeja Guha Majumdar, Aratrika SahaTest-Time Adaptation Corruption Robustness
Heritage Institute of Technology Kolkata, West Bengal, India

Jul 7, 2026J/K move · Enter open · S save

Drift Happens: An Empirical Study of Neural Architecture Robustness to Temporal Distribution Shift

Robin Holzinger, Riccardo Colletti

Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, USA

Distribution Shift Robustness Neural Network Generalization Neural Network Robustness Temporal Distribution Shift

Open paper AskPDF arXiv

Real-world data distributions evolve over time, inducing temporal distribution shift that can substantially degrade the reliability of deployed machine learning systems. However, the extent to which architectural choices and their associated inductive biases affect temporal robustness remains insufficiently understood. We present a systematic empirical comparison of temporal robustness across three heterogeneous, time-indexed domains encompassing image classification, multi-label text classification, and text regression tasks. Using a unified evaluation framework based on temporal drift matrices, we train models on cumulative historical data and evaluate their performance on both earlier and later time periods, thereby quantifying cross-temporal generalization. Our study spans model families ranging from simple multilayer perceptrons and convolutional networks to recurrent networks and pretrained Transformer-based encoders. Collectively, the results show that architectural inductive biases systematically shape temporal robustness: models whose inductive biases lead them to exploit localized, highly discriminative features attain the highest in-distribution accuracy, yet those features are often the ones that change most over time, so these models degrade fastest, while pretrained encoders that draw on coarser, more stable representations drift more gradually. These observations offer practical guidance for selecting architectures for real-world systems subject to temporal drift.

Abstract

Distribution Shift Robustness

Explore similar work

Drift Happens: An Empirical Study of Neural Architecture Robustness to Temporal Distribution Shift

Adversarial Vulnerability Under Temporal Concept Drift: A Longitudinal Study of Android Malware Detection

When Test-Time Adaptation Helps, Harms, or Becomes Inactive: A Condition-Level Study on CIFAR-10-C