Organizations: University of Sydney, Sydney, Australia · Macquarie University, Sydney, Australia · Department of Informatics, University of Oslo, Oslo, Norway · University of Southern Queensland, Toowoomba, Australia · University of Technology Sydney, Sydney, Australia
Tabular foundation models (TFMs) are being applied to Web data streams such as network intrusion detection, where they adapt by updating the labelled rows of their context without retraining. When a past concept returns after leaving the context, a TFM faces a recall-or-relearn decision between restoring stored rows of the concept and relearning it from new labels. We propose MICE (Mixture of In-Context Experts), a training-free learner that cuts the stream into segments at detected changes. It merges a closed segment into a stored context when the TFM's estimated additional error on the segment is within a tolerance, under a capacity limit on stored contexts. A fast level weights the stored contexts and first-in-first-out (FIFO) windows by the log-loss of their predictions on the last labelled batch. We show that, with pure stored contexts and a margin condition, this fast level gains over relearning, in expected accuracy, the normalised positive area between the stored context's accuracy and the TFM's learning curve, up to a first-batch term. A slow level mixes the fast level's prediction with that of the FIFO window of the last labelled rows and bounds the discounted half-Brier regret of MICE against this window at every time. Learning curves measured beforehand predict the ordering of the observed recall gains. MICE outperforms reset, which empties the context at detected drifts, on grid streams with random boundary phases. Merging keeps fewer experts than pools that store every segment, and with fixed segmentation a time-matched snapshot pool matches MICE on the shifted grid. On real streams, MICE has the highest mean accuracy of all policies we compare.
Figures & tables
Figure 1. A frozen TFM filters spam from recent labelled mail, its context. Red flags mark detected changes. When campaign A returns, the FIFO window holds only B and predicts ham. MICE recalls A’s stored context and classifies A’s mail as spam. Here, recall avoids relearning the hard campaign A. Both methods learn the new, easy campaign C quickly. Schematic. The schematic mail timeline shows campaign A returning after B. A new campaign C follows the returning A. Red flags mark detected changes. Below the timeline, three rows compare the FIFO window with MICE. In an ordinary week both classify A's mail as spam. When A returns, the FIFO window holds only B's mail and predicts ham. MICE recalls a stored context of A and predicts spam. Both methods learn the new campaign C fast and predict spam.
Figure 2. Three aspects of the recall-or-relearn decision. A and B are two concepts of one stream. (a) When A returns, the FIFO window has discarded A’s rows and the TFM predicts A’s batch with B’s rule. (b) A context that straddles an unknown boundary mixes conflicting rules. (c) What recall is worth: the TFM relearns the easy concept from a few rows but the hard one only from many, and the red rows mark the relearning that recall avoids. Schematic. Fig. 3 measures each aspect. Under the purity and margin conditions of Thm. 4.1 , the fast level’s expected accuracy gain over relearning equals the normalised positive area in (c), up to a first-batch term. Panel (a) shows a FIFO window discarding the rows of a concept that later returns. Panel (b) contrasts the predictions from a pure and a mixed context. Panel (c) compares the learning curves of an easy and a hard concept with stored-context accuracy, with the recall area shaded for the hard concept.
Figure 3. Context stress test. (1) Accuracy in the first five batches after a recurrence: FIFO (gray bars) and the oracle-context lower bound of 99% (blue bars), with the reset range of 81–88% shaded. (2) Gain of stored contexts over reset with aligned, shifted and realigned segments, over all batches (blue) and the first five batches after a recurrence (dashed orange), with single streams as pale traces. (3) Learning curves of concepts with 5, 30 and 100 centroids. Three connected diagnostic plots: paired horizontal bars compare FIFO and oracle contexts with a reset reference band, phase curves show loss and recovery of gains as segment alignment changes, and learning curves show accuracy versus context size for three concept complexities.
Figure 4. Overview of MICE (schematic, bottom row illustrative). (1) Segments cut at detected changes join the stored context of lowest estimated recall regret within a tolerance. (2) The fast level weights stored-context and FIFO predictions from one frozen TFM by their log-loss on the last labelled batch. Under pure contexts and the margins of Thm. 4.1 , its expected accuracy gain over relearning equals the area term plus the first-batch term ζ . (3) The slow level mixes the fast-level and FIFO predictions. Its discounted half-Brier regret against FIFO is at most 2ln2 at every time. Three columns, one per stage, are linked by arrows from memory to fast level to slow level. Column 1 (memory): a stream A, A, B, B, A with red flags at detected changes is cut into segments. The last segment joins the stored context AAA, which has the lowest recall regret, and not BB. Its middle panel states the aim of one concept per context: a pure context AAAA is accepted and a mixed context AABB is rejected, and merges happen only within a tolerance. Column 2 (fast level): stored contexts and FIFO windows condition one frozen TFM and receive log-loss weights. Its middle panel plots accuracy against batch for a stored context and for relearning. The gain is the shaded area between the two curves plus a hatched first-batch term zeta. Column 3 (slow level): a discounted half-Brier mix between FIFO and the fast level gives the prediction. Its middle panel shows discounted regret against FIFO below 2 ln 2 at every time. An illustrative bottom row reads, for each stage when A recurs: the segment joins context A, context A takes the weight, and the mix shifts to the fast level.
Real streams (29)
Controlled grid (24)
Gen. (30)
Indep. (9)
Policy
Acc. ↑
F1 ↑
AUC ↑
ECE ↓
Brier ↓
Worst Δ↑
Acc. ↑
F1 ↑
AUC ↑
ECE ↓
Brier ↓
Acc. ↑
Acc. ↑
Trained stream learners
ARF ( Gomes et al., 2017 )
73.4
58.4
85.4
6.0
18.7
− 37.5
67.7
65.6
83.8
11.2
22.9
88.0
98.39
SRP ( Gomes et al., 2019 )
76.6
62.3
87.5
6.0
16.7
− 19.8
69.5
68.1
84.7
8.8
21.6
86.4
98.57
HAT ( Bifet and Gavaldà, 2009 )
64.3
51.1
79.1
16.5
27.4
− 49.6
51.4
49.4
74.5
13.3
32.1
79.2
97.93
Leveraging Bagging ( Bifet et al., 2010 )
72.4
57.3
85.9
5.1
19.1
− 34.6
62.0
60.6
82.4
7.2
24.9
85.9
98.39
Table 1. Main results (%, mean over streams). Worst Δ : lowest accuracy difference to FIFO over the real streams, in points. Indep. (9): the nine independent confirmation segments of Bitcoin Heist and CIC-IDS2017. The top three values per column are shaded.
Figure 5. Learning curves predict the gain of recall, per grid cell (centroids/block length). Dashed lines mark equality. (a) With fixed segments, observed gain over reset of MICE w/o slow level (seeds 10, 11) and of MICE (seeds 20, 21) against the prediction from learning curves measured beforehand. (b) With oracle boundaries, the measured area term of Thm. 4.1 (seeds 40, 41 and 20, 21) and the best expert per batch against the predicted area. Scatter plots compare predicted recall gains with observed gains and compare the predicted area term with its oracle-boundary measurement.
Figure 6. Detected against fixed segmentation. (a) Their difference per stream group. (b) Gain over reset on fresh streams with random boundary phases, test and confirmation seeds. (c) Share of segment closes that are at least 95% pure. The dashed line marks the target share of 90%. Panels compare detected and fixed segmentation across stream groups, recall gains on test and confirmation streams, and segment purity.
Figure 7. MICE minus unmerged pools (archive, snapshot) of sizes 2, 4 and 12 against their time per batch. The dashed line is the time of MICE . (a) Fixed segmentation. (b) Detected segmentation. Accuracy differences between MICE and archive or snapshot pools are plotted against time per batch for fixed and detected segments.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Meaning
(x,y)
a row and its label
B , M
batch size (100 rows), context budget (1000 rows)
C , Ck
a context, the stored context k
fΘ(⋅∣x;C)
the frozen TFM
P , Q
concepts
ϵP(C)
risk on P of the TFM given C
Appendix
Table 2. Notation.
Figure 8. Boundaries inside fixed segments lower the gain of stored contexts over reset, and realigning the segments restores it. (a) Misaligned block schedules. (b) Boundaries shifted by 200 rows, and segments realigned. Per-stream recall gains for aligned, shifted and realigned segment boundaries.
Figure 9. MICE (fixed segments) minus a 3000-row FIFO window on the 29 real streams, with 95% bootstrap intervals. Per-stream accuracy differences between fixed-segmentation MICE and a 3000-row FIFO window, with bootstrap intervals.
Features, classes, concepts
d=10 , 4, 3
Centroids m
5, 30, 100 with Cj∼U([0,1]10)
Weights
uj∼U(0,1) , normalised
Class of centroid j in concept k
λk(j)∼U{0,…,3} , independent
Noise
isotropic Gaussian, s.d. 0.15
Block length L , cycles
500 or 2000 rows, 3
Batch size, M
100, 1000 rows
Appendix
Table 3. Controlled grid: parameters.
Figure 10. Accuracy minus FIFO on each real stream, grouped by evaluation stage (fixed segments). Per-stream accuracy differences of context policies relative to FIFO, grouped by evaluation stage.
Figure 11. MICE minus the archive and the snapshot pool of size 12, fixed segments. (a) Aligned grid. (b) The 29 real streams, with a dotted line at −0.3 points. Per-stream accuracy differences between MICE and unmerged archive or snapshot context pools.
Figure 12. Accuracy against time per batch on three real prefixes for FIFO windows of 1000 to 13 400 rows, the window ensemble and MICE (fixed segments). Peak GPU memory: 0.8 to 26.4 GB for FIFO, 1.0 GB for MICE . Accuracy versus time per batch for different FIFO window sizes, the window ensemble and MICE on three real-stream prefixes.
Tabular foundation models (TFMs) outperform established machine learning models on tabular benchmarks through in-context learning. Building on this success, interest is growing in applying them to data streams, where data arrive continuously and evolve over time. On a stream, a TFM adapts by updating its context rather than its parameters, so its accuracy and cost depend on which examples it keeps and how often it rebuilds its context. We therefore present a systematic study of TFMs on data streams, covering memory management, computational cost, and stream-specific challenges such as concept drift and delayed labels. We find that TFMs achieve the highest predictive performance and that simply retaining the most recent examples is as effective as existing memory management techniques. They also recover faster than streaming learners after drift and keep the highest accuracy under label delay. This accuracy, however, comes at a high serving cost, since a nearly unchanged context is re-encoded at every prediction. These results point to architectural efficiency as the way forward for in-context stream learning.
Nilesh Verma, Daniel Nowak-Assis, Afonso Lourenço +4
AI Institute, University of Waikato, New Zealand · LIP6, CNRS, Sorbonne Universit´e, France · GECAD, Polytechnic of Porto, Portugal
Tabular stream learning requires predictions on sequentially arriving examples under distribution shift. While standard methods adapt by updating model states, tabular foundation models (TFMs) make predictions conditioned on a labeled context in an in-context manner, making them a natural alternative for stream learning. This shifts the challenge from how to update the model to how to manage the context. We propose a future information view that yields three practical requirements for context management: preserve recent examples, retain uncertain examples, and remove redundant examples. We instantiate these requirements as CURE (Context management via Uncertainty-aware admission and Redundancy aware Eviction), a context-managing policy with entropy-gated admission and redundancy-aware eviction. Across seven streams, CURE shows up to 27.0% relative improvement over classical stream learners, remains robust across multiple TFM backbones, and ranks first among other policy variants. Code and datasets are available at https://github.com/morcellinus/CURE-ICML-FMSD.
Jinmo Lee, Doyun Choi, Moongi Choi +1
Department of Computer Science and Engineering, Seoul National University, Seoul, Republic of Korea · The Kim Jaechul Graduate School of AI, KAIST, Daejeon, Republic of Korea.
Production machine learning systems degrade under concept drift, yet practitioners have little principled guidance on when to retrain. Retraining is costly, retraining budgets are finite, and a retrained model does not take effect instantly: training and deployment latency leave a stale model serving predictions while the data continues to move. We present a controlled empirical study of three practical model-refresh policies (periodic retraining, error-threshold triggering, and statistical drift-triggered retraining with ADWIN) against a no-retrain baseline, evaluated under a unified system model that makes retraining budgets and training-plus-deployment latency explicit. Across 3,933 experiment runs spanning three drift regimes, three budget levels, up to five latency levels, three datasets, and two learning modes, we find that the single most consequential design decision is not the retraining policy but whether the deployed model learns incrementally. With per-sample incremental updates, and for the linear online learner with immediate labels studied here, no policy differs from the no-retrain baseline by a practically significant margin in any of 54 paired comparisons, even at extreme latency. Without incremental updates, policy choice separates outcomes by 15-55 percentage points of post-drift accuracy, and simple periodic retraining significantly outperforms both reactive policies under abrupt and gradual drift, while reactive policies retain an advantage only under recurring drift. We document systematic failure modes of reactive policies and a latency-budget queueing interaction that silently halves effective retraining budgets, and release the full simulator, dataset pipelines, and per-run artifacts for reproducibility.