Recall or Relearn? A Mixture of In-Context Experts for Recurring Concept Drift

Zhangkai Wu; Xu Zhang; Zhihong Cui; Yiqi Wang; Yi Zhang

cs.LGOct 27, 2026

Recall or Relearn? A Mixture of In-Context Experts for Recurring Concept Drift

Authors: Zhangkai Wu, Xu Zhang, Zhihong Cui, Yiqi Wang, Yi Zhang

Organizations: University of Sydney, Sydney, Australia · Macquarie University, Sydney, Australia · Department of Informatics, University of Oslo, Oslo, Norway · University of Southern Queensland, Toowoomba, Australia ·

Abstract

Tabular foundation models (TFMs) are being applied to Web data streams such as network intrusion detection, where they adapt by updating the labelled rows of their context without retraining. When a past concept returns after leaving the context, a TFM faces a recall-or-relearn decision between restoring stored rows of the concept and relearning it from new labels. We propose MICE (Mixture of In-Context Experts), a training-free learner that cuts the stream into segments at detected changes. It merges a closed segment into a stored context when the TFM's estimated additional error on the segment is within a tolerance, under a capacity limit on stored contexts. A fast level weights the stored contexts and first-in-first-out (FIFO) windows by the log-loss of their predictions on the last labelled batch. We show that, with pure stored contexts and a margin condition, this fast level gains over relearning, in expected accuracy, the normalised positive area between the stored context's accuracy and the TFM's learning curve, up to a first-batch term. A slow level mixes the fast level's prediction with that of the FIFO window of the last labelled rows and bounds the discounted half-Brier regret of MICE against this window at every time. Learning curves measured beforehand predict the ordering of the observed recall gains. MICE outperforms reset, which empties the context at detected drifts, on grid streams with random boundary phases. Merging keeps fewer experts than pools that store every segment, and with fixed segmentation a time-matched snapshot pool matches MICE on the shifted grid. On real streams, MICE has the highest mean accuracy of all policies we compare.

Figures & tables

Figure 1. A frozen TFM filters spam from recent labelled mail, its context. Red flags mark detected changes. When campaign A returns, the FIFO window holds only B and predicts ham. MICE recalls A’s stored context and classifies A’s mail as spam. Here, recall avoids relearning the hard campaign A. Both methods learn the new, easy campaign C quickly. Schematic. The schematic mail timeline shows campaign A returning after B. A new campaign C follows the returning A. Red flags mark detected changes. Below the timeline, three rows compare the FIFO window with MICE. In an ordinary week both classify A's mail as spam. When A returns, the FIFO window holds only B's mail and predicts ham. MICE recalls a stored context of A and predicts spam. Both methods learn the new campaign C fast and predict spam.

Figure 2. Three aspects of the recall-or-relearn decision. A and B are two concepts of one stream. (a) When A returns, the FIFO window has discarded A’s rows and the TFM predicts A’s batch with B’s rule. (b) A context that straddles an unknown boundary mixes conflicting rules. (c) What recall is worth: the TFM relearns the easy concept from a few rows but the hard one only from many, and the red rows mark the relearning that recall avoids. Schematic. Fig. 3 measures each aspect. Under the purity and margin conditions of Thm. 4.1 , the fast level’s expected accuracy gain over relearning equals the normalised positive area in (c), up to a first-batch term. Panel (a) shows a FIFO window discarding the rows of a concept that later returns. Panel (b) contrasts the predictions from a pure and a mixed context. Panel (c) compares the learning curves of an easy and a hard concept with stored-context accuracy, with the recall area shaded for the hard concept.

Figure 3. Context stress test. (1) Accuracy in the first five batches after a recurrence: FIFO (gray bars) and the oracle-context lower bound of 99% (blue bars), with the reset range of 81–88% shaded. (2) Gain of stored contexts over reset with aligned, shifted and realigned segments, over all batches (blue) and the first five batches after a recurrence (dashed orange), with single streams as pale traces. (3) Learning curves of concepts with 5, 30 and 100 centroids. Three connected diagnostic plots: paired horizontal bars compare FIFO and oracle contexts with a reset reference band, phase curves show loss and recovery of gains as segment alignment changes, and learning curves show accuracy versus context size for three concept complexities.

Figure 4. Overview of MICE (schematic, bottom row illustrative). (1) Segments cut at detected changes join the stored context of lowest estimated recall regret within a tolerance. (2) The fast level weights stored-context and FIFO predictions from one frozen TFM by their log-loss on the last labelled batch. Under pure contexts and the margins of Thm. 4.1 , its expected accuracy gain over relearning equals the area term plus the first-batch term

\zeta

. (3) The slow level mixes the fast-level and FIFO predictions. Its discounted half-Brier regret against FIFO is at most

2\ln 2

at every time. Three columns, one per stage, are linked by arrows from memory to fast level to slow level. Column 1 (memory): a stream A, A, B, B, A with red flags at detected changes is cut into segments. The last segment joins the stored context AAA, which has the lowest recall regret, and not BB. Its middle panel states the aim of one concept per context: a pure context AAAA is accepted and a mixed context AABB is rejected, and merges happen only within a tolerance. Column 2 (fast level): stored contexts and FIFO windows condition one frozen TFM and receive log-loss weights. Its middle panel plots accuracy against batch for a stored context and for relearning. The gain is the shaded area between the two curves plus a hatched first-batch term zeta. Column 3 (slow level): a discounted half-Brier mix between FIFO and the fast level gives the prediction. Its middle panel shows discounted regret against FIFO below 2 ln 2 at every time. An illustrative bottom row reads, for each stage when A recurs: the segment joins context A, context A takes the weight, and the mix shifts to the fast level.

	Real streams (29)						Controlled grid (24)					Gen. (30)	Indep. (9)
Policy	Acc. $\uparrow$	F1 $\uparrow$	AUC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	Worst $\Delta$ $\uparrow$	Acc. $\uparrow$	F1 $\uparrow$	AUC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	Acc. $\uparrow$	Acc. $\uparrow$
Trained stream learners
ARF ( Gomes et al., 2017 )	73.4	58.4	85.4	6.0	18.7	$-$ 37.5	67.7	65.6	83.8	11.2	22.9	88.0	98.39
SRP ( Gomes et al., 2019 )	76.6	62.3	87.5	6.0	16.7	$-$ 19.8	69.5	68.1	84.7	8.8	21.6	86.4	98.57
HAT ( Bifet and Gavaldà, 2009 )	64.3	51.1	79.1	16.5	27.4	$-$ 49.6	51.4	49.4	74.5	13.3	32.1	79.2	97.93
Leveraging Bagging ( Bifet et al., 2010 )	72.4	57.3	85.9	5.1	19.1	$-$ 34.6	62.0	60.6	82.4	7.2	24.9	85.9	98.39

Table 1. Main results (%, mean over streams). Worst

\Delta

: lowest accuracy difference to FIFO over the real streams, in points. Indep. (9): the nine independent confirmation segments of Bitcoin Heist and CIC-IDS2017. The top three values per column are shaded.

Figure 5. Learning curves predict the gain of recall, per grid cell (centroids/block length). Dashed lines mark equality. (a) With fixed segments, observed gain over reset of MICE w/o slow level (seeds 10, 11) and of MICE (seeds 20, 21) against the prediction from learning curves measured beforehand. (b) With oracle boundaries, the measured area term of Thm. 4.1 (seeds 40, 41 and 20, 21) and the best expert per batch against the predicted area. Scatter plots compare predicted recall gains with observed gains and compare the predicted area term with its oracle-boundary measurement.

Figure 6. Detected against fixed segmentation. (a) Their difference per stream group. (b) Gain over reset on fresh streams with random boundary phases, test and confirmation seeds. (c) Share of segment closes that are at least 95% pure. The dashed line marks the target share of 90%. Panels compare detected and fixed segmentation across stream groups, recall gains on test and confirmation streams, and segment purity.

Figure 7. MICE minus unmerged pools (archive, snapshot) of sizes 2, 4 and 12 against their time per batch. The dashed line is the time of MICE . (a) Fixed segmentation. (b) Detected segmentation. Accuracy differences between MICE and archive or snapshot pools are plotted against time per batch for fixed and detected segments.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$(\mathbf{x},y)$	a row and its label
$B$ , $M$	batch size (100 rows), context budget (1000 rows)
$\mathcal{C}$ , $\mathcal{C}_{k}$	a context, the stored context $k$
$f_{\Theta}(\cdot\mid\mathbf{x};\mathcal{C})$	the frozen TFM
$P$ , $Q$	concepts
$\epsilon_{P}(\mathcal{C})$	risk on $P$ of the TFM given $\mathcal{C}$

Appendix

Table 2. Notation.

Figure 8. Boundaries inside fixed segments lower the gain of stored contexts over reset, and realigning the segments restores it. (a) Misaligned block schedules. (b) Boundaries shifted by 200 rows, and segments realigned. Per-stream recall gains for aligned, shifted and realigned segment boundaries.

Figure 9. MICE (fixed segments) minus a 3000-row FIFO window on the 29 real streams, with 95% bootstrap intervals. Per-stream accuracy differences between fixed-segmentation MICE and a 3000-row FIFO window, with bootstrap intervals.

Features, classes, concepts	$d=10$ , 4, 3
Centroids $m$	5, 30, 100 with $C_{j}\sim\mathcal{U}([0,1]^{10})$
Weights	$u_{j}\sim\mathcal{U}(0,1)$ , normalised
Class of centroid $j$ in concept $k$	$\lambda_{k}(j)\sim\mathcal{U}\{0,\dots,3\}$ , independent
Noise	isotropic Gaussian, s.d. 0.15
Block length $L$ , cycles	500 or 2000 rows, 3
Batch size, $M$	100, 1000 rows

Appendix

Table 3. Controlled grid: parameters.

Figure 10. Accuracy minus FIFO on each real stream, grouped by evaluation stage (fixed segments). Per-stream accuracy differences of context policies relative to FIFO, grouped by evaluation stage.

Figure 11. MICE minus the archive and the snapshot pool of size 12, fixed segments. (a) Aligned grid. (b) The 29 real streams, with a dotted line at

-0.3

points. Per-stream accuracy differences between MICE and unmerged archive or snapshot context pools.

Figure 12. Accuracy against time per batch on three real prefixes for FIFO windows of 1000 to 13 400 rows, the window ensemble and MICE (fixed segments). Peak GPU memory: 0.8 to 26.4 GB for FIFO, 1.0 GB for MICE . Accuracy versus time per batch for different FIFO window sizes, the window ensemble and MICE on three real-stream prefixes.

Concept Drift

Concept drift is change over time in the relationship between data and target outcomes, which can erode the accuracy of deployed models.

See all 91 papers

Rethinking Tabular Foundation Models On Data Streams
Oct 4, 2026Nilesh Verma, Daniel Nowak-Assis, Afonso Lourenço +4Tabular Foundation Models Tabular ICL
AI Institute, University of Waikato, New Zealand · LIP6, CNRS, Sorbonne Universit´e, France · GECAD, Polytechnic of Porto, Portugal
Bounded Context Management for Tabular Foundation Models on Stream Learning
Jun 17, 2026Jinmo Lee, Doyun Choi, Moongi Choi +1In-Context Example Selection Tabular Foundation Models
Department of Computer Science and Engineering, Seoul National University, Seoul, Republic of Korea · The Kim Jaechul Graduate School of AI, KAIST, Daejeon, Republic of Korea.
When to Retrain: An Empirical Study of Retraining Policies for Streaming ML Under Concept Drift, Budget, and Latency Constraints
Aug 19, 2026Sawan DasariDrifting Models Concept Drift

Oct 4, 2026J/K move · Enter open · S save

Rethinking Tabular Foundation Models On Data Streams

Nilesh Verma, Daniel Nowak-Assis, Afonso Lourenço, Albert Bifet+3

AI Institute, University of Waikato, New Zealand · LIP6, CNRS, Sorbonne Universit´e, France · GECAD, Polytechnic of Porto, Portugal

Tabular Foundation Models Tabular ICL Streaming Algorithms

Open paper AskPDF arXiv

Tabular foundation models (TFMs) outperform established machine learning models on tabular benchmarks through in-context learning. Building on this success, interest is growing in applying them to data streams, where data arrive continuously and evolve over time. On a stream, a TFM adapts by updating its context rather than its parameters, so its accuracy and cost depend on which examples it keeps and how often it rebuilds its context. We therefore present a systematic study of TFMs on data streams, covering memory management, computational cost, and stream-specific challenges such as concept drift and delayed labels. We find that TFMs achieve the highest predictive performance and that simply retaining the most recent examples is as effective as existing memory management techniques. They also recover faster than streaming learners after drift and keep the highest accuracy under label delay. This accuracy, however, comes at a high serving cost, since a nearly unchanged context is re-encoded at every prediction. These results point to architectural efficiency as the way forward for in-context stream learning.

Abstract

Concept Drift

Explore similar work

Rethinking Tabular Foundation Models On Data Streams

Bounded Context Management for Tabular Foundation Models on Stream Learning

When to Retrain: An Empirical Study of Retraining Policies for Streaming ML Under Concept Drift, Budget, and Latency Constraints