Provenance Guided Incremental Learning Under Evolving Concept Definitions

Ismail Lamaakal

cs.AIAug 24, 2026

Provenance Guided Incremental Learning Under Evolving Concept Definitions

Authors: Ismail Lamaakal

Organizations: Department of Computer Science, Faculty of Applied Sciences Nador, Mohammed Premier University, Oujda, Morocco

Abstract

Learning systems deployed over long periods must adapt not only to statistical changes in incoming data, but also to revisions of the definitions that generate their prediction targets. Conventional concept-drift methods typically infer such changes from observations or prediction errors, even when the underlying policy, rule, or query has been explicitly modified. This paper studies rule-induced concept shift, where the target-defining concept is revised directly, causing previously stored instances to acquire different semantic labels without requiring any change in their observed data. We introduce a provenance-guided incremental learning framework that compiles consecutive concept definitions into a structured rule delta, traces the changed components through historical provenance, certifies records whose previous labels remain valid, and restricts reevaluation to a localized candidate region. Executable revisions are relabeled automatically, ambiguous cases are handled through selective supervision, and the resulting changes are used for incremental predictor repair. A versioned concept memory further supports recurring definitions. We also introduce RuleShift-Bench, spanning financial, demographic, cybersecurity, and graph-structured data with threshold, predicate, logical, relational, recurring, and mixed concept revisions. Across the benchmark, provenance-guided repair attains 92.3% accuracy and 90.2% Macro-F1 while reprocessing 14.7% of the historical collection and retaining 94.6% of affected records. Its average update latency is 179s compared with 993s for complete relabeling and retraining. The results demonstrate that an explicit concept revision can be exploited as a data-maintenance signal, allowing learning systems to update the supervision and predictive state that depend on the change while preserving knowledge that remains valid.

Figures & tables

Fig. 1: Overview of the proposed provenance-guided incremental learning framework under evolving concept definitions. At revision session

t\!\rightarrow\!t+1

, the previous concept definition

Q_{t}

and the revised definition

Q_{t+1}

are first compared by the rule-delta compiler to identify predicates, parameters, logical operators, or relational dependencies that have changed. The provenance analyzer then traces these changes through the historical data and separates records into three groups: certified stable records, whose previous targets remain valid; exactly affected records, whose revised targets can be computed deterministically; and ambiguous records, for which additional supervision is required. Certified stable examples provide a stability buffer, exactly affected examples are relabeled automatically using

Q_{t+1}

, and ambiguous examples are handled through selective annotation. The resulting revision-specific data are used to incrementally repair the deployed predictor from

f_{\theta_{t}}

f_{\theta_{t+1}}

without recomputing the complete historical dataset. Successive revisions are processed in the same manner, while a versioned concept memory stores each concept definition, predicate graph, provenance state, affected-data information, and model state to support efficient future updates and recurring concept definitions.

Dataset	Records	Features	Versions	Thresh.	Pred.	Logic	Relation
PaySim	6,362,620	10	8	✓	✓	✓	–
Census-Income	299,285	40	8	✓	✓	✓	–
UNSW-NB15	2,540,044	47	8	✓	✓	✓	✓
ogbn-arxiv	169,343	128	8	✓	✓	✓	✓

TABLE I: Principal datasets used in RuleShift-Bench. Rule versions include the initial definition and seven subsequent rule-shift protocols.

ID	Shift	Example
R1	Threshold revision	$x>q_{0.75}\rightarrow x>q_{0.60}$
R2	Predicate insertion	$A\rightarrow A\land B$
R3	Predicate deletion	$A\land B\rightarrow A$
R4	Logical rewrite	$A\lor B\rightarrow A\land B$
R5	Relational rewrite	$R_{1}\rightarrow R_{2}$ / path change
R6	Recurring definition	$Q_{1}\rightarrow Q_{2}\rightarrow Q_{1}$

TABLE II: Rule-induced concept-shift protocols.

Method	Acc. (%)	Macro-F1 (%)	Affected Recall (%)	Reprocessed (%)	Labels	Time (s)
Old Predictor	82.7	78.9	0.0	0.0	0	0.0
Full Relabel + Retrain	92.8	90.7	100.0	100.0	0	993
Sliding Window	88.5	85.8	63.2	20.0	0	389
Online Update	87.9	85.1	68.5	12.5	0	241
Replay	89.2	86.7	74.6	17.5	0	325
ADWIN + Retrain	87.6	84.9	66.0	28.4	0	441

TABLE III: Overall comparison across the four RuleShift-Bench data families under rule revisions R1–R5.

Dataset	Records	Full F1	Repair F1	Reprocessed	Full Time	Repair Time
PaySim	6,362,620	91.4	90.9	12.1%	1775 s	181 s
Census-Income	299,285	89.2	88.7	16.3%	154 s	28 s
UNSW-NB15	2,540,044	92.6	92.2	13.8%	903 s	148 s
ogbn-arxiv	169,343	89.5	89.0	16.6%	742 s	121 s

TABLE IV: Dataset-wise comparison between complete recomputation and provenance-guided incremental repair.

Panel of Figure 6 — Fig. 2: Predictive and computational behavior of the evaluated maintenance strategies across the RuleShift-Bench datasets.

Selection	1	2	4	8	16	32
Random	72.4	74.6	77.1	79.8	82.1	83.5
Global uncertainty	75.2	77.9	80.6	83.1	84.9	85.8
Delta-constrained uncertainty	78.6	81.7	84.4	86.9	88.4	89.2

TABLE V: Macro-F1 across different annotation budgets for ambiguous concept revisions.

Method	$Q_{1}$ First	$Q_{1}$ Return	$RG$	Reprocessed	Latency
Full Retrain	91.4	84.8	6.6	100.0%	998 s
Replay	91.2	88.7	2.5	18.0%	315 s
Versioned Repair	91.5	90.6	0.9	12.6%	146 s
Oracle Version Reuse	91.5	91.1	0.4	8.7%	119 s

TABLE VI: Recovery under the recurring concept sequence

Q_{1}\!\rightarrow\!Q_{2}\!\rightarrow\!Q_{3}\!\rightarrow\!Q_{1}

Panel of Figure 9 — Fig. 3: Annotation efficiency and concept recurrence under limited supervision and repeated concept definitions.

Panel of Figure 10 — Fig. 4: Ablation analysis of the proposed framework from predictive and system perspectives.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 12 — Fig. 5: Robustness of provenance-guided incremental maintenance across three predictor families and four data modalities.

Panel of Figure 13 — Fig. 6: Long-horizon behavior under repeated concept recurrence. Accuracy is tracked over twelve revisions together with return-specific recovery gaps and cumulative maintenance cost.

Panel of Figure 14 — Fig. 7: Storage behavior of provenance and versioned concept memory, including state composition, normalized dataset-level overhead, and the benefit of sharing unchanged state across concept versions.

Panel of Figure 15 — Fig. 8: Sensitivity of the framework to the repair-stability balance, stable-buffer size, and provenance granularity.

Panel of Figure 16 — Fig. 9: Matched statistical analysis of predictive differences across maintenance strategies. Intervals report paired Macro-F1 differences, and the accompanying matrix summarizes corrected significance and effect magnitude.

Panel of Figure 17 — Fig. 10: Controlled stress analysis under increasingly global concept revisions, incomplete provenance, non-executable concept components, and highly coupled graph dependencies.

Incremental Learning

Incremental learning studies how models update from sequential data while retaining earlier knowledge as classes, tasks, or domains change.

See all 146 papers

Aug 24, 2026

Continual Visual Learning under Evolving Semantic Concept Shift

Visual foundation models are commonly adapted under the assumption that the appearance of incoming data may change while the semantic meaning of the prediction task remains fixed. In long-lived visual systems, however, taxonomies, policies, and concept definitions can themselves evolve, causing the same visual evidence to require a different interpretation. We study this setting as evolving semantic concept shift and introduce SemReWrite, a framework for selectively updating obsolete visual--semantic mappings while preserving knowledge that remains valid. SemReWrite represents changes between old and revised semantic specifications, combines semantic discrepancy with sparse revised supervision to localize affected visual regions, and uses an input-dependent low-rank rewriting mechanism together with structured semantic memory, preservation, and obsolete-decision suppression. We further introduce EvoShift-Bench, spanning ImageNet, iNaturalist, CUB-200-2011, and DomainNet, with semantic transitions including class split, merge, boundary revision, insertion, partial redefinition, recurrence, and mixed semantic--appearance shift. To explicitly evaluate selective semantic revision, we introduce Rewrite Accuracy (RA) and Preservation Accuracy (PA) for affected and unaffected regions, respectively, Obsolete Retention (OR) for measuring residual outdated semantic associations, and the Selective Revision Score (SRS), which jointly summarizes rewriting and preservation performance. Experiments show that SemReWrite achieves a stronger balance between learning revised semantics and retaining unaffected knowledge than prompt replacement, conventional fine-tuning, parameter-efficient adaptation, and continual-learning strategies.

Ismail Lamaakal, Chaymae Yahyati, Yassine Maleh +2

Department of Computer Science, Faculty of Applied Sciences Nador, Mohammed Premier University, Oujda, Morocco · Laboratory LaSTI, ENSAK, Sultan Moulay Slimane University, Khouribga, Morocco

Continual Learning Vision Foundation Model Adaptation

Ask PDF

Oct 7, 2026

Continual Learning without Continual Training

Continual learning requires models to adapt to new domains and new classes while retaining prior knowledge. Many existing methods rely on continued optimization, using regularization, replay, or parameter expansion to prevent new updates from overwriting previously learned knowledge. Instead, we propose replacing continual training with continual inference: a PFN-based model that is meta-trained, and then frozen, adapting to new classes only by extending an in-context evidence set. Our model, Latent Concept PFN, performs in-context Bayesian inference over a latent concept space that captures semantic structure shared across domains and classes. As each new domain or class arrives, exemplars are added to the memory; adaptation reflects updated posterior beliefs over latent concepts rather than gradient updates. No parameters are changed, reducing forgetting. The same method handles both domain and class incremental continual learning without task identity. Concept annotations are only used during meta-training, acting as a soft anchor on the latent space rather than a fixed bottleneck. Unlike fixed-vocabulary concept methods, the model also handles noisy, ambiguous, or incomplete annotations by combining concept labels with raw input evidence to discover distinctions beyond the predefined concept set. Experiments on class and domain incremental learning datasets demonstrate competitive continual learning performance while learning interpretable latent concepts.

Nikita Narayanan, Ritham Majumdarr, Sonali Parbhoo

Imperial College London

Continual Learning Class-Incremental Learning Domain-Incremental Learning Prior-Data Fitted Networks In-Context Learning

Ask PDF

Aug 19, 2026

When to Retrain: An Empirical Study of Retraining Policies for Streaming ML Under Concept Drift, Budget, and Latency Constraints

Production machine learning systems degrade under concept drift, yet practitioners have little principled guidance on when to retrain. Retraining is costly, retraining budgets are finite, and a retrained model does not take effect instantly: training and deployment latency leave a stale model serving predictions while the data continues to move. We present a controlled empirical study of three practical model-refresh policies (periodic retraining, error-threshold triggering, and statistical drift-triggered retraining with ADWIN) against a no-retrain baseline, evaluated under a unified system model that makes retraining budgets and training-plus-deployment latency explicit. Across 3,933 experiment runs spanning three drift regimes, three budget levels, up to five latency levels, three datasets, and two learning modes, we find that the single most consequential design decision is not the retraining policy but whether the deployed model learns incrementally. With per-sample incremental updates, and for the linear online learner with immediate labels studied here, no policy differs from the no-retrain baseline by a practically significant margin in any of 54 paired comparisons, even at extreme latency. Without incremental updates, policy choice separates outcomes by 15-55 percentage points of post-drift accuracy, and simple periodic retraining significantly outperforms both reactive policies under abrupt and gradual drift, while reactive policies retain an advantage only under recurring drift. We document systematic failure modes of reactive policies and a latency-budget queueing interaction that silently halves effective retraining budgets, and release the full simulator, dataset pipelines, and per-run artifacts for reproducibility.

Sawan Dasari

Drifting Models Concept Drift Streaming Algorithms Incremental Learning

Ask PDF

Abstract

Incremental Learning

Explore similar work

Continual Visual Learning under Evolving Semantic Concept Shift

Continual Learning without Continual Training

When to Retrain: An Empirical Study of Retraining Policies for Streaming ML Under Concept Drift, Budget, and Latency Constraints