cs.AINov 8, 2026

Self-Indexing Latent Memories for Access-Aware Continual Instruction Tuning of LLMs

Authors: Dhruv Dixit, Paritosh Pandey, Snehalkumar 'Neil' S. Gaikwad

Organizations: School of Data and Information Sciences University of North Carolina, Chapel Hill, USA · *Society-Centered AI Lab University of North Carolina, Chapel Hill, USA · Department of Computer Science University of North Carolina, Chapel Hill, USA

Abstract

Continual instruction tuning often preserves old skills with task-private modules. Inference still requires selecting the right module from the prompt alone. We introduce Task-Isolated Latent Adapter Continual Training (TILA-CT), a replay-free method that makes each task module self-indexing. Each task receives a latent memory and adapter bank in a frozen LLM. During adaptation, the memory gathers causal-prefix information and injects task-conditioned context. For access, its persistent state provides the routing prototype used by a shared prompt-only router. On the 8-task CT-Pipeline stream, TILA-CT achieves 65.19 acquisition and 63.98 final macro score, with 1.38 points of average forgetting. A middle-reversed order gives 65.38 acquisition and 64.17 final macro score. Across 6 backbones, final scores remain within 1.5 points of acquisition. Ablations support memory-derived routing keys and causal gathering. Final macro routing accuracy is 93.9%, and oracle routing recovers 1.21 final macro points. These results separate preserved task computation from learned access to it.

Figures & tables

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning

    Jul 26, 2026Reza Rahimi Azghan, Gautham Krishna Gudur, Giulia Pedrielli +2Continual Learning for LLMsContinual LoRA

  2. CaLLM: Continual Adaptation of LLMs via Non-Parametric Adapter Routing

    Oct 27, 2026Bilge Celik, Anna Vettoruzzo, Joaquin VanschorenContinual Learning for LLMsAdapter-Based Continual Learning

  3. Rosetta Memory: Adaptive Memory for Cross-LLM Agents

    Jun 5, 2026Hao Yang, Shiqi Shen, Haoxuan Li +3LLM Agent MemoryPersistent Memory for Language Models