Self-Indexing Latent Memories for Access-Aware Continual Instruction Tuning of LLMs
Organizations: School of Data and Information Sciences University of North Carolina, Chapel Hill, USA · *Society-Centered AI Lab University of North Carolina, Chapel Hill, USA · Department of Computer Science University of North Carolina, Chapel Hill, USA
Abstract
Continual instruction tuning often preserves old skills with task-private modules. Inference still requires selecting the right module from the prompt alone. We introduce Task-Isolated Latent Adapter Continual Training (TILA-CT), a replay-free method that makes each task module self-indexing. Each task receives a latent memory and adapter bank in a frozen LLM. During adaptation, the memory gathers causal-prefix information and injects task-conditioned context. For access, its persistent state provides the routing prototype used by a shared prompt-only router. On the 8-task CT-Pipeline stream, TILA-CT achieves 65.19 acquisition and 63.98 final macro score, with 1.38 points of average forgetting. A middle-reversed order gives 65.38 acquisition and 64.17 final macro score. Across 6 backbones, final scores remain within 1.5 points of acquisition. Ablations support memory-derived routing keys and causal gathering. Final macro routing accuracy is 93.9%, and oracle routing recovers 1.21 final macro points. These results separate preserved task computation from learned access to it.
Figures & tables
| Method | Mean | Gain [95% CI] | Wins | ||||
| Seq-Train | 51.37_{\color[rgb]{0.35,0.35,0.35}\pm 0.13} | 45.43_{\color[rgb]{0.35,0.35,0.35}\pm 0.11} | 6.79 | 22.46 | 7.42 | 18.55 [14.17, 22.14] | 8/8 |
| L2P | 50.43_{\color[rgb]{0.35,0.35,0.35}\pm 0.12} | 45.27_{\color[rgb]{0.35,0.35,0.35}\pm 0.10} | 5.89 | 18.78 | 4.41 | 18.71 [15.72, 21.90] | 8/8 |
| GEM | 48.87_{\color[rgb]{0.35,0.35,0.35}\pm 0.12} | 44.52_{\color[rgb]{0.35,0.35,0.35}\pm 0.11} | 4.97 | 15.39 | 5.21 | 19.46 [14.65, 24.95] | 8/8 |
| A-GEM | 50.61_{\color[rgb]{0.35,0.35,0.35}\pm 0.14} | 46.66_{\color[rgb]{0.35,0.35,0.35}\pm 0.12} | 4.51 | 11.15 | 4.27 | 17.32 [12.35, 23.20] | 8/8 |
| SAPT | 64.19_{\color[rgb]{0.35,0.35,0.35}\pm 0.16} | 60.81_{\color[rgb]{0.35,0.35,0.35}\pm 0.15} | 3.87 | 5.88 | 1.19 | 3.17 [0.61, 6.24] | 6/8 |
| InfLoRA | 63.20_{\color[rgb]{0.35,0.35,0.35}\pm 0.14} | 59.73_{\color[rgb]{0.35,0.35,0.35}\pm 0.13} | 3.97 | 5.61 | 1.76 | 4.25 [2.51, 6.32] | 8/8 |
| Acquisition and reference proximity | Final retention | ||||||
|---|---|---|---|---|---|---|---|
| Backbone | Mean | Max | BWT | ||||
| Llama-3.2-1B | 67.07 | 1.09 | 1.51 | 65.83 | 1.42 | 0.78 | |
| Qwen-1.8B | 65.67 | 1.09 | 1.92 | 64.47 | 1.37 | 0.77 | |
| Llama-3.1-8B | 69.14 | 1.12 | 1.56 | 67.86 | 1.46 | 0.81 | |
| OLMoE-7B | 70.12 | 1.86 | 2.93 | 68.86 | 1.44 | 0.82 | |
| Qwen3-8B-Reasoning | 65.19 | 0.72 | 1.31 | 63.98 | 1.38 | 0.76 | |
| Task | Baseline | ||||
|---|---|---|---|---|---|
| FOMC | GainLoRA | 2.61 | 6.80 | ||
| MeetingBank | SAPT | 1.38 | 3.63 | ||
| ScienceQA | SwitchCIT | 1.37 | 5.21 | ||
| NumGLUE-en | SAPT | 0.84 | 2.30 | ||
| JEC-QA | SAPT | 0.49 | 1.72 |
| ID | Variant | BWT | Route all | Route worst | Route F | Oracle gap | |||
|---|---|---|---|---|---|---|---|---|---|
| A0 | Full TILA-CT | 65.19 | 63.98 | 1.38 | -1.38 | 93.9 | 90.5 | 4.4 | 1.21 |
| A1 | Oracle routing | 65.19 | 65.19 | 0.00 | 0.00 | 100.0 | 100.0 | 0.0 | 0.00 |
| A2 | Independent keys | 65.01 | 63.32 | 1.93 | -1.93 | 90.5 | 85.5 | 6.5 | 1.69 |
| A3 | Standard adapters | 63.53 | 61.72 | 2.07 | -2.07 | 89.0 | 83.5 | 7.3 | 1.80 |
| A4 | Inject only | 64.11 | 62.79 | 1.51 | -1.51 | 92.2 | 87.5 | 5.3 | 1.32 |
| A5 | Fixed | 64.57 | 63.38 | 1.36 | -1.36 | 93.2 | 89.0 | 4.8 | 1.20 |
| Task | Final (%) | Drop (pp) |
|---|---|---|
| FOMC | 92.5 | 7.5 |
| C-STANCE | 95.8 | 3.0 |
| MeetingBank | 92.5 | 5.9 |
| ScienceQA | 90.5 | 7.3 |
| NumGLUE-en | 92.0 | 5.5 |
| JEC-QA | 94.5 | 3.1 |
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Layers | Rank / width | Adapted modules | Parameters |
| TILA-CT (Ours) | 7 | Attn. 384 | Gather and inject | 90.31M |
| Standard MLP (A3) | 7 | Bottleneck 1536 | MLP adapter and independent-key router | 89.17M |
| SAPT | 36 | Rank 32 | Q/K/V/O, MLP | 88.13M |
| MoCL | 36 | Rank 32 | Q/K/V/O, MLP | 87.30M |
| SwitchCIT | 18 | Rank 64 | Q/K/V/O, MLP | 87.29M + classifier |
| QLoRA-CT | 36 | Rank 32 | Q/K/V/O, MLP | 87.29M |
| Setting | Value |
|---|---|
| Adapted layers | 7 decoder layers |
| Routing layer | Before first adapted layer |
| Memory slots | 32 per task |
| Attention width | 384 |
| Attention heads | 8 in each operation |
| Per-head width | 48 |
| Tasks | Added parameters | Backbone fraction |
|---|---|---|
| 8 | 707.78M | 8.63% |
| 20 | 1.77B | 21.51% |
| 50 | 4.41B | 53.91% |
| 100 | 8.82B | 107.60% |
| Method | FOMC | C-St. | Meet. | Sci. | Num. | JEC | Med. | 20Min. |
|---|---|---|---|---|---|---|---|---|
| Seq-Train | 60.23_{\color[rgb]{0.35,0.35,0.35}{\pm 0.36}} | 62.32_{\color[rgb]{0.35,0.35,0.35}{\pm 0.40}} | 48.48_{\color[rgb]{0.35,0.35,0.35}{\pm 0.31}} | 44.91_{\color[rgb]{0.35,0.35,0.35}{\pm 0.27}} | 36.87_{\color[rgb]{0.35,0.35,0.35}{\pm 0.22}} | 24.18_{\color[rgb]{0.35,0.35,0.35}{\pm 0.15}} | 46.47_{\color[rgb]{0.35,0.35,0.35}{\pm 0.29}} | 39.98 |
| L2P | 60.60_{\color[rgb]{0.35,0.35,0.35}{\pm 0.34}} | 59.80_{\color[rgb]{0.35,0.35,0.35}{\pm 0.38}} | 38.36_{\color[rgb]{0.35,0.35,0.35}{\pm 0.25}} | 43.10_{\color[rgb]{0.35,0.35,0.35}{\pm 0.26}} | 38.98_{\color[rgb]{0.35,0.35,0.35}{\pm 0.21}} | 27.10_{\color[rgb]{0.35,0.35,0.35}{\pm 0.17}} | 45.50_{\color[rgb]{0.35,0.35,0.35}{\pm 0.30}} | 48.74 |
| GEM | 58.32_{\color[rgb]{0.35,0.35,0.35}{\pm 0.35}} | 69.93_{\color[rgb]{0.35,0.35,0.35}{\pm 0.44}} | 44.79_{\color[rgb]{0.35,0.35,0.35}{\pm 0.28}} | 43.83_{\color[rgb]{0.35,0.35,0.35}{\pm 0.25}} | 21.48_{\color[rgb]{0.35,0.35,0.35}{\pm 0.13}} | 30.58_{\color[rgb]{0.35,0.35,0.35}{\pm 0.19}} | 50.90_{\color[rgb]{0.35,0.35,0.35}{\pm 0.31}} | 36.35 |
| A-GEM | 60.07_{\color[rgb]{0.35,0.35,0.35}{\pm 0.38}} | 74.76_{\color[rgb]{0.35,0.35,0.35}{\pm 0.45}} | 46.33_{\color[rgb]{0.35,0.35,0.35}{\pm 0.26}} | 46.70_{\color[rgb]{0.35,0.35,0.35}{\pm 0.30}} | 22.00_{\color[rgb]{0.35,0.35,0.35}{\pm 0.12}} | 32.12_{\color[rgb]{0.35,0.35,0.35}{\pm 0.20}} | 52.47_{\color[rgb]{0.35,0.35,0.35}{\pm 0.33}} | 38.85 |
| SAPT | 66.74_{\color[rgb]{0.35,0.35,0.35}{\pm 0.40}} | 73.86_{\color[rgb]{0.35,0.35,0.35}{\pm 0.46}} | 53.50_{\color[rgb]{0.35,0.35,0.35}{\pm 0.31}} | 68.28_{\color[rgb]{0.35,0.35,0.35}{\pm 0.43}} | 55.96_{\color[rgb]{0.35,0.35,0.35}{\pm 0.35}} | 48.13_{\color[rgb]{0.35,0.35,0.35}{\pm 0.29}} | 62.87_{\color[rgb]{0.35,0.35,0.35}{\pm 0.37}} | 57.10 |
| InfLoRA | 63.95_{\color[rgb]{0.35,0.35,0.35}{\pm 0.39}} | 78.67_{\color[rgb]{0.35,0.35,0.35}{\pm 0.47}} | 51.61_{\color[rgb]{0.35,0.35,0.35}{\pm 0.34}} | 66.67_{\color[rgb]{0.35,0.35,0.35}{\pm 0.38}} | 54.04_{\color[rgb]{0.35,0.35,0.35}{\pm 0.33}} | 45.47_{\color[rgb]{0.35,0.35,0.35}{\pm 0.28}} | 59.95_{\color[rgb]{0.35,0.35,0.35}{\pm 0.35}} | 57.44 |
| Comparison | Mean diff. | 95% CI | Task wins |
|---|---|---|---|
| TILA-CT – GoD-MoE | 1.97 | [1.41, 2.58] | 8/8 |
| TILA-CT – GainLoRA | 3.28 | [2.53, 4.15] | 8/8 |
| TILA-CT – SAPT | 3.17 | [0.61, 6.24] | 6/8 |
| Method | FOMC | C-STANCE | MeetingBank | ScienceQA | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SEQ | ISO | SEQ | ISO | SEQ | ISO | SEQ | ISO | |||||
| Seq-train | 74.91 | 74.91 | 0.00 | 74.30 | 77.89 | -3.59 | 56.91 | 60.47 | -3.56 | 50.42 | 58.89 | -8.47 |
| L2P | 73.50 | 73.50 | 0.00 | 71.41 | 74.65 | -3.24 | 43.78 | 46.09 | -2.31 | 48.58 | 52.57 | -3.99 |
| GEM | 68.03 | 68.03 | 0.00 | 81.38 | 85.15 | -3.77 | 50.07 | 54.75 | -4.68 | 48.59 | 53.66 | -5.07 |
| A-GEM | 69.54 | 69.54 | 0.00 | 83.89 | 86.26 | -2.37 | 51.62 | 55.80 | -4.18 | 50.42 | 54.31 | -3.89 |
| SAPT | 73.60 | 73.60 | 0.00 | 81.20 | 82.86 | -1.66 | 57.13 | 58.30 | -1.17 | 73.12 | 73.86 | -0.74 |
| Backbone | FOMC | C-STANCE | MeetingBank | ScienceQA | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SEQ | ISO | SEQ | ISO | SEQ | ISO | SEQ | ISO | |||||
| Llama-3.2-1B | 69.57 | 69.57 | 0.00 | 69.61 | 70.20 | -0.59 | 62.49 | 64.00 | -1.51 | 63.00 | 63.81 | -0.81 |
| Qwen-1.8B | 67.04 | 67.04 | 0.00 | 69.08 | 69.69 | -0.61 | 61.71 | 62.45 | -0.74 | 63.02 | 63.90 | -0.88 |
| Llama-3.1-8B | 71.72 | 71.72 | 0.00 | 71.76 | 72.37 | -0.61 | 64.42 | 65.98 | -1.56 | 64.95 | 65.79 | -0.84 |
| OLMoE-7B | 73.61 | 73.61 | 0.00 | 73.63 | 75.29 | -1.66 | 69.79 | 72.30 | -2.51 | 65.90 | 67.08 | -1.18 |
| Qwen3-8B-R | 76.48 | 76.48 | 0.00 | 88.29 | 88.51 | -0.22 | 55.87 | 56.19 | -0.32 | 69.19 | 69.62 | -0.43 |
| Per-task top-1 routing accuracy (%) | Stage summary (%) | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Stage | Current | Past macro | All macro | Worst task | ||||||||
| After | 100.0 | – | – | – | – | – | – | – | 100.0 | – | 100.0 | 100.0 |
| After | 97.5 | 98.8 | – | – | – | – | – | – | 98.8 | 97.5 | 98.2 | 97.5 |
| After | 96.3 | 98.0 | 98.4 | – | – | – | – | – | 98.4 | 97.2 | 97.6 | 96.3 |
| After | 95.5 | 97.6 | 96.6 | 97.8 | – | – | – | – | 97.8 | 96.6 | 96.9 | 95.5 |
| After | 94.8 | 97.2 | 95.8 | 94.8 | 97.5 | – | – | – | 97.5 | 95.7 | 96.0 | 94.8 |
| Dataset | Splits used | Task type | Output format | Answer extraction | Evaluation metric |
|---|---|---|---|---|---|
| FOMC | Train: 1,984; Test: 496 | English monetary-policy stance classification with 3 classes | One canonical label: HAWKISH , DOVISH , or NEUTRAL | Leading and trailing whitespace and formatting tokens are removed. The output is converted to uppercase and accepted only when exactly one canonical label is present. Source labels are mapped as 0 = DOVISH , 1 = HAWKISH , and 2 = NEUTRAL . | Accuracy. |
| C-STANCE | Train: 33,418; Validation: 7,284; Test: 7,424 | Chinese target-conditioned stance classification with 3 classes | One canonical label: FAVOR , AGAINST , or NEUTRAL | Formatting tokens are removed and the final canonical stance label is extracted. Equivalent forms may be normalized, such as PRO to FAVOR , CON to AGAINST , and NEU to NEUTRAL . Free-form reasoning text is ignored during scoring. | Accuracy. |
| Meeting-Bank | Train: 5,169; Validation: 861; Test: 862 | English meeting-segment abstractive summarization | Free-form English summary | Chat-template artifacts, unintended answer prefixes, and leading or trailing whitespace are removed. The generated summary content and sentence order are otherwise preserved. | ROUGE-L F1 on a 0–100 scale. |
| ScienceQA | Train: 12,726; Validation: 4,241; Test: 4,241 | Multiple-choice science question answering using the text-only subset | One option identifier, usually A through E | The response following the final-answer marker is extracted. Option letters and valid numerical option indices are mapped to the corresponding answer choice. Only one final option is accepted. Image-dependent examples are excluded in the text-only setting, and the post-filter split counts are reported separately. | Answer-option accuracy. |
| NumGLUE-en | Train: 5,000; Test: 2,000; Official splits used by TRACE ( Wang et al., 2023 ) | English numerical and arithmetic reasoning | A short numerical answer, such as an integer, decimal, signed number, or percentage | The response following the final-answer marker is extracted. Whitespace, commas, signs, decimal forms, percentage symbols, units, and trailing punctuation are normalized by the selected evaluator. Intermediate calculations are ignored. | Normalized exact-match accuracy. For categorical NumGLUE subtasks, the official label accuracy is used. |
| JEC-QA | Train: 21,902; Test: 5,273 | Chinese legal-domain multiple-choice question answering with single-answer and multiple-answer questions | A canonical option set, such as A , C , AB , or BCD | The response is converted to uppercase and only valid option identifiers are retained. Duplicate options are removed, and the predicted set is sorted into canonical order. A prediction is correct only when the full predicted option set matches the gold option set exactly. Missing or extra choices are marked incorrect. | Exact option-set accuracy. |
Continual Learning for LLMs
Continual learning for large language models studies how models absorb new knowledge and skills while retaining previously learned capabilities.
See all 252 papers