Large language models deployed in specialized domains must improve in-domain performance without sacrificing general capabilities. Existing parameter-efficient fine-tuning methods are typically always on: their learned perturbations are applied to every input, which can degrade out-of-domain (OOD) performance. We propose Engram Adapter, a framework that repurposes pretraining-time conditional memory as a post-hoc adapter for frozen LLMs. It uses multi-channel matching over local n-gram patterns with explicit occupancy tracking as a lightweight selectivity prior, making residual injection more likely on in-domain inputs while a learned scalar gate suppresses incoherent OOD retrievals. We evaluate on Qwen3-4B and Qwen3-8B with AG-News and MedMCQA as adaptation tasks and OOD benchmarks spanning reasoning, translation, code generation, and legal reasoning. Engram Adapter improves in-domain accuracy while preserving 99.4%--100.1% of average OOD performance; on LegalBench it slightly exceeds the frozen base model on average, whereas comparable always-on baselines degrade sharply. Mechanistic analyses show that although OOD activations are non-zero, gate and projection attenuation reduce residuals to approximately 0.08% of hidden-state norm, yielding small KL drift and negligible accuracy change. These results suggest conditional activation is a promising route toward modular, retention-preserving domain specialization over frozen backbones.
Figures & tables
Figure 1: Conceptual comparison between always-on PEFT and conditional adaptation. LoRA-style methods expose learned low-rank weight updates to all inputs at inference time, while adapter-style methods expose an additional residual branch to all inputs. In contrast, our Engram Adapter conditionally exposes its residual through explicit local-pattern matching: domain-relevant inputs are more likely to activate the adapter, whereas out-of-domain inputs with no or weak matches receive masked or gate-attenuated residuals.
Figure 2: Engram Adapter overview. Four self-attention layers ( L={1,3,7,15} ) receive hook-based adapters sharing hash memory tables and occupancy masks. Input n -grams retrieve masked memory, which is projected and gated before residual injection; only adapter parameters are trainable.
In-Domain
Out-of-Domain (General Capabilities)
Method
AG-News
ARC-C
MedMCQA
FLORES
FLORES
MBPP
Acc.%
Acc.%
Acc.%
BLEU
chrF
pass@1
Base Model
85.14
88.99
53.65
29.85
58.85
66.80
LoRA r=64
89.89 ± 0.50
88.14 ± 0.49
53.72 ± 0.33
28.96 ± 0.44
58.13 ± 0.18
56.67 ± 12.21
LoRA r=32
90.15 ± 0.14
88.11 ± 0.38
54.90 ± 0.27
29.55 ± 0.19
58.61 ± 0.06
64.93 ± 0.84
Adapter
90.54
87.88
53.41
29.91
58.56
56.00
Table 1: AG-News domain adaptation results on Qwen3-4B. LoRA and Engram Adapter results are reported as mean ± std across three random seeds (42, 7783, 114514). For Qwen3-4B, AG-News, ARC-C, MedMCQA, and FLORES are evaluated zero-shot; MBPP uses 3-shot prompting. Bold : best among adapted models; underline : second best. AG-News is the in-domain target task; the remaining four benchmarks measure out-of-domain retention.
In-Domain
Out-of-Domain (General Capabilities)
Method
MedMCQA
ARC-C
AG-News
FLORES
FLORES
MBPP
Acc.%
Acc.%
Acc.%
BLEU
chrF
pass@1
Base Model
53.65
88.99
85.14
29.85
58.85
66.80
LoRA r=64
54.38 ± 0.33
88.22 ± 0.19
86.30 ± 0.24
30.06 ± 0.29
58.56 ± 0.18
64.07 ± 0.34
LoRA r=32
57.10 ± 0.27
88.14 ± 0.14
86.13 ± 0.12
30.17 ± 0.10
58.97 ± 0.01
65.13 ± 0.09
Adapter
55.49
87.97
84.61
29.23
58.82
56.40
Table 2: MedMCQA domain adaptation results on Qwen3-4B. LoRA and Engram Adapter results are reported as mean ± std across three random seeds (42, 7783, 114514). For Qwen3-4B, MedMCQA, ARC-C, AG-News, and FLORES are evaluated zero-shot; MBPP uses 3-shot prompting. Bold : best among adapted models; underline : second best. MedMCQA is the in-domain target task; the remaining four benchmarks measure out-of-domain retention.
AG → Med
Med → AG
Metric
All
Active
All
Active
Mean act.
28.26%
26.28%
Mean gate
0.599
0.640
0.656
0.690
Mean ∥Δu∥/∥u∥
0.078%
0.275%
0.076%
0.290%
P95 ∥Δu∥/∥u∥
0.371%
1.124%
0.348%
1.233%
Table 3: OOD residual perturbation analysis for the two highest-activation cross-domain pairs (seed 7783). All statistics are computed across all tokens and all four injected layers ( L={1,3,7,15} ). “Active” rows condition on tokens where the joint mask fired ( mn=1 for at least one order group).
Metric
AG → Med
Med → AG
Base accuracy
53.65%
85.14%
Adapted accuracy
53.60%
86.57%
Prediction changed
7.55%
2.16%
Negative flip rate
3.80%
0.36%
Positive flip rate
3.75%
1.79%
Mean KL to base
0.088
0.040
Table 4: Output distribution analysis for OOD cross-domain pairs (seed 7783). Prediction change rate measures the fraction of examples where the argmax prediction differs between the base and adapted models. Negative flips are examples that the base model got right but the adapted model got wrong; positive flips are the reverse.
In-Domain
OOD
OOD Avg.
Method
AG-News
MedMCQA
ARC-C
LegalBench
Ret.%
Acc.%
Acc.%
Acc.%
Avg. Acc.%
Base Model
79.49
55.51
89.77
71.91
100.00
LoRA r=64
90.29
57.92
90.53
44.67
89.1
LoRA r=64 (lr= 10−5 )
86.96
57.33
89.68
61.08
96.0
LoRA r=64 (lr= 10−6 )
82.18
53.60
89.52
68.54
97.2
Table 5: AG-News domain adaptation results on Qwen3-8B. All methods are trained on 10,000 AG-News examples with seed 7783. AG-News is the in-domain target task; MedMCQA, ARC-Challenge, and LegalBench measure out-of-domain retention. All reported 8B baselines use seed 7783, and all Qwen3-8B evaluations use zero-shot prompting. OOD Avg. Ret. averages relative retention over MedMCQA, ARC-Challenge, and LegalBench.
Figure 3: Adaptation–retention trade-off on Qwen3-8B. Each point represents a method trained on AG-News; the x -axis is in-domain accuracy and the y -axis is average OOD retention over MedMCQA, ARC-Challenge, and LegalBench (100% = base model). The Engram Adapter is the only adapted method that combines a substantial in-domain improvement with ≥ 100% average OOD retention. LoRA variants achieve higher in-domain accuracy in some settings but lower OOD retention, while TALR preserves more OOD performance at the cost of little in-domain improvement. All LoRA variants are r =64.
Table 6: Experimental setup summary. LoRA and Engram Adapter report mean ± std across three seeds on 4B; non-LoRA non-Engram 4B baselines and all reported 8B baselines use seed 7783. For Qwen3-4B, all non-MBPP evaluations use zero-shot prompting and MBPP uses 3-shot prompting; Qwen3-8B evaluations use zero-shot prompting.
Hyperparameter
Default
Description
inject_layers
(1,3,7,15)
Layer indices with adapters
ngram_orders
(2,3)
Suffix n -gram sizes
prev_ngram_orders
(3,)
Previous-context sizes
num_hashes
4
Hash functions per lookup
table_size
90,007
Buckets per hash table
per_hash_dim
128 / 256
Embedding dim per hash (4B / 8B)
Appendix
Table 7: Hyperparameter defaults.
Property
LoRA
Engram Adapter
Adapted path
Always-on ΔW=BA
Additive residual over frozen θ
Activation condition
Always on
Occupancy-conditioned n -gram matches
OOD behavior
Full ΔW applied
Gate-attenuated ( < 0.1% residual)
Retention mechanism
None (empirical only)
Gate + mask (empirical)
False positive control
N/A
Selectivity prior tunable via k and M
Composability
Requires merging / switching
Separate adapter module over frozen backbone
Appendix
Table 8: Structural comparison of LoRA and Engram Adapter.
Method
ARC-C
MedMCQA
FLORES (BLEU)
MBPP
Avg. Ret.
LoRA r=64
99.04 ± 0.55
100.12 ± 0.62
97.01 ± 1.48
84.83 ± 18.28
95.25 ± 4.95
LoRA r=32
99.01 ± 0.43
102.32 ± 0.50
98.98 ± 0.62
97.21 ± 1.25
99.38 ± 0.51
Adapter
98.75
99.55
100.20
83.83
95.58
DoRA r=32
99.04
101.73
100.77
89.52
97.77
PiSSA r=32
22.34
60.07
0.00
0.00
20.60
(IA)3
96.65
99.27
71.09
84.43
87.87
Appendix
Table 9: Average OOD retention (%) for AG-News domain adaptation (4B). LoRA and Engram Adapter results are mean ± std across three seeds. Higher is better; 100% = perfect preservation.
Method
ARC-C
AG-News
FLORES (BLEU)
MBPP
Avg. Ret.
LoRA r=64
99.14 ± 0.21
101.36 ± 0.28
100.69 ± 0.96
95.91 ± 0.51
99.28 ± 0.20
LoRA r=32
99.04 ± 0.16
101.16 ± 0.14
101.06 ± 0.34
97.50 ± 0.14
99.69 ± 0.06
Adapter
98.85
99.38
97.92
84.43
95.15
DoRA r=32
99.23
99.55
100.23
85.33
96.09
PiSSA r=32
30.68
0.00
0.00
0.00
7.67
(IA)3
70.66
99.34
70.10
86.83
81.74
Appendix
Table 10: Average OOD retention (%) for MedMCQA domain adaptation (4B). LoRA and Engram Adapter results are mean ± std across three seeds. Higher is better; 100% = perfect preservation.
Task
Base
Engram
LoRA64
lr= 10−5
lr= 10−6
TALR
abercrombie
36.84
40.00
36.84
23.16
37.89
32.63
consumer_contracts_qa
90.40
90.91
69.70
92.68
92.17
92.68
contract_nli_confid.
78.05
78.05
74.39
79.27
78.05
89.02
contract_nli_limited
82.69
83.17
65.87
81.25
83.17
81.25
contract_nli_return
81.82
81.82
75.76
83.33
81.82
87.88
contract_qa
97.50
97.50
97.50
97.50
97.50
98.75
Appendix
Table 11: LegalBench results on Qwen3-8B (trained on AG-News, evaluated OOD on 12 legal reasoning tasks). Values are task accuracies (%).
Figure 4: OOD perturbation cascade on the two highest-activation cross-domain pairs. Despite ≈ 27% token-level activation, the gate suppresses residual perturbation to < 0.08% of hidden-state norm, yielding minimal KL divergence and negligible accuracy change.
Signal
Med.
ARC-C
Legal
Engram n -gram density
0.60
0.17
0.27
TF-IDF max cosine
16.90
24.06
31.53
Qwen embedding retrieval
0.00
0.00
0.10
Appendix
Table 12: Matched-coverage OOD false-accept rate (%) for domain-gate signals. All thresholds are selected at the 5th percentile of held-out AG-News scores, corresponding to approximately 95% in-domain acceptance. Lower is better.
AG → MedMCQA
Med → AG-News
Type
Variant
Mean%
P95%
Mean%
P95%
—
Baseline
0.275
1.124
0.290
1.233
Gate
g=1 (disabled)
0.505 ↑
2.124 ↑
0.509 ↑
1.990 ↑
no_dot
0.273
1.086
0.286
1.105
Appendix
Table 13: Gate ablation on the two highest-activation OOD pairs. Metrics are mean and P95 residual ratio ( ∥Δu∥/∥u∥ , %) among activated tokens. ↑ indicates degradation (higher OOD perturbation); ↓ indicates improvement.
Method
Params
Train profile
Peak mem.
Base Model
0
—
—
LoRA r=64
132.1M
20.88s / 47.88 ex/s
23.2GB
LoRA r=32
66.1M
19.85s / 50.38 ex/s
22.4GB
Engram Adapter
154.0M
19.80s / 50.50 ex/s
17.5GB
Appendix
Table 14: Controlled supplementary efficiency profile on Qwen3-4B AG-News. Training numbers use an AG-News-1k one-epoch profile on a single NVIDIA RTX 6000D with batch size 8, cutoff length 1024, bf16, and AdamW.
Latency Overhead
Activation Rate
Adapter
Eval Set
Forward
Generation
n2
n3
p3
AG-News
AG-News (in-domain)
+12.3%
—
0.500
0.356
0.376
AG-News
MedMCQA
+39.9%
—
0.219
0.126
0.137
AG-News
ARC-C
+34.5%
—
0.194
0.052
0.062
AG-News
MBPP
+1.45%
+21.8%
0.397
0.221
0.221
MedMCQA
MedMCQA (in-domain)
+36.6%
—
0.717
0.601
0.610
Appendix
Table 15: Original benchmark-dependent inference overhead and n -gram activation profile on Qwen3-4B. Forward overhead measures mean prefill latency increase; generation overhead is reported only for MBPP (fixed 128-token output). Activation columns report the fraction of token positions where the joint mask fires for each n -gram channel. For in-domain rows (shaded), generation overhead is not reported because output-length differences confound the comparison. FLORES-200 forward overhead is omitted due to high variance across adapters ( SD=25.5% , n=2 ). These measurements were collected in the final L20 evaluation environment and should not be directly compared with the controlled RTX 6000D profile in Table 14 .
Adapting Large Language Models (LLMs) to specialized domains often incurs an alignment tax, as fine-tuning on domain-specific tasks can cause catastrophic forgetting and substantially degrade performance on general tasks. We propose MemSFT, which mitigates the alignment tax by decoupling domain specialization from backbone parameter updates through a plug-and-play parametric memory. The memory is trained to imitate the behavior of a non-parametric retriever operating over domain data, thereby memorizing knowledge and patterns that would otherwise be accessed through retrieval. Once trained on a specific domain, the memory can be reused across LLMs of different sizes. During generation, a learned router dynamically fuses the output distributions of the memory and backbone at each decoding step, allowing domain expertise to be invoked selectively. Across biology, geoscience, and law, evaluations with models ranging from Qwen3-8B to Qwen3-235B-A22B show that MemSFT consistently improves domain performance with negligible degradation in general performance, whereas full SFT suffers severe forgetting on general tasks. Overall, our results demonstrate a practical path to decoupling general model capabilities from domain-specific knowledge at the parameter level, thereby equipping LLMs with new specialized capabilities without compromising their general capabilities.
Jiarui Wang, Xiang Shi, Jiaqi Cao +8
LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai AI Laboratory · School of Computer Science, Shanghai Jiao Tong University +1
Adapting LLMs to new domains causes forgetting because standard methods (e.g., full fine-tuning, LoRA) inject new directions into the weight space. We show that forgetting is governed by one algebraic property: whether the update preserves the column span of the pretrained weight matrix (Proposition 1). We propose GAIN, the simplest multiplicative alternative (W_new = S * W), which satisfies this by construction and can be absorbed into existing weights for zero inference cost. Across five models (774M to 70B) adapted sequentially over eight domains, GAIN improves earlier-domain perplexity by 7-13%, while LoRA degrades it by 18-36%. GAIN matches replay-augmented LoRA without storing prior data and dominates EWC on the forgetting-adaptation Pareto front. While LoRA can only reduce forgetting by sacrificing in-domain adaptation, GAIN achieves both with no domain boundaries and no regularization. The principle generalises: (IA)^3, an independent multiplicative method, also improves earlier domains.
Hengshuai Yao, Xing Chen, Ahmed Murtadha +1
Sapient Intelligence · Department of Computing Science, University of Alberta
Supervised fine-tuning (SFT) is the de facto standard for adapting large language models (LLMs) to target domains, but it often degrades the model's general capabilities, a phenomenon known as catastrophic forgetting. Existing approaches typically modify the SFT loss to mitigate forgetting, but they inevitably operate along a domain-generality trade-off. In this work, we step outside this trade-off by decoupling the two capabilities at the model level: we keep the original base model for general capability, and selectively invoke the SFT expert only when domain-specific knowledge is required. Specifically, we propose CPR (Critical-Point Routing), a token-level routing framework between a base model and its expert derivative, based on critical tokens where the base model fails but the expert succeeds. We train a lightweight hierarchical router that estimates the expert-call probability per token, and pair it with a tailored inference procedure that combines momentum smoothing and threshold gating. Across diverse model-domain configurations, CPR achieves state-of-the-art across all settings, surpassing SFT expert by 1.4-5.5% in domain performance while recovering its general-capability drop from 3.4-14.5% to at most 0.5%, with minimal overhead from invoking the expert on only one-third of tokens.