When to Adapt: Conditional Memory Adapters for Retention-Preserving Domain Specialization

Jiayu Hou; Lei Wang

cs.CLAug 29, 2026

When to Adapt: Conditional Memory Adapters for Retention-Preserving Domain Specialization

Authors: Jiayu Hou, Lei Wang

Organizations: University of Electronic Science and Technology of China, Chengdu, China

Abstract

Large language models deployed in specialized domains must improve in-domain performance without sacrificing general capabilities. Existing parameter-efficient fine-tuning methods are typically always on: their learned perturbations are applied to every input, which can degrade out-of-domain (OOD) performance. We propose Engram Adapter, a framework that repurposes pretraining-time conditional memory as a post-hoc adapter for frozen LLMs. It uses multi-channel matching over local n-gram patterns with explicit occupancy tracking as a lightweight selectivity prior, making residual injection more likely on in-domain inputs while a learned scalar gate suppresses incoherent OOD retrievals. We evaluate on Qwen3-4B and Qwen3-8B with AG-News and MedMCQA as adaptation tasks and OOD benchmarks spanning reasoning, translation, code generation, and legal reasoning. Engram Adapter improves in-domain accuracy while preserving 99.4%--100.1% of average OOD performance; on LegalBench it slightly exceeds the frozen base model on average, whereas comparable always-on baselines degrade sharply. Mechanistic analyses show that although OOD activations are non-zero, gate and projection attenuation reduce residuals to approximately 0.08% of hidden-state norm, yielding small KL drift and negligible accuracy change. These results suggest conditional activation is a promising route toward modular, retention-preserving domain specialization over frozen backbones.

Figures & tables

Figure 1: Conceptual comparison between always-on PEFT and conditional adaptation. LoRA-style methods expose learned low-rank weight updates to all inputs at inference time, while adapter-style methods expose an additional residual branch to all inputs. In contrast, our Engram Adapter conditionally exposes its residual through explicit local-pattern matching: domain-relevant inputs are more likely to activate the adapter, whereas out-of-domain inputs with no or weak matches receive masked or gate-attenuated residuals.

Figure 2: Engram Adapter overview. Four self-attention layers ( $\mathcal{L}=\{1,3,7,15\}$ ) receive hook-based adapters sharing hash memory tables and occupancy masks. Input $n$ -grams retrieve masked memory, which is projected and gated before residual injection; only adapter parameters are trainable.

	In-Domain	Out-of-Domain (General Capabilities)
Method	AG-News	ARC-C	MedMCQA	FLORES	FLORES	MBPP
	Acc.%	Acc.%	Acc.%	BLEU	chrF	pass@1
Base Model	85.14	88.99	53.65	29.85	58.85	66.80
LoRA $r\!=\!64$	89.89 $\pm$ 0.50	88.14 $\pm$ 0.49	53.72 $\pm$ 0.33	28.96 $\pm$ 0.44	58.13 $\pm$ 0.18	56.67 $\pm$ 12.21
LoRA $r\!=\!32$	90.15 $\pm$ 0.14	88.11 $\pm$ 0.38	54.90 $\pm$ 0.27	29.55 $\pm$ 0.19	58.61 $\pm$ 0.06	64.93 $\pm$ 0.84
Adapter	90.54	87.88	53.41	29.91	58.56	56.00

Table 1: AG-News domain adaptation results on Qwen3-4B. LoRA and Engram Adapter results are reported as mean

\pm

std across three random seeds (42, 7783, 114514). For Qwen3-4B, AG-News, ARC-C, MedMCQA, and FLORES are evaluated zero-shot; MBPP uses 3-shot prompting. Bold : best among adapted models; underline : second best. AG-News is the in-domain target task; the remaining four benchmarks measure out-of-domain retention.

	In-Domain	Out-of-Domain (General Capabilities)
Method	MedMCQA	ARC-C	AG-News	FLORES	FLORES	MBPP
	Acc.%	Acc.%	Acc.%	BLEU	chrF	pass@1
Base Model	53.65	88.99	85.14	29.85	58.85	66.80
LoRA $r\!=\!64$	54.38 $\pm$ 0.33	88.22 $\pm$ 0.19	86.30 $\pm$ 0.24	30.06 $\pm$ 0.29	58.56 $\pm$ 0.18	64.07 $\pm$ 0.34
LoRA $r\!=\!32$	57.10 $\pm$ 0.27	88.14 $\pm$ 0.14	86.13 $\pm$ 0.12	30.17 $\pm$ 0.10	58.97 $\pm$ 0.01	65.13 $\pm$ 0.09
Adapter	55.49	87.97	84.61	29.23	58.82	56.40

Table 2: MedMCQA domain adaptation results on Qwen3-4B. LoRA and Engram Adapter results are reported as mean

\pm

std across three random seeds (42, 7783, 114514). For Qwen3-4B, MedMCQA, ARC-C, AG-News, and FLORES are evaluated zero-shot; MBPP uses 3-shot prompting. Bold : best among adapted models; underline : second best. MedMCQA is the in-domain target task; the remaining four benchmarks measure out-of-domain retention.

	AG $\to$ Med		Med $\to$ AG
Metric	All	Active	All	Active
Mean act.	28.26%		26.28%
Mean gate	0.599	0.640	0.656	0.690
Mean $\\|\Delta u\\|/\\|u\\|$	0.078%	0.275%	0.076%	0.290%
P95 $\\|\Delta u\\|/\\|u\\|$	0.371%	1.124%	0.348%	1.233%

Table 3: OOD residual perturbation analysis for the two highest-activation cross-domain pairs (seed 7783). All statistics are computed across all tokens and all four injected layers (

\mathcal{L}=\{1,3,7,15\}

). “Active” rows condition on tokens where the joint mask fired (

m_{n}=1

for at least one order group).

Metric	AG $\to$ Med	Med $\to$ AG
Base accuracy	53.65%	85.14%
Adapted accuracy	53.60%	86.57%
Prediction changed	7.55%	2.16%
Negative flip rate	3.80%	0.36%
Positive flip rate	3.75%	1.79%
Mean KL to base	0.088	0.040

Table 4: Output distribution analysis for OOD cross-domain pairs (seed 7783). Prediction change rate measures the fraction of examples where the argmax prediction differs between the base and adapted models. Negative flips are examples that the base model got right but the adapted model got wrong; positive flips are the reverse.

	In-Domain	OOD			OOD Avg.
Method	AG-News	MedMCQA	ARC-C	LegalBench	Ret.%
	Acc.%	Acc.%	Acc.%	Avg. Acc.%
Base Model	79.49	55.51	89.77	71.91	100.00
LoRA $r\!=\!64$	90.29	57.92	90.53	44.67	89.1
LoRA $r\!=\!64$ (lr= $10^{-5}$ )	86.96	57.33	89.68	61.08	96.0
LoRA $r\!=\!64$ (lr= $10^{-6}$ )	82.18	53.60	89.52	68.54	97.2

Table 5: AG-News domain adaptation results on Qwen3-8B. All methods are trained on 10,000 AG-News examples with seed 7783. AG-News is the in-domain target task; MedMCQA, ARC-Challenge, and LegalBench measure out-of-domain retention. All reported 8B baselines use seed 7783, and all Qwen3-8B evaluations use zero-shot prompting. OOD Avg. Ret. averages relative retention over MedMCQA, ARC-Challenge, and LegalBench.

Figure 3: Adaptation–retention trade-off on Qwen3-8B. Each point represents a method trained on AG-News; the $x$ -axis is in-domain accuracy and the $y$ -axis is average OOD retention over MedMCQA, ARC-Challenge, and LegalBench (100% = base model). The Engram Adapter is the only adapted method that combines a substantial in-domain improvement with $\geq$ 100% average OOD retention. LoRA variants achieve higher in-domain accuracy in some settings but lower OOD retention, while TALR preserves more OOD performance at the cost of little in-domain improvement. All LoRA variants are $r$ =64.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Train task	Train size	Eval	OOD evaluation	Baselines
Qwen3-4B	AG-News	1,000	0-shot; MBPP 3-shot	ARC-C, MedMCQA, FLORES, MBPP	LoRA $r\in\{32,64\}$ (3 seeds), DoRA, PiSSA, Adapter, $(IA)^{3}$
Qwen3-4B	MedMCQA	1,000	0-shot; MBPP 3-shot	ARC-C, AG-News, FLORES, MBPP	LoRA $r\in\{32,64\}$ (3 seeds), DoRA, PiSSA, Adapter, $(IA)^{3}$
Qwen3-8B	AG-News	10,000	zero-shot	MedMCQA, ARC-C, LegalBench	LoRA $r=64$ , LoRA lr= $10^{-5}$ , LoRA lr= $10^{-6}$ , TALR

Appendix

Table 6: Experimental setup summary. LoRA and Engram Adapter report mean

\pm

std across three seeds on 4B; non-LoRA non-Engram 4B baselines and all reported 8B baselines use seed 7783. For Qwen3-4B, all non-MBPP evaluations use zero-shot prompting and MBPP uses 3-shot prompting; Qwen3-8B evaluations use zero-shot prompting.

Hyperparameter	Default	Description
inject_layers	$(1,3,7,15)$	Layer indices with adapters
ngram_orders	$(2,3)$	Suffix $n$ -gram sizes
prev_ngram_orders	$(3,)$	Previous-context sizes
num_hashes	$4$	Hash functions per lookup
table_size	$90{,}007$	Buckets per hash table
per_hash_dim	$128$ / $256$	Embedding dim per hash (4B / 8B)

Appendix

Table 7: Hyperparameter defaults.

Property	LoRA	Engram Adapter
Adapted path	Always-on $\Delta W=BA$	Additive residual over frozen $\theta$
Activation condition	Always on	Occupancy-conditioned $n$ -gram matches
OOD behavior	Full $\Delta W$ applied	Gate-attenuated ( $<$ 0.1% residual)
Retention mechanism	None (empirical only)	Gate + mask (empirical)
False positive control	N/A	Selectivity prior tunable via $k$ and $M$
Composability	Requires merging / switching	Separate adapter module over frozen backbone

Appendix

Table 8: Structural comparison of LoRA and Engram Adapter.

Method	ARC-C	MedMCQA	FLORES (BLEU)	MBPP	Avg. Ret.
LoRA $r\!=\!64$	99.04 $\pm$ 0.55	100.12 $\pm$ 0.62	97.01 $\pm$ 1.48	84.83 $\pm$ 18.28	95.25 $\pm$ 4.95
LoRA $r\!=\!32$	99.01 $\pm$ 0.43	102.32 $\pm$ 0.50	98.98 $\pm$ 0.62	97.21 $\pm$ 1.25	99.38 $\pm$ 0.51
Adapter	98.75	99.55	100.20	83.83	95.58
DoRA $r\!=\!32$	99.04	101.73	100.77	89.52	97.77
PiSSA $r\!=\!32$	22.34	60.07	0.00	0.00	20.60
$(IA)^{3}$	96.65	99.27	71.09	84.43	87.87

Appendix

Table 9: Average OOD retention (%) for AG-News domain adaptation (4B). LoRA and Engram Adapter results are mean

\pm

std across three seeds. Higher is better; 100% = perfect preservation.

Method	ARC-C	AG-News	FLORES (BLEU)	MBPP	Avg. Ret.
LoRA $r\!=\!64$	99.14 $\pm$ 0.21	101.36 $\pm$ 0.28	100.69 $\pm$ 0.96	95.91 $\pm$ 0.51	99.28 $\pm$ 0.20
LoRA $r\!=\!32$	99.04 $\pm$ 0.16	101.16 $\pm$ 0.14	101.06 $\pm$ 0.34	97.50 $\pm$ 0.14	99.69 $\pm$ 0.06
Adapter	98.85	99.38	97.92	84.43	95.15
DoRA $r\!=\!32$	99.23	99.55	100.23	85.33	96.09
PiSSA $r\!=\!32$	30.68	0.00	0.00	0.00	7.67
$(IA)^{3}$	70.66	99.34	70.10	86.83	81.74

Appendix

Table 10: Average OOD retention (%) for MedMCQA domain adaptation (4B). LoRA and Engram Adapter results are mean

\pm

std across three seeds. Higher is better; 100% = perfect preservation.

Task	Base	Engram	LoRA64	lr= $10^{-5}$	lr= $10^{-6}$	TALR
abercrombie	36.84	40.00	36.84	23.16	37.89	32.63
consumer_contracts_qa	90.40	90.91	69.70	92.68	92.17	92.68
contract_nli_confid.	78.05	78.05	74.39	79.27	78.05	89.02
contract_nli_limited	82.69	83.17	65.87	81.25	83.17	81.25
contract_nli_return	81.82	81.82	75.76	83.33	81.82	87.88
contract_qa	97.50	97.50	97.50	97.50	97.50	98.75

Appendix

Table 11: LegalBench results on Qwen3-8B (trained on AG-News, evaluated OOD on 12 legal reasoning tasks). Values are task accuracies (%).

Figure 4: OOD perturbation cascade on the two highest-activation cross-domain pairs. Despite

\approx

27% token-level activation, the gate suppresses residual perturbation to

<

0.08% of hidden-state norm, yielding minimal KL divergence and negligible accuracy change.

Signal	Med.	ARC-C	Legal
Engram $n$ -gram density	0.60	0.17	0.27
TF-IDF max cosine	16.90	24.06	31.53
Qwen embedding retrieval	0.00	0.00	0.10

Appendix

Table 12: Matched-coverage OOD false-accept rate (%) for domain-gate signals. All thresholds are selected at the 5th percentile of held-out AG-News scores, corresponding to approximately 95% in-domain acceptance. Lower is better.

		AG $\to$ MedMCQA		Med $\to$ AG-News
Type	Variant	Mean%	P95%	Mean%	P95%
—	Baseline	0.275	1.124	0.290	1.233
Gate	$g=1$ (disabled)	0.505 $\uparrow$	2.124 $\uparrow$	0.509 $\uparrow$	1.990 $\uparrow$
Gate	no_dot	0.273	1.086	0.286	1.105

Appendix

Table 13: Gate ablation on the two highest-activation OOD pairs. Metrics are mean and P95 residual ratio (

\|\Delta u\|/\|u\|

, %) among activated tokens.

\uparrow

indicates degradation (higher OOD perturbation);

\downarrow

indicates improvement.

Method	Params	Train profile	Peak mem.
Base Model	0	—	—
LoRA $r\!=\!64$	132.1M	20.88s / 47.88 ex/s	23.2GB
LoRA $r\!=\!32$	66.1M	19.85s / 50.38 ex/s	22.4GB
Engram Adapter	154.0M	19.80s / 50.50 ex/s	17.5GB

Appendix

Table 14: Controlled supplementary efficiency profile on Qwen3-4B AG-News. Training numbers use an AG-News-1k one-epoch profile on a single NVIDIA RTX 6000D with batch size 8, cutoff length 1024, bf16, and AdamW.

		Latency Overhead		Activation Rate
Adapter	Eval Set	Forward	Generation	n2	n3	p3
AG-News	AG-News (in-domain)	+12.3%	—	0.500	0.356	0.376
AG-News	MedMCQA	+39.9%	—	0.219	0.126	0.137
AG-News	ARC-C	+34.5%	—	0.194	0.052	0.062
AG-News	MBPP	+1.45%	+21.8%	0.397	0.221	0.221
MedMCQA	MedMCQA (in-domain)	+36.6%	—	0.717	0.601	0.610

Appendix

Table 15: Original benchmark-dependent inference overhead and

n

-gram activation profile on Qwen3-4B. Forward overhead measures mean prefill latency increase; generation overhead is reported only for MBPP (fixed 128-token output). Activation columns report the fraction of token positions where the joint mask fires for each

n

-gram channel. For in-domain rows (shaded), generation overhead is not reported because output-length differences confound the comparison. FLORES-200 forward overhead is omitted due to high variance across adapters (

\mathrm{SD}=25.5\%

n=2

). These measurements were collected in the final L20 evaluation environment and should not be directly compared with the controlled RTX 6000D profile in Table 14 .

Domain Adaptation for LLMs

Domain adaptation for large language models tailors their knowledge and behavior to specialized fields, languages, and tasks.

See all 93 papers

MemSFT: Mitigating Alignment Tax with an External Parametric Memory
Jul 28, 2026Jiarui Wang, Xiang Shi, Jiaqi Cao +8LLM Alignment Memory-Augmented Language Models
LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai AI Laboratory · School of Computer Science, Shanghai Jiao Tong University +1
GAIN: Multiplicative Modulation for Domain Adaptation
Apr 6, 2026Hengshuai Yao, Xing Chen, Ahmed Murtadha +1Domain Adaptation for LLMs Continual Learning for LLMs
Sapient Intelligence · Department of Computing Science, University of Alberta
CPR for LLMs: Critical-Point Routing against Catastrophic Forgetting in Domain Adaptation
Aug 31, 2026Kwangmin Ki, Yunhun Nam, Jongheon Jeong +1LLM Fine-Tuning Domain Adaptation for LLMs
Korea University · Yonsei University

Jul 28, 2026J/K move · Enter open · S save

MemSFT: Mitigating Alignment Tax with an External Parametric Memory

Jiarui Wang, Xiang Shi, Jiaqi Cao, Rubin Wei+7

LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai AI Laboratory · School of Computer Science, Shanghai Jiao Tong University+1

LLM Alignment Memory-Augmented Language Models Memorization in Language Models Domain Adaptation LLM Fine-Tuning Domain Adaptation for LLMs

Open paper AskPDF arXiv

Adapting Large Language Models (LLMs) to specialized domains often incurs an alignment tax, as fine-tuning on domain-specific tasks can cause catastrophic forgetting and substantially degrade performance on general tasks. We propose MemSFT, which mitigates the alignment tax by decoupling domain specialization from backbone parameter updates through a plug-and-play parametric memory. The memory is trained to imitate the behavior of a non-parametric retriever operating over domain data, thereby memorizing knowledge and patterns that would otherwise be accessed through retrieval. Once trained on a specific domain, the memory can be reused across LLMs of different sizes. During generation, a learned router dynamically fuses the output distributions of the memory and backbone at each decoding step, allowing domain expertise to be invoked selectively. Across biology, geoscience, and law, evaluations with models ranging from Qwen3-8B to Qwen3-235B-A22B show that MemSFT consistently improves domain performance with negligible degradation in general performance, whereas full SFT suffers severe forgetting on general tasks. Overall, our results demonstrate a practical path to decoupling general model capabilities from domain-specific knowledge at the parameter level, thereby equipping LLMs with new specialized capabilities without compromising their general capabilities.

Abstract

Domain Adaptation for LLMs

Explore similar work

MemSFT: Mitigating Alignment Tax with an External Parametric Memory

GAIN: Multiplicative Modulation for Domain Adaptation

CPR for LLMs: Critical-Point Routing against Catastrophic Forgetting in Domain Adaptation