Brainstacks: Cross-Domain Cognitive Capabilities via Frozen MoE-LoRA Stacks for Continual LLM Learning

Mohammad R. Abu Ayyash

cs.CLApr 1, 2026

Brainstacks: Cross-Domain Cognitive Capabilities via Frozen MoE-LoRA Stacks for Continual LLM Learning

Authors: Mohammad R. Abu Ayyash

Organizations: Brains Build Research, Ramallah, Palestine

Abstract

We present Brainstacks, a modular architecture for continual multi-domain fine-tuning of large language models that packages domain expertise as frozen adapter stacks composing additively on a shared frozen base at inference. Five interlocking components: (1) MoE-LoRA with Shazeer-style noisy top-2 routing across all seven transformer projections under QLoRA 4-bit quantization with rsLoRA scaling; (2) an inner loop performing residual boosting by freezing trained stacks and adding new ones; (3) an outer loop training sequential domain-specific stacks with curriculum-ordered dependencies; (4) null-space projection via randomized SVD constraining new stacks to subspaces orthogonal to prior directions, achieving zero forgetting in isolation; (5) an outcome-based sigmoid meta-router trained on empirically discovered domain-combination targets that selectively weights stacks, enabling cross-domain composition. Two boundary experiments: (6) PSN pretraining on a randomly initialized model; (7) per-domain RL (DPO/GRPO) validating compatibility with post-SFT alignment. Validated on TinyLlama-1.1B (4 domains, 9 stacks) and Gemma 3 12B IT (5 domains, 10 stacks), MoE-LoRA achieves 2.5x faster convergence than parameter-matched single LoRA, residual boosting breaks through the single-stack ceiling, and the routed system recovers generation quality destroyed by ungated stack accumulation. The central finding: the outcome-based router discovers that domain stacks encode transferable cognitive primitives (instruction-following clarity, numerical reasoning, procedural logic, chain-of-thought structure) rather than domain-specific knowledge, with medical prompts routing to chat+math stacks in 97% of cases despite zero medical data in those stacks.

Figures & tables

Method	Params (M)	Train Loss	Val Loss	Time (min)
Single LoRA ( $r{=}64$ )	50.5	0.932	0.874	9.5
MoE-LoRA ( $4\times r{=}16$ )	53.6	7.444	0.872	20.2

Table 1: MoE-LoRA vs Single LoRA on TinyLlama-1.1B (tatsu-lab/alpaca).

Figure 2: Validation loss curves for Single LoRA (rank 64) vs MoE-LoRA (4 experts, rank 16). MoE-LoRA converges faster per step despite 2x wall-clock overhead.

Figure 3: Single LoRA vs Brainstacks residual boosting on chat domain. Single LoRA plateaus at 0.8741 after 400 steps. Brainstacks breaks through this ceiling via frozen stacked residuals, reaching 0.8531 after 3 rounds (1200 cumulative steps)—a 2.4% relative improvement. Each round adds a new 53M-parameter stack that learns what the previous frozen stacks left uncaptured.

Domain	Stacks	Val Losses (per round)	Final Loss	Time (min)
Chat	3	2.587, 1.305, 1.303	0.853	49.2
Code	2	0.953, 0.505, 0.493	0.493	97.5
Medical	2	1.526, 0.671, 0.663	0.663	136.3
Math	2	1.627, 0.695, 0.696	0.696	202.5

Table 2: TinyLlama 4-domain training results. 9 stacks across 4 domains.

Figure 4: Ungated interference matrix (with null-space projection). Each row shows all-domain validation losses after training through that domain. Gray cells indicate domains not yet trained. Chat degrades from 0.853 to 1.778 as stacks accumulate; medical suffers the worst interference at 1.984. These elevated values are not forgetting—frozen weights are unchanged—but magnitude accumulation from ungated cross-stack interference.

Figure 5: Ungated interference matrix (without null-space projection). Compared to Figure 4 , all final-stage losses are higher: chat 1.839 (vs 1.778), code 1.512 (vs 1.452), medical 2.066 (vs 1.984), math 1.184 (vs 1.041). Without null-space protection, interference accumulates faster across domains.

Eval	After Code	After Medical	After Math
Domain	(NoNS $\to$ NS)	(NoNS $\to$ NS)	(NoNS $\to$ NS)
Chat	$1.507\to 1.477$	$1.468\to 1.453$	$1.839\to 1.778$
Code	$0.788\to 0.819$	$0.814\to 0.816$	$1.512\to 1.452$
Medical	—	$1.350\to 1.234$	$2.066\to 1.984$
Math	—	—	$1.184\to 1.041$

Table 3: Progressive forgetting matrix: NoNS

\to

NS comparison.

Figure 6: Null-space projection effect on TinyLlama (4 domains). Left: per-domain loss difference (with NS minus without NS); green = NS reduced interference, red = NS cost. Right: chat domain val loss over sequential training. Null space consistently protects prior domains, with the gap widening as more domains stack, after math, NS saves 0.061 on chat (1.778 vs 1.839).

Figure 7: Expert activation heatmap across domains. Approximately uniform utilization confirms load-balance loss effectiveness.

Figure 8: Cross-domain cosine similarity of principal subspace directions. Low values indicate effective orthogonal separation.

Figure 9: Singular value spectrum per domain stack, showing the concentration of information in top directions.

Figure 10: Per-layer orthogonality between domain stacks across transformer layers.

Benchmark	Base	Routed	Delta
HellaSwag	0.670	0.650	$-$ 0.020
ARC-Easy	0.510	0.515	$+$ 0.005
ARC-Challenge	0.525	0.495	$-$ 0.030
TruthfulQA	0.350	0.370	$+$ 0.020
MMLU	0.450	0.435	$-$ 0.015
GSM8K	0.665	0.665	0.000

Table 4: Zero-shot benchmark results on Gemma 3 12B IT (200 samples each).

Figure 11: Zero-shot benchmark comparison on Gemma 3 12B IT (200 samples each). Base model (blue) vs Brainstacks routed with meta-router selective gating (green). The routed system maintains competitive performance across all benchmarks, with no catastrophic degradation from stack accumulation. Differences at this sample size fall within sampling noise.

Continual Learning for LLMs

Continual learning for large language models studies how models absorb new knowledge and skills while retaining previously learned capabilities.

See all 231 papers

Jun 30, 2026

Learning to Select, Not Relearn: Hard-Routed Mixtures of Reasoning LoRAs

Composing independently trained LoRA adapters into a single large language model is useful for multi-domain adaptation, especially when the original training data cannot be shared. A common approach is to use MoE-style routing over LoRA experts, but for frozen pretrained adapters, soft weighted combinations can change the unit-scale additive update under which each LoRA module was originally trained. We propose Hard-Routed MoR-LoRA, a two-stage framework for composing frozen reasoning LoRA experts through unit-scale hard selection. First, domain-specific LoRA adapters are trained independently using reinforcement learning from verifiable feedback to obtain reasoning experts. Then, all experts are frozen, reasoning traces are distilled from them, and only a lightweight shared router together with a small attention LoRA is trained for integration. The router selects exactly one expert per token using hard top-1 routing, while a straight-through estimator enables gradient-based training. Experiments across five benchmarks, multiple model scales, and additional model families show that Hard-Routed MoR-LoRA preserves expert behavior while requiring substantially fewer trainable parameters than soft-routing mixture baselines. Our analysis further shows that normalized soft mixtures often concentrate most routing mass on a single expert, suggesting that hard unit-scale routing provides a simple and efficient abstraction for frozen LoRA expert composition.

Seyed Alireza Molavi, Zhan Su, Yan Hu +3

1Halmstad University, Halmstad, Sweden · 2The Chinese University of Hong Kong, Shenzhen, China

Mixture-of-Experts Language Models Low-Rank Adaptation LLM Routing

Ask PDF

Apr 6, 2026

GAIN: Multiplicative Modulation for Domain Adaptation

Adapting LLMs to new domains causes forgetting because standard methods (e.g., full fine-tuning, LoRA) inject new directions into the weight space. We show that forgetting is governed by one algebraic property: whether the update preserves the column span of the pretrained weight matrix (Proposition 1). We propose GAIN, the simplest multiplicative alternative (W_new = S * W), which satisfies this by construction and can be absorbed into existing weights for zero inference cost. Across five models (774M to 70B) adapted sequentially over eight domains, GAIN improves earlier-domain perplexity by 7-13%, while LoRA degrades it by 18-36%. GAIN matches replay-augmented LoRA without storing prior data and dominates EWC on the forgetting-adaptation Pareto front. While LoRA can only reduce forgetting by sacrificing in-domain adaptation, GAIN achieves both with no domain boundaries and no regularization. The principle generalises: (IA)^3, an independent multiplicative method, also improves earlier domains.

Hengshuai Yao, Xing Chen, Ahmed Murtadha +1

Sapient Intelligence · Department of Computing Science, University of Alberta

Domain Adaptation for LLMs Continual Learning for LLMs Catastrophic Forgetting

Ask PDF

Apr 20, 2026

Train Separately, Merge Together: Modular Post-Training with Mixture-of-Experts

Extending a fully post-trained language model with new domain capabilities is fundamentally limited by monolithic training paradigms: retraining from scratch is expensive and scales poorly, while continued training often degrades existing capabilities. We present BAR (Branch-Adapt-Route), which trains independent domain experts, each through its own mid-training, supervised finetuning, and reinforcement learning pipeline, and composes them via a Mixture-of-Experts architecture with lightweight router training. Unlike retraining approaches that mix all domains and require full reprocessing for any update (with cost scaling quadratically), BAR enables updating individual experts independently with linear cost scaling and no degradation to existing domains. At the 7B scale, with experts for math, code, tool use, and safety, BAR achieves an overall score of 49.1 (averaged across 7 evaluation categories), matching or exceeding re-training baselines (47.8 without mid-training, 50.5 with). We further show that modular training provides a structural advantage: by isolating each domain, it avoids the catastrophic forgetting that occurs when late-stage RL degrades capabilities from earlier training stages, while significantly reducing the cost and complexity of updating or adding a domain. Together, these results suggest that decoupled, expert-based training is a scalable alternative to monolithic retraining for extending language models.

Jacob Morrison, Sanjay Adhikesaven, Akshita Bhagia +3

University of Washington · Allen Institute for AI · University of California, Berkeley

Continual Learning for LLMs Mixture-of-Experts Language Models Language Model Post-Training Fine-Tuning Language Models Mixture-of-Experts Fine-Tuning

Ask PDF

Abstract

Continual Learning for LLMs

Explore similar work

Learning to Select, Not Relearn: Hard-Routed Mixtures of Reasoning LoRAs

GAIN: Multiplicative Modulation for Domain Adaptation

Train Separately, Merge Together: Modular Post-Training with Mixture-of-Experts