cs.CLApr 1, 2026

Brainstacks: Cross-Domain Cognitive Capabilities via Frozen MoE-LoRA Stacks for Continual LLM Learning

Authors: Mohammad R. Abu Ayyash

Organizations: Brains Build Research, Ramallah, Palestine

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We present Brainstacks, a modular architecture for continual multi-domain fine-tuning of large language models that packages domain expertise as frozen adapter stacks composing additively on a shared frozen base at inference. Five interlocking components: (1) MoE-LoRA with Shazeer-style noisy top-2 routing across all seven transformer projections under QLoRA 4-bit quantization with rsLoRA scaling; (2) an inner loop performing residual boosting by freezing trained stacks and adding new ones; (3) an outer loop training sequential domain-specific stacks with curriculum-ordered dependencies; (4) null-space projection via randomized SVD constraining new stacks to subspaces orthogonal to prior directions, achieving zero forgetting in isolation; (5) an outcome-based sigmoid meta-router trained on empirically discovered domain-combination targets that selectively weights stacks, enabling cross-domain composition. Two boundary experiments: (6) PSN pretraining on a randomly initialized model; (7) per-domain RL (DPO/GRPO) validating compatibility with post-SFT alignment. Validated on TinyLlama-1.1B (4 domains, 9 stacks) and Gemma 3 12B IT (5 domains, 10 stacks), MoE-LoRA achieves 2.5x faster convergence than parameter-matched single LoRA, residual boosting breaks through the single-stack ceiling, and the routed system recovers generation quality destroyed by ungated stack accumulation. The central finding: the outcome-based router discovers that domain stacks encode transferable cognitive primitives (instruction-following clarity, numerical reasoning, procedural logic, chain-of-thought structure) rather than domain-specific knowledge, with medical prompts routing to chat+math stacks in 97% of cases despite zero medical data in those stacks.

Figures & tables

Figure 1
Figure 1: Domain subspace directions at layer 24 ( q_proj ). Each arrow is a top-3 principal direction extracted via SVD. Domains point in distinct orientations, confirming the null-space projection forces orthogonal subspace separation. Math (red) and reasoning (orange) share partial overlap (cosine similarity 0.54), consistent with shared training data sources.
MethodParams (M)Train LossVal LossTime (min)
Single LoRA ( r=64r{=}64 )50.50.9320.8749.5
MoE-LoRA ( 4×r=164\times r{=}16 )53.67.4440.87220.2
Table 1: MoE-LoRA vs Single LoRA on TinyLlama-1.1B (tatsu-lab/alpaca).
Figure 2: Validation loss curves for Single LoRA (rank 64) vs MoE-LoRA (4 experts, rank 16). MoE-LoRA converges faster per step despite 2x wall-clock overhead.
Figure 3: Single LoRA vs Brainstacks residual boosting on chat domain. Single LoRA plateaus at 0.8741 after 400 steps. Brainstacks breaks through this ceiling via frozen stacked residuals, reaching 0.8531 after 3 rounds (1200 cumulative steps)—a 2.4% relative improvement. Each round adds a new 53M-parameter stack that learns what the previous frozen stacks left uncaptured.
DomainStacksVal Losses (per round)Final LossTime (min)
Chat32.587, 1.305, 1.3030.85349.2
Code20.953, 0.505, 0.4930.49397.5
Medical21.526, 0.671, 0.6630.663136.3
Math21.627, 0.695, 0.6960.696202.5
Table 2: TinyLlama 4-domain training results. 9 stacks across 4 domains.
Figure 4: Ungated interference matrix (with null-space projection). Each row shows all-domain validation losses after training through that domain. Gray cells indicate domains not yet trained. Chat degrades from 0.853 to 1.778 as stacks accumulate; medical suffers the worst interference at 1.984. These elevated values are not forgetting—frozen weights are unchanged—but magnitude accumulation from ungated cross-stack interference.
Figure 5: Ungated interference matrix (without null-space projection). Compared to Figure 4 , all final-stage losses are higher: chat 1.839 (vs 1.778), code 1.512 (vs 1.452), medical 2.066 (vs 1.984), math 1.184 (vs 1.041). Without null-space protection, interference accumulates faster across domains.
EvalAfter CodeAfter MedicalAfter Math
Domain(NoNS →\to NS)(NoNS →\to NS)(NoNS →\to NS)
Chat1.507→1.4771.507\to 1.4771.468→1.4531.468\to 1.4531.839→1.7781.839\to 1.778
Code0.788→0.8190.788\to 0.8190.814→0.8160.814\to 0.8161.512→1.4521.512\to 1.452
Medical—1.350→1.2341.350\to 1.2342.066→1.9842.066\to 1.984
Math——1.184→1.0411.184\to 1.041
Table 3: Progressive forgetting matrix: NoNS →\to NS comparison.
Figure 6: Null-space projection effect on TinyLlama (4 domains). Left: per-domain loss difference (with NS minus without NS); green = NS reduced interference, red = NS cost. Right: chat domain val loss over sequential training. Null space consistently protects prior domains, with the gap widening as more domains stack, after math, NS saves 0.061 on chat (1.778 vs 1.839).
Figure 7: Expert activation heatmap across domains. Approximately uniform utilization confirms load-balance loss effectiveness.
Figure 8: Cross-domain cosine similarity of principal subspace directions. Low values indicate effective orthogonal separation.
Figure 9: Singular value spectrum per domain stack, showing the concentration of information in top directions.
Figure 10: Per-layer orthogonality between domain stacks across transformer layers.
BenchmarkBaseRoutedDelta
HellaSwag0.6700.650−- 0.020
ARC-Easy0.5100.515++ 0.005
ARC-Challenge0.5250.495−- 0.030
TruthfulQA0.3500.370++ 0.020
MMLU0.4500.435−- 0.015
GSM8K0.6650.6650.000
Table 4: Zero-shot benchmark results on Gemma 3 12B IT (200 samples each).
Figure 11: Zero-shot benchmark comparison on Gemma 3 12B IT (200 samples each). Base model (blue) vs Brainstacks routed with meta-router selective gating (green). The routed system maintains competitive performance across all benchmarks, with no catastrophic degradation from stack accumulation. Differences at this sample size fall within sampling noise.
More in this topic

Continual Learning for LLMs

Continual learning for large language models studies how models absorb new knowledge and skills while retaining previously learned capabilities.

See all 231 papers

Explore similar work

CardsList

  1. Learning to Select, Not Relearn: Hard-Routed Mixtures of Reasoning LoRAs

    Jun 30, 2026Seyed Alireza Molavi, Zhan Su, Yan Hu +3Mixture-of-Experts Language ModelsLow-Rank Adaptation

    1Halmstad University, Halmstad, Sweden · 2The Chinese University of Hong Kong, Shenzhen, China

  2. GAIN: Multiplicative Modulation for Domain Adaptation

    Apr 6, 2026Hengshuai Yao, Xing Chen, Ahmed Murtadha +1Domain Adaptation for LLMsContinual Learning for LLMs

    Sapient Intelligence · Department of Computing Science, University of Alberta

  3. Train Separately, Merge Together: Modular Post-Training with Mixture-of-Experts

    Apr 20, 2026Jacob Morrison, Sanjay Adhikesaven, Akshita Bhagia +3Continual Learning for LLMsMixture-of-Experts Language Models

    University of Washington · Allen Institute for AI · University of California, Berkeley