Agents unlock new capabilities through Switching LoRA Adapters as a Tool (SLAaaT)

Kenneth Ge

cs.LGAug 17, 2026

Agents unlock new capabilities through Switching LoRA Adapters as a Tool (SLAaaT)

Authors: Kenneth Ge

Organizations: Independent San Francisco, CA, United States of America

Abstract

Post-training can unlock new capabilities and improve performance on specialized tasks, but sometimes at the cost of catastrophic forgetting in other domains. This poses a problem in long agent trajectories that compose different capabilities. We reject this tradeoff by giving an agent a tool to switch between specialized LoRA adapters mid-trace. To test its effectiveness, we compose two synthetic coding tasks that are logically simple but require specialization. We find that this allows the model to solve problems it previously could not, that the model is able to switch autonomously (and find a new strategy that beats our human heuristic baseline on one task), and that this incurs an up to an 18x reduction in capability tax compared to an agent using only one specialized adapter. Our approach also substantially outperforms spawning subagents in both task capabilities (solving 4 of our hardest tasks versus none) and token usage (46.1x fewer tokens in some scenarios).

Figures & tables

Figure 1: The agent switches its own LoRA adapter mid-trajectory, activating the right specialist for each subtask.

Condition	json–python (control)	fauxjson–python	json–fauxthon	keystone (both)	Total /200
base	42	0	22	0	64
fauxjson (pinned)	0	0	0	0	0
fauxthon (pinned)	19	0	11	0	30
fused	0	0	0	0	0
arrow (token-level)	41	0	2	0	43
subagent	38	2	19	0	59

Table 1: Task passes (out of 50 per cell) across the four notation bindings. Shading scales with pass rate. Rows group into single-configuration arms, routed arms over the fixed {fauxjson, fauxthon} library, and the frontier reference.

Figure 2: Retention of general capabilities. Pass rate on random frozen subsets of IFEval, HumanEval+, MATH-500, and BFCL-lite for each condition; the black bar marks the base model (equivalently, the human heuristic). The fused and fauxthon adapters pay a large capability tax, most severely on HumanEval+ (3.7% and 7.3%), whereas the routed conditions (autoswitch, subagent, arrow) stay close to base. This is the “capability tax” referred to throughout Section 3 .

Subtask	base	fauxjson	fauxthon	fused
fauxjson translation /50	0	23	0	44
fauxthon program tests /500	0	0	290	360

Table 2: Testing our four single-configuration conditions on just fauxjson and fauxthon tasks outside of agentic tool calling confounds. Shading scales with score (higher is better).

Condition	passed: mean	passed: median	failed: mean	failed: median
autoswitch (116 passes)	1,793	834	2,138	797
subagent (59 passes)	5,242	1,071	39,407	36,731

Table 3: Output tokens per task attempt. Shading scales with token count (more tokens is worse).

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Base model	Qwen3.6-35B-A3B
LoRA rank	16
Learning rate	$1\times 10^{-4}$ , 3% linear warmup, cosine decay to 0
Optimizer	Adam
Batch size	128 conversations
Epochs	1
Fauxjson training rows	20,000

Appendix

Table 4: LoRA training hyperparameters.

Long-Horizon LLM Agents

Long-horizon large language model agents plan, use tools, and preserve task state across extended, multi-step interactions.

See all 446 papers

WeAct: Dynamic Weight Switching as Agent Actions
Oct 19, 2026Rui Shi, Xunjian Yin, Jiacheng Sang +2LLM Agent Training Continual Learning for LLM Agents
Carnegie Mellon University · City University of Hong Kong · Duke University +1
New LoRA Skills Should Read but Never Write
Sep 25, 2026Zeyan Li, Panqi Yang, Qirong Guo +5Low-Rank Adaptation Continual Learning for LLMs
Shanghai Jiao Tong University · Xi’an Jiaotong University · The Hong Kong University of Science and Technology (Guangzhou) +1
Skill-to-LoRA: From Using Skills to Learning Behaviors for Token-Efficient LLM Agents
Jun 15, 2026Tianyi Zhang, Zhonghao QiAI Coding Agents LLM Agent Skill Learning
1The Chinese University of Hong Kong

Oct 19, 2026J/K move · Enter open · S save

WeAct: Dynamic Weight Switching as Agent Actions

Rui Shi, Xunjian Yin, Jiacheng Sang, Zichuan Liu+1

Carnegie Mellon University · City University of Hong Kong · Duke University+1

LLM Agent Training Continual Learning for LLM Agents Continual LoRA Low-Rank Adaptation

Open paper AskPDF arXiv

Long-horizon agents must learn heterogeneous capabilities from interleaved trajectories. Updating shared parameters to learn new capabilities can degrade previously acquired ones, resulting in catastrophic forgetting. As a trajectory unfolds, different stages may require different capabilities, yet conventional joint training and static adapter merging maintain a fixed parameter configuration throughout execution. We introduce WeAct (Weight-level Internal Actions), which trains independent capability-specific LoRA adapters over a frozen backbone and adds adapter selection to the agent's action space. These internal actions use the model's native tool-calling interface for on-demand capability composition in a shared context. Across eight deep-research benchmarks, Qwen3-32B with WeAct improves average accuracy by 6.2% relative to full fine-tuning counterparts despite using less compute, and by 27.5% relative to the strongest static LoRA-merging method. WeAct also supports modular continual adaptation, by adding a new code computation adapter to expand the agent's capabilities and updating the existing writing adapter to improve report quality. Both updates leave the backbone and other adapters unchanged, supporting modular continual adaptation in agents.

Abstract

Long-Horizon LLM Agents

Explore similar work

WeAct: Dynamic Weight Switching as Agent Actions

New LoRA Skills Should Read but Never Write

Skill-to-LoRA: From Using Skills to Learning Behaviors for Token-Efficient LLM Agents