WeAct: Dynamic Weight Switching as Agent Actions

Rui Shi; Xunjian Yin; Jiacheng Sang; Zichuan Liu; Shuyan Zhou

cs.LGOct 19, 2026

WeAct: Dynamic Weight Switching as Agent Actions

Authors: Rui Shi, Xunjian Yin, Jiacheng Sang, Zichuan Liu, Shuyan Zhou

Organizations: Carnegie Mellon University · City University of Hong Kong · Duke University · Meta MSL

Abstract

Long-horizon agents must learn heterogeneous capabilities from interleaved trajectories. Updating shared parameters to learn new capabilities can degrade previously acquired ones, resulting in catastrophic forgetting. As a trajectory unfolds, different stages may require different capabilities, yet conventional joint training and static adapter merging maintain a fixed parameter configuration throughout execution. We introduce WeAct (Weight-level Internal Actions), which trains independent capability-specific LoRA adapters over a frozen backbone and adds adapter selection to the agent's action space. These internal actions use the model's native tool-calling interface for on-demand capability composition in a shared context. Across eight deep-research benchmarks, Qwen3-32B with WeAct improves average accuracy by 6.2% relative to full fine-tuning counterparts despite using less compute, and by 27.5% relative to the strongest static LoRA-merging method. WeAct also supports modular continual adaptation, by adding a new code computation adapter to expand the agent's capabilities and updating the existing writing adapter to improve report quality. Both updates leave the backbone and other adapters unchanged, supporting modular continual adaptation in agents.

Figures & tables

Figure 1: Internal actions extend the agent’s action space. WeAct selects LoRA adapters internally and interacts with the environment through external actions.

Figure 2: WeAct overview. (a) Capability learning builds the adapter library. (b) Tool-mediated inference switches adapters through internal tool calls and interacts with the environment through external tools. B/P/S/W nodes denote base/Plan/Search/Writing states; solid and dashed arrows indicate selected and alternative execution paths.

Figure 3: Capability retention under sequential training. Detailed results are reported in Appendix Table 7 .

Method	General	Multi-Hop QA			Search
	GAIA	2Wiki	Bam- boogle	MuSiQue	Web WalkerQA	xbench- DS	Browse Comp	FRAMES	Avg.
Prompt & Multi-Agent-Based
ReAct	35.92 $\pm$ 4.73	46.20 $\pm$ 1.58	53.60 $\pm$ 4.46	13.98 $\pm$ 0.71	27.35 $\pm$ 1.71	35.00 $\pm$ 4.77	4.00 $\pm$ 1.39	48.67 $\pm$ 1.74	33.09 $\pm$ 1.09
Least-to-Most	37.86 $\pm$ 4.78	47.40 $\pm$ 1.58	48.00 $\pm$ 4.47	14.40 $\pm$ 0.71	27.21 $\pm$ 1.71	39.00 $\pm$ 4.88	5.00 $\pm$ 1.54	51.70 $\pm$ 1.74	33.82 $\pm$ 1.10
ManuSearch	41.75 $\pm$ 4.86	53.50 $\pm$ 1.58	60.80 $\pm$ 4.37	16.84 $\pm$ 0.76	28.38 $\pm$ 1.73	45.00 $\pm$ 4.97	7.50 $\pm$ 1.86	62.14 $\pm$ 1.69	39.49 $\pm$ 1.12
Single-Model Weight Adaptation

Table 1: Results on eight benchmarks (%). Subscripts give nominal task-based binomial standard-error approximations. Bold and underlined values denote the best and second-best scores per benchmark.

Figure 4: SearchAtlas trace diagnostics. Query utilization: fraction of queries supporting the answer; constraint coverage: fraction of question constraints covered by those queries; PK

\rightarrow

answer: share of answers with unsupported content attributed to prior knowledge. Error bars are 95% task-bootstrap intervals.

Figure 5: Accuracy versus historical mean cumulative model-path time on four Search benchmarks. Times include saved retries.

Variant	MegaScience	SciBench	DS-1000	$\Delta$ GAIA
w/o LoRA-code	26.5	40.0	14.0	0.0
w/ LoRA-code	29.0	40.5	16.0	+1.0

Table 2: Results of code-adapter insertion.

\Delta

GAIA is relative to the original

K=3

system.

Method		ResearchQA	DeepResearch Bench
LoRA-write	hard	58.33	26.51
LoRA-write	soft	54.14	27.92
LoRA-write++	hard	60.72 (+2.40)	27.71 (+1.21)
LoRA-write++	soft	55.97 (+1.83)	28.52 (+0.61)

Table 3: Writing refresh scores. Bold values show gains over Original.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Use	Source	Count
Main	MiroVerse-v0.1	5,887
Main	OpenSeeker-v1-Data	1,075
Main total		6,962
Code	MiroVerse-MegaScience	3,413
Writing++	DR-Tulu / OpenScholar	1,000

Appendix

Table 4: Training-data sources. Main-corpus counts refer to source trajectories before capability splitting; Code and Writing++ counts refer to capability samples used in Section 4.5 .

Policy change	Changed	Spans (before $\to$ after)
Strengthen targeted verification	4/80	Plan $20\to 20$ ; Verif. $21\to 21$
Narrow recap; broaden verification	36/78	Recap $30\to 12$ ; Verif. $9\to 41$
Revise tool/span rules	11/78	Evid. $19\to 19$ ; Verif. $38\to 37$

Appendix

Table 5: Policy sensitivity on fixed inputs. Changed : relabeled/total assistant messages, including base-only transitions. Slices overlap and are not independent.

Figure 7: Nominal training compute. (a) Total recipe compute, with WeAct split by capability. (b) Dense and causal-attention FLOPs per token. Error bars show length-statistic uncertainty (details in text).

Figure 8: Accuracy versus model-path time on General and Multi-hop QA. Points pair Table 1 accuracy with historical mean cumulative time, including retries. Colors and markers identify methods; line styles group benchmarks. Time axes are logarithmic in seconds.

Figure 9: Historical task-record wall-clock distributions. Colors and markers identify methods; line styles identify benchmarks. Each panel uses its own logarithmic time axis in seconds and retains all tails. Curves include all valid task-record times and saved retries, regardless of correctness; logged wall time can include post-answer judging.

Benchmark	$N$	ReAct	Full SFT	Single LoRA	TIES-R64	WeAct -soft	WeAct -hard
GAIA	103	0.00	16.50	26.21	14.56	4.85	1.94
2Wiki	1,000	0.70	1.00	6.40	4.70	1.00	0.60
WebWalkerQA	680	0.29	9.26	11.32	12.79	9.26	3.82

Appendix

Table 6: Tasks with saved retries (%). The denominator is the task count

N

for each method and benchmark.

Figure 10: Runtime decomposition on WebWalkerQA with 50 runs per method. Bars show stage-aligned means. The ellipsis denotes repeated capability switching; all durations are retained. ManuSearch* is a historical reference with unmatched deployment and tool conditions.

	Plan			Search			Writing
Adapter organization	G	L	%	G	L	%	G	L	%
Single-LoRA (rank 32)	14.28	1.67	11.7	6.95	1.57	22.6	3.05	1.41	46.2
Single-LoRA (rank 96)	14.31	1.82	12.7	6.81	1.68	24.7	3.06	1.50	49.2
Random partition	13.36	1.32	9.9	6.16	0.96	15.6	2.58	1.09	42.4
Capability adapters	14.22	0.00	0.0	6.84	0.00	0.0	3.09	0.00	0.0

Appendix

Table 7: Acquired gain (G) and lost gain (L) in points of content-token accuracy, and fraction of acquired gain lost (%), pooled over the four orders in which each capability precedes another. Fractions use unrounded means.

Figure 11: Retention during subsequent training. Panels track the first capability in each order on monitoring trajectories; shading marks later stages, and 100% denotes the gain acquired in its own stage.

Figure 12: Gradient alignment and magnitude in the Single-LoRA pilot. (a) Pairwise cosines and (b) gradient norms at ten checkpoints of a rank-96 Single-LoRA run on frozen Qwen3-32B . Most pairwise cosines are positive, while gradient magnitudes differ across capabilities.

Figure 13: Local mismatch on capability-matched activations. Curves show trajectory-weighted layer means of Eq. 10 ; shading gives pointwise 95% trajectory-bootstrap confidence intervals. The dashed zero line is Individual’s identity reference. Full Additive uses no final rank compression.

Figure 14: Paired target-token NLL gaps relative to Individual. Points show validation records, boxes show medians and interquartile ranges, and diamonds mark means with 95% confidence intervals.

Method	QU $\uparrow$	Cov $\uparrow$	PK $\downarrow$
ReAct	0.365	0.336	0.590
Least-to-Most	0.386	0.360	0.600
ManuSearch	0.305	0.385	0.520
Single LoRA	0.315	0.332	0.630
Full SFT	0.410	0.478	0.530
TIES-R64	0.293	0.311	0.680

Appendix

Table 8: SearchAtlas diagnostics by method over the 100 tasks of Figure 4 . Values are task means; QU = query utilization, Cov = question-constraint coverage (answer graph), PK = share of answers with a prior-knowledge-to-answer edge. Bold marks the best value per column (lowest for PK).

	MegaScience	SciBench	DS-1000	GAIA
#Tasks	200	575	100	103

Appendix

Table 9: The number of selected tasks used for evaluating code insertion tasks. The MegaScience and DS-1000 tasks were randomly sampled based on a balanced consideration of cost and representativeness.

Method	MMLU- Pro	IFEval	MT- Bench	Struct Flow	Human Eval
Base	58.03	77.82	9.3875	98.12	90.85
Full SFT	$-19.99$	$-19.48$	$-0.93$	$-1.60$	$-6.86$
DARE-TIES	$-0.55$	$-4.61$	$+0.53$	$-0.95$	$-1.34$
Single LoRA	$-1.15$	$\bm{-3.80}$	$0.00$	$-0.76$	$-5.37$
WeAct -hard	$\bm{+1.60}$	$-4.51$	$\bm{+0.80}$	$-0.08$	$-2.69$
WeAct -soft	$+1.21$	$-4.28$	$\bm{+0.80}$	$\bm{-0.04}$	$\bm{0.00}$

Appendix

Table 10: General-capability retention (external tools disabled). Base: scores; others:

100(S/S_{\mathrm{Base}}-1)\%

. Green/red mark the best/worst changes per column, including ties.

Rubric	WeAct -Hard			WeAct -Soft
	Original	LoRA-write++	$\Delta$	Original	LoRA-write++	$\Delta$
Comprehensiveness	25.32	26.33	$+1.01$	26.50	26.78	$+0.28$
Depth	22.46	23.59	$+1.13$	23.82	24.19	$+0.37$
Instruction following	29.65	31.22	$+1.57$	31.36	32.45	$+1.09$
Readability	32.33	33.81	$+1.48$	33.32	35.10	$+1.77$

Appendix

Table 11: DeepResearchBench rubric scores before and after the Writing refresh (higher is better).

\Delta

is the reported score-point change from Original to LoRA-write++ .

Evaluation Set	Rows	NLL		Perplexity
	Rows	Original	LoRA-write++	Original	LoRA-write++
OpenScholar validation	205	1.27	1.02	3.56	2.76
Historical Writing validation	200	0.46	0.50	1.59	1.66

Appendix

Table 12: Supervised-token fit before and after the Writing refresh (lower is better).

LLM Agent Training

Large language model agent training teaches models to plan, use tools, and learn from multi-step interaction with environments.

See all 581 papers

Agents unlock new capabilities through Switching LoRA Adapters as a Tool (SLAaaT)
Aug 17, 2026Kenneth GeLong-Horizon LLM Agents LLM Agent Skill Learning
Independent San Francisco, CA, United States of America
WHALE: A Simple Recipe for Joint Harness-Weight Optimization
Aug 31, 2026Haechan Kim, Yoonho Lee, Gisang Lee +2Agent Harness Optimization LLM Agent Training
KRAFTON · KAIST · Stanford University
Position: Modular Memory is the Key to Continual Learning Agents
Mar 2, 2026Vaggelis Dorovatas, Malte Schwerin, Andrew D. Bagdanov +21Continual Learning for LLM Agents Memory-Augmented Agents
Toyota Motor Europe · University of Bremen · University of Florence +17

Aug 17, 2026J/K move · Enter open · S save

Agents unlock new capabilities through Switching LoRA Adapters as a Tool (SLAaaT)

Kenneth Ge

Independent San Francisco, CA, United States of America

Long-Horizon LLM Agents LLM Agent Skill Learning Catastrophic Forgetting Low-Rank Adaptation

Open paper AskPDF arXiv

Post-training can unlock new capabilities and improve performance on specialized tasks, but sometimes at the cost of catastrophic forgetting in other domains. This poses a problem in long agent trajectories that compose different capabilities. We reject this tradeoff by giving an agent a tool to switch between specialized LoRA adapters mid-trace. To test its effectiveness, we compose two synthetic coding tasks that are logically simple but require specialization. We find that this allows the model to solve problems it previously could not, that the model is able to switch autonomously (and find a new strategy that beats our human heuristic baseline on one task), and that this incurs an up to an 18x reduction in capability tax compared to an agent using only one specialized adapter. Our approach also substantially outperforms spawning subagents in both task capabilities (solving 4 of our hardest tasks versus none) and token usage (46.1x fewer tokens in some scenarios).

Abstract

LLM Agent Training

Explore similar work

Agents unlock new capabilities through Switching LoRA Adapters as a Tool (SLAaaT)

WHALE: A Simple Recipe for Joint Harness-Weight Optimization

Position: Modular Memory is the Key to Continual Learning Agents