Learning an Anchored Prompt Space for Continual Adaptation of Large Language Models

Rongguang Ye; Zhan Zhuang; Yichen Wu; Ming Tang; Kede Ma

cs.CLSep 26, 2026

Learning an Anchored Prompt Space for Continual Adaptation of Large Language Models

Authors: Rongguang Ye, Zhan Zhuang, Yichen Wu, Ming Tang, Kede Ma

Organizations: City University of Hong Kong · Southern University of Science and Technology · Harvard University

Abstract

Continually adapting large language models requires acquiring new knowledge while preserving previously learned capabilities. Although jointly adapting model parameters and task-specific soft prompts is promising, it has two key limitations: previously learned prompts may lose effectiveness as the model evolves, and relationships among prompts that could support cross-task transfer are not explicitly learned. We propose Learning an Anchored Prompt Space (LAPS), which preserves historical prompt effectiveness and learns relationships among task-specific soft prompts to facilitate positive transfer. Historical prompts are first aligned with the updated model through self-distillation. LAPS then constructs an anchored prompt space whose vertices correspond to learned task-specific soft prompts and whose intermediate geometry is shaped by learnable Bézier control prompts. Once this space is learned, LAPS identifies the best-performing prompt for each task on its validation set, allowing the selected prompt to draw on knowledge acquired from related tasks. Experiments on the TRACE benchmark across three Qwen3 model scales show that LAPS consistently outperforms existing distillation-based, prompt-based, and joint prompt-parameter adaptation methods, improving overall task performance while reducing forgetting.

Figures & tables

Figure 1: Overview of LAPS. (a) After adapting to the current task $\mathcal{T}_{t}$ , LAPS aligns each historical prompt $\mathbf{P}_{j,t-1}$ to the updated model, producing $\mathbf{P}_{j,t}$ through on-policy self-distillation. (b) The aligned prompts $\{\mathbf{P}_{j,t}\}_{j\leq t}$ are fixed as simplex anchors, while non-vertex Bézier control prompts $\mathbf{C}_{\bm{\alpha},t}$ are trained with cross-task replay losses to shape the space between anchors. (c) With the final model fixed, Bayesian optimization searches validation scores over simplex coordinates $\bm{\lambda}$ and returns the selected soft prompt $\mathbf{Z}_{T}(\bm{\lambda}_{j,T}^{\star})$ for each observed task.

Method	C-STANCE	FOMC	MeetingBank	Py150	ScienceQA	NG-cm	NG-ds	20Minuten	Average $\uparrow$	BWT $\uparrow$
Qwen3-1.7B
Initial	$36.3$	$32.7$	$18.4$	$2.5$	$63.4$	$70.2$	$30.2$	$35.8$	$36.2$	N/A
SFT	$\underline{53.1}$	$58.5$	$20.1$	$52.6$	$76.7$	$46.3$	$53.8$	$38.8$	$50.0$	$-9.3$
Replay	$52.8$	$62.4$	$46.6$	$56.6$	$73.4$	$51.1$	$53.2$	$38.9$	$54.4$	$-4.1$
OPR-RU	$50.4$	$58.2$	$47.0$	$54.7$	$77.1$	$\underline{55.7}$	$51.7$	$38.8$	$54.2$	$-4.9$
OPR-SC	$50.7$	$51.9$	$38.6$	$55.0$	$77.7$	$47.5$	$52.6$	$38.7$	$51.6$	$-7.6$

Table 1: Performance across Qwen3 model scales. The best and second best results at each model scale are shown in bold and underlined , respectively.

Figure 2: Task-wise continual adaptation trajectories with Qwen3-1.7B. Each panel tracks one task after it is learned, and later points show its performance after subsequent adaptation stages. Drops indicate forgetting as the model learns new tasks. The gray dashed line marks the transition from sequential adaptation to the validation-guided prompt selection introduced in LAPS.

Table 5

Method	Qwen3-1.7B						Qwen3-4B
	Length = 32		Length = 64		Length = 128		Length = 32		Length = 64		Length = 128
	Average	BWT	Average	BWT	Average	BWT	Average	BWT	Average	BWT	Average	BWT
ProgPrompt	$52.1$	$\mathbf{0.0}$	$49.2$	$\mathbf{0.0}$	$51.2$	$0.0$	$57.0$	$\mathbf{0.0}$	$57.4$	$\mathbf{0.0}$	$57.5$	$\mathbf{0.0}$
ProMoT	$58.0$	$-4.4$	$55.8$	$-4.6$	$48.6$	$-6.9$	$63.4$	$-2.0$	${65.5}$	$-0.2$	${62.6}$	$-3.2$
LAPS (Ours)	$\mathbf{60.5}$	$-1.2$	$\mathbf{59.5}$	$-1.6$	$\mathbf{58.5}$	$\mathbf{0.7}$	$\mathbf{66.2}$	$-0.3$	$\mathbf{65.6}$	$-0.4$	$\mathbf{64.7}$	${-1.3}$

Table 4: Effect of the soft prompt length. Results are averaged over eight evaluations.

Table 7

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Replay	Backbone	Soft Prompt	# Trainable Parameters (M)
Method	Replay	Backbone	Soft Prompt	Qwen3-1.7B	Qwen3-4B	Qwen3-8B
SFT	✗	✓	✗	$1{,}720.6$	$4{,}022.5$	$8{,}190.7$
Replay	✓	✓	✗	$1{,}720.6$	$4{,}022.5$	$8{,}190.7$
LwF	✗	✓	✗	$1{,}720.6$	$4{,}022.5$	$8{,}190.7$
OPR-RU	✓	✓	✗	$1{,}720.6$	$4{,}022.5$	$8{,}190.7$
OPR-SC	✓	✓	✗	$1{,}720.6$	$4{,}022.5$	$8{,}190.7$

Appendix

Table 7: Comparison of trainable components and parameter counts.

Dataset	Source	Average Length	Metric	Language	Size
C-STANCE	Social Media	109	Accuracy	Chinese	5,000
FOMC	Finance	76	Accuracy	English	5,000
MeetingBank	Meeting	3,550	ROUGE-L	English	5,000
Py150	GitHub	814	Edit Similarity	English	5,000
ScienceQA	Science	285	Accuracy	English	5,000
NumGLUE-cm	Math	50	Accuracy	English	5,000

Appendix

Table 8: Statistics of the eight continual adaptation tasks used in our experiments.

Model	Target Task	$\bm{\lambda}_{1}$	$\bm{\lambda}_{2}$	$\bm{\lambda}_{3}$	$\bm{\lambda}_{4}$	$\bm{\lambda}_{5}$	$\bm{\lambda}_{6}$	$\bm{\lambda}_{7}$	$\bm{\lambda}_{8}$
Qwen3-1.7B	C-STANCE	0.3313	0.2977	0.0001	0.0134	0.0646	0.2884	0.0000	0.0045
	FOMC	0.2173	0.3009	0.0164	0.0002	0.4052	0.0170	0.0000	0.0430
	MeetingBank	0.0000	0.0011	0.9967	0.0000	0.0011	0.0000	0.0003	0.0006
	Py150	0.0000	0.0000	0.0000	0.9962	0.0004	0.0014	0.0000	0.0015
	ScienceQA	0.0112	0.0000	0.0000	0.0000	0.9445	0.0012	0.0007	0.0424
	NumGLUE-cm	0.0004	0.0260	0.0016	0.0059	0.0087	0.8868	0.0636	0.0070

Appendix

Table 9: Task-specific simplex coordinates selected by Bayesian optimization.

Figure 5: Visualization of the learned anchored prompt space with and without non-vertex control prompts on Qwen3-1.7B and Qwen3-4B. The values in green indicate the cumulative performance improvement over the eight tasks after space search.

Method	20Minuten	NG-ds	NG-cm	ScienceQA	Py150	MeetingBank	FOMC	C-STANCE	Average $\uparrow$	BWT $\uparrow$
Qwen3-1.7B
SFT	$39.1$	$\mathbf{64.2}$	$54.2$	$18.2$	$58.6$	$43.8$	$52.3$	$50.8$	$47.6$	$-17.1$
Replay	$39.5$	$59.0$	$\mathbf{59.3}$	$68.8$	$58.8$	$56.0$	$62.3$	$51.4$	$56.9$	$-7.7$
ProgPrompt	$38.5$	$49.2$	$37.7$	$70.9$	$50.5$	$47.7$	$60.6$	$50.1$	$50.6$	$\mathbf{0.0}$
ProMoT	$39.7$	$61.4$	$30.6$	$75.6$	$59.4$	$\mathbf{61.9}$	$\mathbf{64.7}$	$\mathbf{52.2}$	$55.7$	$-4.2$
LAPS (Ours)	$\mathbf{39.8}$	$56.3$	$50.9$	$\mathbf{76.5}$	$\mathbf{60.3}$	$61.7$	$62.4$	$51.3$	$\mathbf{57.4}$	$-2.7$

Appendix

Table 10: Continual learning performance under the reverse task order (20Minuten

\rightarrow

NumGLUE-ds

\rightarrow

NumGLUE-cm

\rightarrow

ScienceQA

\rightarrow

Py150

\rightarrow

MeetingBank

\rightarrow

FOMC

\rightarrow

C-STANCE).

Continual Learning for LLMs

Continual learning for large language models studies how models absorb new knowledge and skills while retaining previously learned capabilities.

See all 229 papers

From a Prompt to Repertoires: Evolving Functional REpertoires Enable LLM Continual Learning
Oct 8, 2026Fengyuan Liu, Yue Wang, Hangxi Guo +4Continual Learning for LLMs Prompt Learning
The Chinese University of Hong Kong, Shenzhen · Shanghai AI Laboratory · University of Science and Technology of China +2
Generalizable Self-Evolving Memory for Automatic Prompt Optimization
Mar 23, 2026Guanbao Liang, Yuanchen Bei, Sheng Zhou +5Automatic Prompt Optimization Language Model Self-Improvement
Zhejiang University · University of Illinois Urbana-Champaign · Alibaba Group
Training Prompt Matters: State-Adaptive Optimization for Robust Fine-Tuning
Jun 1, 2026Wenhang Shi, Yiren Chen, Shuqing Bian +5Continual Learning for LLMs Prompt Learning
School of Information, Renmin University of China, Beijing, China · Peking University, Beijing, China · Tencent, Beijing, China

Oct 8, 2026J/K move · Enter open · S save

From a Prompt to Repertoires: Evolving Functional REpertoires Enable LLM Continual Learning

Fengyuan Liu, Yue Wang, Hangxi Guo, Fengyuan Liu+3

The Chinese University of Hong Kong, Shenzhen · Shanghai AI Laboratory · University of Science and Technology of China+2

Continual Learning for LLMs Prompt Learning Prompt Optimization Catastrophic Forgetting

Open paper AskPDF arXiv

Continual learning remains challenging for large language models, which must enable models to acquire new skills and knowledge without degrading existing capabilities. Existing approaches typically address this challenge by carefully designing how model parameters are updated. In contrast, prompt optimization avoids costly parameter updates while achieving competitive or even superior performance to reinforcement learning methods such as GRPO on individual knowledge-intensive and reasoning tasks. This raises a natural question: Can prompt optimization, as an efficient adaptation approach, be directly applied to continual learning? Our analysis shows that, under sequential task adaptation, it suffers from catastrophic forgetting, while optimized prompts accumulate rules that overfit to local task distributions. To address these limitations, we propose Evolving Functional REpertoires (EFRE), which replaces a single prompt with a repertoire of functions that evolves as new tasks arrive: compatible updates refine existing functions, while conflicting updates trigger the emergence of new ones. On a three-task continual-learning stream, EFRE achieves a final average performance 7.50 percentage points higher than GRPO. Moreover, after adaptation to the Bio task, its performance on FinQA decreases by only 1.56 percentage points, compared with 25.10 percentage points for the base prompt optimization method. We further instantiate EFRE in a minimal agent system and observe consistent improvements across different backbone models. Overall, these results demonstrate EFRE's strong performance in continual learning for large language models and highlight its substantial potential for continual learning in advanced agent systems.

Abstract

Continual Learning for LLMs

Explore similar work

From a Prompt to Repertoires: Evolving Functional REpertoires Enable LLM Continual Learning

Generalizable Self-Evolving Memory for Automatic Prompt Optimization

Training Prompt Matters: State-Adaptive Optimization for Robust Fine-Tuning