Prefix Tuning

Follow this topic Ask about this topic

Latest papers

10 papers3 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Prefix-Tuning for Arbitrary Output Sequences on Pretrained Transformers
Oct 16, 2026Peter Cho-Ho Lam, Ziyi Wang, Zirui ZhouPrefix Tuning Parameter-Efficient Fine-Tuning
Huawei Technologies Canada Co., Ltd.
Not All Drift Is a Mechanism: Tracing Prompt Adaptation in LLMs
Oct 13, 2026Mikhail Aleksandrov, Nail Bashirov, Maksim Ivanov +6Prefix Tuning LLM Prompting
BRAIn Lab
When Can Prefixes Compile LoRA? Exact Resource-Capped Tests for Frozen Attention
Sep 29, 2026Joyanta Jyoti Mondal, Ibne Farabi ShihabPrefix Tuning Low-Rank Adaptation
Department of Computer and Information Sciences, University of Delaware, USA · Department of Computer Science, Iowa State University, USA
SkillSmith: Learning to Compose Parametric Skills and Textual Knowledge
Jul 29, 2026Lucio M. Dery, Benedict Aaron Tjandra, Siavash Samiei +4Knowledge Augmentation for Language Models Prompt Tuning
Google DeepMind
Distill to Detect: Exposing Stealth Biases in LLMs through Cartridge Distillation
Jul 1, 2026Shayan Talaei, Abhinav Chinta, Devvrit Khatri +3LLM Auditing Language Model Distillation
1Stanford University · 3Foundation AI–Cisco Systems Inc. · University of Texas at Austin
GuideCAD: A Lightweight Multimodal Framework for 3D CAD Model Generation via Prefix Embedding
Jun 5, 2026Minseong Kim, Jinyeong Park, Sungho Park +13D Asset Generation Parametric CAD Modeling
Convergence Research Center for Insect Vectors Incheon, South Korea · Department of Computer Science and Engineering Incheon National University Incheon, 22012, South Korea · Department of Computer Science and Engineering Center for Brain-Machine Interface Incheon National University Incheon, South Korea
Beyond Retrieval: Learning Compact User Representations for Scalable LLM Personalization
Jun 3, 2026Heng Cao, Fan Zhang, Jian Yao +8LLM Personalization Prompt Tuning
*Microsoft · †The Hong Kong Polytechnic University · ⋄Shanghai International Studies University +1
How You Begin is How You Reason: Driving Exploration in RLVR via Prefix-Tuned Priors
May 9, 2026Yifan Xu, Junren Chen, Yifan ChenRL for Language Model Reasoning Prompt Tuning
Hong Kong Baptist University · University of Maryland, College Park
Enhancing Continual Learning of Vision-Language Models via Dynamic Prefix Weighting
Apr 20, 2026Hyeonseo Jang, Hyuk Kwon, Kibok LeeClass-Incremental Learning Vision-Language Models
Yonsei University
Towards Dynamic Dense Retrieval with Routing Strategy
Feb 26, 2026Zhan Su, Fengran Mo, Jinghan Zhang +4Prefix Tuning Dense Retrieval
Université de Montréal, Montréal, Québec, Canada · Clemson University, Clemson, South Carolina, USA · University of Washington, America

Oct 16, 2026J/K move · Enter open · S save

Prefix-Tuning for Arbitrary Output Sequences on Pretrained Transformers

Peter Cho-Ho Lam, Ziyi Wang, Zirui Zhou

Huawei Technologies Canada Co., Ltd.

Prefix Tuning Parameter-Efficient Fine-Tuning Autoregressive Transformer

Open paper AskPDF arXiv

Prefix-tuning is a parameter-efficient fine-tuning method for large language models (LLMs) that steers model outputs by prepending task-specific prefix representations to the input sequence. Despite its practical success, the theoretical understanding of the expressiveness of prefixes remains incomplete. In particular, the following question remains open: Given a pretrained autoregressive Transformer, for any input-output sequence pair

(\mathtt{x},\mathtt{y})

, does there always exist a prefix

\mathcal{S}

such that prepending

\mathcal{S}

\mathtt{x}

induces the model to generate the target output

\mathtt{y}

? In this work, we prove that under the assumption that

\mathtt{y}

is in the range of the pretrained Transformer, such a prefix

\mathcal{S}

exists and can be explicitly computed by solving multiple linear systems. We further show that this assumption is necessary, in the sense that such a prefix may not exist when the assumption fails, thereby providing a complete answer to the above question. Our analysis is constructive and the obtained theoretical results hold under fairly general assumptions on model architecture. Extensive experiments on GPT-2 and Qwen3-4B demonstrate that our algorithm produces prefixes with strong control over model outputs.