When Can Prefixes Compile LoRA? Exact Resource-Capped Tests for Frozen Attention

Joyanta Jyoti Mondal; Ibne Farabi Shihab

cs.LGSep 29, 2026

When Can Prefixes Compile LoRA? Exact Resource-Capped Tests for Frozen Attention

Authors: Joyanta Jyoti Mondal, Ibne Farabi Shihab

Organizations: Department of Computer and Information Sciences, University of Delaware, USA · Department of Computer Science, Iowa State University, USA

Abstract

Can a fixed continuous prefix replace a given low-rank adapter while the attention head stays frozen? In this research, we show that the answer depends on the adapter's target through three conditions. First, observability: at one causal readout, every independent key--value prefix sees the content only through the query, attention partition, and value numerator, so a target that differs on two inputs with equal summaries incurs an error floor at every prefix length; norm caps extend this floor to nearly equal summaries. Second, realizability: at a common query, any prefix reduces exactly to two aggregate variables, and the norm-capped optimum is an attained second-order-cone program, also after a fixed output projection; it places two equal-norm rank-one value updates on opposite sides of compilability. Third, implementation: under affine query exposure, $2r$ signed slots approximate a rank- $r$ value update, but their values grow as $O(ε^{-3/2})$ , and the construction passes all 400 tolerance checks in float64 yet only 38 in bfloat16. A first-layer GPT-2 readout with fixed token and position meets the common-query condition without clamping activations; at three such heads, the capped optimum leaves 18.4% to 74.2% of the projected adapter effect uncompiled, with a head-dependent value--query ordering. All claims concern local approximation at one head, not whole-network equivalence.

Figures & tables

Figure 1: Overview. Stage I : a prefix sees the content only through

\Sigma=(q,Z,N)

, so a target that differs on an equal-

\Sigma

pair keeps an error floor at every prefix length (Section 4 ). Stage II : at a common query, any prefix reduces to two aggregates, and the capped optimum is an exact SOCP (Section 5 ). The compile test combines both; a compiled prefix must still be checked at the target precision (Section 6 ). Measured values are from Section 7 .

Tolerance	Float64 error	Float32 error	Bfloat16 error	Value norm
$10^{-1}$	$4.27\!\times\!10^{-2}$	$4.27\!\times\!10^{-2}$	$9.32\!\times\!10^{-2}$	$2.38\!\times\!10^{2}$
$10^{-2}$	$4.39\!\times\!10^{-3}$	$4.39\!\times\!10^{-3}$	$1.93\!\times\!10^{-1}$	$7.30\!\times\!10^{3}$
$10^{-3}$	$4.40\!\times\!10^{-4}$	$4.37\!\times\!10^{-4}$	$1.10\!\times\!10^{0}$	$2.30\!\times\!10^{5}$
$10^{-4}$	$4.40\!\times\!10^{-5}$	$8.29\!\times\!10^{-5}$	$1.32\!\times\!10^{0}$	$7.28\!\times\!10^{6}$
$10^{-6}$	$4.40\!\times\!10^{-7}$	$1.09\!\times\!10^{-3}$	$1.32\!\times\!10^{0}$	$7.28\!\times\!10^{9}$

Table 1: Direct

2r

-slot construction, rank two. Each error is the mean over 20 seeds of the maximum over 10,004 test points. These are not standard errors or uniform-supremum estimates. The last column is the mean maximum prefix-value norm. Full seed-level results and sample SDs accompany the numerical data.

Figure 2: Finite precision and the signed construction. (a) Mean sampled maximum error at rank two over 20 seeds; the dotted line is the requested tolerance. (b) Mean maximum prefix-value norm against the

\epsilon^{-3/2}

rate of Theorem 11 . (c) Tolerance passes per rank among 100 seed–tolerance cases. The prefix keeps

2r

slots throughout.

Figure 3: Structural obstructions. (a) Same-head witness of Theorem 9 : the rank-one query update has minimax error

\eta_{\alpha}

, while the value update compiles exactly. (b) Exact capped optimum of Corollary 7 against the exact slot count; markers are Table 5 (

K=4

). (c) Near-fiber lower bound of Theorem 4 with the best attained errors of Table 2 .

Query perturbation $e$	Proved lower bound	Best attained error	Mean attained error
0	$0.8509$	$0.8509$	$0.8510\,\pm\,0.0001$
0.001	$0.8488$	$0.8507$	$0.8508\,\pm\,0.0000$
0.01	$0.8297$	$0.8457$	$0.8467\,\pm\,0.0005$
0.1	$0.6384$	$0.7315$	$0.7346\,\pm\,0.0040$

Table 2: Near-fiber experiment with 16 slots, key norm at most two, and value norm at most one. The last column is the mean

\pm

sample SD of best-iterate errors over ten optimizer initializations on the same pair, not independent datasets.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
Attention interface
$d,\ d_{k},\ d_{v},\ d_{o}$	input, key, value, and output dimensions
$W^{Q},W^{K},W^{V},W^{O}$	frozen query, key, value, and output-projection weights
$X=(x_{1},\ldots,x_{j})$	content states up to the readout position $j$
$q,\ k_{i},\ v_{i},\ s_{i}$	readout query, content keys and values, scaled logits
$Z_{X},\ N_{X},\ h_{X}$	partition function, value numerator, frozen head output

Appendix

Table 3: Notation, grouped by the part of the paper that introduces each symbol.

Rank	Float64	Float32	Bfloat16
1	$100/100$	$80/100$	$20/100$
2	$100/100$	$80/100$	$17/100$
4	$100/100$	$69/100$	$1/100$
8	$100/100$	$60/100$	$0/100$

Appendix

Table 4: Tolerance passes among 100 seed–tolerance cases per rank and precision. Every precision uses the same analytic construction and test points.

Slots	Contracting target $a_{0}=.5$	Amplifying target $a_{0}=1.5$
1	$0.0000$	$0.5180$
4	$0.0000$	$0.5683$
16	$0.0000$	$0.7266$
64	$0.0396$	$1.0396$

Appendix

Table 5: Exact best worst-case error on the matched two-input domain, with per-slot key cap

K=4

. Both unprefixed errors equal

0.5

. The constructed prefix attains the bounded optimum; no optimizer performance is involved.

Head	$d_{k}$	$d_{o}$	Fit	Evaluation	$m$	$K$	$V$
0	64	768	128	128	4	8.40	6.10
4	64	768	128	128	4	10.20	7.30
8	64	768	128	128	4	9.10	6.70

Appendix

Table 6: Interface and resource inputs for the first-layer GPT-2 evaluation.

d_{k}

is head width,

d_{o}

is residual width, and

K,V

are physical per-slot norm caps. Fitting and evaluation contexts are distinct design partitions.

Head	Target	$r$	$D^{G}$	$E_{\rm lo}$	$E_{\rm hi}$	Learned best	$E_{\rm hi}/D^{G}$
0	Value	1	0.0826	0.015198	0.015198	0.017263	0.184
0	Value	4	0.1462	0.059503	0.059503	0.063889	0.407
0	Query	1	0.0918	0.057925	0.057926	0.061139	0.631
0	Query	4	0.1584	0.085852	0.085853	0.092189	0.542
4	Value	1	0.0735	0.050200	0.050201	0.052038	0.683
4	Value	4	0.1327	0.098463	0.098463	0.102444	0.742

Appendix

Table 7: Pretrained-head comparison. All errors use the selected output block

G=W^{O}

and exactly 4 slots. The lower and upper conic endpoints differ by at most

10^{-6}

; the last column divides the upper endpoint by the adapter effect

D^{G}

Figure 4: First-layer GPT-2 heads under the capped test with four slots and

G=W^{O}

. Each column is a head (0, 4, 8), target (V value, Q query), and rank. (a) Conic optimum and best learned prefix on the fitting domain, divided by the adapter effect

D^{G}

. (b) Worst-case error on held-out evaluation contexts, divided by the evaluation-domain effect.

Head	Target	$r$	Conic upper	Learned best	Learned median	Relative gap
0	Value	1	0.015198	0.017263	0.020485	0.025
0	Value	4	0.059503	0.063889	0.069737	0.030
0	Query	1	0.057926	0.061139	0.064903	0.035
0	Query	4	0.085853	0.092189	0.095990	0.040
4	Value	1	0.050201	0.052038	0.055346	0.025
4	Value	4	0.098463	0.102444	0.108549	0.030

Appendix

Table 8: Same-domain optimizer comparison. The relative gap is

(E_{\rm best}-E_{\rm hi})/D^{G}

. Best and median summarize the ten restarts; no uncertainty interval or independent-dataset claim is attached to them.

Head	Target	$r$	Evaluation oracle	Conic transfer	Learned transfer
0	Value	1	0.209	0.274	0.321
0	Value	4	0.435	0.501	0.550
0	Query	1	0.662	0.729	0.780
0	Query	4	0.567	0.632	0.679
4	Value	1	0.711	0.777	0.826
4	Value	4	0.773	0.840	0.891

Appendix

Table 9: Held-out transfer comparison. All values are worst-case projected errors divided by the evaluation-domain adapter effect. The oracle uses evaluation targets; the transferred prefixes do not. These ratios are not a generalization guarantee.

Target	Scale	$m$	$K$	$V$	Exact-count	Learned	At most
Value	0.5	1	4.20	3.05	0.624	0.653	0.624
Value	0.5	4	4.20	3.05	0.624	0.658	0.624
Value	0.5	16	4.20	3.05	0.670	0.728	0.624
Value	1.0	1	8.40	6.10	0.407	0.436	0.407
Value	1.0	4	8.40	6.10	0.407	0.441	0.407
Value	1.0	16	8.40	6.10	0.407	0.465	0.407

Appendix

Table 10: Rank-four resource sweep. Errors are normalized by the fixed target effect. The at-most values include all smaller counts and the zero-slot option; equal count-1 and count-4 optima give the displayed minima, and no better intermediate count is assumed.

Prefix Tuning

See all 10 papers

May 29, 2026

Finer Parameter Steps for Low-Rank PEFT: A Controlled Study with CP Tensor Adapters

Low-rank adapters are usually compared by sweeping a small set of ranks, but the rank also fixes the resolution of the parameter budget. For a

2048{\times}2048

OPT attention projection, increasing LoRA by one rank stores

4096

trainable scalars, leaving large gaps between feasible low-budget adapter sizes. This paper asks whether a tensorized adapter with finer capacity increments changes the observed accuracy--budget trade-off. We instantiate this question with fixed-component canonical polyadic (CP) tensor adapters. Under a

32{\times}64{\times}32{\times}64

tensorization, one normalized CP component stores

193

trainable scalars per projection, about

21

times smaller than one LoRA rank step. We compare CP adapters and LoRA on OPT-1.3B across SST-2, RTE, and BoolQ under matched target modules, training protocol, data caps, and seed schedules. CP trains stably and fills the gaps between LoRA ranks, but the effect is task-dependent: SST-2 reaches an early low-budget plateau, BoolQ benefits from additional CP components before saturating slightly below LoRA, and RTE remains LoRA-favored. Finer parameter steps are therefore useful for diagnosing PEFT budget sensitivity, but they do not by themselves guarantee a better accuracy--budget curve.

Xinjue Wang, Xiuheng Wang, Yejun Zhang +3

Department of Information and Communications Engineering, Aalto University, 02150, Espoo, Finland · Université de Lorraine, CNRS, CRAN, France · Department of Computer Science, Aalto University, 02150, Espoo, Finland +1

Tensor Decomposition Adapter Tuning Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Ask PDF

May 7, 2026

Rethinking Adapter Placement: A Dominant Adaptation Module Perspective

Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning method that places trainable low-rank adapters into frozen pre-trained models. Recent studies show that using fewer LoRA adapters may still maintain or even improve performance, but existing methods still distribute adapters broadly, leaving where to place a limited number of adapters to maximize performance largely open. To investigate this, we introduce PAGE (Projected Adapter Gradient Energy), a gradient-based sensitivity probe that estimates the initial trainable gradient energy available to each candidate LoRA adapter. Surprisingly, we find that PAGE is highly concentrated on a single shallow FFN down-projection across two model families and four downstream tasks. We term this module the dominant adaptation module and show that its layer index is architecture-dependent but task-stable. Motivated by this finding, we propose DomLoRA, a placement method that places a single adapter at the dominant adaptation module. With only 0.7% of vanilla LoRA's trainable parameters, DomLoRA outperforms it on average across downstream tasks, including instruction following, mathematical reasoning, coding, and multi-turn conversation. This method also matches or improves other LoRA variants and reduces training time by up to 2.74

\times

compared with broad placement, supporting the dominant adaptation module perspective as a practical placement guideline.

Suoxin Zhang, Run He, Di Fang +3

South China University of Technology, China · Zhejiang University, China

LLM Fine-Tuning Adapter Tuning Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Ask PDF

Oct 16, 2026

Prefix-Tuning for Arbitrary Output Sequences on Pretrained Transformers

Prefix-tuning is a parameter-efficient fine-tuning method for large language models (LLMs) that steers model outputs by prepending task-specific prefix representations to the input sequence. Despite its practical success, the theoretical understanding of the expressiveness of prefixes remains incomplete. In particular, the following question remains open: Given a pretrained autoregressive Transformer, for any input-output sequence pair

(\mathtt{x},\mathtt{y})

, does there always exist a prefix

\mathcal{S}

such that prepending

\mathcal{S}

\mathtt{x}

induces the model to generate the target output

\mathtt{y}

? In this work, we prove that under the assumption that

\mathtt{y}

is in the range of the pretrained Transformer, such a prefix

\mathcal{S}

exists and can be explicitly computed by solving multiple linear systems. We further show that this assumption is necessary, in the sense that such a prefix may not exist when the assumption fails, thereby providing a complete answer to the above question. Our analysis is constructive and the obtained theoretical results hold under fairly general assumptions on model architecture. Extensive experiments on GPT-2 and Qwen3-4B demonstrate that our algorithm produces prefixes with strong control over model outputs.

Peter Cho-Ho Lam, Ziyi Wang, Zirui Zhou

Huawei Technologies Canada Co., Ltd.

Prefix Tuning Parameter-Efficient Fine-Tuning Autoregressive Transformer

Ask PDF

Abstract

Prefix Tuning

Explore similar work

Finer Parameter Steps for Low-Rank PEFT: A Controlled Study with CP Tensor Adapters

Rethinking Adapter Placement: A Dominant Adaptation Module Perspective

Prefix-Tuning for Arbitrary Output Sequences on Pretrained Transformers