When Can Prefixes Compile LoRA? Exact Resource-Capped Tests for Frozen Attention

Joyanta Jyoti Mondal; Ibne Farabi Shihab

cs.LGSep 29, 2026

When Can Prefixes Compile LoRA? Exact Resource-Capped Tests for Frozen Attention

Authors: Joyanta Jyoti Mondal, Ibne Farabi Shihab

Organizations: Department of Computer and Information Sciences, University of Delaware, USA · Department of Computer Science, Iowa State University, USA

Abstract

Can a fixed continuous prefix replace a given low-rank adapter while the attention head stays frozen? In this research, we show that the answer depends on the adapter's target through three conditions. First, observability: at one causal readout, every independent key--value prefix sees the content only through the query, attention partition, and value numerator, so a target that differs on two inputs with equal summaries incurs an error floor at every prefix length; norm caps extend this floor to nearly equal summaries. Second, realizability: at a common query, any prefix reduces exactly to two aggregate variables, and the norm-capped optimum is an attained second-order-cone program, also after a fixed output projection; it places two equal-norm rank-one value updates on opposite sides of compilability. Third, implementation: under affine query exposure, $2r$ signed slots approximate a rank- $r$ value update, but their values grow as $O(ε^{-3/2})$ , and the construction passes all 400 tolerance checks in float64 yet only 38 in bfloat16. A first-layer GPT-2 readout with fixed token and position meets the common-query condition without clamping activations; at three such heads, the capped optimum leaves 18.4% to 74.2% of the projected adapter effect uncompiled, with a head-dependent value--query ordering. All claims concern local approximation at one head, not whole-network equivalence.

Figures & tables

Figure 1: Overview. Stage I : a prefix sees the content only through

\Sigma=(q,Z,N)

, so a target that differs on an equal-

\Sigma

pair keeps an error floor at every prefix length (Section 4 ). Stage II : at a common query, any prefix reduces to two aggregates, and the capped optimum is an exact SOCP (Section 5 ). The compile test combines both; a compiled prefix must still be checked at the target precision (Section 6 ). Measured values are from Section 7 .

Tolerance	Float64 error	Float32 error	Bfloat16 error	Value norm
$10^{-1}$	$4.27\!\times\!10^{-2}$	$4.27\!\times\!10^{-2}$	$9.32\!\times\!10^{-2}$	$2.38\!\times\!10^{2}$
$10^{-2}$	$4.39\!\times\!10^{-3}$	$4.39\!\times\!10^{-3}$	$1.93\!\times\!10^{-1}$	$7.30\!\times\!10^{3}$
$10^{-3}$	$4.40\!\times\!10^{-4}$	$4.37\!\times\!10^{-4}$	$1.10\!\times\!10^{0}$	$2.30\!\times\!10^{5}$
$10^{-4}$	$4.40\!\times\!10^{-5}$	$8.29\!\times\!10^{-5}$	$1.32\!\times\!10^{0}$	$7.28\!\times\!10^{6}$
$10^{-6}$	$4.40\!\times\!10^{-7}$	$1.09\!\times\!10^{-3}$	$1.32\!\times\!10^{0}$	$7.28\!\times\!10^{9}$

Table 1: Direct

2r

-slot construction, rank two. Each error is the mean over 20 seeds of the maximum over 10,004 test points. These are not standard errors or uniform-supremum estimates. The last column is the mean maximum prefix-value norm. Full seed-level results and sample SDs accompany the numerical data.

Figure 2: Finite precision and the signed construction. (a) Mean sampled maximum error at rank two over 20 seeds; the dotted line is the requested tolerance. (b) Mean maximum prefix-value norm against the

\epsilon^{-3/2}

rate of Theorem 11 . (c) Tolerance passes per rank among 100 seed–tolerance cases. The prefix keeps

2r

slots throughout.

Figure 3: Structural obstructions. (a) Same-head witness of Theorem 9 : the rank-one query update has minimax error

\eta_{\alpha}

, while the value update compiles exactly. (b) Exact capped optimum of Corollary 7 against the exact slot count; markers are Table 5 (

K=4

). (c) Near-fiber lower bound of Theorem 4 with the best attained errors of Table 2 .

Query perturbation $e$	Proved lower bound	Best attained error	Mean attained error
0	$0.8509$	$0.8509$	$0.8510\,\pm\,0.0001$
0.001	$0.8488$	$0.8507$	$0.8508\,\pm\,0.0000$
0.01	$0.8297$	$0.8457$	$0.8467\,\pm\,0.0005$
0.1	$0.6384$	$0.7315$	$0.7346\,\pm\,0.0040$

Table 2: Near-fiber experiment with 16 slots, key norm at most two, and value norm at most one. The last column is the mean

\pm

sample SD of best-iterate errors over ten optimizer initializations on the same pair, not independent datasets.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
Attention interface
$d,\ d_{k},\ d_{v},\ d_{o}$	input, key, value, and output dimensions
$W^{Q},W^{K},W^{V},W^{O}$	frozen query, key, value, and output-projection weights
$X=(x_{1},\ldots,x_{j})$	content states up to the readout position $j$
$q,\ k_{i},\ v_{i},\ s_{i}$	readout query, content keys and values, scaled logits
$Z_{X},\ N_{X},\ h_{X}$	partition function, value numerator, frozen head output

Appendix

Table 3: Notation, grouped by the part of the paper that introduces each symbol.

Rank	Float64	Float32	Bfloat16
1	$100/100$	$80/100$	$20/100$
2	$100/100$	$80/100$	$17/100$
4	$100/100$	$69/100$	$1/100$
8	$100/100$	$60/100$	$0/100$

Appendix

Table 4: Tolerance passes among 100 seed–tolerance cases per rank and precision. Every precision uses the same analytic construction and test points.

Slots	Contracting target $a_{0}=.5$	Amplifying target $a_{0}=1.5$
1	$0.0000$	$0.5180$
4	$0.0000$	$0.5683$
16	$0.0000$	$0.7266$
64	$0.0396$	$1.0396$

Appendix

Table 5: Exact best worst-case error on the matched two-input domain, with per-slot key cap

K=4

. Both unprefixed errors equal

0.5

. The constructed prefix attains the bounded optimum; no optimizer performance is involved.

Head	$d_{k}$	$d_{o}$	Fit	Evaluation	$m$	$K$	$V$
0	64	768	128	128	4	8.40	6.10
4	64	768	128	128	4	10.20	7.30
8	64	768	128	128	4	9.10	6.70

Appendix

Table 6: Interface and resource inputs for the first-layer GPT-2 evaluation.

d_{k}

is head width,

d_{o}

is residual width, and

K,V

are physical per-slot norm caps. Fitting and evaluation contexts are distinct design partitions.

Head	Target	$r$	$D^{G}$	$E_{\rm lo}$	$E_{\rm hi}$	Learned best	$E_{\rm hi}/D^{G}$
0	Value	1	0.0826	0.015198	0.015198	0.017263	0.184
0	Value	4	0.1462	0.059503	0.059503	0.063889	0.407
0	Query	1	0.0918	0.057925	0.057926	0.061139	0.631
0	Query	4	0.1584	0.085852	0.085853	0.092189	0.542
4	Value	1	0.0735	0.050200	0.050201	0.052038	0.683
4	Value	4	0.1327	0.098463	0.098463	0.102444	0.742

Appendix

Table 7: Pretrained-head comparison. All errors use the selected output block

G=W^{O}

and exactly 4 slots. The lower and upper conic endpoints differ by at most

10^{-6}

; the last column divides the upper endpoint by the adapter effect

D^{G}

Figure 4: First-layer GPT-2 heads under the capped test with four slots and

G=W^{O}

. Each column is a head (0, 4, 8), target (V value, Q query), and rank. (a) Conic optimum and best learned prefix on the fitting domain, divided by the adapter effect

D^{G}

. (b) Worst-case error on held-out evaluation contexts, divided by the evaluation-domain effect.

Head	Target	$r$	Conic upper	Learned best	Learned median	Relative gap
0	Value	1	0.015198	0.017263	0.020485	0.025
0	Value	4	0.059503	0.063889	0.069737	0.030
0	Query	1	0.057926	0.061139	0.064903	0.035
0	Query	4	0.085853	0.092189	0.095990	0.040
4	Value	1	0.050201	0.052038	0.055346	0.025
4	Value	4	0.098463	0.102444	0.108549	0.030

Appendix

Table 8: Same-domain optimizer comparison. The relative gap is

(E_{\rm best}-E_{\rm hi})/D^{G}

. Best and median summarize the ten restarts; no uncertainty interval or independent-dataset claim is attached to them.

Head	Target	$r$	Evaluation oracle	Conic transfer	Learned transfer
0	Value	1	0.209	0.274	0.321
0	Value	4	0.435	0.501	0.550
0	Query	1	0.662	0.729	0.780
0	Query	4	0.567	0.632	0.679
4	Value	1	0.711	0.777	0.826
4	Value	4	0.773	0.840	0.891

Appendix

Table 9: Held-out transfer comparison. All values are worst-case projected errors divided by the evaluation-domain adapter effect. The oracle uses evaluation targets; the transferred prefixes do not. These ratios are not a generalization guarantee.

Target	Scale	$m$	$K$	$V$	Exact-count	Learned	At most
Value	0.5	1	4.20	3.05	0.624	0.653	0.624
Value	0.5	4	4.20	3.05	0.624	0.658	0.624
Value	0.5	16	4.20	3.05	0.670	0.728	0.624
Value	1.0	1	8.40	6.10	0.407	0.436	0.407
Value	1.0	4	8.40	6.10	0.407	0.441	0.407
Value	1.0	16	8.40	6.10	0.407	0.465	0.407

Appendix

Table 10: Rank-four resource sweep. Errors are normalized by the fixed target effect. The at-most values include all smaller counts and the zero-slot option; equal count-1 and count-4 optima give the displayed minima, and no better intermediate count is assumed.

Prefix Tuning

See all 10 papers

Finer Parameter Steps for Low-Rank PEFT: A Controlled Study with CP Tensor Adapters
May 29, 2026Xinjue Wang, Xiuheng Wang, Yejun Zhang +3Tensor Decomposition Adapter Tuning
Department of Information and Communications Engineering, Aalto University, 02150, Espoo, Finland · Université de Lorraine, CNRS, CRAN, France · Department of Computer Science, Aalto University, 02150, Espoo, Finland +1
Rethinking Adapter Placement: A Dominant Adaptation Module Perspective
May 7, 2026Suoxin Zhang, Run He, Di Fang +3LLM Fine-Tuning Adapter Tuning
South China University of Technology, China · Zhejiang University, China
Prefix-Tuning for Arbitrary Output Sequences on Pretrained Transformers
Oct 16, 2026Peter Cho-Ho Lam, Ziyi Wang, Zirui ZhouPrefix Tuning Parameter-Efficient Fine-Tuning
Huawei Technologies Canada Co., Ltd.

May 29, 2026J/K move · Enter open · S save

Finer Parameter Steps for Low-Rank PEFT: A Controlled Study with CP Tensor Adapters

Xinjue Wang, Xiuheng Wang, Yejun Zhang, Sergiy A. Vorobyov+2

Department of Information and Communications Engineering, Aalto University, 02150, Espoo, Finland · Université de Lorraine, CNRS, CRAN, France · Department of Computer Science, Aalto University, 02150, Espoo, Finland+1

Tensor Decomposition Adapter Tuning Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Low-rank adapters are usually compared by sweeping a small set of ranks, but the rank also fixes the resolution of the parameter budget. For a

2048{\times}2048

OPT attention projection, increasing LoRA by one rank stores

4096

trainable scalars, leaving large gaps between feasible low-budget adapter sizes. This paper asks whether a tensorized adapter with finer capacity increments changes the observed accuracy--budget trade-off. We instantiate this question with fixed-component canonical polyadic (CP) tensor adapters. Under a

32{\times}64{\times}32{\times}64

tensorization, one normalized CP component stores

193

trainable scalars per projection, about

21

times smaller than one LoRA rank step. We compare CP adapters and LoRA on OPT-1.3B across SST-2, RTE, and BoolQ under matched target modules, training protocol, data caps, and seed schedules. CP trains stably and fills the gaps between LoRA ranks, but the effect is task-dependent: SST-2 reaches an early low-budget plateau, BoolQ benefits from additional CP components before saturating slightly below LoRA, and RTE remains LoRA-favored. Finer parameter steps are therefore useful for diagnosing PEFT budget sensitivity, but they do not by themselves guarantee a better accuracy--budget curve.

Abstract

Prefix Tuning

Explore similar work

Finer Parameter Steps for Low-Rank PEFT: A Controlled Study with CP Tensor Adapters

Rethinking Adapter Placement: A Dominant Adaptation Module Perspective

Prefix-Tuning for Arbitrary Output Sequences on Pretrained Transformers