PREFACE: Separating Shared Corrections from Prefix Conditioning in PEFT
Abstract
Does input-conditioned parameter-efficient fine-tuning (PEFT) improve because it uses the current input, or because it adds a useful learned correction? We investigate this distinction with PREFACE, a bounded channel controller for PEFT-adapted projections that requires no test-time gradients. The controller reads observed context; in multiple-choice likelihood scoring, its state is independent of the candidate continuation. Our central study freezes IA3, tuned LoRA, and DoRA adapters and matches controller architecture, initialization, batches, and updates across current-prefix and learned-global controls. Across ARC-Challenge and SciQ, current-prefix control improves all six mean accuracies over frozen adapters, but none of its paired contrasts with learned-global control excludes zero. Replacing its corrections with their training-prefix mean changes accuracy by at most 0.34 percentage points; input-varying corrections contribute only 1.8-2.8% of correction RMS. On ARC test, frozen-adapter current/global controls score 25.68%/25.77%, distinct from the earlier jointly trained 23.1% to 26.4% gain over IA3. We derive and evaluate static compilation of fixed corrections, separating deployment simplification from statistical equivalence. Broader GPT-2, Pythia, and Qwen experiments show family-, task-, and scoring-dependent gains, with modest recent-model effects. The study provides an intervention-based evaluation of conditional PEFT: improvements over a static adapter do not by themselves demonstrate a benefit from input matching.
Figures & tables
| Task | Adapter | Frozen | Global | Current | Fixed mean | Random | Shuffled |
|---|---|---|---|---|---|---|---|
| ARC-C | IA3 | 24.4 0.3 | 27.6 1.9 | 28.2 0.8 | 28.5 0.7 | 28.4 0.7 | 28.3 0.7 |
| ARC-C | LoRA | 32.6 2.7 | 33.9 1.6 | 34.1 1.2 | 34.2 1.4 | 34.1 1.2 | 34.3 1.2 |
| ARC-C | DoRA | 34.3 4.0 | 35.1 1.2 | 35.9 1.5 | 36.0 1.5 | 36.0 1.4 | 35.9 1.5 |
| SciQ | IA3 | 51.5 0.9 | 53.0 0.6 | 52.9 0.6 | 52.9 0.6 | 53.0 0.7 | 53.0 0.7 |
| SciQ | LoRA | 51.4 1.6 | 52.5 1.0 | 52.7 1.0 | 52.8 0.9 | 52.8 0.9 | 52.8 0.9 |
| SciQ | DoRA | 50.6 0.8 | 53.6 0.6 | 53.5 0.6 | 53.3 0.6 | 53.3 0.6 | 53.3 0.6 |
| Task | Adapter | Current frozen [95%] | Current global [95%] | Global contrast [B12] |
|---|---|---|---|---|
| ARC-C | IA3 | +3.79 [0.00, 7.47] | +0.56 [-1.45, 3.01] | [-2.34, 4.35] |
| ARC-C | LoRA | +1.56 [-1.78, 4.91] | +0.22 [-1.67, 2.34] | [-2.68, 3.68] |
| ARC-C | DoRA | +1.56 [-2.34, 5.91] | +0.78 [-1.67, 2.79] | [-3.46, 3.90] |
| SciQ | IA3 | +1.40 [-0.80, 3.60] | -0.03 [-0.87, 0.80] | [-1.30, 1.23] |
| SciQ | LoRA | +1.33 [-0.70, 3.30] | +0.23 [-0.60, 1.10] | [-1.07, 1.67] |
| SciQ | DoRA | +2.87 [1.00, 4.77] | -0.10 [-0.90, 0.67] | [-1.37, 1.10] |
| Accuracy (%, three seeds) | Latency (ms/question) | ||||||
|---|---|---|---|---|---|---|---|
| Task | Adapter | Current | Fixed mean | Compiled | Frozen | Current | Compiled |
| ARC-C | IA3 | 28.21 | 28.54 | 28.54 | 8.57 | 18.90 | 8.43 |
| ARC-C | LoRA | 34.11 | 34.23 | 34.23 | 8.47 | 18.29 | 8.48 |
| ARC-C | DoRA | 35.90 | 36.01 | 36.01 | 8.47 | 18.30 | 8.44 |
| SciQ | IA3 | 52.93 | 52.93 | 52.93 | 8.14 | 18.16 | 8.19 |
| SciQ | LoRA | 52.73 | 52.80 | 52.80 | 8.17 | 18.23 | 8.19 |
| Evaluation | IA3 | PREFACE | Paired [95%] | |
|---|---|---|---|---|
| DistilGPT-2 / ARC test | 1,172 | 23.1 0.4 | 26.4 1.4 | +3.36 [1.39, 5.26] |
| Qwen3.5 / ARC val. | 299 | 47.4 0.7 | 47.9 0.4 | +0.56 [-0.56, 1.67] |
| Qwen3.5 / SciQ val. | 1,000 | 84.7 0.8 | 85.1 0.9 | +0.40 [0.03, 0.80] |
| Qwen3.5 / PIQA val. | 1,838 | 70.2 0.4 | 70.4 0.5 | +0.15 [-0.20, 0.47] |
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
| PEFT family | Setting | Same-family comparison | Params | Seeds | Static | PREFACE | |
| Headline ARC-Challenge gains | |||||||
| IA3 | DistilGPT-2 / ARC-C | IA3 PREFACE-IA3-small | 147,540 | 3 | 24.4 0.3 | 28.7 2.8 | +4.2 |
| IA3 | GPT-2 small / ARC-C | IA3 PREFACE-IA3-small | 239,700 | 5 | 24.5 1.5 | 29.0 2.3 | +4.5 |
| IA3 | Pythia-410M / ARC-C | IA3 PREFACE-IA3-gated | 467,052 | 3 | 31.5 1.5 | 34.0 0.8 | +2.5 |
| IA3 | Qwen2.5-1.5B / ARC-C | IA3 PREFACE-IA3-gated | 1,164,912 | 3 | 51.5 1.0 | 53.1 0.2 | +1.6 |
| LoRA | DistilGPT-2 / ARC-C | LoRA-tuned PREFACE-LoRA | 270,632 | 3 | 32.6 2.7 | 34.3 0.7 | +1.8 |
| PEFT family | Protocol | Static adapter | PREFACE | |
|---|---|---|---|---|
| IA3 | CP text | 24.4 0.3 | 28.7 2.8 | +4.2 [0.1, 8.1] |
| LoRA | CP text | 32.6 2.7 | 34.3 0.7 | +1.8 [-1.4, 5.1] |
| DoRA | CP text | 34.3 4.0 | 34.4 3.2 | +0.1 [-3.1, 3.5] |
| IA3 | FTP+prefill | 19.0 0.8 | 23.0 2.3 | +4.0 [1.3, 6.8] |
| LoRA | FTP+prefill | 25.9 1.7 | 26.2 0.4 | +0.3 [-1.9, 2.6] |
| DoRA | FTP+prefill | 25.2 1.8 | 25.9 1.8 | +0.7 [-1.7, 3.0] |
| Method | Trainable params | Seeds | Sum | Mean |
|---|---|---|---|---|
| DistilGPT-2 | 0 | 3 | 40.3 0.1 | 40.5 0.1 |
| LoRA-r8 (initial) | 184,320 | 3 | 27.1 1.2 | 34.0 2.1 |
| LoRA-tuned | 368,640 | 3 | 43.5 0.7 | 51.4 1.6 |
| DoRA | 188,928 | 3 | 40.4 2.1 | 50.6 0.8 |
| IA3 | 18,432 | 3 | 42.0 0.5 | 51.5 0.9 |
| PREFACE-IA3-shared | 270,632 | 3 | 41.9 0.8 | 52.4 0.9 |
| Base adapter | Static | PREFACE | Paired delta |
|---|---|---|---|
| IA3 | 81.7 0.5 | 81.4 0.1 | -0.3 [-0.8, 0.2] |
| LoRA | 75.0 2.8 | 75.0 2.8 | 0.0 [-0.1, 0.0] |
| Dataset | Score | IA3 | PREFACE | Hierarchical 95% interval | ||
|---|---|---|---|---|---|---|
| ARC-Challenge | sum | 299 | 43.03 1.35 | 43.14 1.00 | +0.11 | [-0.67, 0.89] |
| ARC-Challenge | mean | 299 | 47.38 0.70 | 47.94 0.39 | +0.56 | [-0.56, 1.67] |
| SciQ | sum | 1,000 | 82.30 0.98 | 82.63 0.81 | +0.33 | [0.07, 0.67] |
| SciQ | mean | 1,000 | 84.73 0.76 | 85.13 0.85 | +0.40 | [0.03, 0.80] |
| PIQA | sum | 1,838 | 69.44 0.08 | 69.73 0.17 | +0.29 | [0.00, 0.58] |
| PIQA | mean | 1,838 | 70.24 0.44 | 70.38 0.54 | +0.15 | [-0.20, 0.47] |
| Protocol | Controller-visible prefix | Scored tokens | Shared-prefix handling |
|---|---|---|---|
| CP text | Question/context only | Candidate answer text | None; one prefix representation is reused across choices. |
| FTP+prefill | Question plus visible options | First discriminative answer token | Tokens shared by all answer strings are prefixed but not scored. |
| MCP label | Question plus labeled options | Next option-label token | No answer-text continuation is exposed to the controller. |
| MCP text | Question plus labeled options | Candidate answer text | Each candidate text is masked from the controller. |
| Options-trained | Question plus labeled options | Candidate answer text | Training and evaluation use the same options-visible template. |
| Base adapter | Protocol | Static adapter | PREFACE | |
|---|---|---|---|---|
| IA3 | MCP label | 21.3 0.2 | 21.4 0.0 | +0.1 [0.0, 0.4] |
| LoRA | MCP label | 24.3 0.5 | 23.2 2.6 | -1.1 [-4.9, 2.7] |
| DoRA | MCP label | 26.4 2.9 | 25.9 1.3 | -0.6 [-3.6, 2.2] |
| IA3 | MCP text | 29.2 0.8 | 28.9 4.0 | -0.3 [-5.0, 4.7] |
| LoRA | MCP text | 31.8 4.1 | 33.0 3.9 | +1.2 [-1.3, 3.8] |
| DoRA | MCP text | 33.2 4.5 | 32.9 2.0 | -0.3 [-4.8, 3.8] |
| Base adapter | Backbone | Same-family comparison | Params | Seeds | Base | PREFACE | |
|---|---|---|---|---|---|---|---|
| IA3 | DistilGPT-2 | IA3 PREFACE-IA3-small | 147,540 | 3 | 24.4 0.3 | 28.7 2.8 | +4.2 |
| LoRA | DistilGPT-2 | LoRA-tuned PREFACE-LoRA | 270,632 | 3 | 32.6 2.7 | 34.3 0.7 | +1.8 |
| DoRA | DistilGPT-2 | DoRA PREFACE-DoRA | 270,632 | 3 | 34.3 4.0 | 34.4 3.2 | +0.1 |
| IA3 | GPT-2 small | IA3 PREFACE-IA3-small | 239,700 | 5 | 24.5 1.5 | 29.0 2.3 | +4.5 |
| IA3 | GPT-2 medium | IA3 PREFACE-IA3-small | 565,332 | 3 | 32.2 1.0 | 33.6 5.7 | +1.3 |
| IA3 | Pythia-410M | IA3 PREFACE-IA3-gated | 467,052 | 3 | 31.5 1.5 | 34.0 0.8 | +2.5 |
| Method | Params | ARC Acc. | Paired evidence |
|---|---|---|---|
| IA3 | 18,432 | 24.4 0.3 | – |
| LoReFT-minimal | 18,552 | 28.8 0.6 | – |
| PREFACE-LoReFT | 14,004 | 29.0 1.3 | +0.2 [-0.9, 1.7] vs LoReFT |
| PARA/PEDRO-small | 60,056 | 30.5 2.3 | – |
| PREFACE-PARA/PEDRO-small | 13,908 | 31.5 1.7 | +1.0 [0.0, 2.5] vs PARA/PEDRO |
| PREFACE-IA3-small | 147,540 | 28.7 2.8 | -1.9 [-7.9, 3.3] vs PARA/PEDRO |
| Controller | Params | Acc. | vs IA3 | Summary path |
|---|---|---|---|---|
| Static IA3 | 18k | 24.4 0.3 | – | static scale |
| MeanMLP | 148k | 28.7 2.8 | +4.2 | mean MLP |
| TokenMLP | 148k | 29.2 2.5 | +4.8 | shared MLP mean |
| Attention pooling | 203k | 27.9 0.8 | +3.5 | learned weighted mean |
| Self-attention | 149k | 28.5 2.8 | +4.1 | self-attn mean |
| TokenCNN | 240k | 28.7 2.9 | +4.2 | Conv1D mean |
| Family | Setting | Reference prefix variant | Ref. | Prefix | [95%] | Seed-level evidence |
| Mult. | SciQ | IA3 PREFACE-IA3-shared | 51.5 | 52.5 | +1.0 [-0.5, 2.5] | 2/3 seeds; boot ; McNemar diag. |
| Mult. | ARC | IA3 PREFACE-IA3-small | 24.4 | 28.7 | +4.2 [0.1, 8.1] | 3/3 seeds; boot ; McNemar diag. |
| Mult. | WinoGrande | IA3 PREFACE-IA3-small | 48.8 | 49.8 | +1.0 [-0.6, 2.5] | 3/3 seeds; boot ; McNemar diag. |
| Mult. | ARC, GPT-2 | IA3 PREFACE-IA3-small | 24.5 | 29.0 | +4.5 [1.0, 8.3] | 5/5 seeds; boot ; McNemar diag. |
| Mult. | ARC, GPT-2-M | IA3 PREFACE-IA3-small | 32.2 | 33.6 | +1.3 [-4.3, 7.6] | 1/3 seeds; boot ; McNemar diag. |
| Mult. | ARC, Pythia | IA3 PREFACE-IA3-gated | 31.5 | 34.0 | +2.5 [0.7, 4.2] | 3/3 seeds; boot ; McNemar diag. |
| Model identifier | Revision | Precision |
|---|---|---|
| distilgpt2 | 2290a62682d06624634c1f46a6ad5be0f47f38aa | float32 |
| gpt2 | 607a30d783dfa663caf39e06633721c8d4cfcd7e | float32 |
| gpt2-medium | 6dcaa7a952f72f9298047fd5137cd6e4f05f41da | float32 |
| EleutherAI/pythia-410m-deduped | c4fc8d586d62df497f1f9b69d66d3ca419992d3e | float32 |
| Qwen/Qwen2.5-0.5B-Instruct | 7ae557604adf67be50417f59c2c2f167def9a775 | float16 |
| Qwen/Qwen2.5-1.5B-Instruct | 989aa7980e4cf806f80c7fef2b1adb7bc71aa306 | float16 |
| Method | Params | ms/q | q/s | choices/s | Peak MB |
|---|---|---|---|---|---|
| Frozen | 0 | 51.6 | 19.4 | 77.1 | 377.8 |
| IA3 | 18k | 52.5 | 19.0 | 75.7 | 377.9 |
| LoRA-tuned | 184k | 60.6 | 16.5 | 65.7 | 378.5 |
| DoRA | 189k | 58.0 | 17.2 | 68.6 | 378.5 |
| PREFACE-IA3-small | 148k | 75.0 | 13.3 | 53.1 | 378.5 |
| PREFACE-LoRA | 271k | 85.6 | 11.7 | 46.5 | 379.7 |
| Control source | Pooled acc. | Current minus control | 95% interval |
|---|---|---|---|
| IA3 (no controller) | 67.74 | +0.11 | [-0.37, 0.59] |
| Learned global | 67.74 | +0.11 | [-0.37, 0.59] |
| Current prefix | 67.86 | – | – |
| Within-task shuffle | 67.82 | +0.04 | [-0.30, 0.37] |
| Cross-task shuffle | 67.74 | +0.11 | [-0.22, 0.45] |
| Random vector | 67.93 | -0.07 | [-0.48, 0.30] |
| Family | Method | Prefix source | Params | Acc. | Seed-level evidence vs. same-family reference |
|---|---|---|---|---|---|
| Mult. | IA3 | static | 18k | 24.4 0.3 | – |
| Mult. | Static basis (92k) | learned global | 92k | 27.5 3.0 | +3.1 [-0.7, 6.6]; 2/3 seeds; boot |
| Mult. | PREFACE-IA3-small | current prefix | 148k | 28.7 2.8 | +4.2 [0.2, 8.0]; 3/3 seeds; boot |
| Mult. | PREFACE-IA3-small | random prefix | 148k | 27.6 1.7 | +3.2 [0.0, 6.4]; 3/3 seeds; boot |
| Mult. | PREFACE-IA3-small | shuffled prefix | 148k | 28.8 2.7 | +4.3 [0.4, 8.1]; 3/3 seeds; boot |
| LoRA | LoRA-tuned | static | 184k | 32.6 2.7 | – |
| Family | Method | Prefix source | Params | Acc. | Diagnostic note | |
|---|---|---|---|---|---|---|
| Mult. | IA3 | static | 18k | 27.2 0.7 | – | Static low-parameter multiplicative reference. |
| Mult. | Parameter-matched static | learned global | 147k | 29.7 2.5 | +2.45 [0.33, 4.57] | Adds scale capacity without input conditioning. |
| Mult. | PREFACE-IA3-small | current prefix | 148k | 30.5 1.6 | +3.34 [0.45, 6.24] | Prefix-controlled scale; +0.89 [-1.78, 3.57] vs. the parameter-matched static control. |
| Mult. | PREFACE-IA3-small | random prefix | 148k | 28.4 0.3 | current +2.12 [-0.45, 4.68] | Replaces the context representation with fixed random vectors. |
| Mult. | PREFACE-IA3-small | shuffled prefix | 148k | 30.9 1.6 | current -0.33 [-0.78, 0.00] | Uses donor-prefix summaries; not worse than current-prefix states. |
| Mean | P95 | Sum | Mean | Sum net/flips | Mean net/flips | Seeds | |
|---|---|---|---|---|---|---|---|
| 0.05 | 0.0297 | 0.0491 | +1.00 [-0.33, 2.45] | -0.11 [-1.45, 1.23] | +9 / 69 | -1 / 80 | 3 |
| 0.20 | 0.0662 | 0.1776 | +2.01 [0.00, 4.12] | +2.01 [-2.01, 5.80] | +18 / 147 | +18 / 234 | 3 |
| 1.00 | 0.1199 | 0.3852 | +1.78 [-0.56, 4.12] | +1.56 [-1.11, 4.46] | +16 / 185 | +14 / 312 | 3 |
| Task | Adapter | Current frozen B12 | B36 |
|---|---|---|---|
| ARC-C | IA3 | [-2.01, 9.36] | [-2.79, 10.03] |
| ARC-C | LoRA | [-3.46, 6.69] | [-4.24, 7.36] |
| ARC-C | DoRA | [-4.35, 8.36] | [-5.02, 9.36] |
| SciQ | IA3 | [-1.87, 4.80] | [-2.30, 5.30] |
| SciQ | LoRA | [-1.90, 4.40] | [-2.37, 4.90] |
| SciQ | DoRA | [0.00, 5.83] | [-0.40, 6.30] |
| Task | Adapter | Current bounded | Unbounded |
|---|---|---|---|
| ARC-C | IA3 | 28.2 0.8 | 27.9 0.8 |
| ARC-C | LoRA | 34.1 1.2 | 34.1 1.5 |
| ARC-C | DoRA | 35.9 1.5 | 34.8 1.5 |
| SciQ | IA3 | 52.9 0.6 | 52.7 0.4 |
| SciQ | LoRA | 52.7 1.0 | 52.6 1.3 |
| SciQ | DoRA | 53.5 0.6 | 53.6 1.0 |
| Task | Frozen IA3 | Global | Current | Within shuffle | Cross shuffle | Train mean |
|---|---|---|---|---|---|---|
| ARC-C | 21.63 | 22.41 | 23.63 | 23.60 | 23.52 | 23.97 |
| SciQ | 49.27 | 49.77 | 49.60 | 49.76 | 49.74 | 49.77 |
| PIQA | 59.07 | 59.18 | 58.92 | 58.93 | 58.94 | 58.89 |
| WinoGrande | 49.38 | 49.38 | 49.54 | 49.64 | 49.41 | 49.46 |
| Macro | 44.84 | 45.18 | 45.42 | 45.48 | 45.40 | 45.52 |
| Task | Adapter | Mode | Latency median [min,max] ms | Questions/s | Incremental peak MiB |
|---|---|---|---|---|---|
| ARC-C | IA3 | frozen | 8.57 [8.43, 10.73] | 116.7 | 34.8 |
| ARC-C | IA3 | dynamic | 18.90 [18.35, 25.23] | 52.9 | 35.0 |
| ARC-C | IA3 | fixed-unfolded | 9.30 [9.22, 9.48] | 107.6 | 35.1 |
| ARC-C | IA3 | compiled | 8.43 [8.41, 10.93] | 118.6 | 34.8 |
| ARC-C | LoRA | frozen | 8.47 [8.46, 13.64] | 118.0 | 34.8 |
| ARC-C | LoRA | dynamic | 18.29 [18.07, 18.61] | 54.7 | 35.0 |
| Method | Params | ARC-C | SciQ | Note |
|---|---|---|---|---|
| IA3 | 18,432 | 24.4 0.3 | 51.5 0.9 | static scale |
| Prompt tuning | 15,360 | 27.4 2.3 | 49.0 0.3 | 20 virtual tokens |
| Prefix tuning | 1,302,656 | 31.7 2.5 | 32.3 0.6 | projected prefix |
| PREFACE-IA3-small | 147,540 | 28.7 2.8 | 52.6 0.5 | IA3 + prefix controller |
Parameter-Efficient Fine-Tuning
Parameter-efficient fine-tuning adapts pretrained models by updating small parameter subsets or lightweight modules instead of all weights.
See all 646 papers