cs.LGMar 16, 2026
SaveW2T: LoRA Weights Already Know What They Can Do
Organizations: University of Surrey · Stanford University · University of Notre Dame
Abstract
Each LoRA checkpoint compactly stores task-specific updates in low-rank weight matrices, offering an efficient way to adapt large language models to new tasks and domains. In principle, these weights already encode what the adapter does and how well it performs. In this paper, we ask whether this information can be read directly from the weights, without running the base model or accessing training data. A key obstacle is that a single LoRA update can be factorized in infinitely many ways. Without resolving this ambiguity, models trained on the factors may fit the particular factorization rather than the underlying update. To this end, we propose \methodfull, which maps each LoRA update to a provably canonical form via QR decomposition followed by SVD, so that all equivalent factorizations share the same representation. The resulting components are then tokenized and processed by a Transformer to produce a weight-space embedding. Across language and vision LoRA collections, W2T achieves strong results on attribute classification, performance prediction, and adapter retrieval, demonstrating that LoRA weights reliably indicate model behavior once factorization ambiguity is removed. Code is available at https://github.com/xiaolonghan2000/Weight2Token.
Figures & tables
Figure 1: LoRA updates already encode downstream behaviors, but reading them requires a symmetry-aware representation. (a) LoRA checkpoints trained from raw data are treated as data objects for downstream tasks. (b) Symmetry-agnostic processing of raw LoRA factors. (c) Symmetry-aware decomposition of LoRA updates into canonical components.
Figure 2: Overview of the proposed W2T modeling framework for LoRA weight space. (a) LoRA factors are transformed into a canonical representation via symmetry-aware QR–SVD decomposition, yielding rank-wise components under a fixed deterministic convention. (b) Each rank-wise component is projected into a rank token through modulated tokenization. (c) Rank tokens within each position are seen as a set and aggregated via -guided pooling. (d) Position tokens are globally modeled to produce a weight-space embedding for downstream tasks.
| Algorithm | GoEmotions-LoRA | CelebA-LoRA | CUB-LoRA | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Macro-F1 | Micro-F1 | AUROC | Macro-F1 | Micro-F1 | AUROC | Macro-F1 | Micro-F1 | AUROC | |
| MLP | |||||||||
| CNN | |||||||||
| ViT | |||||||||
| GLNet | |||||||||
| W2T (Ours) | 5.67 | 24.84 | 61.25 | 50.38 | 75.02 | 89.64 | 18.34 | 43.24 | 73.72 |
Table 1: Attribute classification results (%). The best value in each column is highlighted in bold .
| Method | Split | MAE | RMSE | Pearson | Spearman |
|---|---|---|---|---|---|
| MLP | Valid | ||||
| Test | |||||
| CNN | Valid | ||||
| Test | |||||
| ViT | Valid | ||||
| Test |
Table 2: Performance prediction results (%) on ARC-LoRA. Best values are in bold .
| Method | Average | ARC-C | BoolQ | GSM8K | MBPP |
| RawCos | 33.41 | ||||
| MLP | |||||
| CNN | |||||
| ViT | |||||
| GLNet | |||||
| W2T (Ours) | 65.71 | 99.14 | 50.76 | 87.96 |
Table 3: Adapter retrieval results (NDCG@10, %). All learned encoders are trained on ARC-Easy-LoRA performance prediction and transferred to retrieval without retrieval-specific tuning. RawCos is a training-free baseline that ranks adapters by cosine similarity in raw LoRA weight space. Best values are in bold .
Figure 3: Local retrieval neighborhood for a 8-shot BoolQ query. Each subplot shows the same query (star) amid a shared pool. W2T places relevant adapters (dark) closest to the query with top-1 hit and rel@5=5, while baselines surface more irrelevant neighbors.
Figure 4: Ablation on attribute classification. Results are reported on CelebA-LoRA. Removing QR–SVD causes the largest performance drop, while removing rank-level or position-level modeling leads to smaller but consistent degradations.
Figure 5: Practical GL( ) invariance. The horizontal axis is perturbation strength in random GL( ) reparameterization (larger means stronger perturbation). Left: embedding drift under increasing perturbation. Right: prediction consistency under the same perturbation schedule.
| Property | Value |
|---|---|
| Singular-value gap | |
| Update gap | |
| Factor cosine (mean) | |
| Time (ms), SVD / QR–SVD | |
| Speedup |
Table 4: Practical equivalence of QR-based core SVD and direct SVD. Unless noted otherwise, values are per-pair medians; Factor cosine reports the mean of per-pair minimum subspace cosines.
Figure 6: Scaling behavior from a training-data ratio of 0.7 onward. Markers are raw Macro-F1 values. Dashed curves are 3-point centered moving-average trends. The lower panel reports pointwise . Over this displayed range, W2T follows a steeper upward trend than GLNet.
| Method | Macro-F1 | Micro-F1 | mAUROC |
|---|---|---|---|
| MLP | |||
| CNN | |||
| ViT | |||
| GLNet | |||
| W2T (Ours) | 48.06 | 73.84 | 89.23 |
| Oracle (SD1.5) |
Table 5: Base-model transfer on CelebA-LoRA (%). All models are trained on SD1.4 LoRA checkpoints and evaluated zero-shot on SD1.5 at the same rank ( ). Oracle denotes the in-domain W2T result on SD1.5 under the same protocol.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Raw data LoRA | Key settings | Size | Checkpoint label |
| CelebA | one identity one LoRA; 21 images per identity | q/v; ; ; lr 1e-4/3e-4/1e-3/3e-3; steps 100/133/167/200 | 10,177 | 40-d identity-level attribute vector |
| CUB | one image one LoRA | q/v; ; ; lr 1e-4/3e-4/1e-3/3e-3; steps 100/150/200/250 | 11,788 | 312-d image-level attribute vector |
| GoEmotions | one train sample one LoRA | q/v; ; ; lr 5e-6–3e-4; epoch 1–4; dropout 0–0.1 | 20,000 | 28-d emotion multi-hot vector |
| ARC-Easy | one full-data trial one LoRA | q/v; ; ; lr 5e-6–3e-4; epoch 1–4; dropout 0–0.1 | 10,000 | checkpoint accuracy |
| ARC-Challenge BoolQ GSM8K MBPP | Full-datasets runs gallery LoRAs | q/v; ; ; lr 5e-6–3e-4; epoch 1–4; dropout 0–0.1 | 1,296 | task identity |
| few-shot runs query LoRAs | q/v; ; ; lr 2e-4; epoch 3; dropout 0.05; shots 1/8/16/64/128/256; 25 queries per shot | 600 | task identity |
Table 6: Dataset-level summary of LoRA construction in Appendix B. Shared backbone choices are described in the surrounding text.
| Attribute | W2T | Image-based | Summary | |||
| (ours) | DMM-CNN | PS-MCNN-LC | GNAS | Best img. | Gain | |
| Wearing Necklace | 96.41 | +7.43 | ||||
| Narrow Eyes | 95.10 | +6.03 | ||||
| Big Lips | 76.47 | +3.34 | ||||
| Straight Hair | 88.24 | +2.28 | ||||
| Rosy Cheeks | 98.37 | +1.45 | ||||
Table 7: Selected CelebA attributes (%) where W2T is strongest or near-best against image-based methods. Gain is computed against the best image baseline in each row.
| W2T | GLNet | MLP | CNN | ViT | |
| Params (M) | 17.75 | 23.16 | 51.06 | 77.75 | 39.91 |
Table 8: Parameter counts of the representative CelebA models. Values are measured from the saved experiment checkpoints.
Figure 7: Two-dimensional visualization of W2T weight-space embeddings for four highly separable CelebA attributes ( Gray Hair , High Cheekbones , Wearing Hat , and Wearing Necklace ). Each point represents one LoRA checkpoint and colors denote binary labels. Clear class-level clustering emerges across different attribute types, showing that LoRA weight space encodes semantically aligned structure.
| Shots | Rel. cost | Hit@1 (%) | NDCG@10 (%) |
|---|---|---|---|
| 1 | 38.00 | 42.77 | |
| 8 | 63.00 | 60.94 | |
| 16 | 62.00 | 61.90 | |
| 64 | 69.00 | 65.71 | |
| 128 | 65.00 | 63.65 | |
| 256 | 71.00 | 72.03 |
Table 9: W2T retrieval quality under varying query-shot budgets. Relative cost is normalized by the 256-shot setting.
Figure 8: Marginal MRR gain between successive query-shot levels under W2T retrieval. Early budget increases produce the largest returns, while later increments yield smaller and more dataset-dependent improvements.