Heirloom: Carrying a Phone's Personalization Across Base-Model Generations
Authors: Zhuomin Chen, Jingchao Ni, Qingkai Dong, Xu Zheng, Gabriel Hübner Lucchesi, Mo Sha, Wei Cheng, Dongsheng Luo
Organizations: Florida International University, Miami, FL, USA · University of Houston, Houston, TX, USA · University of Connecticut, Storrs, CT, USA · Independent Researcher, USA · NEC Laboratories America, Inc., Princeton, NJ, USA · Singapore Management University, Singapore
On-device language models can learn a user's preferences through adapters trained on private data. However, a base-model upgrade may change the layers or vocabulary, leaving the personal adapter incompatible. Replacing it with a non-personal adapter loses much of the personalization, while retraining requires retaining the user's labeled history. We present Heirloom, a system that transfers a personal adapter to a new base model on the phone without retaining labels. Heirloom uses the old personal model as a teacher, collecting its predictions on stored unlabeled inputs, vendor-provided prompts, or prompts generated by the new base. It then unloads the old base and trains an adapter for the new model to match those predictions, keeping only one base model in memory at a time. We evaluate Heirloom across three generation upgrades from two model families, an architecture-preserving refresh, and three personalization tasks. On movie tagging, migration from stored inputs maintains pre-upgrade accuracy and differs from labeled retraining by at most 1.7 percentage points across the three upgrades. On a phone's neural processing unit (NPU), a fixed-length migration for the 4B upgrade takes 38 minutes and consumes 20% of the battery end to end, making the stored-input setting feasible during overnight charging.
Figures & tables
Figure 1: Heirloom migrates a personal adapter across a base-model upgrade. (a) The old personal model, the user’s adapter on the old base. (b) The vendor ships a new base that is incompatible with the old adapter, and retraining needs labels the phone did not keep. (c) Heirloom trains the new adapter to match the old model’s outputs on the stored inputs.
Model
Layers
Attn. / linear
Vocabulary
Qwen3-1.7B
28
28 / 0
151,936
Qwen3.5-2B
24
6 / 18
248,320
Qwen3-4B
36
36 / 0
151,936
Qwen3.5-4B
32
8 / 24
248,320
Phi-3.5-mini
32
32 / 0
32,064
Phi-4-mini
32
32 / 0
200,064
Table 1: Base models of the three upgrades and the refresh, from each model’s configuration. Linear-attention layers are Gated DeltaNet layers Yang et al. (2025b) .
Figure 2: The step from Qwen3-1.7B to Qwen3.5-2B, drawn from the configurations in Table 1 . The adapter attaches to the projections of each of the old base’s attention layers. The new base has fewer layers, three of every four of them linear-attention layers with other projections, and a larger vocabulary whose token ids differ. The feed-forward projections keep their shape.
Figure 3: Per-user accuracy before and after each upgrade. Crosses, the new base with the vendor’s adapter, fall below the diagonal for most users. Dots, Heirloom with stored inputs, lie on both sides of it. One point per user, four folds pooled.
Figure 4: The old personal model’s outputs on past inputs largely reproduce the user’s labels. Left, for each user, the share of stored inputs whose teacher top label is the user’s tag (dots), its accuracy on unseen films (circles) and the non-personal teacher’s share (crosses). Right, the teacher’s top-label probability by prompt source. 2B upgrade, fold 0, 19 users.
Figure 5: Heirloom at a base-model upgrade. Phase 1 runs the old personal model once over the prompts available at the user’s retention level and stores its outputs. Phase 2 trains a new adapter on the new base, starting from the vendor’s adapter, to reproduce those outputs. The strip below shows the one base in memory at each step.
2B upgrade
4B upgrade
Phi upgrade
Qwen3-1.7B → 3.5-2B
Qwen3-4B → 3.5-4B
Phi-3.5 → Phi-4-mini
Method
Acc.
Δ
Flips
Acc.
Δ
Flips
Acc.
Δ
Flips
Before the upgrade
54.3
58.9
57.0
No personalization
No adapter
19.9
−34.5
75
34.3
−24.5
53
26.3
−30.8
62
Vendor’s adapter
38.9
−15.4
47
46.7
−12.1
38
43.3
−13.7
38
Table 2: Movie-tagging accuracy and negative flip rates across three model upgrades (19 users, four folds). Δ is the mean per-user accuracy difference from the old personal model, in percentage points. Flips is the negative flip rate (%). Dashes indicate inapplicable comparisons.
Figure 6: With stored inputs and no labels, Heirloom keeps the pre-upgrade accuracy on all three upgrades. Dots are the mean per-user difference to the old personal model over 19 users and four folds, bars its 95% interval. The filter baselines reported in Tables 2 and 15 , together with methods outside the plotted range in Tables 2 and 3 , are omitted.
2B upgrade
4B upgrade
Phi upgrade
Qwen3-1.7B → 3.5-2B
Qwen3-4B → 3.5-4B
Phi-3.5 → Phi-4-mini
Method
Acc.
Δ
Flips
Acc.
Δ
Flips
Acc.
Δ
Flips
Before the upgrade
54.3
58.9
57.0
Nothing kept
Heirloom , self-generated
53.1
−1.3
16
56.7
−2.2
15
54.9
−2.1
15
Heirloom , 128 public prompts
53.3
−1.0
16
56.6
−2.2
16
54.4
−2.6
14
Table 3: Movie-tagging results for alternative prompt sources and retention budgets (19 users, four folds). Δ and Flips are defined in Table 2 . The shaded row repeats the stored-input result.
Figure 7: On the 2B and Phi upgrades, Heirloom with stored inputs changes fewer of the answers the old model got right than either retraining baseline. Left bars give the negative flip rate of Tables 2 and 3 , and right bars the share of the old model’s wrong answers now right, averaged over 19 users and four folds.
Figure 8: Held-out selection (stars) lands within a point of the best single length. Lines show accuracy relative to the old personal model after 2 to 16 epochs on all prompts, per prompt source, averaged over 19 users and four folds.
Method
Acc.
Δ [95% CI]
Flips
Before the upgrade
58.9
–
–
Copy the old adapter
51.6
−7.2 [−9.5, −5.2]
23
Heirloom , stored inputs
58.1
−0.8 [−2.2, +0.7]
13
Heirloom , self-generated
55.7
−3.2 [−5.2, −1.3]
16
Retraining, fixed recipe
58.3
−0.5 [−2.3, +1.1]
17
Vendor’s adapter
44.5
−14.3 [−19.6, −9.6]
37
Table 4: Accuracy and negative flips following the Qwen3-4B refresh (19 users, four folds). Δ is the difference from the old personal model, with 95% confidence intervals. Retraining uses eight epochs at a learning rate of 10−4
Δ [95% CI] against
Method
Acc.
pre-upgrade
retraining
Flips
MAE
All 100 users
Before the upgrade
73.8
–
–
–
0.28
Retraining, fixed recipe
75.1
+1.3 [+0.8, +1.8]
–
12
0.26
Retraining, 128 stored inputs
73.8
0.0 [−0.6, +0.5]
−1.3 [−1.7, −1.0]
14
0.28
Heirloom , 128 stored inputs
72.9
−0.9 [−1.4, −0.5]
−2.2 [−2.8, −1.8]
12
0.29
Table 5: Product rating on the 4B upgrade (Qwen3-4B → Qwen3.5-4B), four folds, all 100 users and the first 20. Δ is the per-user difference in points with its 95% interval, against the old personal model and against retraining on the whole labeled history. MAE is the mean absolute error of the score. Each group’s last row compares Heirloom with retraining at 128 inputs each.
Figure 9: Product rating on the 4B upgrade, 100 users. Left, the rows of Table 5 as mean per-user differences to the old personal model with 95% intervals, its vendor’s adapter and no-adapter rows below this range. Right, per user, Heirloom with all stored inputs against Heirloom at the 128-input cap, both relative to the old personal model.
Personal
1.7B → 3.5-2B
4B → 3.5-4B
data kept
Method
R-L
Δ
R-L
Δ
–
Before the upgrade
12.66
–
13.41
–
inputs + labels
Retraining, fixed recipe
13.11
+0.45
14.62
+1.21
Retraining, best per user
13.35
+0.69
14.76
+1.35
inputs
Heirloom , 512 stored
12.31
−0.35
13.63
+0.22
Heirloom , 128 stored
11.61
−1.05
13.38
−0.03
Table 6: Headline generation on the two Qwen upgrades, 10 users. R-L is ROUGE-L F1 × 100 and Δ its difference to the old personal model. Retraining uses the user’s labeled history (510 to 1000 articles), while the stored-input rows use at most 512 or 128 inputs, without labels. Best per user is each user’s setting chosen on that user’s test set.
Figure 10: Headline generation, ROUGE-L relative to the old personal model against training epochs on all prompts, 10 users. Lines are the cells of Table 8 , Heirloom by prompt set and teacher, and retraining at its recipe’s learning rate. Stars are the held-out selection for 512 stored inputs. Each value subtracts the old personal model’s R-L of Table 6 .
Minutes
Tokens/s
Energy
Peak memory
Method
Phase 1
Phase 2
Total
phase 2
kJ
% battery
(GB)
Δ
2B upgrade, Qwen3-1.7B → Qwen3.5-2B
Stored inputs, selected length
1.0
71.8
72.7
77
29.3
30
2.6
–
Stored inputs, 8 epochs
1.0
20.0
21.0
77
8.3
8.6
2.6
+0.1
Retraining with labels
–
20.0
20.0
79
8.2
8.5
2.5
–
4B upgrade, Qwen3-4B → Qwen3.5-4B
Table 7: Cost of one migration on the OnePlus 15 for the user with the median number of stored inputs (83). Phase 1 is the old personal model’s teacher pass and phase 2 the training of the new adapter on the new base. Minutes are measured on the charger, and energy is in kJ and as a share of the battery. Δ is the accuracy at the fixed length minus that at the selected length, from the GPU runs.
Figure 11: On the two phones, every measured migration from stored inputs fits in an 8-hour overnight charge. Left, minutes on the charger, as phase 1, selection run and final training. Right, the energy from the per-token rates, idle draw subtracted, as a share of the battery. Bars cut at the axis end continue in Tables 7 and 20 .
Figure 12: The 4B stored-input migration on the OnePlus 15 at a fixed eight epochs, followed by labeled retraining, was measured end to end on battery. Table 7 reports the corresponding values. Top, the charge counter as a share of the full charge. Bottom, available memory from each job’s step log, with the idle reading and the old base’s weights marked.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Row
Pair
1
2
4
8
16
Heirloom , 512 stored
2B
12.31 10
12.37
12.18
12.18
12.32
4B
13.63 10
13.64
13.51
13.76
13.35
Heirloom , 128 stored
2B
11.52 8
12.03 2
11.74
11.79
11.67
4B
13.34 9
13.44 1
12.75
12.73
12.98
Heirloom , public
2B
12.12 10
11.74
11.56
11.40
11.17
4B
13.33 10
12.63
12.88
12.85
12.53
Appendix
Table 8: Headline generation, ROUGE-L F1 × 100 over 10 users at every training length (epochs). Distillation rows show the run on all prompts at each checkpoint, with a superscript counting the users for whom held-out selection picked that length. Retraining rows show every learning rate and length run. lr denotes learning rate.
Epochs chosen (users)
Method
R-L
Δ
Δ to raw text [95% CI]
1
2
4
8
16
2B upgrade, Qwen3-1.7B → Qwen3.5-2B
Raw text, rerun
12.25
−0.41
–
10
0
0
0
0
+ ULD, λ=1.5 , k=1024
12.50
−0.16
+0.25 [−0.39, +0.91]
1
8
1
0
0
+ ULD, λ=0.5 , k=1024
12.44
−0.23
+0.19 [−0.18, +0.61]
6
4
0
0
0
4B upgrade, Qwen3-4B → Qwen3.5-4B
Appendix
Table 9: Universal logit distillation (ULD) on headline generation, ROUGE-L F1 × 100, 10 users. The rows are Heirloom ’s text distillation from at most 512 stored inputs, rerun for this comparison, and the same with the ULD term at weight λ . Δ is the difference to the old personal model and, with its 95% interval, to the raw-text row. Epochs chosen counts the users for whom held-out selection picked each length.
Table 22
1 epoch
2 epochs
4 epochs
8 epochs
16 epochs
Upgrade
Rate
Val.
Test
Val.
Test
Val.
Test
Val.
Test
Val.
Test
Movie tagging, retraining with labels
Qwen3-1.7B → Qwen3.5-2B
5×10−5
57.4
58.1
57.7
57.0
57.9 ∗
56.7
57.9
56.9
10−4
54.6
54.5
52.8
55.7
55.6
56.1
54.9
56.0
2×10−4
48.7
52.0
49.7
52.1
54.1
52.6
5×10−4
37.4
39.3
36.9
39.4
36.9
38.8
Appendix
Table 12: Retraining with labels at every cell of its grid, accuracy × 100. Val. is the pooled validation accuracy of the first fold’s users, each trained on four fifths of the history and scored on the rest, and Test the accuracy over all four folds, averaged over users. The asterisk marks the fixed recipe (ties go to fewer epochs, then the lower rate).
Adapter rank
Method
8
16
32
Vendor’s adapter
36.5
36.2
34.5
Before the upgrade
54.9
55.2
54.9
Retraining, rank-16 recipe
53.9
55.2
56.4
Retraining, re-selected recipe
54.6
55.2
56.0
Heirloom , stored inputs, at 2×10−4
56.6
54.3
50.3
Appendix
Table 13: Adapter rank on the 2B upgrade, first fold, 19 users, accuracy × 100. Retraining uses the rank-16 and the re-selected recipe, and Heirloom trains at 2×10−4 and at the rate held-out selection picks. Δ is Heirloom at the selected rate minus retraining, with its 95% interval.
Method
Acc.
Δ [95% CI]
Flips
No adapter
26.3
−30.8 [−36.9, −25.6]
62
Vendor’s adapter
43.3
−13.7 [−19.2, −8.8]
38
Mapped adapter
8.9
−48.1 [−55.3, −41.0]
90
Mapped, then fine-tuned on 100 labels
26.2
−30.8 [−34.8, −27.1]
66
Vendor’s adapter, the same fine-tuning
57.7
+0.7 [−1.1, +2.5]
16
Fresh LoRA, the same fine-tuning
47.6
−9.4 [−11.9, −7.1]
33
Appendix
Table 14: LoRASuite mapping on the Phi upgrade (19 users, four folds). Accuracy and negative flip rates are reported before and after fine-tuning with up to 100 labeled inputs. Fine-tuning starts from the mapped adapter, the vendor adapter, or a fresh LoRA. Δ and its 95% confidence interval are relative to the old personal model.
Method
Acc.
Δ old
Δ unfiltered
Flips
2B upgrade, Qwen3-1.7B → 3.5-2B
Self-generated, unfiltered (rerun)
52.9
−1.4 [−3.1, +0.2]
16
Trans-LoRA filter
53.1
−1.3 [−2.8, +0.3]
+0.2 [−0.8, +1.1]
15
TuneShift-KD filter
49.8
−4.6 [−6.8, −2.3]
−3.1 [−4.6, −1.7]
21
4B upgrade, Qwen3-4B → 3.5-4B
Self-generated, unfiltered (rerun)
56.8
−2.1 [−3.3, −0.9]
14
Appendix
Table 15: Adapted synthetic-prompt filters on movie tagging (19 users, four folds). Δ old compares with the pre-upgrade personal model, while Δ unfiltered compares with unfiltered synthetic prompts. Both differences include 95% confidence intervals.
Per-user adapter, from scratch
Per-user adapter, from the vendor’s
Old base
No adapter
Vendor’s adapter
Acc.
Δ
up/down/tie
Acc.
Δ
up/down/tie
Movie tagging
Qwen3-1.7B
21.4
39.2
45.4
+6.2
13/6/1
52.6
+13.3
15/4/1
Qwen3-4B
28.4
42.9
48.2
+5.3
11/8/1
57.6
+14.7
16/2/2
Phi-3.5-mini
31.1
40.8
48.4
+7.6
12/6/2
55.3
+14.6
16/3/1
Product rating
Appendix
Table 16: Personal gain on each old base before any upgrade, one split per user, 20 users per task. The columns give accuracy × 100 without an adapter, with the vendor’s adapter, and with per-user adapters trained from scratch or from the vendor’s adapter. Δ is the per-user adapter’s difference to the vendor’s adapter, and up/down/tie counts the users above, below and level with it.
Qwen3-1.7B → Qwen3.5-2B
Qwen3-4B → Qwen3.5-4B
Phi-3.5-mini → Phi-4-mini
Method
Acc.
Δ
Acc.
Δ
Acc.
Δ
Before the upgrade
48.1
–
51.2
–
51.4
–
Retraining, fixed recipe
46.8
−1.3
51.7
+0.6
48.3
−3.1
Heirloom , stored inputs
48.9
+0.8
48.8
−2.4
51.0
−0.4
Heirloom , self-generated
47.3
−0.8
50.2
−0.9
49.1
−2.3
Non-personal teacher
38.1
−10.0
42.5
−8.7
37.6
−13.8
Appendix
Table 17: Movie-tagging results for the excluded user with 3,190 films (three upgrades, four folds). Δ is the accuracy difference from the old personal model.
Personal
data kept
Method
Acc.
Δ
95% CI
Flips
MAE
Qwen3-4B → Qwen3.5-4B
–
Personal model before the upgrade
74.9
–
–
0.26
inputs + labels
Retrieval, 8 labeled examples in the prompt
69.4
−5.4
[−8.9, −2.6]
21
0.32
Retraining with labels, fixed recipe
76.8
+1.9
[+0.6, +3.3]
10
0.24
Retraining with labels, per-user tuning
76.5
+1.6
[+0.4, +2.9]
11
0.24
Appendix
Table 18: Full product-rating results for the first 20 users on the 4B and Phi upgrades (four folds). Δ and its 95% confidence interval are relative to the old personal model. Fixed-16-epoch rows report accuracy only. The 128-input retraining baseline uses a separately selected recipe.
Row
Pair
2
4
8
16
Heirloom , 128 stored
4B
73.0 13
73.9 21
75.1 20
75.3 26
Phi
74.2 13
75.3 26
75.8 17
76.0 24
Heirloom , all stored
4B
74.4 15
75.7 18
76.3 18
76.4 29
Phi
75.6 22
77.0 12
77.3 22
77.5 24
Heirloom , public
4B
73.2 3
74.0 17
74.1 26
74.0 34
Phi
74.5 6
74.7 13
75.2 33
74.9 28
Appendix
Table 19: Heirloom ’s accuracy × 100 on product rating after each number of training epochs, for the first 20 users over four folds. A superscript counts the user-folds whose held-out selection picked that length. The 4B upgrade is Qwen3-4B → Qwen3.5-4B and the Phi upgrade Phi-3.5-mini → Phi-4-mini.
Minutes
Tokens/s
Energy
Peak memory
Method
Phase 1
Phase 2
Total
phase 2
kJ
% battery
(GB)
Δ
2B upgrade, Qwen3-1.7B → Qwen3.5-2B, OnePlus 13T
Stored inputs, selected length
1.1
70.9
72.0
78
58.7
91
2.6
–
Stored inputs, 8 epochs
1.1
19.7
20.8
78
16.7
26
2.6
+0.1
Retraining with labels
–
20.6
20.6
77
16.8
26
2.6
–
Public prompts, 4 epochs
5.6
59.9
65.5
81
54.6
84
2.8
−0.3
Appendix
Table 20: Migration costs on the OnePlus 13T for the 2B upgrade, using the same user and measurement protocol as Table 7 . Time is measured while charging. Energy is estimated from battery measurements with idle consumption excluded. Δ is the GPU accuracy difference between fixed and selected training lengths.
Multi-billion-parameter LLMs now run on phones for inference, and training them on the device would personalize them without user data leaving the phone. Prior work has measured individual training steps of such models on phones, but not complete training runs, and not whether adapters trained on the device improve personalization. We present the first systematic characterization of a multi-billion-parameter LLM fine-tuned on a mobile device, covering memory, per-step time, thermal behavior, and energy. An iPhone 17 Pro can fine-tune a 3B-parameter LLM to a typical user within one battery charge, and the resulting adapters improve personalization as much as adapters trained on a server. Sustained training throttles the phone to about half its initial throughput, and none of the pausing or burst schedules we tested recovers it. Nearly all of each training step is spent in the frozen base model, most of it in the backward pass, which nine of the ten other runtimes we audited do not accelerate. Apple's MLX had a kernel for it that was never dispatched and was incorrect, and our repair, now merged upstream, trains an adapter 1.47x faster on a third less energy. On-device fine-tuning is feasible on current phones, and making it efficient requires runtimes and operating systems to treat training as a first-class workload.
Organizations maintain task-specific adapters for open-weight language models, and each new base-model release forces a migration decision: retain existing specialists, port adapters, refresh from preserved behavior, or retrain. Prior transfer work evaluates isolated model pairs, without studying these choices across real model release sequences. We present UpgradeBench, a decision-driven longitudinal benchmark covering four consecutive Qwen releases, one continuation checkpoint, six tasks, and two model sizes, augmented by OLMo checkpoints with known training lineage. The benchmark disentangles three core questions: whether a new checkpoint improves fixed-recipe retrained specialist performance, whether specialization assets transfer across versions, and what recovery resources are usable. We observe upgrade gains differ across task-scale-release episodes: some retrained baselines improve while others stay within training noise, with durability ranging from under one release interval for text-to-SQL to over fourteen months for intent classification. Direct adapter copying depends neither on architecture nor model family: on OLMo, retention drops from 0.88-0.99 at 46B-token continued pretraining to zero at 2.9T tokens; annealing and model souping introduce no extra harm, with portability decaying with continued-pretraining distance. Given preserved input data, teacher relabeling recovers target-base specialists without fresh gold annotations, though compute savings are not guaranteed. Simulating a fixed decision policy over 33 upgrade episodes yields 0.37pp mean quality regret with zero behavioral regressions at one-third the compute and label cost of full retraining. A lightweight CKA probe over 256 prompts predicts cross-version adapter portability (Spearman 0.74 across eight model pairs). We release per-example predictions, cost logs, split manifests, and evaluation code.
Ye Chen, Weining Zhang
Alibaba Group · Cheung Kong Graduate School of Business
On-device fine-tuning lets a large language model (LLM) learn from its user's private data without uploading it. Low-rank adaptation (LoRA) reduces the trainable parameters, but each first-order training step still multiplies by the frozen base-model weights during the forward pass, activation recomputation, and input-gradient computation. A phone's neural processing unit (NPU) already holds these weights for inference; using it for training also requires computing input gradients, which its inference interface does not support. We present a system that runs all three frozen-weight products on the NPU while keeping the training graph, adapters, and optimizer on the CPU. Its input-gradient kernels reuse the resident weight tiles and the forward dequantizer, avoiding a transposed weight copy and ensuring forward and backward use the same effective weights. To preserve usable gradients, the system selects precision by gradient sensitivity and applies a per-row range guard for half-precision (FP16) outputs. Its runtime fuses the remaining CPU operators and overlaps NPU execution with adapter computation, while resumable execution and thermal pacing let training proceed as a background job. We evaluate it on two Qualcomm Snapdragon platforms using four LLMs from the Qwen and Llama families. Our workloads cover nightly personalization with labels supplied on the phone and the public LaMP-2 benchmark; comparisons include an optimized CPU trainer, a phone-GPU trainer, forward-only training, and GPU-trained references.
Xu Zheng, Wei Cheng, Zhuomin Chen +4
Florida International University, Miami, FL, USA · NEC Laboratories America, Princeton, NJ, USA · North Carolina State University, Raleigh, NC, USA +2