Always-on virtual assistants increasingly personalize on-device to new user-specific commands via few-shot fine-tuning (e.g., LoRA), but this adaptation is evaluated almost exclusively on whether it learns the new commands, not on whether the assistant can still recognize an input that matches none of its intents - the out-of-scope (OOS) rejection problem. We show that naive few-shot LoRA personalization degrades OOS-rejection AUROC (area under the ROC curve) on every one of five public intent-classification benchmarks (mean drop 0.055--0.258, five seeds each), while in-scope accuracy is preserved. A scorer-invariant check shows part of this drop is an artifact of the standard confidence scorer's own miscalibration under adaptation, though a smaller, genuine effect remains under any scorer; we trace it to a measurable shrinkage of the embedding margin around known-intent regions (pooled Spearman rho=0.61$). A deployment-cheap fix is scoring rejection by Mahalanobis distance instead of softmax confidence without extra training/retraining recovers most of the lost quality on 3 of 5 datasets at no accuracy cost, beats two further OOD scoring baselines, and generalizes across shot count, LoRA rank, and a second base encoder. We report honestly where the fix falls short and release all code for reproducibility.
Figures & tables
Fig. 1: Naive few-shot LoRA personalization degrades OOS-rejection AUROC on every one of 5 intent benchmarks (mean over 5 seeds), while in-scope accuracy is preserved.
Fig. 2: Out-of-scope rejection AUROC before and after naive few-shot LoRA personalization, mean ± std over 5 seeds. Every dataset drops; the 95% confidence interval on the drop excludes zero in every case (Table I ).
Fig. 3: Embedding-margin shift (Mahalanobis distance from known-class centroids to out-of-scope points, minus intra-class distance, before minus after personalization) versus OOS-rejection AUROC drop, one point per (dataset, seed). Pooled Spearman ρ=0.61 ( p=0.0012 , n=25 ).
Dataset
Margin shift (Mahalanobis)
Spearman ρ (Mahalanobis)
Spearman ρ (cosine)
CLINC150
2063.7 ± 163.3
0.60 ( p =0.28, n =5)
0.70 ( p =0.19)
Banking77
466.0 ± 87.1
-0.10 ( p =0.87, n =5)
0.50 ( p =0.39)
HWU64
641.8 ± 92.1
0.40 ( p =0.50, n =5)
0.90 ( p =0.04)
SNIPS
4820.1 ± 2455.3
0.00 ( p =1.00, n =5)
-0.30 ( p =0.62)
MASSIVE
430.3 ± 250.1
0.50 ( p =0.39, n =5)
0.30 ( p =0.62)
Pooled (all datasets)
—
0.61 ( p =0.0012, n =25)
0.76 ( p =0.0000)
TABLE II: Embedding-margin shift and its correlation with AUROC drop.
Fig. 4: After-personalization OOS-rejection AUROC across all seven tested method configurations. Full fine-tuning is a diagnostic control, not a proposed fix (it defeats the on-device, parameter-efficient premise of this paper). Both Mahalanobis-scorer variants (purple/grey) beat naive LoRA (blue) and the calibration-replay-based combined method (pink) on every dataset; the orthogonal-projection-LoRA-only and calibration-replay-only components (green/orange) do not reliably beat naive LoRA alone.
Dataset
Naive LoRA
Full FT
+Mahalanobis
+OPLoRA+Mahalanobis
CLINC150
0.886 ± 0.046
0.947 ± 0.005
0.970 ± 0.001
0.971 ± 0.001
Banking77
0.806 ± 0.077
0.831 ± 0.060
0.878 ± 0.021
0.885 ± 0.015
HWU64
0.769 ± 0.076
0.810 ± 0.033
0.868 ± 0.039
0.868 ± 0.037
SNIPS
0.687 ± 0.116
0.861 ± 0.028
0.874 ± 0.046
0.877 ± 0.051
MASSIVE
0.787 ± 0.018
0.808 ± 0.035
0.847 ± 0.038
0.853 ± 0.034
TABLE III: After-personalization OOS-rejection AUROC, mean ± std ( n=5 seeds), key configurations. Full results (all 7 methods) in Fig. 4 and the released artifact.
Dataset
Naive LoRA
Combined-Maha
Chance level
Naive enrichment
CLINC150
0.681 ± 0.119
0.302 ± 0.150
0.033
20.4 ×
Banking77
0.535 ± 0.175
0.316 ± 0.145
0.077
7.0 ×
HWU64
0.550 ± 0.137
0.165 ± 0.122
0.093
5.9 ×
SNIPS
0.896 ± 0.176
0.679 ± 0.341
0.333
2.7 ×
MASSIVE
0.320 ± 0.142
0.246 ± 0.129
0.098
3.3 ×
TABLE IV: Fraction of false accepts (OOS test points scored in-scope) predicted as a new intent, naive LoRA vs. combined_maha (mean ± std, n=5 seeds), against the chance level under a uniform prediction over all classes.
Dataset
MSP drop
Maha-fixed drop
Energy AUROC
ODIN AUROC
Combined-Maha AUROC
CLINC150
0.078 ± 0.047
0.004 ± 0.000
0.922 ± 0.016
0.916 ± 0.018
0.971 ± 0.001
Banking77
0.067 ± 0.046
0.027 ± 0.009
0.801 ± 0.055
0.807 ± 0.057
0.885 ± 0.015
HWU64
0.097 ± 0.063
0.023 ± 0.011
0.786 ± 0.088
0.781 ± 0.089
0.868 ± 0.037
SNIPS
0.258 ± 0.139
0.091 ± 0.029
0.841 ± 0.060
0.829 ± 0.094
0.877 ± 0.051
MASSIVE
0.055 ± 0.037
0.027 ± 0.013
0.803 ± 0.029
0.805 ± 0.030
0.853 ± 0.034
TABLE V: Robustness checks: how much of the naive-LoRA drop is scorer choice (columns 2–3, mean ± std, n=5 seeds), and combined_maha versus two further OOD-scoring baselines (columns 4–6, after-personalization AUROC). ODIN: temperature-scaling half only.
Fig. 5: After-personalization OOS-rejection AUROC, naive LoRA vs. combined_maha , pooled across all 5 datasets ( n=25 ), across shot count, LoRA rank, and base encoder. The fix holds (non-overlapping 95% CI, higher AUROC, no accuracy cost) on 8 of 9 tested conditions; only k=1 misses the CI bar, directionally favorable but the noisiest cell in the sweep.