Test-Time Prompt Tuning (TPT) adapts large pretrained models to unseen target distribution shifts by updating only lightweight prompt tokens. Existing TPT methods are effective, but they mainly optimize each prompt update locally. Across test-time steps, prompts updated at the current step are reused for the corresponding predictions and updates, so noisy or biased unlabeled objectives can accumulate and destabilize the prompt-update trajectory. We identify two failure modes: magnitude expansion, where updates move the prompt too far, and directional drift, where consecutive updates point in inconsistent directions. To analyze and control these failures, we propose Test-Time Prompt-Agnostic Decomposition (TPD), which characterizes observed prompt-update trajectories from a dynamical-systems view and decomposes base prompt updates into magnitude and direction. TPD computes a spectral radius to measure update expansion and decomposes update directions into persistent, oscillatory, and residual components. Building on this decomposition, Adaptive Koopman Control (AKC) regulates update magnitude by shrinking updates under expansive recent dynamics, while Hankel Update Router (HUR) refines update direction by preserving persistent components and suppressing oscillatory and residual components. Together, AKC and HUR produce a stabilized prompt update. As a prompt-agnostic framework, TPD can be plugged into visual and text TPT methods without modifying the backbone, prompt architecture, or adaptation loss. Experiments on 15 datasets show that TPD consistently improves 10 baselines, with accuracy gains of 3--8% for visual prompts and over 2% for text prompts.
Figures & tables
Figure 1: (a) The accuracy of ViT visual and CLIP text prompts in episodic modes, averaged over four ImageNet variants. Gains denote TPD improvements. The zoomed windows show accuracy, magnitude (spectral radius ρ ), and direction ( ℓ2 distance between consecutive update vectors, decomposed in (b)) in unstable and stable regimes. (b) TPD decomposes each prompt update into persistent, oscillatory, and residual components, with the spectral radius measuring local expansion. (c) Stability-performance relation summarized by spectral radius and persistent ratio.
Figure 2: Overview of TPD . The base prompt update ut from visual or text TPT is delay-embedded with a Hankel window and projected into a local Hankel subspace. AKC regulates update magnitude and HUR refines update direction. The controlled magnitude and refined direction are then combined into a stabilized prompt update gtuthur , which is applied to the prompt state at step t .
Method
ImageNet
ImageNet-V2
ImageNet-S
ImageNet-A
ImageNet-R
OOD Mean
Mean
ViT-based visual prompt tuning
ViT-B/16
81.44
70.85
35.44
30.92
47.48
46.17
53.23
VPT + TPD
78.16 +3.15
64.55 +5.32
32.77 +3.25
19.17 +19.81
42.14 +4.33
39.66 +8.18
47.36 +7.17
E2VPT + TPD
78.35 +2.69
64.09 +4.93
32.89 +2.70
18.17 +20.79
42.07 +3.94
39.31 +8.09
47.11 +7.01
VFPT + TPD
78.71 +2.62
65.34 +4.75
33.16 +2.81
22.05 +18.68
42.81 +3.55
40.84 +7.45
48.41 +6.48
BPT + TPD
81.30 +2.87
70.48 +2.45
28.62 +9.93
38.97 +4.23
43.68 +5.39
45.44 +5.51
52.61 +4.97
Table 1: Comparisons on the domain generalization setting with visual and text TPT methods. OOD Mean averages the 4 variants excluding ImageNet, while Mean averages ImageNet and the 4 variants. Values are the baseline accuracy with TPD gains.
Method
Caltech101
OxfordPets
StanfordCars
Flowers102
Food101
Aircraft
SUN397
DTD
EuroSAT
UCF101
Mean
ViT-based visual prompt tuning
ViT-B/16
86.71
92.20
57.84
66.23
82.78
22.06
56.07
48.19
49.91
65.08
62.71
VPT + TPD
92.65 +2.72
86.89 +2.49
60.68 +2.13
64.86 +3.23
85.14 +1.31
21.21 +3.25
59.92 +3.87
38.19 +4.06
20.91 +11.26
59.28 +2.73
58.97 +3.71
E2VPT + TPD
92.29 +2.01
86.89 +2.75
60.03 +2.08
63.85 +3.96
84.55 +1.26
20.79 +3.51
59.07 +3.48
37.93 +5.33
17.29 +11.67
58.75 +2.81
58.14 +3.89
VFPT + TPD
92.76 +2.36
87.33 +2.71
62.01 +2.41
67.77 +1.98
85.18 +1.43
21.69 +3.33
59.31 +2.64
40.53 +3.43
23.54 +12.66
60.26 +1.43
60.04 +3.44
BPT + TPD
79.51 +13.89
68.26 +14.45
49.88 +11.15
64.82 +2.33
67.45 +15.21
21.47 +4.15
48.50 +7.84
41.46 +1.96
57.42 +1.71
50.52 +9.59
54.93 +8.23
Table 2: Comparisons on the cross-dataset setting with visual and text TPT methods. All methods are evaluated on 10 downstream datasets without training. Values are baseline accuracy with TPD gains.
VPT
TPT
Variant
DG
CSD
×
DG
CSD
×
Baseline
39.66
58.97
1.00
60.81
65.10
1.00
Simple stabilization rules
+ Update clipping
41.92
59.54
1.11
61.36
65.47
1.06
+ EMA update
42.74
59.86
0.99
61.65
65.82
1.01
+ Norm control
43.65
61.03
1.07
61.72
65.74
1.03
Table 3: Comparison with simple stabilization methods on visual and text TPT methods. DG and CSD denote the accuracy means over four ImageNet variants and 10 cross-datasets. × : per-step wall-clock overhead relative to the baseline.
VPT
TPT
Variant
DG
CSD
DG
CSD
Baseline
39.66
58.97
60.81
65.10
Ablation study of Koopman operator modes
Full
43.58
60.86
61.80
66.38
Diagonal (+ AKC )
45.92
61.74
62.56
67.21
Ablation study of projection matrix modes
Table 4: Ablation study of TPD on visual and text TPT methods. DG and CSD denote the accuracy means over four ImageNet variants and 10 cross-datasets.
Figure 3: (a), (b) show Koopman spectra of visual prompt trajectories. The spectra are estimated post-hoc from generated update trajectories, and the black circle denotes the unit circle. (c), (d) show component ratios and spectral radius ρ over test steps.
Figure 4: (a), (b) CKA similarity between text-prompt representations across test steps after smoothing. (c) Koopman spectral radius estimated from prompt updates, the estimate is post-hoc for TPT and used online by AKC . (d) Persistent ratio of HUR versus subsequent accuracy gain. Each gray point denotes one short temporal window. The red dashed curve is a piecewise linear fitted trend line.
Figure 5: (a) ECE on ImageNet-A and ImageNet-R, with separate vertical scales. (b) Accuracy across batch sizes. Pink and blue in (a,b) denote each baseline and its TPD variant, respectively. (c) Proportions of stable ( ρt≤1 , light green) and unstable ( ρt>1 , light orange) TPD update steps, where ρt denotes the estimated spectral radius; each stacked bar sums to 100% .
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
Variant
ImageNet
ImageNet-V2
ImageNet-Sketch
ImageNet-A
ImageNet-R
OoD Mean 4
Baseline
78.16
64.55
32.77
19.17
42.14
39.66
+ Update clipping
78.84
65.52
34.09
22.95
45.11
41.92
+ EMA update
79.35
65.98
36.82
24.44
43.71
42.74
+ Norm control
79.76
66.66
34.26
25.60
48.09
43.65
+ Low-pass filtering
80.14
68.29
34.76
28.43
45.65
44.28
VPT+ TPD
81.31
69.87
36.02
38.98
46.47
47.84
Appendix
Table A1: Per-dataset simple stabilization results for VPT under domain generalization. OoD Mean 4 averages ImageNet-V2, ImageNet-Sketch, ImageNet-A, and ImageNet-R.
Variant
Caltech
Pets
Cars
Flowers
Food
Aircraft
SUN
DTD
EuroSAT
UCF
Mean
Baseline
92.65
86.89
60.68
64.86
85.14
21.21
59.92
38.19
20.91
59.28
58.97
+ Update clipping
92.94
87.35
61.27
65.16
85.85
21.93
60.57
38.48
21.78
60.03
59.54
+ EMA update
93.27
87.46
61.07
65.33
85.67
22.47
61.00
38.77
22.79
60.80
59.86
+ Norm control
93.10
87.71
63.33
66.75
86.59
26.40
60.40
40.54
24.25
61.19
61.03
+ Low-pass filtering
93.39
89.29
62.23
67.32
86.20
21.99
62.55
39.07
25.15
61.93
60.91
VPT+ TPD
95.37
89.38
62.81
68.09
86.45
24.46
63.79
42.25
32.17
62.01
62.68
Appendix
Table A2: Per-dataset simple stabilization results for VPT under cross-dataset transfer. Mean averages the ten downstream datasets.
Variant
ImageNet
ImageNet-V2
ImageNet-Sketch
ImageNet-A
ImageNet-R
OOD Mean
Baseline
68.98
63.45
47.94
54.77
77.06
60.81
+ Update clipping
69.90
64.12
48.40
55.09
77.84
61.36
+ EMA update
69.86
63.80
49.82
55.25
77.74
61.65
+ Norm control
70.11
63.64
50.05
55.80
77.39
61.72
+ Low-pass filtering
70.62
64.49
49.02
56.03
79.38
62.23
TPT+ TPD
71.19
65.98
49.62
56.73
79.98
63.08
Appendix
Table A3: Per-dataset simple stabilization results for TPT under domain generalization. OOD Mean averages the four shifted variants excluding ImageNet.
Variant
Caltech
Pets
Cars
Flowers
Food
Aircraft
SUN
DTD
EuroSAT
UCF
Mean
Baseline
94.16
87.79
66.87
68.98
84.67
24.78
65.50
47.75
42.44
68.04
65.10
+ Update clipping
94.48
88.06
67.12
69.63
84.88
25.17
65.59
48.52
42.78
68.49
65.47
+ EMA update
94.24
88.53
67.47
69.75
85.13
26.04
65.91
48.58
43.85
68.73
65.82
+ Norm control
94.60
88.77
67.84
68.98
85.26
25.80
65.77
49.00
42.93
68.49
65.74
+ Low-pass filtering
94.56
88.66
68.09
69.83
85.58
26.34
66.37
51.18
43.11
68.57
66.23
TPT+ TPD
96.32
90.31
68.55
71.06
88.32
27.19
68.51
51.98
46.03
72.00
68.03
Appendix
Table A4: Per-dataset simple stabilization results for TPT under cross-dataset transfer. Mean averages the ten downstream datasets.
Table A7: Per-dataset projection ablation for VPT. (a) Domain generalization.
Variant
Caltech
Pets
Cars
Flowers
Food
Aircraft
SUN
DTD
EuroSAT
UCF
Mean
Baseline
92.65
86.89
60.68
64.86
85.14
21.21
59.92
38.19
20.91
59.28
58.97
Plain projection
93.87
86.92
61.57
65.41
85.84
23.01
61.29
39.95
24.33
60.03
60.22
VPT+ HUR (Hankel)
95.21
89.26
64.99
67.07
86.85
25.26
64.69
40.37
26.81
61.30
62.18
VPT+ TPD
95.37
89.38
62.81
68.09
86.45
24.46
63.79
42.25
32.17
62.01
62.68
Appendix
Table A7: Per-dataset projection ablation for VPT. (b) Cross-dataset transfer.
Variant
ImageNet
ImageNet-V2
ImageNet-Sketch
ImageNet-A
ImageNet-R
OOD Mean
Baseline
68.98
63.45
47.94
54.77
77.06
60.81
Plain projection
69.46
64.31
48.65
54.99
77.78
61.43
TPT+ HUR (Hankel)
70.77
65.31
50.24
56.77
79.05
62.84
TPT+ TPD
71.19
65.98
49.62
56.73
79.98
63.08
Appendix
Table A8: Per-dataset projection ablation for TPT. (a) Domain generalization.
Variant
Caltech
Pets
Cars
Flowers
Food
Aircraft
SUN
DTD
EuroSAT
UCF
Mean
Baseline
94.16
87.79
66.87
68.98
84.67
24.78
65.50
47.75
42.44
68.04
65.10
Plain projection
94.77
88.17
67.42
70.20
85.43
26.04
66.19
48.76
43.56
69.52
66.01
TPT+ HUR (Hankel)
95.25
89.07
67.98
71.74
86.68
27.18
68.60
53.01
45.59
70.34
67.54
TPT+ TPD
96.32
90.31
68.55
71.06
88.32
27.19
68.51
51.98
46.03
72.00
68.03
Appendix
Table A8: Per-dataset projection ablation for TPT. (b) Cross-dataset transfer.
Variant
ImageNet
ImageNet-V2
ImageNet-Sketch
ImageNet-A
ImageNet-R
OoD 4
Baseline
78.16
64.55
32.77
19.17
42.14
39.66
w/o Pres.
81.33
69.32
35.99
36.51
46.41
47.06
w/o Osc.
81.36
69.38
36.48
36.65
46.61
47.28
w/o Res.
81.02
69.08
35.25
36.35
46.68
46.84
VPT+ TPD
81.31
69.87
36.02
38.98
46.47
47.84
Appendix
Table A9: Per-dataset HUR routing ablation for VPT. (a) Domain generalization.
Variant
Caltech
Pets
Cars
Flowers
Food
Aircraft
SUN
DTD
EuroSAT
UCF
Mean
Baseline
92.65
86.89
60.68
64.86
85.14
21.21
59.92
38.19
20.91
59.28
58.97
w/o Pres.
95.42
88.96
62.50
67.97
85.98
24.54
63.06
41.19
31.81
61.49
62.29
w/o Osc.
95.66
88.09
62.19
68.21
86.34
24.00
61.89
41.78
33.26
62.28
62.37
w/o Res.
94.69
88.96
62.16
67.84
86.01
23.85
63.08
43.03
30.81
61.64
62.21
VPT+ TPD
95.37
89.38
62.81
68.09
86.45
24.46
63.79
42.25
32.17
62.01
62.68
Appendix
Table A9: Per-dataset HUR routing ablation for VPT. (b) Cross-dataset transfer.
Variant
ImageNet
ImageNet-V2
ImageNet-Sketch
ImageNet-A
ImageNet-R
OOD Mean
Baseline
68.98
63.45
47.94
54.77
77.06
60.81
w/o Pres.
70.54
65.86
49.05
56.71
80.17
62.95
w/o Osc.
71.25
65.58
49.42
56.36
80.00
62.84
w/o Res.
71.70
66.02
49.22
56.36
79.61
62.80
TPT+ TPD
71.19
65.98
49.62
56.73
79.98
63.08
Appendix
Table A10: Per-dataset HUR routing ablation for TPT. (a) Domain generalization.
Variant
Caltech
Pets
Cars
Flowers
Food
Aircraft
SUN
DTD
EuroSAT
UCF
Mean
Baseline
94.16
87.79
66.87
68.98
84.67
24.78
65.50
47.75
42.44
68.04
65.10
w/o Pres.
96.27
90.35
68.00
70.65
88.08
27.06
68.88
51.65
45.48
70.90
67.73
w/o Osc.
96.92
89.89
68.10
70.36
89.18
26.16
68.99
52.01
46.30
70.82
67.87
w/o Res.
96.47
89.62
68.19
70.24
87.38
27.48
67.96
51.24
46.06
70.79
67.54
TPT+ TPD
96.32
90.31
68.55
71.06
88.32
27.19
68.51
51.98
46.03
72.00
68.03
Appendix
Table A10: Per-dataset HUR routing ablation for TPT. (b) Cross-dataset transfer.
Figure 6: Sensitivity to the Hankel window on domain-generalization datasets. Each heatmap reports per-dataset performance of VPT+ TPD or TPT+ TPD while sweeping the delay length K and the number of Hankel columns M . Colors are normalized within each panel, with warmer colors indicating higher accuracy. Both prompt types show a broad high-performing region at short-to-moderate windows, supporting the default K=3,M=8 .
Figure 7: Sensitivity to the Hankel window on cross-dataset transfer. The same (K,M) sweep is evaluated on ten downstream datasets. The high-performing region remains broad across datasets and prompt types, while excessively short or long histories are less reliable.
Figure 8: Sensitivity of AKC on domain-generalization datasets. Each heatmap sweeps the spectral-radius threshold ρˉ and gain floor gmin for VPT+ TPD or TPT+ TPD . Colors are normalized within each panel. Performance is strongest in a broad region around ρˉ=1.0 with a small gain floor, indicating that AKC benefits from suppressing locally expansive dynamics without over-damping stable updates.
Figure 9: Sensitivity of AKC on cross-dataset transfer. The ρˉ – gmin sweep shows that moderate stability control is robust across datasets, whereas a large gain floor weakens the controller and overly strict settings may suppress useful adaptation.
Figure 10: Sensitivity of HUR routing weights on domain-generalization datasets. Each heatmap sweeps the triangular grid 0≤βn≤βc≤1 , where βc controls the oscillatory component and βn controls the residual component; the masked region has βn>βc . Warmer colors indicate higher per-panel accuracy. The high-performing region favors small residual weights and moderate oscillatory attenuation, supporting selective routing rather than using the raw update unchanged.
Figure 11: Sensitivity of HUR routing weights on cross-dataset transfer. The triangular βc – βn sweep is consistent across VPT+ TPD and TPT+ TPD : retaining persistent components while attenuating oscillatory and residual components yields a broad high-performing region, whereas weights near one reduce HUR to weak routing and degrade performance.
Figure 12: Persistent ratio versus accuracy gain for TPT+ TPD . Each point corresponds to a local trajectory segment, and the red dashed line shows the fitted trend for each dataset.
Stream
Method
Overall
V2
Sketch
First-10
Abrupt
VPT
52.90±0.37
64.52
29.67
47.28
VPT+TPD
54.32±0.32
65.78
31.40
48.78
Mixed
VPT
46.10±0.25
65.09
27.11
N/A
VPT+TPD
47.63±0.19
66.19
29.08
N/A
Rapid Alternation
VPT
45.22±0.51
63.82
26.62
45.82
VPT+TPD
49.66±0.27
67.64
31.69
50.09
Appendix
Table A11: Accuracy (%) under nonstationary streams. Overall reports mean ± standard deviation across five paired test orders; the other accuracy columns report the corresponding means.
Method
Stream
P
O
R
Accuracy
VPT
Abrupt
17.54
0.71
81.75
52.90→54.32
VPT
Mixed
12.83
0.48
86.69
46.10→47.63
VPT
Rapid Alternation
5.09
1.26
93.65
45.22→49.66
TPT
Abrupt
3.30
0.45
96.25
57.31→57.92
TPT
Mixed
3.16
0.44
96.40
54.85→55.58
TPT
Rapid Alternation
3.19
0.46
96.35
55.09→55.82
Appendix
Table A12: Component ratios and accuracy (%) under nonstationary shifts. P, O, and R denote persistent, oscillatory, and residual fractions. Accuracy is reported as Base → Base+TPD.
VPT
TPT
Method
ImageNet-A
ImageNet-R
ImageNet-A
ImageNet-R
Base
19.25±1.97
42.32±0.61
54.44±1.12
77.01±2.05
Base+TPD
38.73±1.28
46.54±0.54
56.62±1.16
79.63±1.89
Appendix
Table A13: Accuracy (%) as mean ± standard deviation across three random seeds ( 4827 , 7316 , and 9054 ). Base denotes the corresponding VPT or TPT method without TPD.
VPT
TPT
Controller
ImageNet-A
ImageNet-R
ImageNet-A
ImageNet-R
Base
19.17
42.14
54.77
77.06
Adaptive Clipping
23.52
41.92
54.85
77.31
Variance-aware Scaling
22.76
43.65
55.19
77.26
TPD
38.98
46.47
56.73
79.98
Appendix
Table A14: Accuracy (%) of adaptive magnitude controllers using the same base updates and evaluation streams. Base denotes the corresponding VPT or TPT method without a controller.
Test-time adaptation (TTA) has emerged as a popular paradigm for improving the performance of vision-language models (e.g., CLIP) on downstream tasks. Among existing CLIP-based TTA methods, Test-Time Prompt Tuning (TPT) is a pioneering work that optimizes textual prompts using multiple test-time augmentations and remains a strong baseline to date. In this work, we revisit TPT and reveal that its optimization can be interpreted as implicitly learning from self-generated pseudo labels. Building on this perspective, we propose a unified self-ensembling framework (USE) that ensures consistency between the optimization and inference stages. During optimization, we introduce a simple yet effective self-ensembling (SE) strategy that emphasizes the test image itself over its augmented views adaptively to obtain more reliable pseudo labels. To fully exploit the potential of augmentations, we further apply the same strategy at inference time, unifying the objectives of both stages. Notably, SE can also act as a lightweight optimization-free TTA method. Extensive experiments across multiple datasets demonstrate that SE and USE outperform their counterparts, respectively. Furthermore, SE yields consistent performance gains when integrated with existing TTA methods. The code is available at https://github.com/sirujiang/USE.
Siru Jiang, Jian Liang, Ran He +1
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences +1
Test-time training (TTT) adapts a pretrained model to each prompt via parameter updates, improving accuracy under pretraining-to-test distribution shifts. Yet, its performance often suffers from instability and sensitivity to hyperparameters such as update steps and subspace. We explain this behavior through a decision-theoretic lens, treating TTT as implicit Bayesian inference in the kernel regime. Under a Gaussian process benchmark, we show that TTT reduces prediction error when updates are spectrally matched to the prompt's signal-to-noise ratio and aligned with query-relevant eigen-directions. This perspective underpins the following results: (1) we show when fixed update steps and subspaces fail under distribution shifts, motivating adaptive strategies; (2) we prove that selecting update steps via prompt evidence admits a PAC-Bayes guarantee against overfitting; and (3) we characterize the Bayes-optimal update subspace under a linear-Gaussian correction model, yielding a scoring rule for selecting Transformer blocks and heads. Our theory helps explain the empirical instability of TTT, taking a step toward principled guidance for when, how far, and which directions to adapt.
Tomoya Wakayama
RIKEN Center for Advanced Intelligence Project (AIP), Tokyo, Japan.
Test-Time Prompt Tuning (TPT) adapts vision-language models using augmented views, but its effectiveness is hindered by the challenge of determining which views are beneficial. Standard entropy-based filtering relies on the internal confidence scores of the model, which are often miscalibrated under distribution shift, assigning high confidence to irrelevant crops or background regions while ignoring semantic content. To address this, we propose a dual-modality anchor-guided framework that grounds view selection in semantic evidence. We introduce a text anchor from attribute-rich descriptions, to provide fine-grained class semantics, and an adaptive image anchor that captures evolving test-time statistics. Using these anchors, we filter views based on alignment and confidence, ensuring that only informative views guide adaptation. Moreover, we treat the anchors as auxiliary predictive heads and combine their predictions with the original output in a confidence-weighted ensemble, yielding a stable supervision signal for prompt updates. Extensive experiments on 15 benchmark datasets demonstrate new state-of-the-art performance, highlighting the contribution of anchor-guided supervision as a foundation for robust prompt updates.
Jungwon Choi, Eunwoo Kim
School of Computer Science and Engineering, Chung-Ang University