Test-Time Prompt-Agnostic Decomposition

Junze Wang; Lei Fan; Dezheng Zhang; Donglin Di; Yang Song; Sidong Liu; Cong Cong

cs.CVOct 26, 2026

Test-Time Prompt-Agnostic Decomposition

Authors: Junze Wang, Lei Fan, Dezheng Zhang, Donglin Di, Yang Song, Sidong Liu, Cong Cong

Organizations: University of Science and Technology Beijing · University of New South Wales · Sympflux Co., Ltd. · Macquarie University

Abstract

Test-Time Prompt Tuning (TPT) adapts large pretrained models to unseen target distribution shifts by updating only lightweight prompt tokens. Existing TPT methods are effective, but they mainly optimize each prompt update locally. Across test-time steps, prompts updated at the current step are reused for the corresponding predictions and updates, so noisy or biased unlabeled objectives can accumulate and destabilize the prompt-update trajectory. We identify two failure modes: magnitude expansion, where updates move the prompt too far, and directional drift, where consecutive updates point in inconsistent directions. To analyze and control these failures, we propose Test-Time Prompt-Agnostic Decomposition ( $\mathtt{TPD}$ ), which characterizes observed prompt-update trajectories from a dynamical-systems view and decomposes base prompt updates into magnitude and direction. $\mathtt{TPD}$ computes a spectral radius to measure update expansion and decomposes update directions into persistent, oscillatory, and residual components. Building on this decomposition, Adaptive Koopman Control ( $\mathtt{AKC}$ ) regulates update magnitude by shrinking updates under expansive recent dynamics, while Hankel Update Router ( $\mathtt{HUR}$ ) refines update direction by preserving persistent components and suppressing oscillatory and residual components. Together, $\mathtt{AKC}$ and $\mathtt{HUR}$ produce a stabilized prompt update. As a prompt-agnostic framework, $\mathtt{TPD}$ can be plugged into visual and text TPT methods without modifying the backbone, prompt architecture, or adaptation loss. Experiments on 15 datasets show that $\mathtt{TPD}$ consistently improves 10 baselines, with accuracy gains of $3$ -- $8\%$ for visual prompts and over $2\%$ for text prompts.

Figures & tables

Figure 1: (a) The accuracy of ViT visual and CLIP text prompts in episodic modes, averaged over four ImageNet variants. Gains denote

\mathtt{TPD}

improvements. The zoomed windows show accuracy, magnitude (spectral radius

\rho

), and direction (

\ell_{2}

distance between consecutive update vectors, decomposed in (b)) in unstable and stable regimes. (b)

\mathtt{TPD}

decomposes each prompt update into persistent, oscillatory, and residual components, with the spectral radius measuring local expansion. (c) Stability-performance relation summarized by spectral radius and persistent ratio.

Figure 2: Overview of $\mathtt{TPD}$ . The base prompt update $\mathbf{u}_{t}$ from visual or text TPT is delay-embedded with a Hankel window and projected into a local Hankel subspace. $\mathtt{AKC}$ regulates update magnitude and $\mathtt{HUR}$ refines update direction. The controlled magnitude and refined direction are then combined into a stabilized prompt update $g_{t}\mathbf{u}_{t}^{\mathrm{hur}}$ , which is applied to the prompt state at step $t$ .

ViT-based visual prompt tuning
Method	ImageNet	ImageNet-V2	ImageNet-S	ImageNet-A	ImageNet-R	OOD Mean	Mean
ViT-B/16	81.44	70.85	35.44	30.92	47.48	46.17	53.23
VPT + $\mathtt{TPD}$	78.16 +3.15	64.55 +5.32	32.77 +3.25	19.17 +19.81	42.14 +4.33	39.66 +8.18	47.36 +7.17
E2VPT + $\mathtt{TPD}$	78.35 +2.69	64.09 +4.93	32.89 +2.70	18.17 +20.79	42.07 +3.94	39.31 +8.09	47.11 +7.01
VFPT + $\mathtt{TPD}$	78.71 +2.62	65.34 +4.75	33.16 +2.81	22.05 +18.68	42.81 +3.55	40.84 +7.45	48.41 +6.48
BPT + $\mathtt{TPD}$	81.30 +2.87	70.48 +2.45	28.62 +9.93	38.97 +4.23	43.68 +5.39	45.44 +5.51	52.61 +4.97

Table 1: Comparisons on the domain generalization setting with visual and text TPT methods. OOD Mean averages the 4 variants excluding ImageNet, while Mean averages ImageNet and the 4 variants. Values are the baseline accuracy with

\mathtt{TPD}

gains.

ViT-based visual prompt tuning
Method	Caltech101	OxfordPets	StanfordCars	Flowers102	Food101	Aircraft	SUN397	DTD	EuroSAT	UCF101	Mean
ViT-B/16	86.71	92.20	57.84	66.23	82.78	22.06	56.07	48.19	49.91	65.08	62.71
VPT + $\mathtt{TPD}$	92.65 +2.72	86.89 +2.49	60.68 +2.13	64.86 +3.23	85.14 +1.31	21.21 +3.25	59.92 +3.87	38.19 +4.06	20.91 +11.26	59.28 +2.73	58.97 +3.71
E2VPT + $\mathtt{TPD}$	92.29 +2.01	86.89 +2.75	60.03 +2.08	63.85 +3.96	84.55 +1.26	20.79 +3.51	59.07 +3.48	37.93 +5.33	17.29 +11.67	58.75 +2.81	58.14 +3.89
VFPT + $\mathtt{TPD}$	92.76 +2.36	87.33 +2.71	62.01 +2.41	67.77 +1.98	85.18 +1.43	21.69 +3.33	59.31 +2.64	40.53 +3.43	23.54 +12.66	60.26 +1.43	60.04 +3.44
BPT + $\mathtt{TPD}$	79.51 +13.89	68.26 +14.45	49.88 +11.15	64.82 +2.33	67.45 +15.21	21.47 +4.15	48.50 +7.84	41.46 +1.96	57.42 +1.71	50.52 +9.59	54.93 +8.23

Table 2: Comparisons on the cross-dataset setting with visual and text TPT methods. All methods are evaluated on 10 downstream datasets without training. Values are baseline accuracy with

\mathtt{TPD}

gains.

	VPT			TPT
Variant	DG	CSD	$\times$	DG	CSD	$\times$
Baseline	39.66	58.97	1.00	60.81	65.10	1.00
Simple stabilization rules
+ Update clipping	41.92	59.54	1.11	61.36	65.47	1.06
+ EMA update	42.74	59.86	0.99	61.65	65.82	1.01
+ Norm control	43.65	61.03	1.07	61.72	65.74	1.03

Table 3: Comparison with simple stabilization methods on visual and text TPT methods. DG and CSD denote the accuracy means over four ImageNet variants and 10 cross-datasets.

\times

: per-step wall-clock overhead relative to the baseline.

	VPT		TPT
Variant	DG	CSD	DG	CSD
Baseline	39.66	58.97	60.81	65.10
Ablation study of Koopman operator modes
Full	43.58	60.86	61.80	66.38
Diagonal (+ $\mathtt{AKC}$ )	45.92	61.74	62.56	67.21
Ablation study of projection matrix modes

Table 4: Ablation study of

\mathtt{TPD}

on visual and text TPT methods. DG and CSD denote the accuracy means over four ImageNet variants and 10 cross-datasets.

Figure 3: (a), (b) show Koopman spectra of visual prompt trajectories. The spectra are estimated post-hoc from generated update trajectories, and the black circle denotes the unit circle. (c), (d) show component ratios and spectral radius

\rho

over test steps.

Figure 4: (a), (b) CKA similarity between text-prompt representations across test steps after smoothing. (c) Koopman spectral radius estimated from prompt updates, the estimate is post-hoc for TPT and used online by $\mathtt{AKC}$ . (d) Persistent ratio of $\mathtt{HUR}$ versus subsequent accuracy gain. Each gray point denotes one short temporal window. The red dashed curve is a piecewise linear fitted trend line.

Figure 5: (a) ECE on ImageNet-A and ImageNet-R, with separate vertical scales. (b) Accuracy across batch sizes. Pink and blue in (a,b) denote each baseline and its

\mathtt{TPD}

variant, respectively. (c) Proportions of stable (

\rho_{t}\leq 1

, light green) and unstable (

\rho_{t}>1

, light orange)

\mathtt{TPD}

update steps, where

\rho_{t}

denotes the estimated spectral radius; each stacked bar sums to

100\%

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Variant	ImageNet	ImageNet-V2	ImageNet-Sketch	ImageNet-A	ImageNet-R	OoD Mean 4
Baseline	78.16	64.55	32.77	19.17	42.14	39.66
+ Update clipping	78.84	65.52	34.09	22.95	45.11	41.92
+ EMA update	79.35	65.98	36.82	24.44	43.71	42.74
+ Norm control	79.76	66.66	34.26	25.60	48.09	43.65
+ Low-pass filtering	80.14	68.29	34.76	28.43	45.65	44.28
VPT+ $\mathtt{TPD}$	81.31	69.87	36.02	38.98	46.47	47.84

Appendix

Table A1: Per-dataset simple stabilization results for VPT under domain generalization. OoD Mean 4 averages ImageNet-V2, ImageNet-Sketch, ImageNet-A, and ImageNet-R.

Variant	Caltech	Pets	Cars	Flowers	Food	Aircraft	SUN	DTD	EuroSAT	UCF	Mean
Baseline	92.65	86.89	60.68	64.86	85.14	21.21	59.92	38.19	20.91	59.28	58.97
+ Update clipping	92.94	87.35	61.27	65.16	85.85	21.93	60.57	38.48	21.78	60.03	59.54
+ EMA update	93.27	87.46	61.07	65.33	85.67	22.47	61.00	38.77	22.79	60.80	59.86
+ Norm control	93.10	87.71	63.33	66.75	86.59	26.40	60.40	40.54	24.25	61.19	61.03
+ Low-pass filtering	93.39	89.29	62.23	67.32	86.20	21.99	62.55	39.07	25.15	61.93	60.91
VPT+ $\mathtt{TPD}$	95.37	89.38	62.81	68.09	86.45	24.46	63.79	42.25	32.17	62.01	62.68

Appendix

Table A2: Per-dataset simple stabilization results for VPT under cross-dataset transfer. Mean averages the ten downstream datasets.

Variant	ImageNet	ImageNet-V2	ImageNet-Sketch	ImageNet-A	ImageNet-R	OOD Mean
Baseline	68.98	63.45	47.94	54.77	77.06	60.81
+ Update clipping	69.90	64.12	48.40	55.09	77.84	61.36
+ EMA update	69.86	63.80	49.82	55.25	77.74	61.65
+ Norm control	70.11	63.64	50.05	55.80	77.39	61.72
+ Low-pass filtering	70.62	64.49	49.02	56.03	79.38	62.23
TPT+ $\mathtt{TPD}$	71.19	65.98	49.62	56.73	79.98	63.08

Appendix

Table A3: Per-dataset simple stabilization results for TPT under domain generalization. OOD Mean averages the four shifted variants excluding ImageNet.

Variant	Caltech	Pets	Cars	Flowers	Food	Aircraft	SUN	DTD	EuroSAT	UCF	Mean
Baseline	94.16	87.79	66.87	68.98	84.67	24.78	65.50	47.75	42.44	68.04	65.10
+ Update clipping	94.48	88.06	67.12	69.63	84.88	25.17	65.59	48.52	42.78	68.49	65.47
+ EMA update	94.24	88.53	67.47	69.75	85.13	26.04	65.91	48.58	43.85	68.73	65.82
+ Norm control	94.60	88.77	67.84	68.98	85.26	25.80	65.77	49.00	42.93	68.49	65.74
+ Low-pass filtering	94.56	88.66	68.09	69.83	85.58	26.34	66.37	51.18	43.11	68.57	66.23
TPT+ $\mathtt{TPD}$	96.32	90.31	68.55	71.06	88.32	27.19	68.51	51.98	46.03	72.00	68.03

Appendix

Table A4: Per-dataset simple stabilization results for TPT under cross-dataset transfer. Mean averages the ten downstream datasets.

Variant	ImageNet	ImageNet-V2	ImageNet-Sketch	ImageNet-A	ImageNet-R	OoD 4
Baseline	78.16	64.55	32.77	19.17	42.14	39.66
Full Koopman	79.74	69.39	35.83	23.73	45.36	43.58
VPT+ $\mathtt{AKC}$ (diag.)	80.94	69.70	38.33	28.47	47.17	45.92
VPT+ $\mathtt{TPD}$	81.31	69.87	36.02	38.98	46.47	47.84

Appendix

Table A5: Per-dataset Koopman operator ablation for VPT. (a) Domain generalization.

Variant	Caltech	Pets	Cars	Flowers	Food	Aircraft	SUN	DTD	EuroSAT	UCF	Mean
Baseline	92.65	86.89	60.68	64.86	85.14	21.21	59.92	38.19	20.91	59.28	58.97
Full Koopman	93.67	88.14	61.30	66.34	85.88	24.63	62.44	39.78	24.84	61.56	60.86
VPT+ $\mathtt{AKC}$ (diag.)	94.24	89.78	62.33	66.83	86.03	25.23	62.15	40.96	26.95	62.94	61.74
VPT+ $\mathtt{TPD}$	95.37	89.38	62.81	68.09	86.45	24.46	63.79	42.25	32.17	62.01	62.68

Appendix

Table A5: Per-dataset Koopman operator ablation for VPT. (b) Cross-dataset transfer.

Variant	ImageNet	ImageNet-V2	ImageNet-Sketch	ImageNet-A	ImageNet-R	OOD Mean
Baseline	68.98	63.45	47.94	54.77	77.06	60.81
Full Koopman	69.86	63.73	49.87	55.83	77.76	61.80
TPT+ $\mathtt{AKC}$ (diag.)	71.28	64.36	50.52	56.28	79.09	62.56
TPT+ $\mathtt{TPD}$	71.19	65.98	49.62	56.73	79.98	63.08

Appendix

Table A6: Per-dataset Koopman operator ablation for TPT. (a) Domain generalization.

Variant	Caltech	Pets	Cars	Flowers	Food	Aircraft	SUN	DTD	EuroSAT	UCF	Mean
Baseline	94.16	87.79	66.87	68.98	84.67	24.78	65.50	47.75	42.44	68.04	65.10
Full Koopman	95.29	88.91	68.05	70.44	85.86	26.52	66.46	48.17	44.65	69.47	66.38
TPT+ $\mathtt{AKC}$ (diag.)	95.42	89.18	68.77	70.60	88.29	26.43	68.19	50.30	44.95	69.97	67.21
TPT+ $\mathtt{TPD}$	96.32	90.31	68.55	71.06	88.32	27.19	68.51	51.98	46.03	72.00	68.03

Appendix

Table A6: Per-dataset Koopman operator ablation for TPT. (b) Cross-dataset transfer.

Variant	ImageNet	ImageNet-V2	ImageNet-Sketch	ImageNet-A	ImageNet-R	OoD 4
Baseline	78.16	64.55	32.77	19.17	42.14	39.66
Plain projection	79.17	67.46	34.53	23.41	43.83	42.31
VPT+ $\mathtt{HUR}$ (Hankel)	81.10	69.84	38.60	26.25	49.21	45.98
VPT+ $\mathtt{TPD}$	81.31	69.87	36.02	38.98	46.47	47.84

Appendix

Table A7: Per-dataset projection ablation for VPT. (a) Domain generalization.

Variant	Caltech	Pets	Cars	Flowers	Food	Aircraft	SUN	DTD	EuroSAT	UCF	Mean
Baseline	92.65	86.89	60.68	64.86	85.14	21.21	59.92	38.19	20.91	59.28	58.97
Plain projection	93.87	86.92	61.57	65.41	85.84	23.01	61.29	39.95	24.33	60.03	60.22
VPT+ $\mathtt{HUR}$ (Hankel)	95.21	89.26	64.99	67.07	86.85	25.26	64.69	40.37	26.81	61.30	62.18
VPT+ $\mathtt{TPD}$	95.37	89.38	62.81	68.09	86.45	24.46	63.79	42.25	32.17	62.01	62.68

Appendix

Table A7: Per-dataset projection ablation for VPT. (b) Cross-dataset transfer.

Variant	ImageNet	ImageNet-V2	ImageNet-Sketch	ImageNet-A	ImageNet-R	OOD Mean
Baseline	68.98	63.45	47.94	54.77	77.06	60.81
Plain projection	69.46	64.31	48.65	54.99	77.78	61.43
TPT+ $\mathtt{HUR}$ (Hankel)	70.77	65.31	50.24	56.77	79.05	62.84
TPT+ $\mathtt{TPD}$	71.19	65.98	49.62	56.73	79.98	63.08

Appendix

Table A8: Per-dataset projection ablation for TPT. (a) Domain generalization.

Variant	Caltech	Pets	Cars	Flowers	Food	Aircraft	SUN	DTD	EuroSAT	UCF	Mean
Baseline	94.16	87.79	66.87	68.98	84.67	24.78	65.50	47.75	42.44	68.04	65.10
Plain projection	94.77	88.17	67.42	70.20	85.43	26.04	66.19	48.76	43.56	69.52	66.01
TPT+ $\mathtt{HUR}$ (Hankel)	95.25	89.07	67.98	71.74	86.68	27.18	68.60	53.01	45.59	70.34	67.54
TPT+ $\mathtt{TPD}$	96.32	90.31	68.55	71.06	88.32	27.19	68.51	51.98	46.03	72.00	68.03

Appendix

Table A8: Per-dataset projection ablation for TPT. (b) Cross-dataset transfer.

Variant	ImageNet	ImageNet-V2	ImageNet-Sketch	ImageNet-A	ImageNet-R	OoD 4
Baseline	78.16	64.55	32.77	19.17	42.14	39.66
w/o Pres.	81.33	69.32	35.99	36.51	46.41	47.06
w/o Osc.	81.36	69.38	36.48	36.65	46.61	47.28
w/o Res.	81.02	69.08	35.25	36.35	46.68	46.84
VPT+ $\mathtt{TPD}$	81.31	69.87	36.02	38.98	46.47	47.84

Appendix

Table A9: Per-dataset

\mathtt{HUR}

routing ablation for VPT. (a) Domain generalization.

Variant	Caltech	Pets	Cars	Flowers	Food	Aircraft	SUN	DTD	EuroSAT	UCF	Mean
Baseline	92.65	86.89	60.68	64.86	85.14	21.21	59.92	38.19	20.91	59.28	58.97
w/o Pres.	95.42	88.96	62.50	67.97	85.98	24.54	63.06	41.19	31.81	61.49	62.29
w/o Osc.	95.66	88.09	62.19	68.21	86.34	24.00	61.89	41.78	33.26	62.28	62.37
w/o Res.	94.69	88.96	62.16	67.84	86.01	23.85	63.08	43.03	30.81	61.64	62.21
VPT+ $\mathtt{TPD}$	95.37	89.38	62.81	68.09	86.45	24.46	63.79	42.25	32.17	62.01	62.68

Appendix

Table A9: Per-dataset

\mathtt{HUR}

routing ablation for VPT. (b) Cross-dataset transfer.

Variant	ImageNet	ImageNet-V2	ImageNet-Sketch	ImageNet-A	ImageNet-R	OOD Mean
Baseline	68.98	63.45	47.94	54.77	77.06	60.81
w/o Pres.	70.54	65.86	49.05	56.71	80.17	62.95
w/o Osc.	71.25	65.58	49.42	56.36	80.00	62.84
w/o Res.	71.70	66.02	49.22	56.36	79.61	62.80
TPT+ $\mathtt{TPD}$	71.19	65.98	49.62	56.73	79.98	63.08

Appendix

Table A10: Per-dataset

\mathtt{HUR}

routing ablation for TPT. (a) Domain generalization.

Variant	Caltech	Pets	Cars	Flowers	Food	Aircraft	SUN	DTD	EuroSAT	UCF	Mean
Baseline	94.16	87.79	66.87	68.98	84.67	24.78	65.50	47.75	42.44	68.04	65.10
w/o Pres.	96.27	90.35	68.00	70.65	88.08	27.06	68.88	51.65	45.48	70.90	67.73
w/o Osc.	96.92	89.89	68.10	70.36	89.18	26.16	68.99	52.01	46.30	70.82	67.87
w/o Res.	96.47	89.62	68.19	70.24	87.38	27.48	67.96	51.24	46.06	70.79	67.54
TPT+ $\mathtt{TPD}$	96.32	90.31	68.55	71.06	88.32	27.19	68.51	51.98	46.03	72.00	68.03

Appendix

Table A10: Per-dataset

\mathtt{HUR}

routing ablation for TPT. (b) Cross-dataset transfer.

Panel of Figure 6 — Figure 6: Sensitivity to the Hankel window on domain-generalization datasets. Each heatmap reports per-dataset performance of VPT+ $\mathtt{TPD}$ or TPT+ $\mathtt{TPD}$ while sweeping the delay length $K$ and the number of Hankel columns $M$ . Colors are normalized within each panel, with warmer colors indicating higher accuracy. Both prompt types show a broad high-performing region at short-to-moderate windows, supporting the default $K=3,M=8$ .

Panel of Figure 7 — Figure 7: Sensitivity to the Hankel window on cross-dataset transfer. The same $(K,M)$ sweep is evaluated on ten downstream datasets. The high-performing region remains broad across datasets and prompt types, while excessively short or long histories are less reliable.

Panel of Figure 8 — Figure 8: Sensitivity of $\mathtt{AKC}$ on domain-generalization datasets. Each heatmap sweeps the spectral-radius threshold $\bar{\rho}$ and gain floor $g_{\min}$ for VPT+ $\mathtt{TPD}$ or TPT+ $\mathtt{TPD}$ . Colors are normalized within each panel. Performance is strongest in a broad region around $\bar{\rho}=1.0$ with a small gain floor, indicating that $\mathtt{AKC}$ benefits from suppressing locally expansive dynamics without over-damping stable updates.

Panel of Figure 9 — Figure 9: Sensitivity of $\mathtt{AKC}$ on cross-dataset transfer. The $\bar{\rho}$ – $g_{\min}$ sweep shows that moderate stability control is robust across datasets, whereas a large gain floor weakens the controller and overly strict settings may suppress useful adaptation.

Panel of Figure 10 — Figure 10: Sensitivity of $\mathtt{HUR}$ routing weights on domain-generalization datasets. Each heatmap sweeps the triangular grid $0\leq\beta_{n}\leq\beta_{c}\leq 1$ , where $\beta_{c}$ controls the oscillatory component and $\beta_{n}$ controls the residual component; the masked region has $\beta_{n}>\beta_{c}$ . Warmer colors indicate higher per-panel accuracy. The high-performing region favors small residual weights and moderate oscillatory attenuation, supporting selective routing rather than using the raw update unchanged.

Panel of Figure 11 — Figure 11: Sensitivity of $\mathtt{HUR}$ routing weights on cross-dataset transfer. The triangular $\beta_{c}$ – $\beta_{n}$ sweep is consistent across VPT+ $\mathtt{TPD}$ and TPT+ $\mathtt{TPD}$ : retaining persistent components while attenuating oscillatory and residual components yields a broad high-performing region, whereas weights near one reduce $\mathtt{HUR}$ to weak routing and degrade performance.

Figure 12: Persistent ratio versus accuracy gain for TPT+

\mathtt{TPD}

. Each point corresponds to a local trajectory segment, and the red dashed line shows the fitted trend for each dataset.

Stream	Method	Overall	V2	Sketch	First-10
Abrupt	VPT	$52.90\pm 0.37$	64.52	29.67	47.28
	VPT+TPD	$54.32\pm 0.32$	65.78	31.40	48.78
Mixed	VPT	$46.10\pm 0.25$	65.09	27.11	N/A
	VPT+TPD	$47.63\pm 0.19$	66.19	29.08	N/A
Rapid Alternation	VPT	$45.22\pm 0.51$	63.82	26.62	45.82
	VPT+TPD	$49.66\pm 0.27$	67.64	31.69	50.09

Appendix

Table A11: Accuracy (%) under nonstationary streams. Overall reports mean

\pm

standard deviation across five paired test orders; the other accuracy columns report the corresponding means.

Method	Stream	P	O	R	Accuracy
VPT	Abrupt	17.54	0.71	81.75	$52.90\rightarrow 54.32$
VPT	Mixed	12.83	0.48	86.69	$46.10\rightarrow 47.63$
VPT	Rapid Alternation	5.09	1.26	93.65	$45.22\rightarrow 49.66$
TPT	Abrupt	3.30	0.45	96.25	$57.31\rightarrow 57.92$
TPT	Mixed	3.16	0.44	96.40	$54.85\rightarrow 55.58$
TPT	Rapid Alternation	3.19	0.46	96.35	$55.09\rightarrow 55.82$

Appendix

Table A12: Component ratios and accuracy (%) under nonstationary shifts. P, O, and R denote persistent, oscillatory, and residual fractions. Accuracy is reported as Base

\rightarrow

Base+TPD.

	VPT		TPT
Method	ImageNet-A	ImageNet-R	ImageNet-A	ImageNet-R
Base	$19.25\pm 1.97$	$42.32\pm 0.61$	$54.44\pm 1.12$	$77.01\pm 2.05$
Base+TPD	$38.73\pm 1.28$	$46.54\pm 0.54$	$56.62\pm 1.16$	$79.63\pm 1.89$

Appendix

Table A13: Accuracy (%) as mean

\pm

standard deviation across three random seeds (

4827

7316

, and

9054

). Base denotes the corresponding VPT or TPT method without TPD.

	VPT		TPT
Controller	ImageNet-A	ImageNet-R	ImageNet-A	ImageNet-R
Base	19.17	42.14	54.77	77.06
Adaptive Clipping	23.52	41.92	54.85	77.31
Variance-aware Scaling	22.76	43.65	55.19	77.26
TPD	38.98	46.47	56.73	79.98

Appendix

Table A14: Accuracy (%) of adaptive magnitude controllers using the same base updates and evaluation streams. Base denotes the corresponding VPT or TPT method without a controller.

Test-Time Adaptation

Test-time adaptation updates or recalibrates pretrained models at inference to handle distribution shifts, often without target labels.

See all 318 papers

USE: A Unified Self-Ensembling Framework for Test-Time Prompt Tuning
Jul 4, 2026Siru Jiang, Jian Liang, Ran He +1Ensemble Learning VLM Adaptation
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences +1
A Decision-Theoretic View of Test-Time Training: When, How Far, and Which Directions to Adapt
Jun 14, 2026Tomoya WakayamaTest-Time Training
RIKEN Center for Advanced Intelligence Project (AIP), Tokyo, Japan.
Dual-Modality Anchor-Guided Filtering for Test-time Prompt Tuning
Apr 14, 2026Jungwon Choi, Eunwoo KimVLM Robustness VLM Adaptation
School of Computer Science and Engineering, Chung-Ang University

Jul 4, 2026J/K move · Enter open · S save

USE: A Unified Self-Ensembling Framework for Test-Time Prompt Tuning

Siru Jiang, Jian Liang, Ran He, Tieniu Tan

School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences+1

Ensemble Learning VLM Adaptation Test-Time Adaptation of VLMs Pseudo-Labeling Test-Time Augmentation Test-Time Prompt Tuning

Open paper AskPDF arXiv

Test-time adaptation (TTA) has emerged as a popular paradigm for improving the performance of vision-language models (e.g., CLIP) on downstream tasks. Among existing CLIP-based TTA methods, Test-Time Prompt Tuning (TPT) is a pioneering work that optimizes textual prompts using multiple test-time augmentations and remains a strong baseline to date. In this work, we revisit TPT and reveal that its optimization can be interpreted as implicitly learning from self-generated pseudo labels. Building on this perspective, we propose a unified self-ensembling framework (USE) that ensures consistency between the optimization and inference stages. During optimization, we introduce a simple yet effective self-ensembling (SE) strategy that emphasizes the test image itself over its augmented views adaptively to obtain more reliable pseudo labels. To fully exploit the potential of augmentations, we further apply the same strategy at inference time, unifying the objectives of both stages. Notably, SE can also act as a lightweight optimization-free TTA method. Extensive experiments across multiple datasets demonstrate that SE and USE outperform their counterparts, respectively. Furthermore, SE yields consistent performance gains when integrated with existing TTA methods. The code is available at https://github.com/sirujiang/USE.

Abstract

Test-Time Adaptation

Explore similar work

USE: A Unified Self-Ensembling Framework for Test-Time Prompt Tuning

A Decision-Theoretic View of Test-Time Training: When, How Far, and Which Directions to Adapt

Dual-Modality Anchor-Guided Filtering for Test-time Prompt Tuning