Technical note on: Zero-Training Feature-Space Alignment via Information Geometry

Behraj Khan; Tahir Qasim Syed; Syed Ahmad Chan Bukhari

cs.CVSep 29, 2026

Technical note on: Zero-Training Feature-Space Alignment via Information Geometry

Authors: Behraj Khan, Tahir Qasim Syed, Syed Ahmad Chan Bukhari

Organizations: School of Mathematics and Computer Science, Institute of Business Administration Karachi, Pakistan · Division of Computer Science, Mathematics and Science, St. John’s University, USA

Abstract

Deep vision models often degrade under distribution shift. Test-time adaptation can improve robustness but typically requires iterative optimization, hyperparameter tuning, and multiple forward-backward passes. We propose Zero-Training Fisher Geometry Alignment (ZFGA), a closed-form method that improves robustness under covariate shift without modifying model parameters. ZFGA is based on the observation that distribution shifts distort feature-space geometry. It estimates the Fisher information matrix of the predictive distribution with respect to feature embeddings and applies a linear transformation that aligns test-feature Fisher geometry with a reference geometry computed from clean data. This provides a natural-gradient-inspired preconditioning step in feature space. We evaluate ZFGA on CIFAR-10-C and ImageNet-C using ResNet-50, DINO ViT-S/16, and CLIP ViT-B/32. ZFGA consistently improves over zero-shot inference across all three models, although it is not the strongest method for every model. Covariance whitening performs better on ResNet-50, while Fisher whitening is statistically indistinguishable from ZFGA on CLIP. Across six training-free and gradient-based alternatives (covariance whitening, Fisher whitening, TENT, T3A, LAME, and AdaNPC), ZFGA is the only method that does not substantially harm any of the three model families. The Fisher geometry distortion is also positively correlated with ZFGA gain (Pearson r = 0.366, p = 0.017), providing preliminary evidence that geometric misalignment contributes to robustness degradation. ZFGA requires only forward passes and matrix operations at inference time, offering a lightweight and deterministic alternative to optimization-based test-time adaptation.

Figures & tables

Figure 1: Overview of Zero-Training Fisher Geometry Alignment (ZFGA). Clean reference images and corrupted test images are processed by a frozen encoder $z=f_{\theta}(x)$ to obtain feature embeddings. The corresponding predictive distributions yield empirical Fisher information matrices $\hat{\mathbf{I}}{\mathrm{ref}}$ and $\hat{\mathbf{I}}{\mathrm{te}}$ , whose Frobenius discrepancy $\Delta_{\mathrm{F}}=\lVert\hat{\mathbf{I}}{\mathrm{ref}}-\hat{\mathbf{I}}{\mathrm{te}}\rVert_{\mathrm{F}}$ quantifies geometric distortion under covariate shift. ZFGA constructs the closed-form alignment matrix $\mathbf{A}=(\hat{\mathbf{I}}{\mathrm{te}}+\epsilon\mathbf{I})^{-1/2}(\hat{\mathbf{I}}{\mathrm{ref}}+\epsilon\mathbf{I})^{1/2}$ and transforms the test embedding as $z^{\prime}=\mathbf{A}z$ . Classification then uses the fixed temperature-scaled cosine model $p(y\mid x)\propto\exp(\tau,z^{\prime\top}t_{y})$ with frozen prototypes ${t_{y}}$ . Zero-training: frozen encoder and prototypes, no parameter updates, and only forward passes and matrix operations at inference time.

ResNet-50
Model	Frozen	Cov.	Fisher	ZFGA	TENT / TENT-LN
Acc. $\pm$ std	68.50 $\pm$ 0.30	49.54 $\pm$ 1.17	65.49 $\pm$ 0.97	74.08 $\pm$ 0.26	73.31 $\pm$ 0.70
$\Delta$	–	$-18.96$	$-3.01$	+5.58	$+4.81$
DINOv3 ViT-S/16
Acc. $\pm$ std	84.89 $\pm$ 0.74	33.76 $\pm$ 1.21	88.20 $\pm$ 0.92	84.97 $\pm$ 0.69	84.89 $\pm$ 0.74
$\Delta$	–	$-51.13$	+3.31	$+0.08$	$+0.00$

Table 1: Main Results on CIFAR-10-C (Severities 2–3). Mean accuracy (%)

\pm

std over 3 seeds, averaged over 7 corruption types.

\Delta

is gain over the frozen baseline. Best non-TENT result per model in bold . DINO/CLIP use TENT-LN (LayerNorm variant); see Appendix .3 .

Model	Zero-shot	T3A	LAME	AdaNPC	ZFGA
ResNet-50	59.53 $\pm$ 0.75	60.16 $\pm$ 0.60	58.08 $\pm$ 0.60	64.49 $\pm$ 0.76	60.25 $\pm$ 0.74
DINOv3 ViT-S/16	67.60 $\pm$ 1.10	66.18 $\pm$ 0.99	64.02 $\pm$ 0.97	69.66 $\pm$ 0.78	67.74 $\pm$ 1.06
CLIP ViT-B/32	78.70 $\pm$ 0.98	79.57 $\pm$ 0.48	77.93 $\pm$ 1.32	74.52 $\pm$ 1.38	78.98 $\pm$ 1.02

Table 2: Comparison with training-free/backprop-free baselines on CIFAR-10-C (Severities 2–3). Mean accuracy (%)

\pm

std over 3 seeds. All methods evaluated on identical encoders, corruption subset, and severities as Table 1 .

Model	Zero-shot	Cov. Whit.	Fisher Whit.	ZFGA
ResNet-50	27.82	29.80	28.88	30.18
DINOv3 ViT-S/16	38.18	$15.61_{\scriptscriptstyle(-59.1\%)}$	37.77	38.44
CLIP ViT-B/32	54.45	$32.49_{\scriptscriptstyle(-40.3\%)}$	54.82	55.02

Table 3: Results on ImageNet-C (Severity 3 only, restricted to 3 corruption types). Mean top-1 accuracy (%) averaged over Gaussian Noise, Motion Blur, and Contrast; single-run point estimates, no seed variance reported (see Section 4.2 ). Best results in bold .

Method	ResNet-50	DINOv3 ViT-S/16	CLIP ViT-B/32
Frozen baseline	750 $\pm$ 135 ms	6315 $\pm$ 122 ms	1590 $\pm$ 41 ms
Cov. Whitening	1537 $\pm$ 459 ms	6356 $\pm$ 129 ms	1610 $\pm$ 43 ms
Fisher Whitening	1645 $\pm$ 708 ms	6322 $\pm$ 121 ms	—
ZFGA	1565 $\pm$ 370 ms	6324 $\pm$ 120 ms	1606 $\pm$ 42 ms
TENT / TENT-LN	17405 $\pm$ 839 ms	19545 $\pm$ 250 ms	3717 $\pm$ 42 ms
TENT adapted params	53,120 (0.226%)	19,200 (0.089%)	39,936 (0.045%)

Table 4: Inference latency per 512-sample batch (mean

\pm

std over 5 runs, single GPU). Adapted-parameter counts are for the gradient-based baseline.

Figure 2: Correlation between Fisher distortion and ZFGA effectiveness. Across models and corruptions (severities 2–3), larger geometric distortion (x-axis:

\Delta_{\text{F}}

) is weakly associated with greater ZFGA gain over zero-shot baseline (y-axis). Pearson

r=0.366

p=0.017

(

r^{2}\approx 0.13

); this pooled correlation is preliminary evidence and should not be read as a strong or precisely quantified effect.

Figure 3: Fisher matrix eigenvalue spectrum. Top 50 eigenvalues for clean (green solid) vs. corrupted (red dashed) Fisher matrices on Gaussian noise severity 2. ResNet shows the largest spectrum shift (highest condition number) of the three models shown, DINO an intermediate shift, and CLIP the smallest shift. This ordering is descriptive of the three models studied and is not independently statistically tested.

Figure 4: Ablation studies. (Left) Batch size sensitivity: ZFGA accuracy on CLIP with Gaussian noise (sev. 2) stabilizes around $n=128$ –256. (Middle) Regularization $\epsilon$ : stable across $[10^{-6},10^{-3}]$ , showing robustness to hyperparameter choice. (Right) Severity sensitivity: ZFGA gain vs. corruption severity for Gaussian noise only (not averaged over all seven CIFAR-10-C corruption types, and not directly comparable to the multi-corruption headline numbers in Table 1 ). ResNet shows an inverted-U pattern peaking at severity 3 (+4.30%), where geometric distortion appears maximal without catastrophic information loss. Comparatively robust models (DINO, CLIP) show flatter profiles.

Corruption	Sev.	Zero-shot	Cov.	Fisher	TENT	T3A	LAME	AdaNPC	ZFGA
Gaussian noise	2	42.2	48.2	38.2	43.0	43.4	40.8	54.3	43.4
Gaussian noise	3	31.2	35.0	28.3	31.2	33.1	27.1	41.6	32.0
Motion blur	2	57.8	59.6	58.5	57.1	58.3	56.1	56.5	58.3
Motion blur	3	46.5	48.5	48.9	46.5	49.8	45.7	45.2	47.9
Defocus blur	2	70.8	74.1	69.4	71.2	69.1	67.6	76.5	71.1
Defocus blur	3	61.1	65.9	63.0	60.6	63.0	61.3	64.7	61.8

Table 5: ResNet-50: per-corruption accuracy (%) on CIFAR-10-C, severities 2–3. Mean over 3 seeds per cell.

Corruption	Sev.	Zero-shot	Cov.	Fisher	TENT	T3A	LAME	AdaNPC	ZFGA
Gaussian noise	2	41.3	21.7	47.7	40.5	49.9	30.5	23.7	42.0
Gaussian noise	3	26.0	16.7	31.4	26.0	39.4	17.5	11.3	26.8
Motion blur	2	68.0	13.0	65.1	69.3	64.1	65.9	73.6	68.1
Motion blur	3	63.0	12.3	59.1	64.5	59.4	59.4	65.2	63.1
Defocus blur	2	76.9	16.9	74.7	76.4	72.3	73.1	80.5	77.0
Defocus blur	3	71.4	16.1	68.8	72.5	67.0	68.2	73.8	71.5

Table 6: DINOv3 ViT-S/16: per-corruption accuracy (%) on CIFAR-10-C, severities 2–3. Mean over 3 seeds per cell.

Corruption	Sev.	Zero-shot	Cov.	Fisher	TENT	T3A	LAME	AdaNPC	ZFGA
Gaussian noise	2	59.5	26.0	62.7	60.2	60.0	52.1	22.1	61.7
Gaussian noise	3	45.8	26.8	50.8	46.6	46.7	34.2	12.0	48.2
Motion blur	2	79.0	39.6	80.6	79.1	80.7	80.1	81.4	79.0
Motion blur	3	71.3	34.2	73.0	72.1	73.5	65.2	70.0	71.7
Defocus blur	2	86.3	38.4	86.2	87.0	88.7	88.7	89.8	86.2
Defocus blur	3	83.5	39.3	83.7	84.1	85.4	85.3	86.6	83.7

Table 7: CLIP ViT-B/32: per-corruption accuracy (%) on CIFAR-10-C, severities 2–3. Mean over 3 seeds per cell.

Explore similar work

Dual Distribution Estimation for Zero-shot Noisy Test-Time Adaptation with VLMs
Jun 24, 2026Wenjie Zhu, Yabin Zhang, Liang Xu +3Out-Of-Distribution Detection Vision-Language Model Adaptation
The Hong Kong Polytechnic University · Harbin Institute of Technology (Shenzhen) · Shanghai Jiao Tong University +1
Ramen: Robust Test-Time Adaptation of Vision-Language Models with Active Sample Selection
Apr 23, 2026Wenxuan Bao, Yanjun Zhao, Xiyuan Yang +1Vision-Language Model Adaptation Stable Test-Time Adaptation
University of Illinois Urbana-Champaign
AGC: Adaptive Geodesic Correction for Adversarial Robustness on Vision-Language Models
May 15, 2026Zhiwei Li, Jiacheng Xue, Weining Wang +4Contrastive Language-Image Pre-Training Model Data Augmentation
1NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · School of Computer Science and Engineering, Central South University · University of Chinese Academy of Sciences

Jun 24, 2026, cs.CVJ/K move · Enter open · S save

Dual Distribution Estimation for Zero-shot Noisy Test-Time Adaptation with VLMs

Wenjie Zhu, Yabin Zhang, Liang Xu, Xin Jin+2

The Hong Kong Polytechnic University · Harbin Institute of Technology (Shenzhen) · Shanghai Jiao Tong University+1

Out-Of-Distribution Detection Vision-Language Model Adaptation Test-Time Adaptation Test-Time Training Fast Inference Negative Results

Open paper AskPDF arXiv

While test-time adaptation (TTA) empowers vision-language models to adapt without costly retraining, it remains highly vulnerable to out-of-distribution (OOD) outliers prevalent in real-world applications. This discrepancy motivates Noisy TTA (NTTA), an online task to filter noisy OOD samples on the fly while maximizing in-distribution (ID) classification accuracy. Existing zero-shot NTTA approaches typically rely on test-time discriminative training, leading to overconfident misclassifications and significantly degraded inference efficiency. To address these limitations, we propose a novel framework named Dual Distribution Estimation (DDE), shifting the zero-shot NTTA paradigm from instance-level learning to training-free Gaussian distribution modeling. DDE incorporates two novel modules: Positive Feature Distribution Estimation (PFDE) and Negative Label Distribution Estimation (NLDE). PFDE explicitly models class-wise inclusion and exclusion Gaussian distributions to formulate a calibrated contrastive score, robustly enhancing ID accuracy. In parallel, NLDE improves OOD identification by explicitly modeling the negative label distribution to mine highly discriminative labels, effectively mitigating spurious correlations. Extensive experiments show that on the large-scale ImageNet benchmark, DDE achieves an improvement of 3.70% in harmonic mean accuracy and reduces the FPR95 for OOD detection by 6.20%, while ensuring highly scalable and efficient online inference. Furthermore, DDE is zero-shot and training-free, demonstrating remarkable robustness in data-scarce scenarios. Codes are available at https://github.com/ZhuWenjie98/DDE.

Abstract

Explore similar work

Dual Distribution Estimation for Zero-shot Noisy Test-Time Adaptation with VLMs

Ramen: Robust Test-Time Adaptation of Vision-Language Models with Active Sample Selection

AGC: Adaptive Geodesic Correction for Adversarial Robustness on Vision-Language Models