The Geometry of Robustness: Optimizing Loss Landscape Curvature and Feature Manifold Alignment for Robust Finetuning of Vision-Language Models

Shivang Chopra; Shaunak Halbe; Chengyue Huang; Brisa Maneechotesuwan; Zsolt Kira

cs.CVMar 28, 2026

The Geometry of Robustness: Optimizing Loss Landscape Curvature and Feature Manifold Alignment for Robust Finetuning of Vision-Language Models

Authors: Shivang Chopra, Shaunak Halbe, Chengyue Huang, Brisa Maneechotesuwan, Zsolt Kira

Organizations: Georgia Institute of Technology

Abstract

Fine-tuning approaches for Vision-Language Models (VLMs) face a critical three-way trade-off between In-Distribution (ID) accuracy, Out-of-Distribution (OOD) generalization, and adversarial robustness. Existing robust fine-tuning strategies resolve at most two axes of this trade-off. Generalization-preserving methods retain ID/OOD performance but leave models vulnerable to adversarial attacks, while adversarial training improves robustness to targeted attacks but degrades ID/OOD accuracy. Our key insight is that the robustness trade-off stems from two geometric failures: sharp, anisotropic minima in parameter space and unstable feature representations that deform under perturbation. To address this, we propose GRACE (Gram-aligned Robustness via Adaptive Curvature Estimation), a unified fine-tuning framework that jointly regularizes the parameter-space curvature and feature-space invariance for VLMs. Grounded in Robust PAC-Bayes theory, GRACE employs adaptive weight perturbations scaled by local curvature to promote flatter minima, combined with a feature alignment loss that maintains representation consistency across clean, adversarial, and OOD inputs. On ImageNet fine-tuning of CLIP models, GRACE simultaneously improves ID accuracy by 10.8%, and adversarial accuracy by 13.5% while maintaining 57.0% OOD accuracy (vs. 57.4% zero-shot baseline). Geometric analysis confirms that GRACE converges to flatter minima without feature distortion across distribution shifts, providing a principled step toward generalized robustness in foundation VLMs.

Figures & tables

Figure 1: The VLM robustness three-way tradeoff. Existing robust fine-tuning strategies resolve at most two of {ID, OOD, adversarial} robustness simultaneously, leaving a gap in generalized robustness. GRACE is designed to close this gap.

Method	(A) Proximity	(B) Sharpness	(C) Stability
	$\\|\theta-\theta_{0}\\|^{2}$	$\text{Tr}(\nabla^{2}R)$	$d_{H\Delta H}$
Vanilla FT	✗	✗	✗
WiSE-FT [ 36 ]	✓	✗	✗
FLYP [ 5 ]	✓	✗	$\sim$
TPGM [ 28 ]	✓	✗	✗
SPD [ 29 ]	✓	✗	✗

Table 1: Decomposition of robust fine-tuning methods by optimization objectives. Methods addressing each term explicitly through their loss function or training procedure are marked (✓), those with implicit or partial coverage are marked (

\sim

), and those without explicit optimization are marked ( ✗ ).

Figure 2: (a) Feature Distribution Analysis : 3D projection of image features for in-distribution ( $f_{\text{ID}}$ ), OOD ( $f_{\text{OOD}}$ ), and PGD adversarial inputs ( $f_{\text{Adv}}$ ) of the same class. (b) Loss Landscape Analysis : 3D/2D loss slices around the converged solutions for each method, using shared perturbation directions.

	Cosine Alignment		$\Delta$ LID (↓)
Method	ID $\to$ OOD	ID $\to$ Adv	ID $\to$ OOD	ID $\to$ Adv
FT	0.84	0.48	+2.7	+6.5
WiSE-FT (S1)	0.91	0.57	+1.1	+5.6
TeCoA (S2)	0.78	0.72	+4.7	+2.9
GRACE	0.89	0.85	+1.8	+2.5

Table 2: Class-conditional feature-space stability. Cosine similarity between ID and shifted class centroids (left) and change in LID relative to ID (right). Lower

\Delta

LID implies more stable local manifolds.

Method	$\lambda_{\max}\,(\times 10^{4})$	$\\|H\\|_{F}/\sqrt{d}\penalty\ (\times 10^{2})$
FT	3.5	0.89
WiSE-FT (S1)	3.3	0.78
TeCoA (S2)	1.8	0.52
GRACE	1.6	0.43

Table 3: Parameter-space sharpness. Top Hessian eigenvalue and normalized Frobenius norm.

Figure 3: Low-rank adaptation and perturbation in GRACE. Frozen pretrained weights $W(\theta_{0})$ (blue) are updated only through low-rank LoRA adapters (orange). LAR-AWP injects additional low-rank perturbations (red) in the same subspace.

Figure 4: LAR-AWP rank curriculum. A diagonal rank mask controls the effective perturbation rank per layer. Curvature estimates $h_{W}$ (from mini-batch gradients) are used to assign higher perturbation ranks to sharper layers, focusing smoothing where the loss landscape is steepest.

Figure 5: Gram-volume feature alignment. For each input, GRACE compares clean, adversarial, and LAR-AWP-perturbed image embeddings via a small Gram matrix. The Gram-volume loss encourages these three vectors to remain close to each other (low volume) while preserving separation across different classes.

	ID		Domain Shift						Natural Adversarial				Statistics
Method	ImageNet		ImageNet-V2		ImageNet-S		ImageNet-R		ImageNet-A		ImageNet-A-Plus		OOD Avg.		Nat Adv Avg
	Clean	Adv	Clean	Adv	Clean	Adv	Clean	Adv	Clean	Adv	Clean	Adv	Clean	Adv	Clean	Adv
CLIP	63.35	0.00	56.48	0.00	41.44	0.00	68.83	0.01	32.69	0.00	37.92	0.00	55.58	0.00	35.30	0.00
Vanilla FT	74.86	0.00	63.75	0.00	43.34	0.00	60.85	0.00	22.47	0.00	29.12	0.00	55.98	0.00	25.80	0.00
WISE-FT	70.20	0.00	61.06	0.00	43.34	0.00	63.78	0.00	32.60	0.00	39.74	0.00	56.06	0.00	36.17	0.00
FLYP	72.15	0.00	62.22	0.00	42.22	0.00	64.38	0.01	30.06	0.00	37.89	0.00	56.27	0.00	33.98	0.00

Table 4: OOD Results on ImageNet. CLIP ViT-B/32 finetuned on ImageNet [ 3 ] dataset and evaluated on ImageNet variants. The numbers are top-1 accuracy (%). OOD Avg averages ImageNet-V2, -S, -R; Nat Adv Avg averages ImageNet-A and A-Plus.

	Zero shot Datasets																Statistics
	CalTech101		Cars		DTD		EuroSAT		FGVC		Flowers		Oxford Pets		STL-10		ZS Avg.
Method	Clean	Adv	Clean	Adv	Clean	Adv	Clean	Adv	Clean	Adv	Clean	Adv	Clean	Adv	Clean	Adv	Clean	Adv
CLIP	80.47	0.00	51.47	0.00	41.43	0.00	45.27	0.00	15.93	0.00	60.18	0.00	84.00	0.00	95.71	0.00	59.31	0.00
Vanilla FT	79.85	0.00	45.44	0.00	40.26	0.00	41.18	0.00	14.16	0.00	55.42	0.00	84.30	0.00	97.03	0.00	57.21	0.00
WISE-FT	81.47	0.00	53.18	0.00	42.07	0.00	46.57	0.00	16.08	0.00	60.66	0.00	85.90	0.00	97.26	0.00	60.40	0.00
FLYP	80.77	0.00	33.72	0.00	38.56	0.00	39.72	0.00	11.76	0.00	50.18	0.00	84.21	0.00	96.00	0.00	54.37	0.00

Table 5: Clean and adversarial evaluation on zero-shot image classification datasets (ViT-B/32). Models are trained on ImageNet; all other datasets are zero-shot. ZS Avg is the mean across the 8 datasets.

Method	ID	Average	Average	Harmonic
		OOD	Adversarial	Mean
CLIP	63.35	57.44	8.82	20.46
Vanilla FT	74.86	56.59	8.95	21.01
WISE-FT	70.20	58.05	9.04	21.11
FLYP	72.15	55.32	8.49	20.03
TPGM	74.13	57.56	8.89	20.92

Table 6: Unified summary across settings (ViT-B/32). CLIP ViT-B/32 fine-tuned on ImageNet-1K and evaluated across ID, OOD, PGD adversarial (AutoAttack/APGD-CE), and natural adversarial (ImageNet-A/A-Plus average).

Method	ID	Average	Average	Harmonic
		OOD	Adversarial	Mean
CLIP	63.3	57.4	8.8	20.4
LoRA-FT	72.8	55.0	8.2	19.4
LoRA-SPD	73.0	56.0	8.5	19.5
LoRA-TeCoA	60.0	45.0	22.5	37.5
VPT-PMG-AFT	70.0	52.0	22.7	38.6

Table 7: Comparison of GRACE with other LoRA-based fine-tuning approaches. CLIP ViT-B/32 fine-tuned on ImageNet-1K.

Variant	ID	Average	Average	Harmonic
		OOD	Adversarial	Mean
CLIP	63.3	57.4	8.8	20.4
LoRA-FT (no regs)	72.8	55.0	8.2	19.4
+ GV only	72.0	56.5	8.6	20.2
+ LAR-AWP (no curriculum)	71.0	53.0	17.2	32.9
+ LAR-AWP (with curriculum)	72.5	54.0	22.2	38.7

Table 8: Ablation Study. CLIP ViT-B/32 fine-tuned on ImageNet-1K. Avg is the simple mean of the four columns.

Figure 6: Pareto Curve. GRACE (Red) achieves +7 performance gain while being 1.4

\times

faster than prior adversarial methods.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Layerwise curvature anisotropy in CLIP. Normalized Hutchinson curvature $\kappa_{\ell}$ for each transformer block of CLIP ViT-B/32, ViT-B/16, and ViT-L/14. All models exhibit substantial variation in curvature across depth, indicating strong layerwise geometric heterogeneity.

Figure 8: Schematic of LoRA weight updates. Weight updates are parameterized by low-rank matrices $A$ and $B$ of rank $r$ , constraining deviations from the pre-trained weights to a low-dimensional subspace.

\mathcal{L}_{\text{GRACE}}=\mathcal{L}_{\text{task}}+\lambda_{\text{LAR}}\mathcal{L}_{\text{LAR-AWP}}+\lambda_{\text{GV}}\mathcal{L}_{\text{GV}}.

Appendix

Algorithm 1 GRACE: Unified Robust Fine-Tuning with LoRA, LAR-AWP, and Gram Alignment

Shift	Cosine Similarity	L2 Distance
ID $\to$ OOD	0.35	6.61
ID $\to$ AWP	0.32	6.87

Appendix

Table 9: Feature-space displacement induced by OOD data and AWP. We report cosine similarity and L2 feature distance between ID representations and those produced under OOD samples or adversarial weight perturbations. The similar displacement magnitudes show that AWP produces feature shifts comparable to true OOD drift, supporting its utility as a proxy for robustness stress-testing.

Figure 9: 3D visualization of CLIP feature geometry under different shifts. Using normalized embeddings projected onto the unit sphere, we compare ID samples, natural adversarial (Nat Adv) variants, OOD samples, and features obtained under AWP perturbations. AWP produces feature displacements that closely follow the structure of real OOD and natural adversarial shifts, confirming that curvature-aligned weight perturbations serve as an effective proxy for robustness-critical distribution drift.

Figure 10: Samples from the ImageNet dataset

Hyperparameter	Selected Value
Rank of Weight Updates ( $r_{W}$ )	64
Maximum Rank of Weight Perturbations ( $r_{AWP})$	4
Input Perturbation Strength ( $\epsilon$ )	$4/255$
Input Perturbation Step Size ( $\eta_{1}$ )	$1/255$
Learning Rate ( $\eta_{3}$ )	2e-4
Gradient Threshold Percentile for Weight Perturbations ( $\varphi_{AWP}$ )	80

Appendix

Table 10: List of hyperparameter values used for the fine-tuning experiments.

Figure 11: Evolution of Layerwise AWP Rank During Training. The heatmap shows the adaptive adversarial weight perturbation (AWP) rank assigned to each transformer block across training epochs.

Hyperparameter	Value	ID	Out-Of-Distribution		Adversarial				Statistics
Hyperparameter	Value	ID	Domain Shift	Zero Shot	Adv ID	Adv OOD	Adv ZS	Natural Adv	Avg OOD	Avg Adv
Rank of Weight Updates $(r_{W})$	16	72.80	53.10	58.10	22.10	18.30	18.90	21.10	55.50	20.10
	32	73.60	53.80	59.00	23.90	19.40	20.30	22.00	56.50	21.20
	64	74.21	54.41	59.61	25.44	20.55	21.10	22.67	57.01	22.44
Maximum Rank of Weight Perturbations $(r_{\text{AWP}})$	2	74.30	54.90	59.70	23.10	18.90	19.40	21.90	56.50	20.80
	4	74.21	54.41	59.61	25.44	20.55	21.10	22.67	57.01	22.44

Appendix

Table 11: Hyperparameter sensitivity results for GRACE (ViT-B/32). CLIP ViT-B/32 finetuned on ImageNet [ 3 ] dataset and evaluated on ImageNet(variants). The numbers represent top-1 accuracy.

Method	ID	Out-Of-Distribution		Adversarial				Statistics
		Domain Shift	Zero Shot	Adv ID	Adv OOD	Adv ZS	Natural Adv	Avg OOD	Avg Adv
		(Same Classes)	(Unseen Classes)	( $\epsilon{=}4/255$ )	( $\epsilon{=}4/255$ )	( $\epsilon{=}4/255$ )	ImageNet-A/A-Plus
CLIP	68.35	62.60	64.92	0.00	0.00	0.13	52.48	63.76	13.15
Vanilla FT	81.30	60.43	71.48	0.00	0.00	0.00	39.35	65.96	9.84
WISE-FT [ 36 ]	82.50	66.60	68.68	0.00	0.00	0.00	49.95	67.64	12.49
FLYP [ 5 ]	82.60	64.67	69.91	0.00	0.00	0.00	50.85	67.29	12.71

Appendix

Table 12: Unified summary across settings (ViT-B/16). ID uses ImageNet clean; Domain Shift (Same Classes) is OOD Avg over ImageNet-V2/S/R (clean); Zero Shot is ZS Avg (clean) across 8 datasets; Targeted Adversarial reports (ID/OOD/ZS) accuracies under AutoAttack (APGD-CE,

\epsilon{=}4/255

); Natural Adversarial is the clean average over ImageNet-A/A-Plus.

	ID		Domain Shift						Natural Adversarial				Statistics
Method	ImageNet		ImageNet-V2		ImageNet-S		ImageNet-R		ImageNet-A		ImageNet-A-Plus		OOD Avg.		Nat Adv Avg
	Clean	Adv	Clean	Adv	Clean	Adv	Clean	Adv	Clean	Adv	Clean	Adv	Clean	Adv	Clean	Adv
CLIP [ 24 ]	68.35	0.00	61.90	0.00	48.30	0.00	77.60	0.01	50.10	0.00	54.85	0.00	62.60	0.00	52.48	0.00
Vanilla FT	81.30	0.00	70.60	0.00	45.10	0.00	65.60	0.00	36.60	0.00	42.10	0.00	60.43	0.00	39.35	0.00
WISE-FT [ 36 ]	82.50	0.00	73.10	0.00	51.60	0.00	75.10	0.00	47.60	0.00	52.30	0.00	66.60	0.00	49.95	0.00
FLYP [ 5 ]	82.60	0.00	73.00	0.00	49.60	0.00	71.40	0.01	48.10	0.00	53.60	0.00	64.67	0.00	50.85	0.00

Appendix

Table 13: OOD Results on ImageNet (ViT-B/16). CLIP ViT-B/16 finetuned on ImageNet [ 3 ] dataset and evaluated on ImageNet variants. The numbers are top-1 accuracy (%). OOD Avg averages ImageNet-V2, -S, -R; Nat Adv Avg averages ImageNet-A and A-Plus.

	Zero shot Datasets																Statistics
	CalTech101		Cars		DTD		EuroSAT		FGVC		Flowers		Oxford Pets		STL-10		ZS Avg.
Method	Clean	Adv	Clean	Adv	Clean	Adv	Clean	Adv	Clean	Adv	Clean	Adv	Clean	Adv	Clean	Adv	Clean	Adv
CLIP	91.48	0.41	59.74	0.00	43.85	0.00	54.70	0.00	18.96	0.00	67.42	0.00	86.13	0.00	97.10	0.64	64.92	0.13
Vanilla FT	93.20	0.00	68.40	0.00	52.30	0.00	68.50	0.00	26.80	0.00	75.60	0.00	90.20	0.00	96.80	0.00	71.48	0.00
WISE-FT	92.80	0.00	65.30	0.00	48.60	0.00	62.40	0.00	22.50	0.00	71.80	0.00	88.60	0.00	97.40	0.00	68.68	0.00
FLYP	93.10	0.00	66.80	0.00	50.20	0.00	64.80	0.00	24.30	0.00	73.40	0.00	89.50	0.00	97.20	0.00	69.91	0.00

Appendix

Table 14: Clean and adversarial evaluation on zero-shot image classification datasets (ViT-B/16). Models are trained on ImageNet; all other datasets are zero-shot. ZS Avg is the mean across the 8 datasets.

PEFT of VLMs

See all 13 papers

What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Models
Mar 13, 2026Sen Nie, Jie Zhang, Zhongqi Wang +3VLM Robustness Adversarial Robustness
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China · University of Chinese Academy of Sciences, Beijing 100049, China
Unifying Adversarially Robust Model Experts in Vision-Language Models
Jul 30, 2026Nguyen Duc Thai, Junhao Dong, Sua Qi Rong +2Adversarial Training VLM Robustness
College of Computing and Data Science, Nanyang Technological University, Singapore
AGC: Adaptive Geodesic Correction for Adversarial Robustness on Vision-Language Models
May 15, 2026Zhiwei Li, Jiacheng Xue, Weining Wang +4VLM Robustness Adversarial Robustness
1NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · School of Computer Science and Engineering, Central South University · University of Chinese Academy of Sciences

Mar 13, 2026J/K move · Enter open · S save

What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Models

Sen Nie, Jie Zhang, Zhongqi Wang, Zhaoyang Wei+2

State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China · University of Chinese Academy of Sciences, Beijing 100049, China

VLM Robustness Adversarial Robustness Adversarial Attacks on VLMs VLM Adaptation

Open paper AskPDF arXiv

Achieving adversarial robustness in Vision-Language Models (VLMs) inevitably compromises accuracy on clean data, presenting a long-standing and challenging trade-off. In this work, we revisit this trade-off by investigating a fundamental question: What makes VLMs robust? Through a detailed analysis of adversarially fine-tuned models, we examine how robustness mechanisms function internally and how they interact with clean accuracy. Our analysis reveals that adversarial robustness is not uniformly distributed across network depth. Instead, unexpectedly, it is primarily localized within the shallow layers, driven by a low-frequency spectral bias and input-insensitive attention patterns. Meanwhile, updates to the deep layers tend to undermine both clean accuracy and robust generalization. Motivated by these insights, we propose Adversarial Robustness Adaptation (R-Adapt), a simple yet effective framework that freezes all pre-trained weights and introduces minimal, insight-driven adaptations only in the initial layers. This design achieves an exceptional balance between adversarial robustness and clean accuracy. R-Adapt further supports training-free, model-guided, and data-driven paradigms, offering flexible pathways to seamlessly equip standard models with robustness. Extensive evaluations on 18 datasets and diverse tasks demonstrate our state-of-the-art performance under various attacks. Notably, R-Adapt generalizes efficiently to large vision-language models (e.g., LLaVA and Qwen-VL) to enhance their robustness. Our project page is available at https://summu77.github.io/R-Adapt.

Abstract

PEFT of VLMs

Explore similar work

What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Models

Unifying Adversarially Robust Model Experts in Vision-Language Models

AGC: Adaptive Geodesic Correction for Adversarial Robustness on Vision-Language Models