Reassessing Global Gradient-Norm Imbalance in BLIP Fine-Tuning Across Physical Domains

Kiran Naseer; Samreen Azhar; Dwarikanath Mahapatra

cs.CVSep 20, 2026

Reassessing Global Gradient-Norm Imbalance in BLIP Fine-Tuning Across Physical Domains

Authors: Kiran Naseer, Samreen Azhar, Dwarikanath Mahapatra

Organizations: Department of Computer Science, University of Gujrat, Pakistan · Department of Computer Science, Khalifa University, Abu Dhabi, UAE

Abstract

Imbalanced gradient magnitudes between the visual and language pathways of a vision-language model are often treated as a defect to be corrected. We test that premise for one family of correction, deliberately excluding adaptive, signal-driven schemes (e.g. BalGrad, OGM, PMR, CGGM), which are a mechanistically distinct class outside this study's scope. Measuring the language-to-visual gradient-norm ratio, reported in parameter-normalised form, across nine fine-tuning conditions, three seeds, and three captioning datasets spanning distinct physical domain shifts -- underwater, aerial, radiological -- we find imbalance magnitude varies markedly across domains with no predictable ordering. A plain learning-rate reduction cuts imbalance substantially and lands within a few BLEU points of the best method on every dataset. Staged freezing reduces the ratio on every domain yet never ranks first; a schedule-only control isolates freezing as the cause on one dataset but not the other two. Forcing the two gradient groups to equal magnitude drives per-parameter imbalance close to zero on every domain, yet is both the best result in the study and the worst placement among full fine-tuning methods, on different datasets, with identical settings. Reductions in gradient-norm ratio do not consistently predict captioning performance across domains, and how a given level of balance is reached matters as much as the level itself. As a secondary finding, a commonly reused LoRA configuration applied to BLIP silently adapts zero visual parameters; correcting it improves BLEU-4 on all three datasets.

Figures & tables

Full fine-tuning
Method	UICD	RSICD	ROCOv2
Pretrained (zero-shot)	0.0819	0.0383	0.0024
Naive FT	$0.2420\pm 0.0208$	$0.4463\pm 0.0100$	$0.0215\pm 0.0008$
Low-LR FT	$0.2838\pm 0.0061$	$\mathbf{0.4788\pm 0.0045}$	$0.0238\pm 0.0002$
Joint-Schedule FT	$0.2775\pm 0.0105$	$\mathbf{0.4789\pm 0.0032}$	$0.0244\pm 0.0007$
Staged FT	$0.2920\pm 0.0122$	$0.4695\pm 0.0119$	$0.0221\pm 0.0005$

TABLE I : Best-epoch BLEU-4 (mean

\pm

std over seeds 42, 0, 123) for all nine trained conditions plus the zero-shot baseline, on all three datasets. Best per dataset in bold. ROCOv2 uses a single reference caption per image, so its absolute magnitudes are not comparable to UICD/RSICD; comparisons are meaningful within a column, not across columns (Section VI ). On RSICD the top two conditions differ by

0.0001

, well inside seed noise; both are bolded. Full CIDEr, METEOR, stability and significance results appear in the Supplementary Material.

Converged $R_{t}$ (mean over final ten logged steps)
	Factors applied
Method	Two-phase LR	Freezing	Per-step rescaling
Low-LR FT	no	no	no
Joint-Schedule FT	yes	no	no
Staged FT	yes	yes	no
RMS-Balanced FT	no	no	yes

TABLE II : Mechanism ablation. Four full fine-tuning methods that differ in exactly which factor they apply: a two-phase learning-rate schedule, staged freezing, or per-step gradient rescaling. Converged

R_{t}

is the mean over the last ten logged steps. Reading down each column shows that no single mechanism wins everywhere, and reading across the Staged FT row shows it never wins anywhere. Best-epoch BLEU-4 for these four methods is reported in Table I and is not repeated here.

Explore similar work

May 28, 2026cs.CV

Improving CLIP Adaptation by Breaking Tail Alignment for Source-Free Cross-Domain Few-Shot Learning

Vision-Language Models (VLMs) such as CLIP demonstrate strong zero-shot generalization, but their performance significantly degrades in cross-domain scenarios with scarce target-domain training data (Cross-Domain Few-Shot Learning, CDFSL). In this paper, we focus on the target-domain few-shot finetuning in the CLIP-based CDFSL task. Prevailing finetuning paradigms uniformly align all image patch tokens with their corresponding textual embeddings. However, we find a counterintuitive phenomenon: actively pushing away certain low-similarity image tokens, termed "tail tokens", from their textual embeddings consistently improves target-domain performance. We delve into this phenomenon and provide a novel interpretation: under great domain shifts and scarce training data, the model can hardly extract semantic information from visual inputs; therefore, the common belief of alignment is valid only for tokens already containing sufficient semantic information; for tail tokens, forcing the alignment would lead to excessive overfitting to the scarce training, while breaking the alignment is more useful. Motivated by this, we propose Adaptive Tail-Head Alignment (ATHA), a novel fine-tuning strategy for CLIP that transforms the conventional uniform alignment paradigm to an adaptive alignment paradigm, with both alignment strengthening and weakening. Extensive experiments on four challenging CDFSL benchmarks validate our state-of-the-art performance. Our code is available at https://github.com/shuaiyi308/ATHA.

Shuai Yi, Yixiong Zou, Yuhua Li +1

Institute of Artificial Intelligence, Huazhong University of Science and Technology, Wuhan, China · School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China.

Vision-Language Model Adaptation Contrastive Language-Image Pre-Training Model Few-Shot Learning Domain Adaptation Value Alignment

Ask PDF

Sep 24, 2026cs.CV

Domain Recentering and Confidence-Weighted Prior Calibration for Vision-Language Models

Vision-language models such as CLIP achieve strong zero-shot classification, yet under distribution shift, visual embeddings drift from fixed text embeddings. Training-free calibration avoids the per-sample optimization of prompt learning, but prior feature calibration gives each image the full bias of one hard cluster. We propose Domain Recentering with Confidence Calibration (DRC), a training-free method adapting CLIP from a set of unlabeled target images. DRC fits a Gaussian mixture once and subtracts from each embedding a posterior-weighted average of component means. It then removes residual class preference with a log-prior correction, estimating the prior from confidence-weighted predictions. Among compared methods, DRC achieves the highest average accuracy on cross-domain datasets, exceeding zero-shot CLIP by 4.13 and 5.07 points with ViT-B/16 and ResNet-50, with gains over CLIP also holding under ImageNet distribution shifts.

Youngeun Seol, Jimin Shin, Heeseo Yoon +1

Department of Computer Science and Engineering, Ewha Womans University

Visual Embeddings Logit Lens Confidence Calibration

Ask PDF

Oct 1, 2026cs.CV

Beyond Domain-Level Adaptation: Margin-Oriented Semantic-Appearance Interaction Correction for Personalized Federated Vision-Language Models

Federated parameter-efficient fine-tuning enables distributed clients to adapt pretrained vision-language models without sharing raw data or updating the full backbone. Its effectiveness, however, is limited by domain heterogeneity across clients. Existing personalized methods separate globally shared knowledge from client-specific style, but they largely treat each domain as a class-agnostic transformation. We show that this abstraction is insufficient: the cross-domain displacement associated with a fixed domain varies across semantic classes, and only a subset of these class-domain residuals damages the image-text decision margin. We therefore propose Margin-Oriented Semantic-Appearance Interaction Correction (MOSAIC), which first constructs a decision-aware harmfulness score that measures whether a training-derived class-domain residual favors a competing text prototype over the true class. It then models fine-grained class-domain interactions with a low-rank residual adapter whose class factors and residual basis are globally shared while domain factors remain client-private. An image-conditioned gate further controls candidate-wise correction, and harmful-pair-aware reweighting prioritizes decision-relevant residuals during local optimization. Extensive experiments on Office31, OfficeHome, and DomainNet100 demonstrate that MOSAIC consistently improves macro-client top-1 accuracy across all evaluated domain-shift and joint domain-label-shift settings.

Wentao Yue, Qingyu Mao, Tianyou Lai +2

Vision-Language Model Adaptation Mitigating Domain Shift Domain Adaptation

Ask PDF