Language-Pretraining-Induced Bias: A Strong Foundation for General Vision Tasks

Yaxin Luo; Zhiqiang Shen

cs.CVApr 2, 2026

Language-Pretraining-Induced Bias: A Strong Foundation for General Vision Tasks

Authors: Yaxin Luo, Zhiqiang Shen

Organizations: Department of Machine Learning MBZUAI

Abstract

The ratio of outlier parameters in language pre-training models and vision pre-training models differs significantly, making cross-modality (language and vision) inherently more challenging than cross-domain adaptation. As a result, many prior studies have focused on cross-domain transfer rather than attempting to bridge language and vision modalities, assuming that language pre-trained models are unsuitable for downstream visual tasks due to disparate parameter spaces. Contrary to this assumption, we show that adding a bridge training stage as a modality adaptation learner can effectively align Large Language Model (LLM) parameters with vision tasks. Specifically, we propose a simple yet powerful solution random label bridge training that requires no manual labeling and helps LLM parameters adapt to vision foundation tasks. Moreover, our findings reveal that partial bridge training is often advantageous, as certain layers in LLMs exhibit strong foundational properties that remain beneficial even without fine-tuning for visual tasks. This surprising discovery opens up new avenues for leveraging language pre-trained parameters directly within vision models and highlights the potential of partial bridge training as a practical pathway to cross-modality adaptation.

Figures & tables

Figure 1: In cross-domain adaptation, the data type remains the same, though domains may vary in style or distribution. In contrast, cross-modality adaptation involves fundamentally different feature spaces, alongside variations in style or distribution.

Figure 2: Outlier parameters and weight distributions in models trained on different modalities. Language-pretrained GPT-2 shows a markedly heavier-tailed distribution with numerous large-magnitude “Outlier” weights, whereas vision-pretrained ViT and GPT-2 structure model trained on images exhibit fewer outliers and narrower spreads.

Figure 3: Train and test acc. curves for pretrained vs. scratch GPT-2 on CIFAR-10 using varying random label proportions (0%, 15%, 30%, 100%). Pretrained models consistently show higher accuracy and faster convergence, indicating enhanced robustness to label noise.

Model	CIFAR-10		CIFAR-100		ImageNet-1K
Model	Train-acc1	Test-acc1	Train-acc1	Test-acc1	Train-acc1	Test-acc1
Scratch-GPT-2	98.7	86.0	98.8	58.0	56.4	66.5
Pretrained-GPT-2	99.2 ${}_{\textbf{+0.5}}$	$87.1_{\textbf{+1.1}}$	99.1 ${}_{\textbf{+0.3}}$	$60.4_{\textbf{+2.4}}$	59.2 ${}_{\textbf{+2.8}}$	$68.8_{\textbf{+2.3}}$
Scratch-LLAMA3.2-1B	99.3	79.7	99.7	49.3	66.3	64.0
Pretrained-LLAMA3.2-1B	$99.6_{\textbf{+0.3}}$	$89.4_{\textbf{+9.7}}$	$99.8_{\textbf{+0.1}}$	$64.7_{\textbf{+15.4}}$	$73.8_{\textbf{+7.5}}$	$67.7_{\textbf{+3.7}}$

Table 1: Comparison of language-pretrained GPT-2 weights and scratch-initialized GPT-2 models on CIFAR-10, CIFAR-100 and ImageNet-1K for image classification. ( Note : we only train GPT-2 on ImageNet-1k for 150 Epochs and LLAMA3.2-1B for 50 Epochs because of resource limitations.)

Model	Random Ratio	CIFAR-10		CIFAR-100
Model	Random Ratio	Train-acc1	Test-acc1	Train-acc1	Test-acc1
Scratch-GPT-2	15%	97.4	76.0	98.5	45.2
	30%	75.4	67.8	98.2	32.5
	100%	34.2	2.5	65.9	0.9
Pretrained-GPT-2	15%	$98.6_{\textbf{+1.2}}$	$77.2_{\textbf{+1.2}}$	$99.0_{\textbf{+0.5}}$	$48.1_{\textbf{+2.9}}$
	30%	$97.9_{\textbf{+22.5}}$	$68.7_{\textbf{+0.9}}$	$98.9_{\textbf{+0.7}}$	$33.7_{\textbf{+1.2}}$

Table 2: Performance comparison of language-pretrained and scratch-initialized GPT-2 models under varying ratios of random labels on CIFAR-10 and CIFAR-100. Darker “ ” and “ ” indicates higher Train-acc1 and Test-acc1, respectively.

Figure 4: Loss landscape on CIFAR-10. We visualize a 2D cross-section of the high-dimensional loss surface by plotting $L(\theta_{0}+\alpha d_{1}+\beta d_{2})$ , where $d_{1}=\theta_{T}-\theta_{0}$ is the training direction and $d_{2}$ is a random direction orthogonal to $d_{1}$ (both per-layer normalized). The X-axis and Y-axis correspond to the coefficients $\alpha$ and $\beta$ , respectively (height/color indicates loss). Top : Correct Labels Training. Bottom : 100% Random Labels Training. Left : Scratch GPT-2. Right : Pretrained GPT-2.

Figure 5: t-SNE embeddings for pretrained (top) and scratch-initialized (bottom) GPT-2 models on CIFAR-10 under 100% random labels. Left : Before training, both appear partly disordered. Right : After training, the pretrained model achieves far tighter, more coherent clusters, indicating superior feature separability compared to the scratch model.

Figure 6: Overview of our two-stage Language Bias Bridge Learning framework. In Stage 1, the pretrained LLM is adapted to the target modality under random labels, In Stage 2, a lightweight classifier refines these representations on real labels.

Figure 7: Layer-wise neuron activation ratios for image samples in GPT-2 models. For each block $\ell$ , we record the MLP post-GELU activations $\mathbf{A}^{(\ell)}(x)\in\mathbb{R}^{T\times d_{\mathrm{mlp}}}$ on real images, compute token-averaged neuron responses, and count neuron $k$ as activated by image $x$ if $\bar{a}^{(\ell)}_{k}(x)>0$ . Left : before training. Middle : random-label bridge training. Right : correct-label bridge training. Training increases $r_{\ell}$ , indicating greater utilization of MLP capacity for visual inputs.

	$\mathrm{wd}=0.05$	$\mathrm{wd}=0.1$	$\mathrm{wd}=0.2$
$\mathrm{lr}=5\mathrm{e}{-4}$	$15.1/31.0_{+15.9}$	$10.5/30.3_{+19.8}$	$10.3/23.5_{+13.2}$
$\mathrm{lr}=3\mathrm{e}{-4}$	$17.2/32.4_{+15.2}$	$13.9/31.1_{+17.2}$	$12.6/24.8_{+12.2}$
$\mathrm{lr}=1\mathrm{e}{-3}$	$17.4/35.0_{+17.6}$	$16.6/33.3_{+16.7}$	$13.7/26.8_{+13.1}$

Table 3: Hyperparameter sensitivity study. We sweep learning rate (lr) and weight decay (wd) under the same training protocol and report linear-probe test acc1 on CIFAR-100 (%). Each entry is reported as w/o LBBT / LBBT ; we additionally report the absolute gain as subscript.

Paradigm	CIFAR-10	CIFAR-100	TinyImageNet-200	ImageNet-1K
Paradigm	Train-acc1	Train-acc1	Train-acc1	Train-acc1
Bridge Training	34.2	65.9	94.2	53.7
Language Bias Bridge Training	93.9 ${}_{\textbf{+59.7}}$	98.7 ${}_{\textbf{+32.8}}$	95.3 ${}_{\textbf{+1.1}}$	56.4 ${}_{\textbf{+2.8}}$

Table 4: Random-label bridge training across datasets of varying scale. We report training top-1 accuracy ( Train-acc1 ) on the random bridge labels. Bridge Training denotes training the same GPT-2 architecture on the visual dataset with 100% random labels, starting from randomly initialized weights. Language Bias Bridge Training (LBBT) uses the identical random-label bridge objective, data, and hyperparameters, but initializes from language-pretrained GPT-2 weights.

Paradigm	LLM Backbone	CIFAR-10	CIFAR-100	TinyImageNet-200
Paradigm	LLM Backbone	Train-acc1	Train-acc1	Train-acc1
Bridge Training	Qwen3-8B	88.2	84.2	90.0
Language Bias Bridge Training	Qwen3-8B	$95.9_{\textbf{+7.7}}$	$93.3_{\textbf{+9.1}}$	$92.4_{\textbf{+2.4}}$
Bridge Training	Qwen3-14B	91.6	88.3	91.7
Language Bias Bridge Training	Qwen3-14B	$97.9_{\textbf{+6.3}}$	$97.7_{\textbf{+9.4}}$	$94.3_{\textbf{+3.4}}$

Table 5: LBBT’s Benefits also hold when scale up to larger LLMs.

Model	AP	AP50	AP75	APs	APm	APL
DETR	36.2	58.0	38.4	15.0	39.1	53.8
Language Bias DETR	38.3 ${}_{\textbf{+2.1}}$	59.2 ${}_{\textbf{+1.2}}$	39.6 ${}_{\textbf{+1.2}}$	16.6 ${}_{\textbf{+1.6}}$	41.7 ${}_{\textbf{+2.6}}$	54.2 ${}_{\textbf{+0.4}}$

Table 6: Object Detection Results on COCO 2017 Dataset.

Unstructured SSL Methods
Method	Train - acc1	Test - acc1
Language Bias Bridge Training	21.8	33.4
Rotation Image	25.1	32.2
Structured SSL Methods
MAE	24.5	30.7
SimCLR	16.2	26.2

Table 7: Linear - probe image - classification accuracy on CIFAR-100. (all pretrained on imagenet100 first.)

Figure 8: Partial Bridge Training results under both random-label and correct-label settings. Updating only the first 2 layers already matches the performance of training all layers, and training the first 5 layers surpasses it.

Layers	CIFAR-10		CIFAR-100
Layers	Train-acc1	Test-acc1	Train-acc1	Test-acc1
Correct Labels :
All Layers	99.2	87.1	99.1	60.4
First 1 Layer	90.2	73.9	88.5	49.4
First 2 Layers	97.5	82.6	97.9	54.7
First 5 Layers	98.9	85.5	99.3	59.7

Table 8: Partial Bridge Training. Darker “ ” and “ ” indicates higher Train-acc1 and Test-acc1, respectively. Note that under random-label training, the test accuracy is naturally low because the labels have no true class semantics; hence, the training accuracy is the primary indicator of model’s fitting ability.

Figure 9: Train and test loss curves for DETR. Language-pretrained initialization accelerates convergence and lowers final loss, highlighting its benefit for dense vision tasks.

Model	AED20K		Pascal
Model	miou(ss)	miou(ms)	miou(ss)	miou(ms)
Segmenter - T - Mask/16	37.8	39.1	44.9	45.5
Language Bias Segmenter - T - Mask/16	$38.5_{\textbf{+0.7}}$	$39.5_{\textbf{+0.4}}$	$46.7_{\textbf{+1.8}}$	$47.4_{\textbf{+1.9}}$

Table 9: Semantic Segmentation Results on AED20K and Pascal Context.

Model	LLM Backbone	SQA	TextVQA	GQA	MM-Vet
Scratch-gpt2-encoder-LLaVA1.5-7B	Vicuna-7B	65.5	42.7	38.6	11.1
Pretrained-gpt2-encoder-LLaVA1.5-7B	Vicuna-7B	66.7 ${}_{\textbf{+1.2}}$	44.2 ${}_{\textbf{+1.5}}$	41.2 ${}_{\textbf{+2.6}}$	18.5 ${}_{\textbf{+7.4}}$
Scratch-gpt2-encoder-LLaVA1.6-7B	Mistral-7B	17.7	68.5	47.8	40.9
Pretrained-gpt2-encoder-LLaVA1.6-7B	Mistral-7B	21.0 ${}_{\textbf{+3.3}}$	69.1 ${}_{\textbf{+0.6}}$	49.3 ${}_{\textbf{+1.5}}$	43.2 ${}_{\textbf{+2.3}}$
Scratch-gpt2-encoder-LLaVA1.6-13B	Vicuna-13B	22.5	70.3	51.9	44.3
Pretrained-gpt2-encoder-LLaVA1.6-13B	Vicuna-13B	24.6 ${}_{\textbf{+2.1}}$	71.8 ${}_{\textbf{+1.5}}$	56.1 ${}_{\textbf{+4.2}}$	48.7 ${}_{\textbf{+4.4}}$

Table 10: Comparison on three Multimodal VQA and one Reasoning Benchmarks across different model scales and LLM backbones. ( Pretrained rows are highlighted; subscript denotes absolute gain over the corresponding scratch setting. )

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Weight Parameter Distribution. We compare weight distributions of GPT-2 models trained from scratch ( blue ) and with language-pretrained weights ( orange ) under correct labels (left) and 100% random labels (right). The pretrained model displays a smoother, more heavily tailed distribution, highlighting its ability to adapt to visual data—even with noisy, non-semantic supervision, thanks to latent linguistic priors.

Figure 11: Comparison of weight parameter distributions and outlier counts. For models trained on either random or correct labels, starting from either a pretrained initialization or from scratch. Even under random labels (top row), the pretrained model continues to exhibit stronger performance than its from-scratch counterpart despite producing more high-magnitude outlier parameters. With correct labels (bottom row), the pretrained initialization still outperforms the from-scratch model, though outlier occurrences remain more frequent in the pretrained setup than in the randomly initialized one.

Figure 12: Train and test acc. curves for pretrained vs. scratch GPT-2 on CIFAR-100.

Figure 13: More Partial Training Ablation studies on layers.

Figure 14: Fine-grained parameter outlier and distribution comparison.

Model	mAP	Pass@1	Recall@1	NDCG@1	Pass@5	Recall@5	NDCG@5
Bridge Training	72.3	72.1	72.1	72.1	71.7	88.1	81.3
Language Bias Bridge Training	87.2	86.6	86.6	86.6	86.5	93.5	90.5

Appendix

Table 11: Image Retrieval Results on Tiny-ImageNet200.

Layers	CIFAR-10
Layers	Train-acc1	Test-acc1
Last Layer	43.1	5.2
Last 14 Layers	87.9	6.3
First Layer	69.9	4.3
All Layers	93.9	6.3
First 5 Layers	96.5	6.7

Appendix

Table 12: Extend ablation study on Random Label Partial Bridge Training. Darker “ ” indicates higher Train-acc1 and Test-acc1, respectively. Note that under random-label training, the test accuracy is naturally low because the labels have no true class semantics; hence, the training accuracy is the primary indicator of model’s fitting ability.

Correct Label
Model	Imagenet100		CIFAR-100		CIFAR-10
Model	Train-acc1	Test-acc1	Train-acc1	Test-acc1	Train-acc1	Test-acc1
Pretrained-GPT-2	86.6	76.2	99.1	60.4	99.2	87.1
Pretrained-Chronos-t5	78.7	79.7	94.5	58.1	98.3	88.2
Random Label
Pretrained-GPT-2	94.1	1.5	98.7	0.8	93.9	6.3

Appendix

Table 13: Comparison between Language Pretrained and Time Series Pretrained Bias.

Paradigm	CIFAR-10	CIFAR-100
Paradigm	Test-acc1	Test-acc1
w/o Bridge Training	62.7	35.0
Bridge Training	74.2	49.1
Language Bias Bridge Training	82.3	56.3

Appendix

Table 14: Linear probing image classification performance across CIFAR-10 and CIFAR-100. All models are random label bridge trained on TinyImagenet-200 first.

Metric	Pretrained GPT2	Scratch GPT2
Eigenvalue decay rate	-11.603	-3.691
Kurtosis of eigenvalue distribution	14.511	-1.409
Trace of Hessian	305,404	20,785
Spectral gap	207,779.44	226.10
Participation ratio	0.336	0.778
Noise sensitivity AUC	-0.011	0.003

Appendix

Table 15: Quantitave analysis of loss landscape between Pretrained GPT2 and Scratch GPT2.

Transfer Learning

Transfer learning studies how models reuse knowledge across tasks, domains, or modalities when target data or computation are limited.

See all 417 papers

Beyond Language Modeling: An Exploration of Multimodal Pretraining
Mar 3, 2026Shengbang Tong, David Fan, John Nguyen +18Multimodal Pretraining Unified Multimodal Models
FAIR, Meta · New York University
Enhancing Self-Supervised Visual Representation Learning via Low-Rank Adapted LLMs
Jul 1, 2025Selim Kuzucu, Muhammad Ferjad Naeem, Anna Kukleva +2Vision Transformer Self-Supervised Visual Representation Learning
Max Planck Institute for Informatics, SIC · Google
VisCoP: Visual Probing for Video Domain Adaptation of Vision Language Models
Oct 15, 2025Dominick Reilly, Manish Kumar Govind, Le Xue +1Domain Adaptation VLM Adaptation
University of North Carolina at Charlotte · Elorian AI

Mar 3, 2026J/K move · Enter open · S save

Beyond Language Modeling: An Exploration of Multimodal Pretraining

Shengbang Tong, David Fan, John Nguyen, Ellis Brown+17

FAIR, Meta · New York University

Multimodal Pretraining Unified Multimodal Models Scaling Laws

Open paper AskPDF arXiv

The visual world offers a critical axis for advancing foundation models beyond language. Despite growing interest in this direction, the design space for native multimodal models remains opaque. We provide empirical clarity through controlled, from-scratch pretraining experiments, isolating the factors that govern multimodal pretraining without interference from language pretraining. We adopt the Transfusion framework, using next-token prediction for language and diffusion for vision, to train on diverse data including text, video, image-text pairs, and even action-conditioned video. Our experiments yield four key insights: (i) Representation Autoencoder (RAE) provides an optimal unified visual representation by excelling at both visual understanding and generation; (ii) visual and language data are complementary and yield synergy for downstream capabilities; (iii) unified multimodal pretraining leads naturally to world modeling, with capabilities emerging from general training; and (iv) Mixture-of-Experts (MoE) enables efficient and effective multimodal scaling while naturally inducing modality specialization. Through IsoFLOP analysis, we compute scaling laws for both modalities and uncover a scaling asymmetry: vision is significantly more data-hungry than language. We demonstrate that the MoE architecture harmonizes this scaling asymmetry by providing the high model capacity required by language while accommodating the data-intensive nature of vision, paving the way for truly unified multimodal models.

Abstract

Transfer Learning

Explore similar work

Beyond Language Modeling: An Exploration of Multimodal Pretraining

Enhancing Self-Supervised Visual Representation Learning via Low-Rank Adapted LLMs

VisCoP: Visual Probing for Video Domain Adaptation of Vision Language Models