Language Guidance for Supervised Vision Training: An Empirical Study of Generalization

Shruthi Gowda; Nikita Jain; Elahe Arani; Bahram Zonooz

cs.CVOct 19, 2026

Language Guidance for Supervised Vision Training: An Empirical Study of Generalization

Authors: Shruthi Gowda, Nikita Jain, Elahe Arani, Bahram Zonooz

Organizations: Eindhoven University of Technology

Abstract

Deep neural networks have achieved remarkable success on vision benchmarks, yet they continue to struggle with many generalization challenges. Supervised vision training relies on one-hot labels, which provide limited information about semantic structure and shared attributes between classes. This limited supervision can leave visual representations vulnerable to distribution shifts, spurious correlations, texture bias, adversarial perturbations, and forgetting in sequential learning settings. We study whether pretrained language models can serve as lightweight auxiliary supervision for vision training without requiring paired image-text data, prompt engineering, or contrastive objectives. Specifically, we evaluate two forms of language guidance, Explicit Language Guidance (ExLG) and Implicit Language Guidance (ImLG). We conduct a comprehensive evaluation across six generalization regimes, including in-distribution, out-of-distribution generalization, shortcut and spurious correlation resistance, texture and shape bias, adversarial robustness, and continual learning. Our analyses show that the two mechanisms have complementary strengths, with explicit guidance consistently benefiting in-distribution, low-data performance, and continual learning retention, while implicit guidance is often more useful in shortcut-sensitive settings and under stronger adversarial perturbations. Importantly, both are lightweight and add minimal parameters and training overhead. The analyses characterize when language-derived structure helps supervised vision training and provides a practical roadmap for using off-the-shelf pretrained models from another modality as auxiliary supervision.

Figures & tables

Figure 1: Similarity analysis to show if language context introduced by Explicit or Implicit Language Guidance enhances semantic alignment across domains in DN4IL dataset.

Figure 2: (a) Explicit Language Guidance: Learning visual representations with explicit supervision from language descriptions. (b) Implicit Language Guidance: Learning visual representations via an embedded frozen language block for implicit supervision.

Vision Encoder	ResNet18	ResNet50	ViT
PLM	Sentence Transformer variants	CLIP
Analysis	Datasets
IID	CIFAR10	CIFAR100	TinyImageNet	ImageNet-100
OOD	ImageNet-O	ImageNet-R	ImageNet-A
Shortcut Learning	Tinted-CIFAR10	Skewed-CelebA	WaterBirds
Texture Bias	Stylized TinyImageNet	Texture-Shape Cue-Conflict

Table 1: Summary of all analyses and datasets in this study.

Method	IID			CIFAR-10 Data Efficiency (%)
Method	CIFAR-10	CIFAR-100	TinyImgNet	2	5	10	20
Baseline	94.84 $\pm$ 0.14	76.98 $\pm$ 0.39	58.73 $\pm$ 0.35	45.71 $\pm$ 1.52	55.42 $\pm$ 1.08	67.04 $\pm$ 2.19	79.62 $\pm$ 2.60
ExLG	95.12 $\pm$ 0.05	77.59 $\pm$ 0.08	65.63 $\pm$ 0.26	47.88 $\pm$ 0.53	57.24 $\pm$ 1.95	69.97 $\pm$ 1.87	84.75 $\pm$ 0.61
ImLG	93.41 $\pm$ 0.46	74.10 $\pm$ 0.91	60.02 $\pm$ 0.16	45.03 $\pm$ 2.06	55.53 $\pm$ 2.06	67.82 $\pm$ 1.42	79.03 $\pm$ 0.57

Table 2: In-distribution accuracy and CIFAR-10 sample efficiency. ExLG is most consistent in clean and low-data settings.

Table 5

Method	Tint-CIF10	Skewed-CelebA
Method	Tint-CIF10	Overall	NonBlonde-M	Blonde-F	Blonde-M	NonBlonde-F
Baseline	16.45 $\pm$ 1.81	61.28 $\pm$ 1.21	94.71 $\pm$ 0.08	92.21 $\pm$ 1.02	56.38 $\pm$ 0.39	27.74 $\pm$ 2.29
+ExLG	18.24 $\pm$ 0.60	72.11 $\pm$ 1.28	96.29 $\pm$ 0.30	95.18 $\pm$ 0.31	68.33 $\pm$ 0.98	47.67 $\pm$ 2.31
+ImLG	18.51 $\pm$ 1.04	75.90 $\pm$ 1.79	97.85 $\pm$ 0.65	96.81 $\pm$ 0.11	69.77 $\pm$ 1.03	53.84 $\pm$ 3.38
		WaterBirds
	Method	Overall	Landbird/land	Waterbird/water	Landbird/water	Waterbird/land

Table 5: Shortcut learning on Tinted-CIFAR10, Skewed-CelebA, and Waterbirds. Language-guided models improve robustness to spurious features and conflict groups.

Figure 3: Activation maps of the models on the Skewed-CelebA and Waterbird dataset. Language-guided models focus on the salient features, while conventional methods focus on spurious cues (hair color).

Figure 4: (1) Texture Bias - Analysis on Stylized TinyImageNet across three levels of stylization. (2) Shape vs Texture bias on a conflict cue dataset that has different textures applied on objects

Figure 5: Robustness against PGD-10 adversarial attack on varying strengths ( $\epsilon$ ) on the CIFAR-10

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Comparison of activation maps after each block without ( left ) and with ( right ) implicit language-guided training.

Method	CIFAR10, CIFAR100, Tinted-CIFAR10	Skewed-CelebA, Waterbirds	TinyImageNet
ExLG	$lr=0.1$ $\lambda=15.0$	$lr=0.03$ $\lambda=15.0$	$lr=0.03$ $\lambda=100.0$
ImLG	$lr=0.003$	$lr=0.0001$	$lr=0.003$

Appendix

Table 7: Hyperparameters: All models are trained for 100 epochs using the SGD optimizer, except for implicit methods, which employ the AdamW optimizer due to the inclusion of a transformer block.

Method	Seq-CIFAR10	Seq-TinyImageNet	DN4IL
ExLG	$lr=0.05$ $\lambda=50.0$	$lr=0.05$ $\lambda=100.0$	$lr=0.03$ $\lambda=100.0$
ImLG	$lr=0.001$	$lr=0.0001$	$lr=0.0001$

Appendix

Table 8: Hyperparameters for continual learning analyses: All tasks are trained for

50

epochs with SGD optimizer for ExLG method and AdamW for ImLG.

Figure 8: Few examples of Stylized-TinyImageNet dataset used in Texture vs Shape Analysis

Figure 9: Task-wise performance of class-incremental learning setting on Seq-CIFAR10 dataset with 200 buffer size.

	CIFAR10			CelebA
	Base	ExLG	ImLG	Base	ExLG	ImLG
ResNet50	94.38	97.21	95.86	63.88	74.45	76.58

Appendix

Table 9: Results with a bigger CNN backbone- ResNet50 on one seed

		Cls		Shortcut	OOD			CL
		CIFAR10	TinyImg	CelebA	ImgNet-O	ImgNet-R	ImgNet-A	DN4IL
Base		94.84	58.73	61.28	41.73	10.59	1.92	24.15
ExLG	LM	95.12	65.63	72.11	46.70	14.95	2.94	27.71
ExLG	Larger LM	95.01	65.89	72.93	47.51	14.65	2.92	26.84
	MM-CLIP	95.13	64.62	70.50	45.56	12.58	2.50	25.54

Appendix

Table 10: Results (mean on 3 seeds) with different language models using ExLG method.

		Cls		Shortcut	OOD			CL
		CIFAR10	TinyImg	CelebA	ImgNet-O	ImgNet-R	ImgNet-A	DN4IL
Base		94.84	58.73	61.28	41.73	10.59	1.92	24.15
ImLG	LM	93.41	60.02	75.90	42.20	12.10	2.37	24.22
ImLG	Larger LM	93.78	61.16	77.55	42.12	12.71	2.45	24.51
	MM-CLIP	93.33	57.87	70.50	39.10	10.80	2.06	24.00

Appendix

Table 11: Results (mean on 3 seeds) with different language models using ImLG method.

		CelebA
		Acc	#Trainable Params (M)	Training Time (seconds)
ResNet18	Base	61.28	11.17	575.63
	Ex-LG	72.11	11.17	590.91
	Im-LG	75.90	14.51	586.54
ResNet50	Base	63.88	23.50	4100

Appendix

Table 12: Resource analyses.

ExLG Variant	CIFAR-10	CelebA	DN4IL
Baseline	94.84 $\pm$ 0.14	61.28 $\pm$ 1.21	24.15 $\pm$ 0.34
LM-Rand (random LM weights)	91.09 $\pm$ 1.67	67.25 $\pm$ 1.44	22.27 $\pm$ 1.68
Shuffled descriptions	93.05 $\pm$ 0.04	68.85 $\pm$ 1.16	24.78 $\pm$ 0.23
Simple (class names only)	94.97 $\pm$ 0.11	70.00 $\pm$ 0.36	26.27 $\pm$ 0.78
ExLG	95.12 $\pm$ 0.05	72.11 $\pm$ 1.28	27.71 $\pm$ 0.64

Appendix

Table 13: ExLG ablation on

3

seeds: effect of description quality on classification accuracy.

ImLG Variant	CelebA
Baseline	61.28 $\pm$ 1.21
Projection-only (no LM block)	62.25 $\pm$ 2.81
LM-Rand (random frozen transformer block)	57.41 $\pm$ 8.10
Weight-shuffle (permuted pretrained weights)	70.94 $\pm$ 0.12
ImLG	75.90 $\pm$ 1.79

Appendix

Table 14: ImLG ablation on 3 seeds with projection pipeline, transformer architecture, and pretrained language knowledge. It is reported on CelebA as ImLG shows the most imapct on shortcut learning datasets.

Method	Optimizer	CIFAR-10	CelebA
Baseline	SGD (paper)	94.84 $\pm$ 0.14	61.28 $\pm$ 1.21
Baseline	AdamW	92.33 $\pm$ 0.02	58.00 $\pm$ 2.60
ExLG	SGD (paper)	95.12 $\pm$ 0.05	72.00 $\pm$ 1.28
ExLG	AdamW	91.88 $\pm$ 0.39	62.77 $\pm$ 0.99
ImLG	AdamW (paper)	93.41 $\pm$ 0.46	75.90 $\pm$ 1.79

Appendix

Table 15: Optimizer fairness on

3

seeds: AdamW applied to Baseline and ExLG degrades performance on both datasets. ImLG uses AdamW as an architectural requirement, not an optimisation advantage.

Insertion point	CelebA (mean $\pm$ std)
After layer2	$69.09\pm 5.2$
After layer3	$74.45\pm 2.3$
After layer4 (paper)	$\mathbf{75.90\pm 1.8}$

Appendix

Table 16: ImLG insertion-layer ablation on Skewed-CelebA (3 seeds). The frozen LM block is inserted after the indicated ResNet layer; later layers provide richer, more semantically structured inputs.

OOD Generalization

Out-of-distribution generalization studies how machine-learning models perform when test data, domains, tasks, or environments differ from training.

See all 386 papers

Language-Instructed Vision Embeddings for Controllable and Generalizable Perception
Jun 17, 2026Chengzhi Mao, Xudong Lin, Wen-Sheng ChuVisual Representation Learning Vision Foundation Model Adaptation
Google
Domain Generalization via Text-Anchored Information Bottleneck
Jul 2, 2026Eunyi Lyou, Yunjeong Choi, Junho Lee +1VLM Robustness Information Bottleneck
Seoul National University, Seoul, Republic of Korea
Focusing Where Vision Matters: Selective Training for Large Vision Language Models via Visual Information Gain
Feb 19, 2026Seulbi Lee, Sangheum HwangVision-Language Models Large Vision-Language Models
Department of Data Science, Seoul National University of Science and Technology, Seoul, Republic of Korea. · Currently with Tomocube, Daejeon, Republic of Korea.

Jun 17, 2026J/K move · Enter open · S save

Language-Instructed Vision Embeddings for Controllable and Generalizable Perception

Chengzhi Mao, Xudong Lin, Wen-Sheng Chu

Google

Visual Representation Learning Vision Foundation Model Adaptation VLM Hallucination Mitigation

Open paper AskPDF arXiv

Vision foundation models are typically trained as static feature extractors, placing the burden of task adaptation onto large downstream models. We propose an alternative paradigm: instead of solely feeding visual features into language models, we use language itself to dynamically guide the vision encoder. Our method, Language-Instructed Vision Embeddings (LIVE), leverages language as high-level guidance to produce task-centric embeddings at inference time, removing the need for task-specific retraining. This enables the encoder to focus on contextually relevant aspects of the input, yielding more controllable and generalizable representations. Empirically, LIVE reduces visual hallucinations (+34 points on MMVP), surpasses vision-language models with orders of magnitude more parameters on visual question answering, and generalizes to unseen instructions and tasks -- offering a direct path toward adaptive, instruction-driven visual intelligence.

Abstract

OOD Generalization

Explore similar work

Language-Instructed Vision Embeddings for Controllable and Generalizable Perception

Domain Generalization via Text-Anchored Information Bottleneck

Focusing Where Vision Matters: Selective Training for Large Vision Language Models via Visual Information Gain