Deep neural networks have achieved remarkable success on vision benchmarks, yet they continue to struggle with many generalization challenges. Supervised vision training relies on one-hot labels, which provide limited information about semantic structure and shared attributes between classes. This limited supervision can leave visual representations vulnerable to distribution shifts, spurious correlations, texture bias, adversarial perturbations, and forgetting in sequential learning settings. We study whether pretrained language models can serve as lightweight auxiliary supervision for vision training without requiring paired image-text data, prompt engineering, or contrastive objectives. Specifically, we evaluate two forms of language guidance, Explicit Language Guidance (ExLG) and Implicit Language Guidance (ImLG). We conduct a comprehensive evaluation across six generalization regimes, including in-distribution, out-of-distribution generalization, shortcut and spurious correlation resistance, texture and shape bias, adversarial robustness, and continual learning. Our analyses show that the two mechanisms have complementary strengths, with explicit guidance consistently benefiting in-distribution, low-data performance, and continual learning retention, while implicit guidance is often more useful in shortcut-sensitive settings and under stronger adversarial perturbations. Importantly, both are lightweight and add minimal parameters and training overhead. The analyses characterize when language-derived structure helps supervised vision training and provides a practical roadmap for using off-the-shelf pretrained models from another modality as auxiliary supervision.
Figures & tables
Figure 1: Similarity analysis to show if language context introduced by Explicit or Implicit Language Guidance enhances semantic alignment across domains in DN4IL dataset.
Figure 2: (a) Explicit Language Guidance: Learning visual representations with explicit supervision from language descriptions. (b) Implicit Language Guidance: Learning visual representations via an embedded frozen language block for implicit supervision.
Vision Encoder
ResNet18
ResNet50
ViT
PLM
Sentence Transformer variants
CLIP
Analysis
Datasets
IID
CIFAR10
CIFAR100
TinyImageNet
ImageNet-100
OOD
ImageNet-O
ImageNet-R
ImageNet-A
Shortcut Learning
Tinted-CIFAR10
Skewed-CelebA
WaterBirds
Texture Bias
Stylized TinyImageNet
Texture-Shape Cue-Conflict
Table 1: Summary of all analyses and datasets in this study.
Method
IID
CIFAR-10 Data Efficiency (%)
CIFAR-10
CIFAR-100
TinyImgNet
2
5
10
20
Baseline
94.84 ± 0.14
76.98 ± 0.39
58.73 ± 0.35
45.71 ± 1.52
55.42 ± 1.08
67.04 ± 2.19
79.62 ± 2.60
ExLG
95.12 ± 0.05
77.59 ± 0.08
65.63 ± 0.26
47.88 ± 0.53
57.24 ± 1.95
69.97 ± 1.87
84.75 ± 0.61
ImLG
93.41 ± 0.46
74.10 ± 0.91
60.02 ± 0.16
45.03 ± 2.06
55.53 ± 2.06
67.82 ± 1.42
79.03 ± 0.57
Table 2: In-distribution accuracy and CIFAR-10 sample efficiency. ExLG is most consistent in clean and low-data settings.
Table 5
Method
Tint-CIF10
Skewed-CelebA
Overall
NonBlonde-M
Blonde-F
Blonde-M
NonBlonde-F
Baseline
16.45 ± 1.81
61.28 ± 1.21
94.71 ± 0.08
92.21 ± 1.02
56.38 ± 0.39
27.74 ± 2.29
+ExLG
18.24 ± 0.60
72.11 ± 1.28
96.29 ± 0.30
95.18 ± 0.31
68.33 ± 0.98
47.67 ± 2.31
+ImLG
18.51 ± 1.04
75.90 ± 1.79
97.85 ± 0.65
96.81 ± 0.11
69.77 ± 1.03
53.84 ± 3.38
WaterBirds
Method
Overall
Landbird/land
Waterbird/water
Landbird/water
Waterbird/land
Table 5: Shortcut learning on Tinted-CIFAR10, Skewed-CelebA, and Waterbirds. Language-guided models improve robustness to spurious features and conflict groups.
Figure 3: Activation maps of the models on the Skewed-CelebA and Waterbird dataset. Language-guided models focus on the salient features, while conventional methods focus on spurious cues (hair color).
Figure 4: (1) Texture Bias - Analysis on Stylized TinyImageNet across three levels of stylization. (2) Shape vs Texture bias on a conflict cue dataset that has different textures applied on objects
Figure 5: Robustness against PGD-10 adversarial attack on varying strengths ( ϵ ) on the CIFAR-10
Figure 10
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Comparison of activation maps after each block without ( left ) and with ( right ) implicit language-guided training.
Method
CIFAR10, CIFAR100, Tinted-CIFAR10
Skewed-CelebA, Waterbirds
TinyImageNet
ExLG
lr=0.1λ=15.0
lr=0.03λ=15.0
lr=0.03λ=100.0
ImLG
lr=0.003
lr=0.0001
lr=0.003
Appendix
Table 7: Hyperparameters: All models are trained for 100 epochs using the SGD optimizer, except for implicit methods, which employ the AdamW optimizer due to the inclusion of a transformer block.
Method
Seq-CIFAR10
Seq-TinyImageNet
DN4IL
ExLG
lr=0.05λ=50.0
lr=0.05λ=100.0
lr=0.03λ=100.0
ImLG
lr=0.001
lr=0.0001
lr=0.0001
Appendix
Table 8: Hyperparameters for continual learning analyses: All tasks are trained for 50 epochs with SGD optimizer for ExLG method and AdamW for ImLG.
Figure 8: Few examples of Stylized-TinyImageNet dataset used in Texture vs Shape Analysis
Figure 9: Task-wise performance of class-incremental learning setting on Seq-CIFAR10 dataset with 200 buffer size.
CIFAR10
CelebA
Base
ExLG
ImLG
Base
ExLG
ImLG
ResNet50
94.38
97.21
95.86
63.88
74.45
76.58
Appendix
Table 9: Results with a bigger CNN backbone- ResNet50 on one seed
Cls
Shortcut
OOD
CL
CIFAR10
TinyImg
CelebA
ImgNet-O
ImgNet-R
ImgNet-A
DN4IL
Base
94.84
58.73
61.28
41.73
10.59
1.92
24.15
ExLG
LM
95.12
65.63
72.11
46.70
14.95
2.94
27.71
Larger LM
95.01
65.89
72.93
47.51
14.65
2.92
26.84
MM-CLIP
95.13
64.62
70.50
45.56
12.58
2.50
25.54
Appendix
Table 10: Results (mean on 3 seeds) with different language models using ExLG method.
Cls
Shortcut
OOD
CL
CIFAR10
TinyImg
CelebA
ImgNet-O
ImgNet-R
ImgNet-A
DN4IL
Base
94.84
58.73
61.28
41.73
10.59
1.92
24.15
ImLG
LM
93.41
60.02
75.90
42.20
12.10
2.37
24.22
Larger LM
93.78
61.16
77.55
42.12
12.71
2.45
24.51
MM-CLIP
93.33
57.87
70.50
39.10
10.80
2.06
24.00
Appendix
Table 11: Results (mean on 3 seeds) with different language models using ImLG method.
CelebA
Acc
#Trainable Params (M)
Training Time (seconds)
ResNet18
Base
61.28
11.17
575.63
Ex-LG
72.11
11.17
590.91
Im-LG
75.90
14.51
586.54
ResNet50
Base
63.88
23.50
4100
Appendix
Table 12: Resource analyses.
ExLG Variant
CIFAR-10
CelebA
DN4IL
Baseline
94.84 ± 0.14
61.28 ± 1.21
24.15 ± 0.34
LM-Rand (random LM weights)
91.09 ± 1.67
67.25 ± 1.44
22.27 ± 1.68
Shuffled descriptions
93.05 ± 0.04
68.85 ± 1.16
24.78 ± 0.23
Simple (class names only)
94.97 ± 0.11
70.00 ± 0.36
26.27 ± 0.78
ExLG
95.12 ± 0.05
72.11 ± 1.28
27.71 ± 0.64
Appendix
Table 13: ExLG ablation on 3 seeds: effect of description quality on classification accuracy.
ImLG Variant
CelebA
Baseline
61.28 ± 1.21
Projection-only (no LM block)
62.25 ± 2.81
LM-Rand (random frozen transformer block)
57.41 ± 8.10
Weight-shuffle (permuted pretrained weights)
70.94 ± 0.12
ImLG
75.90 ± 1.79
Appendix
Table 14: ImLG ablation on 3 seeds with projection pipeline, transformer architecture, and pretrained language knowledge. It is reported on CelebA as ImLG shows the most imapct on shortcut learning datasets.
Method
Optimizer
CIFAR-10
CelebA
Baseline
SGD (paper)
94.84 ± 0.14
61.28 ± 1.21
Baseline
AdamW
92.33 ± 0.02
58.00 ± 2.60
ExLG
SGD (paper)
95.12 ± 0.05
72.00 ± 1.28
ExLG
AdamW
91.88 ± 0.39
62.77 ± 0.99
ImLG
AdamW (paper)
93.41 ± 0.46
75.90 ± 1.79
Appendix
Table 15: Optimizer fairness on 3 seeds: AdamW applied to Baseline and ExLG degrades performance on both datasets. ImLG uses AdamW as an architectural requirement, not an optimisation advantage.
Insertion point
CelebA (mean ± std)
After layer2
69.09±5.2
After layer3
74.45±2.3
After layer4 (paper)
75.90±1.8
Appendix
Table 16: ImLG insertion-layer ablation on Skewed-CelebA (3 seeds). The frozen LM block is inserted after the indicated ResNet layer; later layers provide richer, more semantically structured inputs.
Vision foundation models are typically trained as static feature extractors, placing the burden of task adaptation onto large downstream models. We propose an alternative paradigm: instead of solely feeding visual features into language models, we use language itself to dynamically guide the vision encoder. Our method, Language-Instructed Vision Embeddings (LIVE), leverages language as high-level guidance to produce task-centric embeddings at inference time, removing the need for task-specific retraining. This enables the encoder to focus on contextually relevant aspects of the input, yielding more controllable and generalizable representations. Empirically, LIVE reduces visual hallucinations (+34 points on MMVP), surpasses vision-language models with orders of magnitude more parameters on visual question answering, and generalizes to unseen instructions and tasks -- offering a direct path toward adaptive, instruction-driven visual intelligence.
Visual recognition models often fail when deployed in new environments. Domain Generalization (DG) addresses this by learning representations that remain invariant to environment-specific variations. Recent approaches increasingly rely on large vision-language models, assuming that preserving their expressive visual representations improves robustness. However, we show that such visual expressiveness can instead propagate spurious cues that tie representations to the training environments, hindering invariant learning. We therefore discard visual guidance and instead treat the language embedding space as the primary source of domain invariance, naturally acting as an information bottleneck that preserves core semantics while suppressing domain-specific variations. Extensive experiments across diverse backbones exhibit state-of-the-art performance and further analyze what makes guidance effective for robust generalization. These findings shift the focus of DG from improving representations to designing supervision that enforces invariance.
Eunyi Lyou, Yunjeong Choi, Junho Lee +1
Seoul National University, Seoul, Republic of Korea
Large Vision Language Models (LVLMs) have achieved remarkable progress, yet they often suffer from language bias, producing answers without relying on visual evidence. While prior work attempts to mitigate this issue through decoding strategies, architectural modifications, or curated instruction data, they typically lack a quantitative measure of how much individual training samples or tokens actually benefit from the image. In this work, we introduce Visual Information Gain (VIG), a perplexity-based metric that measures the reduction in prediction uncertainty provided by visual input. VIG enables fine-grained analysis at both sample and token levels, effectively highlighting visually grounded elements such as colors, spatial relations, and attributes. Leveraging this, we propose a VIG-guided selective training scheme that prioritizes high-VIG samples and tokens. This approach improves visual grounding and mitigates language bias, achieving superior performance with significantly reduced supervision by focusing exclusively on visually informative samples and tokens.
Seulbi Lee, Sangheum Hwang
Department of Data Science, Seoul National University of Science and Technology, Seoul, Republic of Korea. · Currently with Tomocube, Daejeon, Republic of Korea.