cs.CLMar 8, 2026
SaveCross-Modal Taxonomic Generalization in (Vision-) Language Models
Organizations: Toyota Technological Institute at Chicago · The University of Texas at Austin
Abstract
What is the interplay between semantic representations learned by language models (LM) from surface form alone to those learned from more grounded evidence? We study this question for a scenario where part of the input comes from a different modality -- in our case, in a vision-language model (VLM), where a pretrained LM is aligned with a pretrained image encoder. As a case study, we focus on the task of predicting hypernyms of objects represented in images. We do so in a VLM setup where the image encoder and LM are kept frozen, and only the intermediate mappings are learned. We progressively deprive the VLM of explicit evidence for hypernyms, and test whether knowledge of hypernyms is recoverable from the LM. We find that the LMs we study can recover this knowledge and generalize even in the most extreme version of this experiment (when the model receives no evidence of a hypernym during training). Additional experiments suggest that this cross-modal taxonomic generalization persists under counterfactual image-label mappings only when the counterfactual data have high visual similarity within each category. Taken together, these findings suggest that cross-modal generalization in LMs arises as a result of both coherence in the extralinguistic input and knowledge derived from language cues.
Figures & tables
Figure 1: An instance of our experiments. During training, the projector is deprived of explicit supervision on high-level categories ( hypernyms , e.g., animal ) at various amounts, and is trained to detect the presence (and absence) of lower-level categories (e.g., koala ), keeping the image encoder and the LM backbone frozen. After training, the VLM is tested for generalization to hypernym categories, given previously unseen images.
Figure 2: Depiction of our two ablations, for a sampled toy training set containing several images each of crows, cardinals, and parrots.
Figure 3: Macro F1s of VLMs for predicting the hypernym category for unseen images across LM backbones and image encoder types (DINOv2 vs. SigLIP), in the experiment where all hypernyms are removed. Dashed line indicates majority-label baseline.
Figure 4: Macro F1 on unseen images across LM backbones, LM representations (pre-trained vs. random), and hypernym ablation type (Random vs. Systematic) at different amounts of exposure to hypernym categories, for various test splits. Dashed line indicates the majority-label baseline. Error bands represent 95% confidence intervals across random seeds ( = 3) and categories (hypernyms = 53, leaves = 1216).
Figure 5: Examples of image-leaf mappings resulting from our counterfactual shuffles, in comparison with the original configuration (top). VC indicates the visual coherence of the category under the data configuration.
Figure 6: Macro F1 of models trained on data with different counterfactual shuffles (or lack thereof) at predicting unseen hypernyms, across various amounts of exposure to hypernyms. Dashed line indicates macro F1 of the majority-label baseline. Error bands represent 95% confidence intervals across three random seeds and 53 hypernym categories.
Figure 7: Macro F1 scores on unseen images vs. Visual coherence across the 53 hypernym categories for the Qwen3-1.7B backbone (at 100% ablation). indicates Pearson’s correlation.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | Value |
|---|---|
| Epochs | 5 |
| Batch Size | 16 |
| Number of Layers | 2 |
| Input Dimension | 1024/1152 |
| Output Dimension | 1024/2048 |
| Max Gradient Norm | 1.0 |
Table 1: Training hyperparameters used in our experiments. We adopt a -layer multi-layer perceptron architecture with GeLU activation for all vision-language projectors. The input dimension of our trained MLP projectors is 1024 for DINOv2 and 1152 for SigLIP. The output dimension of our trained MLP projectors is 1024 for Qwen3-0.6B, and 2048 for all other LM backbones. The two MLP layers share the same output dimension across experiments.
Figure 8: Distribution of the number of hypernym categories per leaf in our dataset. Counts (y-axis) are log-scaled.
Figure 9: Number of leaves per hypernym category. Counts (y-axis) are log-scaled.
| Ablation level (%) | # Leaves affected | Data variant | Train set (# I-Q pairs) | Validation set (# I-Q pairs) | Test set (# I-Q pairs) | ||||
|---|---|---|---|---|---|---|---|---|---|
| leaf | seen | unseen | leaf | seen | unseen | ||||
| 100 | 1,222 | Original | 23,250 | 2,028 | – | 3,380 | 9,394 | – | 15,702 |
| Across-category | 23,250 | 2,028 | – | 3,366 | 9,394 | – | 15,680 | ||
| Within-category | 23,250 | 2,028 | – | 3,380 | 9,394 | – | 15,664 | ||
| 90 | 1,148 | Original | 24,840 | 2,028 | 134 | 3,246 | 9,394 | 644 | 15,058 |
| Across-category | 24,818 | 2,028 | 142 | 3,224 | 9,394 | 630 | 15,050 | ||
Table 2: Data statistics under random hypernym ablation levels and counterfactual data variants. “I-Q pair” refers to an image and associated yes-no question. “Seen” and “Unseen” indicate the number of seen and unseen hypernym I-Q pairs, respectively. “# Leaves affected” is the number of leaf categories that have had any I-Q pairs removed, accounting for duplicates (removal under multiple hypernyms), and rounding.
| Ablation level (hypernyms removed) | Data variant | Train set | Validation set | Test set | ||||
|---|---|---|---|---|---|---|---|---|
| leaf | seen | unseen | leaf | seen | unseen | |||
| full | Original | 23,250 | 2,028 | – | 3,380 | 9,394 | – | 15,702 |
| Across-category | 23,250 | 2,028 | – | 3,366 | 9,394 | – | 15,680 | |
| Within-category | 23,250 | 2,028 | – | 3,366 | 9,394 | – | 15,664 | |
| 40 | Original | 29,892 | 2,028 | 564 | 2,816 | 9,394 | 2,684 | 13,018 |
| Across-category | 30,014 | 2,028 | 600 | 2,766 | 9,384 | 2,722 | 12,958 | |
Table 3: Data statistics under systematic hypernym ablation levels and counterfactual data variants. “Seen” and “Unseen” indicate the number of seen and unseen hypernyms, respectively.
| Ablation level | Ablated categories |
|---|---|
| 40 | kitchen appliance, home decor item, medical equipment, plant, protective clothing item, farm animal, mammal, car part, dessert, breakfast food, sea animal, hardware item, school supply, source of light, garden tool, drink, sports equipment, toy, insect, watercraft, footwear, vegetable, piece of jewelry, game, condiment, animal, personal hygiene item, bird, musical, office supply item, item of clothing, piece of women’s clothing, construction equipment, candy, headwear, scientific equipment, home appliance, seafood, safety equipment, container |
| 30 | fastener, condiment, container, kitchen appliance, office supply item, body part, vegetable, kitchen tool, home appliance, outerwear, arts and crafts item, construction equipment, game, insect, breakfast food, home decor item, clothing accessory, food, school supply, fruit, sports equipment, sea animal, piece of women’s clothing, candy, safety equipment, farm animal, dessert, scientific equipment, bird, tool |
| 20 | medical equipment, sea animal, home decor item, fastener, condiment, breakfast food, hardware item, office supply item, toy, seafood, musical, mammal, vehicle, personal hygiene item, protective clothing item, sports equipment, construction equipment, insect, scientific equipment, tool |
| 10 | container, personal hygiene item, dessert, item of clothing, furniture, breakfast food, scientific equipment, garden tool, car part, sea animal |
Table 4: List of ablated categories under different systematic hypernym ablation levels.
Figure 10: Macro F1s of Llama 3.2 models on unseen images across LM backbones, LM representations (pre-trained vs. random), and hypernym ablation type (Random vs. Systematic) at different amounts of exposure to hypernym categories, for various test splits. Dashed line indicates macro F1 of the majority-label baseline. Error bands represent 95% confidence intervals across random seeds ( = 3) and categories (hypernyms = 53, leaves = 1216).
Figure 11: Macro F1s of Llama 3.2 models trained on data with different counterfactual shuffles (or lack thereof) at predicting hypernyms, across various amounts of exposure to hypernym categories. Dashed line indicates macro F1 of the majority-label baseline. Error bands show 95% confidence intervals for three random seeds and 53 hypernym categories.
| Experiment Setting | Macro F1 | Std. Error |
|---|---|---|
| Original | 78.4 | 1.3 |
| Across-category | 50.0 | 0.6 |
| Within-category | 79.8 | 1.2 |
| Majority-label | 39.2 | 0.9 |
Table 5: Macro F1 of models trained with Qwen3-8B as the LM backbone, on the 100% ablation setting, across various shuffles (or lack thereof). Standard error calculated over 3 seeds and 53 hypernym categories.
Figure 12: Macro F1s on unseen images for all four LM backbones across various depths of category organization in our analysis, computed for the 100% ablation experiment. Error bars indicate 95% confidence intervals across three random seeds and number of categories per depth ( = 41; = 9; = 3).
More in this topic
Vision-Language Models
Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.
See all 736 papers