Learning an unfamiliar visual category requires more than adapting a classifier: the model must connect its visual evidence to a useful semantic name. A pre-trained vision-language model can retain discriminative image features while providing a weak textual anchor for the target concept. We address this imbalance through context-grounded semantic anchoring. Our framework, SynerNet, learns concept-specific name embeddings, exposes them to compatible linguistic contexts, and conditions their representations on visual evidence. Difficulty-aware processing and adaptive objective weighting coordinate this adaptation while retaining the pre-trained backbone. On four VISTA-Beyond domains, SynerNet reaches a mean 16-shot accuracy of 74.35%, compared with 72.45% for few-shot name learning. Across 14 domains in the name-masked transfer setting, it improves mean accuracy from 45.17% to 46.74%. Ablations identify concept-specific name embeddings and cross-modal context integration as important components. These results support treating concept acquisition as the construction of visually grounded semantic anchors.
Figures & tables
Figure 1: Visual discrimination need not imply semantic alignment. Circles denote visual evidence and diamonds denote name anchors. Fixed anchors can remain separated from their corresponding visual groups; name learning and visual context ground the anchors. This is a conceptual schematic with generated example thumbnails.
Figure 2: SynerNet architecture. Frozen encoders supply image and text features. Name learning and context exchange form concept-specific text representations. Visual context conditions residual fusion to produce semantic anchors for cosine matching. An auxiliary visual classifier and cross-modal alignment provide the joint objectives. Dashed coral arrows denote adaptation feedback. Example thumbnails and the similarity grid are schematic.
Domain
Method
1-shot
2-shot
4-shot
8-shot
16-shot
Insects / spiders
CoOp
19.3
22.9
26.6
29.8
32.5
CoCoOp
16.4
17.2
17.8
19.9
22.6
CLIP-Adapter
24.3
26.8
29.5
32.8
36.0
FSNL
31.4
35.7
39.7
41.9
43.7
SynerNet
33.8
38.3
41.2
43.5
45.4
Landmark
CoOp
43.8
59.9
80.1
85.8
89.4
Table 1: Few-shot recognition on four VISTA-Beyond domains. Top-1 accuracy (%). Bold indicates the best result within each domain and shot count.
Figure 3: Accuracy and component contributions. (a) Macro-average accuracy across the four few-shot domains, computed from Table 1 . (b) Mean accuracy decrease for each ablation, computed from the five domain scores in Table 2 . The panels summarize different experimental settings and should not be interpreted as sharing the same evaluation population.
Variant
Animals
Architecture
Attire
Food
Pokemon
Mean drop ↓
Full SynerNet
53.9
89.4
82.0
92.9
79.7
—
w/o visual unit
52.1
86.3
78.3
89.8
76.3
3.02
w/o linguistic unit
52.8
87.3
79.8
90.3
77.8
1.98
w/o name unit
50.8
85.8
77.3
88.3
75.3
4.08
w/o coordinator
51.3
86.8
78.3
89.3
76.8
3.08
w/o context exchange
51.5
86.5
78.6
89.5
77.0
2.96
Table 2: Component ablations. Accuracy (%) in the five-domain ablation setting. The final column is the arithmetic mean decrease from the full model, in percentage points.
Domain
OpenCLIP
TransCLIP
ZSNL
Ours
Animals
19.4
21.7
26.4
27.6
Architecture
22.3
24.4
39.1
41.3
Attire
16.8
18.8
34.5
36.4
FolkArt
26.7
29.5
27.8
29.1
Food
8.9
10.4
29.3
30.7
Insects / spiders
16.7
17.9
24.3
26.1
Table 3: Name-masked transfer across 14 domains. Top-1 accuracy (%). Target adaptation uses 16 images per class; names are masked at inference. Bold marks the best score in each row.
Figure 4: Domain-level transfer gains. SynerNet minus ZSNL, computed from Table 3 . The dashed line is the macro-average gain of 1.57 percentage points. Positive gains occur in 13 of the 14 domains.
Vision-language models (VLMs) can follow complex textual instructions, yet they struggle to reason from purely visual context. In particular, current models fail to infer shared concepts from sets of example images and apply them to new inputs. We introduce Visual Concept Inference from Sets (VICIS), a task that evaluates this capability. Given a small context set of images sharing a concept and a query image, the model must generate new images that preserve the context-defined concept while remaining consistent with the query. We show that state-of-the-art VLMs perform poorly on this task, often ignoring the visual context or defaulting to biased generations. To address this gap, we propose a training framework and architecture that learn to infer visual concepts from image sets and extract concept-specific embeddings from queries. Experiments on synthetic data and large-scale ImageNet/WordNet data show that our model generates more accurate and diverse outputs and generalizes to unseen concepts and modalities such as sketches.
Nick Stracke, Kolja Bauer, Stefan Andreas Baumann +3
We study how Vision Transformers ground abstract concepts (e.g., angry) when training data provide limited direct referential evidence. We hypothesize a metonymic grounding mechanism in which abstract predictions are driven by concrete, interpretable anchor concepts (e.g., fire) that bridge visual signals to abstract semantics. By applying Transcoders on CLIP and DINO vision encoders, we recover intermediate features that can be associated with semantic labels for more concrete concepts, and trace their contributions in circuits underlying abstract concept recognition. Experiments on a carefully curated icon dataset reveal structured metonymic circuits, in which perceptual primitives dominate early layers and object-like anchors precede abstract targets. Images containing rendered text instead recruit a distinct perceptual-to-textual route. Causal interventions further validate that metonymic intermediates are functionally involved in grounding abstract concepts.
Jing Ding, Ziqiao Ma, Jiayuan Mao +2
University of Michigan · Vector Institute · University of Pennsylvania +2
Vision-language models (VLMs), like human learners, are frequently exposed to new visual concepts, but how they map novel visual references to language after exposure remains largely underexplored, particularly when those references contradict prior knowledge from pre-training. To study this, we present the Novel Visual References Dataset (NVRD): 19,176 images spanning 90 visual concepts across different levels of visual novelty, each with up to 20 increasingly perturbed versions of the original object to probe generalization. Unlike prior work on visual augmentations of familiar concepts, NVRD comprises entirely novel, open-ended stimuli constructed from scratch, mirroring how humans encounter genuinely new concepts. We evaluate 3 open- and 2 closed-source models alongside 2,400 human judgments for direct human-model comparison, and find that (i) models struggle to acquire novel concepts in-context when they contradict prior knowledge, and (ii) while models and humans show correlated sensitivity to visual perturbations, models significantly overgeneralize, extending learned labels to stimuli that humans reject. We contribute NVRD as a corpus and benchmark for research on visual concept learning in both humans and machines.
Ada Defne Tür, Gaurav Kamath, Joyce Chai +2
McGill University · Mila Quebec AI Institute · University of Michigan - Ann Arbor +1