Seeing Before Naming: Context-Grounded Learning of Novel Visual Concepts

Gabriel Adineera; Jinoh Kim

cs.CVOct 26, 2026

Seeing Before Naming: Context-Grounded Learning of Novel Visual Concepts

Authors: Gabriel Adineera, Jinoh Kim

Organizations: Texas A&M University-Commerce USA

Abstract

Learning an unfamiliar visual category requires more than adapting a classifier: the model must connect its visual evidence to a useful semantic name. A pre-trained vision-language model can retain discriminative image features while providing a weak textual anchor for the target concept. We address this imbalance through context-grounded semantic anchoring. Our framework, SynerNet, learns concept-specific name embeddings, exposes them to compatible linguistic contexts, and conditions their representations on visual evidence. Difficulty-aware processing and adaptive objective weighting coordinate this adaptation while retaining the pre-trained backbone. On four VISTA-Beyond domains, SynerNet reaches a mean 16-shot accuracy of 74.35%, compared with 72.45% for few-shot name learning. Across 14 domains in the name-masked transfer setting, it improves mean accuracy from 45.17% to 46.74%. Ablations identify concept-specific name embeddings and cross-modal context integration as important components. These results support treating concept acquisition as the construction of visually grounded semantic anchors.

Figures & tables

Figure 1: Visual discrimination need not imply semantic alignment. Circles denote visual evidence and diamonds denote name anchors. Fixed anchors can remain separated from their corresponding visual groups; name learning and visual context ground the anchors. This is a conceptual schematic with generated example thumbnails.

Figure 2: SynerNet architecture. Frozen encoders supply image and text features. Name learning and context exchange form concept-specific text representations. Visual context conditions residual fusion to produce semantic anchors for cosine matching. An auxiliary visual classifier and cross-modal alignment provide the joint objectives. Dashed coral arrows denote adaptation feedback. Example thumbnails and the similarity grid are schematic.

Domain	Method	1-shot	2-shot	4-shot	8-shot	16-shot
Insects / spiders	CoOp	19.3	22.9	26.6	29.8	32.5
	CoCoOp	16.4	17.2	17.8	19.9	22.6
	CLIP-Adapter	24.3	26.8	29.5	32.8	36.0
	FSNL	31.4	35.7	39.7	41.9	43.7
	SynerNet	33.8	38.3	41.2	43.5	45.4
Landmark	CoOp	43.8	59.9	80.1	85.8	89.4

Table 1: Few-shot recognition on four VISTA-Beyond domains. Top-1 accuracy (%). Bold indicates the best result within each domain and shot count.

Figure 3: Accuracy and component contributions. (a) Macro-average accuracy across the four few-shot domains, computed from Table 1 . (b) Mean accuracy decrease for each ablation, computed from the five domain scores in Table 2 . The panels summarize different experimental settings and should not be interpreted as sharing the same evaluation population.

Variant	Animals	Architecture	Attire	Food	Pokemon	Mean drop $\downarrow$
Full SynerNet	53.9	89.4	82.0	92.9	79.7	—
w/o visual unit	52.1	86.3	78.3	89.8	76.3	3.02
w/o linguistic unit	52.8	87.3	79.8	90.3	77.8	1.98
w/o name unit	50.8	85.8	77.3	88.3	75.3	4.08
w/o coordinator	51.3	86.8	78.3	89.3	76.8	3.08
w/o context exchange	51.5	86.5	78.6	89.5	77.0	2.96

Table 2: Component ablations. Accuracy (%) in the five-domain ablation setting. The final column is the arithmetic mean decrease from the full model, in percentage points.

Domain	OpenCLIP	TransCLIP	ZSNL	Ours
Animals	19.4	21.7	26.4	27.6
Architecture	22.3	24.4	39.1	41.3
Attire	16.8	18.8	34.5	36.4
FolkArt	26.7	29.5	27.8	29.1
Food	8.9	10.4	29.3	30.7
Insects / spiders	16.7	17.9	24.3	26.1

Table 3: Name-masked transfer across 14 domains. Top-1 accuracy (%). Target adaptation uses 16 images per class; names are masked at inference. Bold marks the best score in each row.

Figure 4: Domain-level transfer gains. SynerNet minus ZSNL, computed from Table 3 . The dashed line is the macro-average gain of 1.57 percentage points. Positive gains occur in 13 of the 14 domains.

Vision-Language Grounding

Vision-Language Grounding links words and phrases to the objects, regions, actions, or moments they describe in images, video, and scenes.

See all 420 papers

Show Me Examples: Inferring Visual Concepts from Image Sets
Jul 2, 2026Nick Stracke, Kolja Bauer, Stefan Andreas Baumann +3Vision-Language Models Conditional Image Generation
Munich Center for Machine Learning · Apple
Metonymic Circuits for Abstract Concept Grounding in Vision Transformers
Oct 3, 2026Jing Ding, Ziqiao Ma, Jiayuan Mao +2VLM Interpretability Vision Transformer
University of Michigan · Vector Institute · University of Pennsylvania +2
Would you still call this Dax? Novel Visual References in VLMs and Humans
Jun 3, 2026Ada Defne Tür, Gaurav Kamath, Joyce Chai +2VLM Evaluation Vision-Language Models
McGill University · Mila Quebec AI Institute · University of Michigan - Ann Arbor +1

Jul 2, 2026J/K move · Enter open · S save

Show Me Examples: Inferring Visual Concepts from Image Sets

Nick Stracke, Kolja Bauer, Stefan Andreas Baumann, Miguel Angel Bautista+2

Munich Center for Machine Learning · Apple

Vision-Language Models Conditional Image Generation In-Context Learning VLM Reasoning Multimodal Understanding Multi-Image Reasoning

Open paper AskPDF arXiv

Vision-language models (VLMs) can follow complex textual instructions, yet they struggle to reason from purely visual context. In particular, current models fail to infer shared concepts from sets of example images and apply them to new inputs. We introduce Visual Concept Inference from Sets (VICIS), a task that evaluates this capability. Given a small context set of images sharing a concept and a query image, the model must generate new images that preserve the context-defined concept while remaining consistent with the query. We show that state-of-the-art VLMs perform poorly on this task, often ignoring the visual context or defaulting to biased generations. To address this gap, we propose a training framework and architecture that learn to infer visual concepts from image sets and extract concept-specific embeddings from queries. Experiments on synthetic data and large-scale ImageNet/WordNet data show that our model generates more accurate and diverse outputs and generalizes to unseen concepts and modalities such as sketches.

Abstract

Vision-Language Grounding

Explore similar work

Show Me Examples: Inferring Visual Concepts from Image Sets

Metonymic Circuits for Abstract Concept Grounding in Vision Transformers

Would you still call this Dax? Novel Visual References in VLMs and Humans