Training-free personalization adapts a frozen vision-language model (VLM) to recognize user-defined concepts from only a few exemplars by retrieving concept entries at inference time. While recent retriever-based methods enrich each concept with descriptive evidence, they largely treat concepts in isolation, so the retrieved information can be plausible yet non-discriminative, especially when the personalized concept set contains highly confusable, fine-grained categories. We argue that personalization is inherently relational: correct recognition depends on the specific distinctions between a concept and its nearest alternatives. To this end, we propose ReCoG, a Relational Concept Graph for retrieval-augmented personalization. ReCoG represents concepts as nodes and augments them with directed edges that explicitly capture discriminative cues between concept pairs, elicited by a frozen VLM during database construction. At inference, ReCoG retrieves a shortlist and resolves ambiguity via explicit pairwise comparisons conditioned on the corresponding relational edges, selecting the concept that consistently wins against competing candidates. We also introduce FINGER benchmark to test personalized recognition on confusable concept sets. Across FINGER, ReCoG substantially outperforms prior methods, with the largest gains in the most confusable regimes. ReCoG also achieves consistently best performance on established personalization benchmarks including captioning and VQA, confirming that relational evidence is broadly beneficial beyond fine-grained settings. Project Page: https://csy-98.github.io/ReCoG/
Figures & tables
Figure 1: Illustration of why relations matter. Left: When VLM describes each concept independently, it produces salient yet generic attributes shared across confusable concepts, leading to ambiguity. Right: When VLM is conditioned on the inter-concept relation ( e.g. , asking what differs between candidates), it can elicit discriminative cues that ease recognition even under fine-grained setting.
Figure 2: Given a set of novel concepts, we use a frozen VLM with a discriminative prompt Pdisc to elicit directed contrastive evidence between concept pairs. For each ordered pair (ci,cj) , the VLM summarizes what makes ci different from cj , producing an edge eij (and vice versa). Collecting these directed edges yields the Relational Concept Graph , which provides explicit inter-concept relations.
Figure 3: The proposed FINGER Benchmark. Left : 20 concepts in CUB-Easy (top) and CUB-Hard (bottom). We also provide the pairwise concept similarity matrices computed from CUB’s visual attribute vectors (darker means higher similarity, i.e. , more difficult). Right : 20 concepts in CLEVR, where each concept corresponds to a specific scene configuration.
Methods
CUB-Easy
CUB-Hard
CLEVR-Default
P.Acc
N.Acc
Wtd.
Recall
P.Acc
N.Acc
Wtd.
Recall
P.Acc
N.Acc
Wtd.
Recall
R2P [ 8 ]
87.9
99.4
93.6
76.1
63.8
98.1
81.0
55.0
61.1
98.0
79.6
24.4
Baseline
50.0
97.3
73.7
65.0
41.8
96.9
69.4
47.9
51.7
97.5
74.6
14.8
ReCoG ( Ours )
96.3
99.8
98.1
80.8
86.3
99.3
92.8
62.1
87.2
99.3
93.3
41.7
Table 1: Recognition and captioning performance on the proposed FINGER benchmark. Baseline denotes the setting using the relation-agnostic description (the same as R2P [ 8 ] ) and performs retrieval in a pairwise manner in Sec. 4.3 . ReCoG uses the relation-aware concept graph and pairwise retrieval.
Figure 5
Methods
MyVLM [ 1 ]
Yo’LLaVA [ 19 ]
PerVA [ 8 ]
P.Acc
N.Acc
Wtd.
Recall
P.Acc
N.Acc
Wtd.
Recall
P.Acc
N.Acc
Wtd.
Recall
MyVLM
96.6
90.9
93.8
0.02
-
-
-
-
66.0
58.5
62.2
-
Yo’LLaVA
97.0
95.7
96.4
0.1
94.9
89.8
92.4
0.2
75.1
69.0
72.0
-
RAP
94.4
98.8
96.6
88.0
86.0
99.2
92.2
79.7
92.9
85.2
89.0
64.1
R2P
96.3
98.5
97.4
91.4
91.1
97.7
94.4
87.1
90.2
92.8
91.8
72.5
ReCoG
98.6
99.9
99.3
92.9
95.0
99.9
97.4
90.4
97.7
99.9
98.8
73.7
Table 3: Recognition/captioning performance on [ 7 , 19 , 8 ] and VQA performance on Yo’LLaVA
Figure 5: Effect of sparse graph construction on CUB-Hard. Left: P.Acc. of ReCoG when relational edges are sparsely constructed. Right: the number of VLM calls required for sparse database construction after removing duplicated concept pairs. Red dashed lines are those of R2P [ 8 ] .
Candidate-wise scoring
Pairwise scoring
Concept-wise descriptive evidence
Method 1 (R2P [ 8 ] ): 63.8%
Method 2: 41.8%
Relational evidence
Method 3: 74.0%
Method 4 (ReCoG): 86.3%
Table 4: Ablation study on different evidence types and scoring procedures on CUB-Hard.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
From Scene
To Scene
Object idx
Attribute
From value
To value
scene00
scene01
2
color
blue
green
scene01
scene02
3
material
metal
rubber
scene02
scene03
3
shape
cylinder
cube
scene02
scene03
4
color
gray
yellow
scene03
scene04
0
color
red
brown
scene04
scene05
2
shape
cube
sphere
Appendix
Table A5: Scene-to-scene attribute change in CLEVR-Default. Each row indicates an actual edit applied when moving from scene t to scene t+1 .
Figure 10
K
Methods
P.Acc / N.Acc
4
R2P
27.8 / 96.2
Baseline
18.0 / 95.7
ReCoG
33.2 / 96.5
Appendix
Table A6: Performance comparison for experiments on changing positional arrangement. Values are reported in Positive Accuracy/Negative Accuracy (%).
Methods
MiniCPM-o-2.6
LLaVA-1.5
P.Acc
N.Acc
Wtd.
P.Acc
N.Acc
Wtd.
Baseline
41.8
96.9
69.4
36.1
96.6
66.4
ReCoG(Ours)
86.3
99.3
92.8
89.4
99.5
94.5
Appendix
Table A7: Comparative results on CUB-Hard: MiniCPM-o-2.6 vs. LLaVA-1.5. Bold indicates the best performance within each model group.
Concept set
R2P P.Acc.
ReCoG P.Acc.
Black
63.8%
86.3%
White
64.5%
90.8%
Brown
69.0%
95.3%
Grey
63.2%
92.1%
Mean ± std.
65.1 ± 2.3%
91.1 ± 3.2%
Appendix
Table A8: Performance comparison across different concept sets based on coloration.
Figure A8: Prompt template for relation-aware database construction.
Figure A9: Prompt template for comparison-based retrieval.
Dataset
R2P P.Acc.
ReCoG P.Acc.
Δ
CelebA
76.1%
94.2%
+18.1
Appendix
Table A9: Personalized recognition performance on CelebA.
Seed
R2P (P.Acc.)
ReCoG (P.Acc.)
23
59.7%
84.2%
42
65.8%
83.7%
123
61.8%
83.2%
2024
72.4%
90.8%
2025
69.2%
87.6%
Mean ± std.
65.8 ± 5.2%
85.9 ± 3.2%
Appendix
Table A10: Sensitivity to support image selection under different random seeds.
Figure A10: Qualitative example comparing R2P-style retrieval (top) and ReCoG (bottom). The query belongs to kie . While the relation-agnostic method is misled by prominent shared attributes and incorrectly favors nba , ReCoG resolves the ambiguity through explicit pairwise comparison with relation-aware evidence and correctly predicts kie .
Figure A11: Qualitative examples comparing R2P-style captioning and ReCoG.
Retrieval-augmented classification uses relevant examples, prototypes, or knowledge at inference time to ground predictions and handle sparse or changing labels.