Recent work reports that vision--language models (VLMs) struggle to establish and maintain stable reference in repeated reference games. Rather than ask which VLM does best, we ask a more basic question: do you need a large pretrained VLM for this at all? On grounding a single director utterance to one of twelve tangram silhouettes, we compare six off-the-shelf VLMs against a transparent baseline that uses no learned visual representation: classical SIFT keypoint matching and a signal-quality index over retrieved images. On identical trials, the transparent baseline matches the strongest VLM (SigLIP-large) and significantly outperforms the other five, including every CLIP and OpenCLIP variant. The baseline additionally retrieves external images, so this is not a matched-information comparison; what it shows is that a learned visual representation is not the bottleneck for this task: given retrieved images, a shape-appropriate classical similarity suffices. Along the way we find that abstract-grounding ability varies widely across VLMs (15--39% top-1; chance 8.33%, humans ≈77--80%), so the weakness is model-specific rather than intrinsic to contrastive pretraining; on the 1{,}013-shape KiloGram benchmark the pattern generalizes for CLIP, with per-shape difficulty tracking human shape-nameability. The pipeline is a classical, inspectable alternative rather than a learned one. We close by sketching how an explicit, inspectable representation of listener-side pact state could carry this approach into interactive multi-turn reference, which we leave to future work. Code available in supplementary material.
Figures & tables
Backbone
Condition
top-1
top-3
top-5
ViT-B/32
raw
17.76
39.86
55.30
+prompt
12.41
34.01
51.46
+rotation
15.24
35.12
50.66
+prompt+rot.
11.10
30.68
49.45
ViT-L/14
raw
26.24
44.40
59.94
+prompt
20.28
39.15
58.02
Table 1: Prompt, augmentation, and scale controls, round-1 top- k (%, n=991). Neither the domain prompt nor best-of-rotations improves either backbone; both reduce accuracy. Model scale is the only factor that helps: ViT-L/14 raw is the best CLIP configuration and is the number carried into Table 3 . Chance is 8.33%.
Condition
Metric
B/32
L/14
Full (1,013-way)
top-1
1.09
1.68
top-5
3.35
6.04
top-10
5.54
8.99
Matched (12-way)
top-1
20.58
25.27
top-5
60.62
64.45
SND corr. (MRR)
ρ
−0.14
−0.26
Table 2: CLIP on KiloGram (Ji et al. , 2022 ) : 1,013 shapes, 10,173 description–shape trials. Full-ranking chance is 0.10 / 0.49 / 0.99% at top-1/5/10; matched 12-way chance is 8.33%. The matched accuracies track the Hawkins CLIP numbers (17.76% B/32 , 26.24% L/14 ; § 4 ). SND correlation is Spearman ρ between per-shape MRR and Shape Naming Divergence; both p<10−5 .
Figure 1: The transparent scoring pipeline. Each crowd-sourced retrieved image and each tangram stimulus is grey-scaled, resized, and aligned to the tangram with a SIFT homography; the Universal Quality Index is then computed on the aligned pair. The tangram’s score g(oi,Iφ) is the best such value over the retrieved set and its rotational and grayscale-inverted augmentations.
Figure 2: UQI as a graded similarity signal. A target tangram (left) scored against five candidate images, with the UQI value shown beneath each (here ranging from 0.007 to 0.617 ). UQI yields a smooth, shape-sensitive score rather than a binary match, and the pipeline ranks the twelve tangrams by the best such score over the retrieved set.
Method
top-1
top-3
top-5
TRACE (this work)
44.3 [37, 52]
65.2 [58, 73]
82.2 [76, 88]
SigLIP-large
43.0 [36, 50]
61.8 [55, 69]
76.4 [70, 82]
SigLIP-base
33.3 [26, 40]
58.8 [52, 66]
70.3 [64, 77]
CLIP L/14 (OpenAI)
26.7 [20, 33]
41.2 [34, 48]
60.0 [53, 67]
CLIP B/32 (OpenAI)
18.8 [13, 25]
40.6 [33, 48]
58.8 [52, 66]
OpenCLIP B/32 (LAION)
17.6 [12, 23]
39.4 [32, 47]
53.3 [46, 61]
Table 3: Single-utterance top- k accuracy on the data ( n=991 : the first-repetition trials for which the retrieval channel returned images, so that every method is scored on identical trials), in % with bootstrap 95% CIs. Chance is 8.33%; rows ordered by top-1. The transparent retrieval channel is statistically indistinguishable from SigLIP-large at every k (overlapping intervals), and the retrieval and SigLIP-large top-1 intervals separate from every CLIP/OpenCLIP variant. Human top-1 on the corpus is ≈ 77–80% (Hawkins et al. , 2020 ) .
vs Model
acc
diff
95% CI
p
SigLIP-large
43.0%
+1.3
[ −8 , +11]
0.82
SigLIP-base
33.3%
+11.0
[+2, +20]
0.02
CLIP L/14
26.7%
+17.6
[+9, +26]
e−4
CLIP B/32
18.8%
+25.5
[+17, +34]
e−8
OpenCLIP B/32
17.6%
+26.7
[+18, +35]
e−9
OpenCLIP L/14
12.7%
+31.6
[+24, +40]
e−12
Table 4: Paired comparison of the transparent retrieval channel (TRACE) against each vision–language model on the data. For each model: its top-1 accuracy (%), the top-1 accuracy difference from TRACE (TRACE − model, in points), the paired bootstrap 95% confidence interval of that difference (2,000 resamples over trials), and the two-sided exact McNemar p -value on the per-trial hit vectors. Positive differences favor TRACE.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Retrieved images
top-1
top-3
top-5
Real
44.3 [37,52]
65.2 [58,74]
82.2 [77,88]
Shuffled
6.1 [2,10]
23.6 [17,30]
38.8 [31,47]
Random
8.5 [4,13]
25.5 [19,32]
41.2 [33,49]
Appendix
Table 5: Retrieval-source ablation on the data, holding the SIFT+UQI scorer fixed and varying only which images each trial is scored against. Real : the trial’s own utterance-conditioned retrievals. Shuffled : another trial’s retrievals (a random derangement). Random : one fixed random sample reused for every trial. Bootstrap 95% CIs (rounded); chance is 8.33%.
Metric (with SIFT alignment)
Top-1
Universal Quality Index (UQI)
41.66%
Structural Similarity (SSIM)
25.8%
Peak Signal-to-Noise (PSNR)
19.7%
Mean Squared Error
18.2%
Mean Absolute Error
17.5%
ERGAS
16.8%
Appendix
Table 6: Ten classical similarity metrics evaluated with SIFT pre-alignment as a config-selection ablation. UQI leads the runner-up (SSIM) by 15.8 points. These are single-pass point estimates without confidence intervals (the retrieval endpoint is now rate-limited; the Limitations section). Chance is 8.33%.
Figure 3: Effect of retrieval breadth. Top-1 accuracy as a function of the number of images scraped per query. Accuracy peaks at six to seven images and declines as additional, less relevant images dilute the retrieved set; the pipeline uses k=7 .