Authors: Frieda Born, Tom Neuhäuser, Lukas Muttenthaler, Brett D. Roads, Bernhard Spitzer, Andrew K. Lampinen, Matt Jones, Klaus-Robert Müller, Michael C. Mozer
Organizations: Technische Universität Berlin · BIFOLD · Max Planck Institute for Human Development · Aignostics · Helmholtz Munich · Technical University of Munich · University College London · Technische Universität Dresden · Google DeepMind · University of Colorado Boulder · Korea University · Max Planck Institute for Informatics
Modern machine learning models typically represent inputs as fixed points in a high-dimensional embedding space. While this approach has been proven powerful for a wide range of downstream tasks, it fundamentally differs from the way humans process information. Because humans are constantly adapting to their environment, they represent objects and their relationships in a highly context-sensitive manner. To address this gap, we propose a method for context-sensitive similarity computation from neural network embeddings, applied to modeling a triplet odd-one-out task with an anchor image serving as simultaneous context. Modeling context enables us to achieve up to a 15% improvement in odd-one-out accuracy over a context-insensitive model. We find that this improvement is consistent across both original and "human-aligned" vision foundation models.
Figures & tables
Figure 1: Qualitative comparison of context-insensitive and context-sensitive models. (a) Example triplet, where the human odd-one-out choice was made anchored in a given context. The context-sensitive model matches the human choice, while the context-insensitive model does not, reflecting the ambiguity of the triplet without context. (b) Representational similarity matrices (RSMs) over a reference image set, showing how the context-sensitive model modulates similarities given the context image in a. (c) PCA plot of the reference set and the three triplet images (same symbols as in a). The horse-cart image shifts to the animal cluster for the context-sensitive model.
Figure 2: Method overview. A foundation model (FM) extracts embeddings for the context and triplet images. The triplet embeddings undergo a context-insensitive transform (CIT), and the context embedding is mapped to a matrix that is used to compute pairwise, context-sensitive similarities (CSS) between the transformed triplet embeddings. The trainable parameters consist of those of the context-insensitive transform and the neural network that maps the context embedding to the matrix.
Figure 3: Odd-one-out accuracy on our triplet-with-context task for original and human-aligned DINOv2, SigLIP, and ViT bases. The context-sensitive models consistently outperform both FM baselines and context-insensitive transforms. Error bars denote 95% confidence intervals computed over held-out test trials. See Appx. F on upper bound estimation.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
FM
CI
CS
↑ FM (%)
↑ CI (%)
DINOv2 (original)
0.536
0.590
0.657
22.4
11.2
DINOv2 (human-aligned)
0.577
0.598
0.687
19.2
14.9
SigLIP (original)
0.471
0.582
0.682
44.9
17.4
SigLIP (human-aligned)
0.571
0.595
0.685
20.0
15.0
ViT (original)
0.386
0.460
0.539
39.6
17.1
ViT (human-aligned)
0.579
0.585
0.672
16.1
14.8
Appendix
Table 1: Triplet odd-one-out accuracies for original and human-aligned foundation model bases on the triplet-with-context task. We report accuracies for the foundation model baselines (FM), context-insensitive models (CI), and context-sensitive models (CS). Relative improvements are computed with respect to the foundation model baselines and context-insensitive models.
Visual representation learning develops image and video features that capture useful structure for recognition, retrieval, spatial reasoning, and action.
School of Data Science, Fudan University, Shanghai · Independent Researcher, London · Department of Information Studies, University College London, London +1