Are In-Context Images Worth 10 Dimensions?

Adhemar de Senneville; Xavier Bou; Jérémy Anger; Rafael Grompone; Gabriele Facciolo

cs.CVSep 29, 2026

Are In-Context Images Worth 10 Dimensions?

Authors: Adhemar de Senneville, Xavier Bou, Jérémy Anger, Rafael Grompone, Gabriele Facciolo

Organizations: Université Paris-Saclay, CNRS, ENS Paris-Saclay, Centre Borelli, Paris, France · Pôle recherche de l’AMIAD, Palaiseau, France · Institut Universitaire de France, Paris, France

Abstract

There has been significant work on understanding the In-Context Learning capabilities of Large Language Models, especially on the induction circuit. For a few-shot classification task, the induction circuit leverages linear representations of each labeled example in-context in order to classify an unlabeled query. However, few works focus on how those linear representations are built in the first place. Leveraging the expressivity of the vision modality compared to text, we uncover a Shared Discriminative Geometry (SDG) inside Large Vision Language Models (LVLMs). It is a low-dimensional space, shared across all image classification tasks, in which in-context images are compressed into linearly separable representations later used to perform classification. We observe that this is the result of the model performing a dimensionality reduction of vision representations in early layers. In order to explain this phenomenon: (1) We show analytically that linear self-attention can perform a dimensionality reduction by projecting in-context data onto its principal components, with each layer implementing one gradient descent step toward this objective. (2) We provide evidence that trained LVLMs reduce the dimensionality of vision representations in early layers via a similar mechanism.

Figures & tables

Panel of Figure 1 — Figure 1: Left (our contribution): The first layers of the LVLM reduce the dimension of image tokens in-context using sample-to-sample comparisons. These representations are transferred to text and later used for in-context learning. Right: Nearest Centroid Classifier (NCC) accuracy across layers for forerunner tokens and vision tokens after and before projecting onto the Text SDG and Vision SDG , respectively.

Figure 2: Training setup. At each layer, we meta-learn the Text SDG projection $\Phi_{\ell}$ and Vision SDG projection $\Phi_{\ell}^{\mathrm{v}}$ so that projected embeddings from the same class cluster around a common direction.

Figure 3: Left: Effect of SDG intervention as a function of the number of top dimensions

D

ablated. Right: Effect of interventions on the top 15 SDG dimensions across shot counts.

Panel of Figure 4 — Figure 4: Left : Effective Dimension across layers of the $\Phi_{\ell}$ (Text SDG) and $\Phi^{\mathrm{v}}_{\ell}$ (Vision SDG). Middle : NCC Accuracy in the top- $k$ SDG dimensions of the input vision tokens, the vision tokens at layer $\ell^{*}\!-\!1$ and forerunner token at $\ell^{*}$ . Right : Text SDG with and without random label assignment.

Ablation effect (3-shot)	$\Delta\%$
(1) 128D- Text SDG at $\ell^{*}\!-\!1$	$-3.2$
(2) 128D- Vision SDG at $\ell^{*}\!-\!1$	$-6.4$
(3) Attention from vision at	$-5.2$
$\ell^{}\!-\!1$ to text at $\ell^{}$
128D- Text SDG at $\ell^{*}$	$-11.4$

Table 1: Ablations from

\ell^{*}\!-\!1

\ell^{*}

Figure 5: Left: Number of Dimensionality Reduction Heads (DR Heads) across layers, as well as the average attention to other images in-context. Middle: NCC Accuracy in the top-

k

SDG dimensions after ablating DR Heads attention. Right: Forerunner NCC accuracy on SST-2.

	Classification			Matching
	Open MI	VLGuard	VizWiz	Matching MI	SugarCrepe	MHaluBench	NaturalBench
Baseline	84.0	63.0	72.0	81.0	57.5	75.0	61.0
128D-SDG Ablation	-11.0	-17.0	-4.0	+6.5	+5.5	+3.0	-2.0
128D-SDG Boost	+11.5	+6.0	+1.5	-4.0	-3.0	-5.5	-1.0

Table 2: Impact of the Text SDG ablation and boost on multimodal benchmarks in the 2-shot setting.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Model performance versus the total number of support examples (0, 2, 4, and 6, corresponding to 0, 1, 2, and 3 shots per class) for two label spaces in the 2-way setting.

	$\{\texttt{Foo},\texttt{Bar}\}$	$\{\texttt{Rud},\texttt{Baz}\}$
Ministral 3 (8B)	48.9	0.0
Qwen 3.5 (9B)	23.3	0.0

Appendix

Table 3: Probability (%) assigned to the other class label at the second in-context example, given the first example’s label (2-way, 1-shot setting).

Figure 7: Histograms of effective dimension for the query (

W_{Q}

), key (

W_{K}

), value (

W_{V}

), and output (

W_{O}

) projection matrices of the model.

Training setup	Support counts used for the loss	Effective dimension of $\Phi_{\ell^{*}\!-\!1}^{\mathrm{v}}$
3-way 3-shot	3, 6, 9	9.69
3-way 3-shot	9	9.68
4-way 4-shot	4, 8, 12, 16	11.08
4-way 4-shot	16	10.79

Appendix

Table 4: Effective dimension of the Vision SDG at

\ell^{*}\!-\!1

under different training setups. Support counts indicate when the query classification loss is computed and exclude the query image.

Figure 8: Ordered eigenvalues of

\Phi_{\ell^{*}}^{\top}\Phi_{\ell^{*}}

(Text SDG, blue) and

(\Phi_{0}^{\mathrm{v}})^{\top}\Phi_{0}^{\mathrm{v}}

(input Vision SDG, green). The Text SDG spectrum drops sharply around rank 10, with approximately 90% of its eigenvalue mass in the top 10 directions; the input Vision SDG decays more gradually.

Figure 9: NCC accuracy of average-pooled vision tokens across layers. NCC Accuracy is computed in 3 different Vision SDG on 3 different datasets. Left : On all 10 datasets combined. Middle : On the Textures dataset. Right : On the Aircraft dataset.

Vision SDG	Effective dimension
training data	at $\ell^{}\!-\!1$ ( $\Phi^{{\mathrm{v}}}_{\ell^{}\!-\!1}$ )
All 10 datasets	9.69
Textures only	9.71
Aircraft only	6.57

Appendix

Table 5: Effective dimension of the Vision SDG at layer

\ell^{*}\!-\!1

when trained on all 10 datasets or on a single dataset.

Figure 10: Layer-wise subspace alignment of the Textures and Aircraft Vision SDGs with the Vision SDG trained on all datasets.

Panel of Figure 11 — Figure 11: Numerical evaluation of Relative Gram Error under different initialization and attention settings.

	Intervention	1-shot	2-shot	3-shot	4-shot	5-shot
LVLM Accuracy		34.4%	77.2%	82.4%	84.2%	85.4%
All	1D to 128D - Control Ablation	-0.4	-1.2	-1.9	-2.5	-2.2
	1D to 15D - SDG Ablation	-7.1	-10.8	-12.9	-12.0	-10.6
	1D to 128D - SDG Ablation	-13.3	-11.3	-11.4	-11.7	-9.7
	16D to 128D - SDG Ablation	-5.7	+1.8	+1.6	+1.4	+0.5
	1D to 15D - SDG Boost	+7.5	+4.7	+3.9	+3.0	+2.9

Appendix

Table 6: Effect of top SDG interventions on model classification performance. Baseline accuracy is reported in percent; intervention effects are changes in percentage points.

Panel of Figure 12 — Figure 12: Left: Number of induction heads across layers. Middle: Head accuracy across layers, with head count per accuracy range. Right: Same as the middle after ablating the SDG at $\ell^{*}$ .

Panel of Figure 14 — Figure 14: Left: Average amount of attention of each head across layers. Attention from the forerunner token to the query image. Right: NCC accuracy at layer 15 per token position after projecting onto the Vision SDG. Columns 0-9 are image patches; the rightmost column contains newline tokens. Blue boxes mark the three patch-token sinks.

	Head 7	Head 8	Head 17
Flowers
Textures
Cars

Appendix

Figure 15: Attention from the Forerunner token to image tokens at layer

\ell^{*}

. Columns correspond to attention heads 7, 8, and 17; rows correspond to the Flowers, Textures and Cars datasets.

Panel of Figure 16 — Figure 16: SDG alignment across layers, measured using $1-\operatorname{BDE}$ . Left: $1-\operatorname{BDE}\bigl((\Phi_{i}^{\mathrm{v}})^{\top}\Phi_{i}^{\mathrm{v}},\,(\Phi_{j}^{\mathrm{v}})^{\top}\Phi_{j}^{\mathrm{v}}\bigr)$ . Middle: $1-\operatorname{BDE}\bigl(\Phi_{i}^{\top}\Phi_{i},\,\Phi_{j}^{\top}\Phi_{j}\bigr)$ . Right: $1-\operatorname{BDE}\bigl(\Phi_{i}^{\top}\Phi_{i},\,(\Phi_{j}^{\mathrm{v}})^{\top}\Phi_{j}^{\mathrm{v}}\bigr)$ . Rows correspond to layer $i$ and columns to layer $j$ .

Figure 17: Visualization of attention from sink token 2 to other in-context images for head 27 at layer 12 (dimensionality reduction head), on a task from the Flowers dataset.

Figure 18: Visualization of attention from sink token 2 to other in-context images for head 27 at layer 12 (dimensionality reduction head), on a task from the Cars dataset.

Figure 19: Visualization of attention from sink token 3 to other in-context images for head 26 at layer 10 (dimensionality reduction head), on a task from the Flowers dataset.

Figure 20: Visualization of attention from sink token 3 to other in-context images for head 26 at layer 10 (dimensionality reduction head), on a task from the Cars dataset.

Model	Attention	Layers	Heads	Total heads	Token dim.	Head dim.	MLP dim.
Ministral 3 (3B)	Full	26	32	832	3072	128	9216
Ministral 3 (8B)	Full	34	32	1088	4096	128	14336
Qwen 3.5 (4B)	Full/linear	32	16/32	128/768	2560	256/128	9216
Qwen 3.5 (9B)	Full/linear	32	16/32	128/768	4096	256/128	12288
Qwen 3 (4B)	Full	36	32	1152	2560	128	9728

Appendix

Table 7: LLM-decoder hyperparameters. The highlighted model is used for the main paper experiments and analyses. Slash-separated values follow the order of attention types. Each Qwen model has 8 full-attention layers and 24 linear-attention layers; head counts refer to query heads for full attention and value heads for linear attention.

Model	$\ell^{*}$	Effective dimension of $\Phi_{\ell^{*}}$
Ministral 3 (3B)	16	10.22
Ministral 3 (8B)	17	10.28
Qwen 3.5 (9B)	19	7.46
Qwen 3.5 (4B)	18	9.18
Qwen 3 (4B)	23	8.92

Appendix

Table 8: Peak NCC layer and Text SDG effective dimension across models.

Panel of Figure 21 — Figure 21: NCC accuracy before and after projection onto the Shared Discriminative Geometry.

Panel of Figure 22 — Figure 22: Effective dimension across layers of $\Phi_{\ell}$ (Text SDG) and $\Phi^{\mathrm{v}}_{\ell}$ (Vision SDG).

Panel of Figure 23 — Figure 23: NCC accuracy in the top- $k$ SDG dimensions of the input vision tokens (Layer $\ell=0$ ) and the forerunner token at $\ell^{*}$ .

Dataset	Question format	Answers
Open MI	This is a	Provided concept names
VLGuard	Original instruction field	harmful / unharmful
VizWiz	Original question field	answerable / unanswerable
Matching MI	Do the two images satisfy the induced relationship?	Yes / No
SugarCrepe	Does this image show ’<caption>’?	Yes / No
MHaluBench	Original claim field	hallucination / non-hallucination

Appendix

Table 9: Question and answer formatting. Matching MI uses two images per example; all other benchmarks use one. Caption-based questions are read directly from the formatted dataset files.

Explore similar work

In-Context Collapse in Vision-Language Models and How to Mitigate it?
Aug 3, 2026Mohammad RostamiVision-Language Foundation Models In-Context Learning
Amazon Generative AI Innovation Center∗, USA.
Enhancing Multimodal In-Context Learning via Inductive-Deductive Reasoning
May 4, 2026Haoyu Wang, Haonan Wang, Yuyan Chen +5Multimodal Reasoning In-Context Learning
Tencent QQ · Fudan University · Cornell University
Beyond Model Size: Probing the Gaps in Visual in-Context Learning by Training a Tiny Model
Jun 9, 2026Sunil Khatri, Steven Landgraf, Markus Ulrich +1Vision Foundation Models Visual In-Context Learning
Karlsruhe Institute of Technology

Aug 3, 2026, cs.CVJ/K move · Enter open · S save

In-Context Collapse in Vision-Language Models and How to Mitigate it?

Mohammad Rostami

Amazon Generative AI Innovation Center∗, USA.

Vision-Language Foundation Models In-Context Learning In-Context Collapse

Open paper AskPDF arXiv

Many-shot in-context learning (ICL) lets vision-language models (VLMs) adapt from image--label demonstrations without weight updates, and is widely assumed to improve as more demonstrations are supplied. We show the opposite: as demonstrations accumulate, a subset of VLMs undergo an \emph{in-context collapse}, a sharp, sometimes catastrophic accuracy drop spanning synthetic classification, natural-image classification, and VQA benchmarks, in some models falling below chance while outputs remain well-formed. Across an open VLM panel (

0.5

B--

11

B) and a frontier model (Claude Sonnet 4.5), the collapse is graded. Two capabilities turn out to be dissociable: robustness to accumulating demonstrations and the ability to learn a novel rule in context, their combinations yield three reproducible regimes. A parameter-matched lesion-and-rescue causally localizes the collapse to the vision-language integration pathway: an adapter on the connector and early/mid layers restores genuine learning (remap accuracy

0.39!\rightarrow!0.91

at 16 shots), while an equal-capacity adapter on the late readout does not. We propose \textsc{CircA}, whose core is a one-time integration vaccine: trained once on one synthetic task, it transfers collapse-resistance to unseen task families (chance

\rightarrow$$0.71

0.60

on CIFAR/Fashion). The layers best for in-context integration are not the layers best for weight-based consolidation, the late readout achieves higher accuracy and less forgetting at fewer parameters. The collapse is an integration failure at the vision--language interface, correctable by a lightweight, transferable intervention.

Abstract

Explore similar work

In-Context Collapse in Vision-Language Models and How to Mitigate it?

Enhancing Multimodal In-Context Learning via Inductive-Deductive Reasoning

Beyond Model Size: Probing the Gaps in Visual in-Context Learning by Training a Tiny Model