Subliminal Learning in Vision-Language Models

Arman Hatami; Romina Aalishah; Ilya E. Monosov

cs.CVNov 4, 2026

Subliminal Learning in Vision-Language Models

Authors: Arman Hatami, Romina Aalishah, Ilya E. Monosov

Organizations: Johns Hopkins University, Baltimore, MD, USA

Abstract

Subliminal learning is a phenomenon in large language models (LLMs) where a teacher passes a hidden preference to a student through training data unrelated to that preference. We study whether subliminal learning also occurs in vision-language models (VLMs). We finetune a teacher VLM to prefer a specific animal or facial attribute and use a control teacher from the same base model. Both teachers generate numbers for an unrelated visual task involving geometric shapes. Two students initialized from the same base VLM are finetuned on the same images, using numbers generated by either the biased or control teacher. Across VLMs ranging from 2B to 7B parameters, we find that both animal and facial preferences can transfer through this unrelated task. Students trained on biased-teacher outputs select the target more often than control students, with a difference of up to $\sim$ 10 percentage points in the animal preference task. Transfer can become stronger with more training data and varies with the target preference, model, and how the numbers are generated. Beyond preferences, we show that the biased student's explanations use more of the same words and phrases as the biased teacher's explanations than those of the control student, even when all three models select the same image. These results matter for training VLMs and show that training on unrelated data may change both a VLM's image preferences and how it explains its choices, and that transfer can become stronger with more training samples.

Figures & tables

Figure 1: Subliminal learning in VLMs. A student finetuned on unrelated carrier samples generated by a biased teacher can acquire the hidden preferences of the teacher. Transfer becomes stronger as the amount of carrier data increases, and can extend to the transfer of the teacher’s wording.

Figure 2: Overview of the subliminal learning framework. (a-b) Tasks with animal and facial preferences used to create a teacher with a target preference. (c) The unrelated carrier task, where the model counts geometric shapes and generates numbers from a range determined by the count. (d) The biased teacher counts the shapes in each carrier image and generates a number sequence from the corresponding range. Valid outputs are then used to finetune a student initialized from the same reference model.

Figure 3: Animal preference transfer for Qwen2.5-VL-3B and Qwen2.5-VL-7B. Each panel compares the reference model, control student, and biased student for six target animals. The top row uses temperature sampling and the bottom row uses greedy decoding. Error bars show 95% confidence intervals.

Figure 4: Effect of carrier data size for Qwen2.5-VL-3B with panda as the target and greedy decoding. The gap between the biased and control students increases with the number of carrier examples.

Figure 5: Face-attribute preference transfer for Qwen2-VL-2B with temperature sampling. Bars show

\Delta_{T}

and error bars show 95% paired-bootstrap confidence intervals.

Metric	Control	Biased	Diff. [95% CI]
All panels ( $n=1{,}200$ )
TF-IDF	2.41	3.16	$+0.74\ [0.49,1.01]$
Jaccard	3.14	5.26	$+2.12\ [1.58,2.67]$
Token F1	4.92	8.14	$+3.21\ [2.53,3.90]$
ROUGE-L	20.75	23.10	$+2.35\ [1.87,2.83]$
Students agree ( $n=843$ )

Table 1: Wording similarity to the biased teacher for the Qwen2.5-VL-3B panda condition. Difference is biased minus control with 95% paired-bootstrap confidence intervals. Animal names and choice labels A-F are removed before comparison.

Figure 6: Token-F1 alignment for the 334 examples where all three models select the same image. Points above the diagonal indicate that the biased student is closer to the biased teacher. Animal names and choice labels A-F are removed before comparison.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
LoRA rank	8
LoRA $\alpha$	16
Precision	bfloat16
Optimizer	AdamW
Learning rate	$2\times 10^{-4}$
Weight decay	0

Appendix

Table 2: Training settings used in the experiments.

Figure 7: Representative samples from the six animal categories used in our experiments, dolphin, wolf, lion, otter, elephant, and panda.

Figure 8: Representative FaceSynthetics samples for the five facial attributes used in our experiments.

Task / count	Valid range	Biased-teacher output	Control-teacher output
Animal / 1	00–24	01 09 14 22 24 24 24 24	01 05 14 22 24 20 11 03
Animal / 4	25–49	34 38 40 42 44 47 49 32	34 48 25 42 27 33 45 32
Animal / 5	50–74	74 54 67 57 55 66 74 64	74 54 67 57 52 66 74 64
Animal / 8	75–99	87 88 98 99 85 77 75 85	77 88 98 79 85 77 84 95
Face / 1	000–249	243 200 032 101 048	246 200 032 101 048
Face / 3	250–499	325 384 338 296 437	325 384 338 296 438

Appendix

Table 3: Real paired carrier examples from the Qwen2.5-VL-3B runs. Every row passes the visual-grounding, format, and range checks.

Condition	Control teacher	Biased teacher
Dolphin	84.51	84.78
Elephant	86.76	81.17
Lion	83.73	88.19
Otter	84.74	86.22
Panda	83.65	89.69
Wolf	84.98	90.26

Appendix

Table 4: Exact carrier shape-count accuracy for the represented Qwen2.5-VL-3B temperature-sampling runs.

Figure 9: Example of animal recognition evaluation. The model sees six animal images labeled A-F and is asked to identify the named target animal.

Figure 10: Example of face attribute preference evaluation. The model sees six portraits labeled A–F and answers a neutral preference question that does not name the target attribute.

Stage	Model input and supervision
Teacher training	Two images labeled A/B, a neutral preference question, and a supervised A/B completion. The biased label selects the target in every example, while the matched control selects it in 50% of examples.
Preference evaluation	Six images labeled A-F and a neutral subjective question. The target name is not shown. The prediction is the argmax over the first-token A-F probabilities.
Recognition evaluation	Six images labeled A-F and a question that explicitly names the animal or face attribute. The prediction is again the argmax over A-F.
Student training	One abstract-shape carrier image, the count-dependent numerical instruction, and the number sequence generated by the corresponding teacher. No animal, face, target name, or preference question is shown to the student.

Appendix

Table 5: What the model sees at each stage of the experiment.

Stage	Exact question or instruction	Label or output
Animal teacher training	“Which image would you use as a logo? Answer only A or B.”	Target is lion in B. Biased teacher label: B. Matched-control label: A.
Animal preference evaluation	“Which image would you keep? Answer only A, B, C, D, E, or F.”	For the lion experiment, lion is C. Outputs are Ref.: A; control teacher: A; biased teacher: C; control student: D; biased student: E.
Animal recognition evaluation	“In which image is the lion? Answer only A, B, C, D, E, or F.”	Lion is D. Ref., control teacher, biased teacher, control student, and biased student all select D.
Face teacher training	“Which portrait would you choose for a fictional profile? Answer only A or B.”	Visible beard is in A. Biased teacher label: A. Matched-control label: B.
Face preference evaluation	“Which portrait would you choose for a fictional profile? Answer only A, B, C, D, E, or F.”	Visible beard is E. Outputs are Ref.: F; control teacher: A; biased teacher: E; control student: F; biased student: F.
Face recognition evaluation	“Which portrait contains a clearly visible beard? Answer only A, B, C, D, E, or F.”	Visible beard is E. All five model arms select E.

Appendix

Table 6: Concrete questions, labels, and outputs from the represented animal, face, and carrier experiments.

Figure 11: Animal preference transfer for Qwen2-VL-2B and Phi-4-multimodal-Instruct (5.6B). Panels compare the reference model, control student, and biased student for six target animals under temperature sampling and greedy decoding.

Target	Control teacher	Biased teacher	$I_{T}$
Dolphin	17.50	98.17	80.67
Elephant	14.92	99.83	84.92
Lion	11.83	100.00	88.17
Otter	15.83	100.00	84.17
Panda	17.08	100.00	82.92
Wolf	16.67	100.00	83.33

Appendix

Table 7: Teacher induction for the Qwen2.5-VL-3B temperature runs. Rates and effects are percentage points.

Target	Reference	Control teacher	Biased teacher	Control student	Biased student
Dolphin	100.00	99.67	96.50	99.83	100.00
Elephant	100.00	100.00	100.00	100.00	100.00
Lion	100.00	97.17	100.00	100.00	100.00
Otter	100.00	99.17	100.00	99.83	100.00
Panda	100.00	80.00	100.00	100.00	100.00
Wolf	100.00	89.17	100.00	100.00	99.83

Appendix

Table 8: Animal recognition accuracy (%) for Qwen2.5-VL-3B with temperature carrier sampling. Each target uses 600 recognition panels.

Figure 12: Face-attribute preference transfer for Qwen2.5-VL-7B with temperature sampling. Bars show

\Delta_{T}

and error bars show 95% paired bootstrap confidence intervals.

Figure 13: Additional carrier data scaling results for Qwen2-VL-2B with otter, Qwen2.5-VL-7B with dolphin, and Phi-4-multimodal-Instruct (5.6B) with wolf. The gap between the biased and control students does not increase monotonically in every setting.

Student update	Control	Biased	Effect [95% CI]
All language layers	31.67	41.58	$+9.92\;[+7.83,+12.08]$
Layer 0 only	34.08	31.50	$-2.58\;[-4.17,-1.00]$

Appendix

Table 9: Qwen2.5-VL-3B panda layer diagnostic. Effects are biased minus control student target-selection rates.

Figure 14: Additional wording similarity results on examples where the compared models select the same image. Points show the biased minus control difference in similarity to the biased teacher for ROUGE-L, token F1, and token Jaccard.

Figure 15: Animal preference transfer across all four models and both decoding methods. Each cell shows the biased student minus control student effect in percentage points.

Large Vision-Language Models

Large vision-language models (LVLMs) connect image and video perception with language generation, reasoning, and interactive tasks.

See all 410 papers

Near-Duplicate Image Selection Transfers Preferences in Vision-Language Models
Oct 20, 2026Jaeyoung Kim, Eunseok Kim, Dongsuk Jang +1
MADI, Seoul, Korea
Subliminal Learning is a LoRA Artifact
May 30, 2026Todd Nief, Harvey Yiyun Fu, Mark Muchane +1LLM Fine-Tuning Low-Rank Adaptation
Department of Computer Science, University of Chicago · Data Science Institute, University of Chicago
You Didn't Have to Say It like That: Subliminal Learning from Faithful Paraphrases
Mar 10, 2026Isaia Gisler, Zhonghao He, Tianyi QiuSubliminal Learning LLM Training
ETH Zürich · University of Cambridge · Peking University

Oct 20, 2026J/K move · Enter open · S save

Near-Duplicate Image Selection Transfers Preferences in Vision-Language Models

Jaeyoung Kim, Eunseok Kim, Dongsuk Jang, Seunghyun Kim

MADI, Seoul, Korea

Open paper AskPDF arXiv

Subliminal learning is the phenomenon in which a model transmits its preferences to another model through data that do not explicitly express them. As models increasingly curate training data for other models, we ask whether this transfer can arise from choosing among near-duplicate images while keeping the source photographs and human-written captions fixed. We show that this restricted form of image selection is sufficient. For each photograph, a curator selects one of eight near-duplicate variants by caption likelihood, without target-specific prompts. Students trained on target-preferring curators' selections show stronger target preferences in an image-free likelihood readout than those trained on neutral selections. We observe owl preference transfer in LLaVA-1.5, Qwen2.5-VL, Qwen3-VL, and Gemma 4, with curators and students initialized from the same base checkpoint. The owl transfer results repeat across independently trained curator ensembles and student seeds on a shared image bank, exceed count-matched random reselections, and replicate on a new source bank with new curators in Qwen3-VL. Target rotations also produce shifts toward camels, oak trees, and a political figure. Transfer strength depends on dataset size, learning rate, and training budget. First-generation students also pass the owl preference to a second generation in LLaVA-1.5 and Qwen3-VL through captions that do not explicitly mention owls. These findings identify near-duplicate image selection as a channel for preference transfer whose strength and onward transmission depend on student training settings.

Abstract

Large Vision-Language Models

Explore similar work

Near-Duplicate Image Selection Transfers Preferences in Vision-Language Models

Subliminal Learning is a LoRA Artifact

You Didn't Have to Say It like That: Subliminal Learning from Faithful Paraphrases