Latent-Lens: Visual Perception in Small Language Models Through Latent Communication
Organizations: Microsoft Applied Sciences Group (ASG)
Abstract
Bridging independent vision-language models (VLMs) and small language models (SLMs) in a modular multi-agent system typically forces them to exchange information as discrete text, which creates a lossy bottleneck for spatial and fine-grained details. This constraint is especially acute for SLMs, whose limited capacity makes them more reliant on the quality of the input signal. Recent latent-communication methods exchange states between language models. How a VLM can pass visual information to a separately pretrained text-only SLM though remains unexplored. We propose Latent-Lens, a lightweight codec that bridges the representation gap between a frozen VLM and a frozen SLM directly in latent space, enabling them to communicate purely through hidden states. The codec consists of a domain encoder that projects hidden states from the VLM into the SLM's embedding space, and an attention decoder formalised as a LoRA that teaches the SLM to attend to these visual prefixes, while training only M parameters (0.2% of the combined model size). On visual question answering (CLEVR, GQA), Latent-Lens achieves % and %, compared with % and % for text communication with the same LoRA budget (+30.2--69.8pp relative to frozen text communication), while on image captioning (Flickr8k) it improves over the same baseline by +16.5 (+18.8 frozen). We also evaluate Latent-Lens with TinyLlama-1.1B and InternVL2-1B, which use different architectures, vocabularies, and hidden dimensions.
Figures & tables
| Condition | Source | Params | CLEVR | GQA | Flickr8k |
|---|---|---|---|---|---|
| (%) | (%) | (BLEU-4) | |||
| VLM only (frozen) | SmolVLM-256M ( HuggingFace, 2024 ) | 0 | 58.6 | 41.8 | 13.9 |
| SLM only (frozen) | Qwen2.5-0.5B ( Alibaba Cloud, 2024 ) | 0 | 1.5 | 18.0 | 11.8 |
| Text communication (frozen) | this work | 0 | 11.2 | 35.2 | 0.1 |
| SLM only + LoRA | Hu et al. (2022) | 0.3M | |||
| Text communication + LoRA | this work | 0.3M |
| VLM | SLM | VLM-only | SLM+LoRA | Latent-Lens |
|---|---|---|---|---|
| SmolVLM-256M | Qwen2.5-0.5B | 38.1 | 33.2 | 60.1 |
| SmolVLM-256M | TinyLlama-1.1B | 38.1 | 38.2 | 54.0 |
| InternVL2-1B | Qwen2.5-0.5B | 51.5 | 33.2 | 57.0 |
| InternVL2-1B | TinyLlama-1.1B | 51.5 | 38.2 | 63.4 |
| Representation | CLEVR (%) | GQA (%) |
|---|---|---|
| Band 1 (early, L0-9) | 59.6 | 43.1 |
| Band 2 (mid, L10-19) | 67.9 | 53.2 |
| Band 3 (late, L20-29) | 70.1 | 57.8 |
| Last layer only | 68.6 | 56.5 |
| Condition | Total Params | GPU Memory | Latency |
|---|---|---|---|
| VLM only | 256M | 1,008 MB | 277 1 ms |
| SLM only (+ LoRA) | 495M | 1,013 MB | 620 36 ms |
| Text communication | 752M | 2,113 MB | 2,137 9 ms |
| Latent-Lens | 752M | 2,477 MB | 781 11 ms |
| Representation | CLEVR (%) | GQA (%) |
|---|---|---|
| Final layer | ||
| Three depth bands |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Layers | Hidden dim | Params | Band ranges ( ) |
|---|---|---|---|---|
| SmolVLM-256M (VLM) | 30 | 576 | 256M | L0-9 / L10-19 / L20-29 |
| InternVL2-1B (VLM) | 24 | 896 | 900M | L0-7 / L8-15 / L16-23 |
| Qwen2.5-0.5B (SLM) | 24 | 896 | 500M | - |
| TinyLlama-1.1B (SLM) | 22 | 2048 | 1.1B | - |
| Component | Params | Description |
|---|---|---|
| Domain encoder | 1,300K | 2-layer MLP: (GELU) |
| Attention decoder (LoRA) | 300K | Rank 4, q/v projections, all 24 layers |
| Total | 1,600K | 0.2% of frozen parameters |
| Question | Gold | VLM resp. | SLM prompt (text communication) |
|---|---|---|---|
| How many objects are either yellow shiny spheres or blue metallic cubes? | 2 | “1.” | A vision model responded: “1.” Based on this, answer: How many objects… |
| How many tiny things are metal cylinders or yellow matte spheres? | 2 | “Metal.” | A vision model responded: “Metal.” Based on this, answer: How many tiny… |
| How many items are there in the described scene? | 8 | “7.” | A vision model responded: “7.” Based on this, answer: How many items… |
| What number of other objects are the same size as the purple rubber cylinder? | 7 | “4.” | A vision model responded: “4.” Based on this, answer: What number of… |
| Question | Gold | VLM resp. | SLM prompt (text communication) |
|---|---|---|---|
| Does the striped shirt look long sleeved and white? | no | “No.” | A vision model responded: “No.” Based on this, answer: Does the striped… |
| Does the table in front of the other table look black and round? | yes | “No.” | A vision model responded: “No.” Based on this, answer: Does the table… |
| Is the toilet paper to the right of the chair resting on a lamp? | no | “Yes.” | A vision model responded: “Yes.” Based on this, answer: Is the toilet paper… |
| Which kind of animal sits atop the chair? | cat | “Cat” | A vision model responded: “Cat” Based on this, answer: Which kind of animal… |
| Gold caption (1 of 5 refs) | VLM response (text-communication channel) |
|---|---|
| The dogs are in the snow in front of a fence. | “In the center of the image, we can see two dogs are fighting. In the background, we can see the snow, fencing, trees, and the sky.” |
| A brown and white dog swimming towards some in the pool. | “In the image, a person is playing with a dog in a pool. The person is wearing athletic shorts and a jersey with the number ‘3’ on it. The dog is brown and white, and it is holding a leash in its…” |
| A man is wearing a Sooners red football shirt and helmet. | “In the image, we can see a person wearing a red sports jersey with the number 28 on it. The person is standing and the background is blurred.” |
| A group of tourists stand around as a lady puts her hand near the mouth of a statue. | “The image depicts a group of five individuals, all of whom are standing or seated, engaged in a dynamic moment. The setting appears to be outdoors, possibly a park or a public space, given the presence of a fence and a river in the background…” |
| CLEVR — Q: “How many items are there in the described scene?” Gold: 8 | |
|---|---|
| VLM only | “7.” |
| SLM only, frozen | “Count…The number of items is not specified in the description…” |
| SLM only + LoRA | “5” |
| Text comm., frozen | “Two. The response indicates that the image was interpreted as two objects…” |
| Text comm. + LoRA | “ 8 ” |
Small Language Models
Small language models are compact neural language systems designed to deliver useful text and reasoning capabilities with limited compute, memory, and energy.
See all 296 papers