Bridging independent vision-language models (VLMs) and small language models (SLMs) in a modular multi-agent system typically forces them to exchange information as discrete text, which creates a lossy bottleneck for spatial and fine-grained details. This constraint is especially acute for SLMs, whose limited capacity makes them more reliant on the quality of the input signal. Recent latent-communication methods exchange states between language models. How a VLM can pass visual information to a separately pretrained text-only SLM though remains unexplored. We propose Latent-Lens, a lightweight codec that bridges the representation gap between a frozen VLM and a frozen SLM directly in latent space, enabling them to communicate purely through hidden states. The codec consists of a domain encoder that projects hidden states from the VLM into the SLM's embedding space, and an attention decoder formalised as a LoRA that teaches the SLM to attend to these visual prefixes, while training only ∼1.6M parameters (0.2% of the combined model size). On visual question answering (CLEVR, GQA), Latent-Lens achieves 81.0±7.9% and 65.4±5.4%, compared with 58.3±5.7% and 61.8±1.0% for text communication with the same LoRA budget (+30.2--69.8pp relative to frozen text communication), while on image captioning (Flickr8k) it improves over the same baseline by +16.5 (+18.8 frozen). We also evaluate Latent-Lens with TinyLlama-1.1B and InternVL2-1B, which use different architectures, vocabularies, and hidden dimensions.
Figures & tables
Figure 1: Latent-Lens architecture. Text communication passes a generated response from the VLM to the SLM. Latent-Lens passes projected VLM hidden states to the SLM. Black borders denote frozen components. The blue border denotes the trained codec.
Condition
Source
Params
CLEVR
GQA
Flickr8k
(%)
(%)
(BLEU-4)
VLM only (frozen)
SmolVLM-256M ( HuggingFace, 2024 )
0
58.6
41.8
13.9
SLM only (frozen)
Qwen2.5-0.5B ( Alibaba Cloud, 2024 )
0
1.5
18.0
11.8
Text communication (frozen)
this work
0
11.2
35.2
0.1
SLM only + LoRA
Hu et al. (2022)
0.3M
42.4±0.2
46.2±1.3
12.7±0.9
Text communication + LoRA
this work
0.3M
58.3±5.7
61.8±1.0
2.4±2.0
Table 1: Main results on CLEVR, GQA (accuracy) and Flickr8k (BLEU-4). Source column lists the paper introducing each baseline. Trained conditions report mean ± sample standard deviation over three runs; frozen conditions are deterministic point estimates.
VLM
SLM
VLM-only
SLM+LoRA
Latent-Lens
SmolVLM-256M
Qwen2.5-0.5B
38.1
33.2
60.1
SmolVLM-256M
TinyLlama-1.1B
38.1
38.2
54.0
InternVL2-1B
Qwen2.5-0.5B
51.5
33.2
57.0
InternVL2-1B
TinyLlama-1.1B
51.5
38.2
63.4
Table 2: Generality across all four VLM × SLM pairings. Score is the mean of CLEVR (%), GQA (%), and Flickr8k (BLEU-4). VLM-only and SLM+LoRA columns are matched-VLM and matched-SLM baselines averaged the same way. These are single-run compatibility results for each pairing. Table 1 reports three-run means for the default SmolVLM/Qwen pair.
Representation
CLEVR (%)
GQA (%)
Band 1 (early, L0-9)
59.6
43.1
Band 2 (mid, L10-19)
67.9
53.2
Band 3 (late, L20-29)
70.1
57.8
Last layer only
68.6
56.5
Table 3: Linear probe accuracy per band on CLEVR and GQA.
Condition
Total Params
GPU Memory
Latency
VLM only
256M
1,008 MB
277 ± 1 ms
SLM only (+ LoRA)
495M
1,013 MB
620 ± 36 ms
Text communication
752M
2,113 MB
2,137 ± 9 ms
Latent-Lens
752M
2,477 MB
781 ± 11 ms
Table 4: Inference latency and GPU memory on a single 24GB GPU (fp16, 20-token generation, averaged over 10 runs).
Representation
CLEVR (%)
GQA (%)
Final layer ×3
71.5±0.9
62.7±1.6
Three depth bands
81.0±7.9
65.4±5.4
Table 5: Comparison of three depth bands with three copies of the final-layer representation. Values are mean ± sample standard deviation over three runs.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Layers
Hidden dim
Params
Band ranges ( B=3 )
SmolVLM-256M (VLM)
30
576
256M
L0-9 / L10-19 / L20-29
InternVL2-1B (VLM)
24
896
900M
L0-7 / L8-15 / L16-23
Qwen2.5-0.5B (SLM)
24
896
500M
-
TinyLlama-1.1B (SLM)
22
2048
1.1B
-
Appendix
Table 6: Model specifications and band assignments.
Component
Params
Description
Domain encoder
1,300K
2-layer MLP: 576→896 (GELU) →896
Attention decoder (LoRA)
300K
Rank 4, q/v projections, all 24 layers
Total
1,600K
0.2% of frozen parameters
Appendix
Table 7: Trainable parameters of Latent-Lens for the Qwen2.5-0.5B pair.
Question
Gold
VLM resp.
SLM prompt (text communication)
How many objects are either yellow shiny spheres or blue metallic cubes?
2
“1.”
A vision model responded: “1.” Based on this, answer: How many objects…
How many tiny things are metal cylinders or yellow matte spheres?
2
“Metal.”
A vision model responded: “Metal.” Based on this, answer: How many tiny…
How many items are there in the described scene?
8
“7.”
A vision model responded: “7.” Based on this, answer: How many items…
What number of other objects are the same size as the purple rubber cylinder?
7
“4.”
A vision model responded: “4.” Based on this, answer: What number of…
Appendix
Table 8: CLEVR examples. Gold answers shown for reference.
Question
Gold
VLM resp.
SLM prompt (text communication)
Does the striped shirt look long sleeved and white?
no
“No.”
A vision model responded: “No.” Based on this, answer: Does the striped…
Does the table in front of the other table look black and round?
yes
“No.”
A vision model responded: “No.” Based on this, answer: Does the table…
Is the toilet paper to the right of the chair resting on a lamp?
no
“Yes.”
A vision model responded: “Yes.” Based on this, answer: Is the toilet paper…
Which kind of animal sits atop the chair?
cat
“Cat”
A vision model responded: “Cat” Based on this, answer: Which kind of animal…
Appendix
Table 9: GQA examples. Gold answers shown for reference.
Gold caption (1 of 5 refs)
VLM response (text-communication channel)
The dogs are in the snow in front of a fence.
“In the center of the image, we can see two dogs are fighting. In the background, we can see the snow, fencing, trees, and the sky.”
A brown and white dog swimming towards some in the pool.
“In the image, a person is playing with a dog in a pool. The person is wearing athletic shorts and a jersey with the number ‘3’ on it. The dog is brown and white, and it is holding a leash in its…”
A man is wearing a Sooners red football shirt and helmet.
“In the image, we can see a person wearing a red sports jersey with the number 28 on it. The person is standing and the background is blurred.”
A group of tourists stand around as a lady puts her hand near the mouth of a statue.
“The image depicts a group of five individuals, all of whom are standing or seated, engaged in a dynamic moment. The setting appears to be outdoors, possibly a park or a public space, given the presence of a fence and a river in the background…”
Appendix
Table 10: Flickr8k examples. Gold is one of the five reference captions per image.
CLEVR — Q: “How many items are there in the described scene?” Gold: 8
VLM only
“7.”
SLM only, frozen
“Count…The number of items is not specified in the description…”
SLM only + LoRA
“5”
Text comm., frozen
“Two. The response indicates that the image was interpreted as two objects…”
Text comm. + LoRA
“ 8 ”
Appendix
Table 11: Qualitative samples comparing baseline outputs on one example per dataset.
Small language models are compact neural language systems designed to deliver useful text and reasoning capabilities with limited compute, memory, and energy.
End-to-end vision-language models (VLMs) bind visual competence to the scale of their language model: as the language model shrinks, perception and reasoning degrade together. We study an embodied scene question-answering (QA) interface in which vision never enters the language model. A frozen perception stack detects and ranges objects; a deterministic semantic serializer compiles the perceived state, errors included, into decision-aligned text; an unmodified text-only large language model (LLM) answers. On a visible-scope-matched, occlusion-audited campus-robot benchmark, under a prospectively frozen criterion, the serialized interface, using detectors fine-tuned in-domain within each fold, outperforms a zero-shot VLM whose language model has the same 7B scale (0.7892 vs 0.7462), with a larger margin at 3B (0.7673 vs 0.6913). Preregistered decoupling experiments show the gain survives paraphrase, attributing it to decision-aligned computation rather than answer-string leakage, while novel judgment vocabularies bound its scope. The advantage grows as the reader shrinks to 1.5B and reverses at 0.5B, and a ground-truth oracle locates the reader-capability floor. Under matched task supervision the interfaces converge: a VLM fine-tuned with low-rank adaptation (LoRA) overtakes the zero-shot system but only ties an equally supervised text reader (0.8441 vs 0.8396, no statistically resolved difference), and both routes remain perception-bound. Reported perception parameters are comparable to those of the VLM's vision tower, and total compute is not smaller.
Cong Xu, Ravi Sankar
iCONS Lab, Department of Electrical and Computer Engineering, University of South Florida, Tampa, FL 33620 USA
Embedding-level scaling expands the capacity of language models through lookup-based memory rather than by increasing model depth, width, or dense computation. Extending this idea to vision-language models (VLMs) is challenging because visual features are continuous, spatially structured, and query-dependent, unlike discrete text tokens that naturally support n-gram lookup. We introduce V-LUMEN, a visual lookup memory for VLMs. V-LUMEN discretizes visual features with a frozen codebook, builds spatially aware memory keys through spatial aggregation-based hashing, and modulates them with text-conditioned relevance signals; all addressing is non-parametric, and the retrieved embeddings are fused into the backbone through a gated residual pathway. On five reasoning-intensive benchmarks, V-LUMEN substantially improves a frozen Qwen3-VL-2B backbone and transfers zero-shot to three benchmarks whose images are disjoint from the training data. At a matched parameter budget, moving part of a LoRA or MoE adapter's capacity into V-LUMEN preserves accuracy on two backbones, indicating that external visual memory is an independent capacity axis rather than a stronger adaptation method. Because lookup cost is constant in table size and incurred only once at prefill, this capacity grows in storage rather than in per-token compute. Code is available at https://github.com/miso-choi/V-Lumen.
The emerging paradigm of "thinking with images" embeds visual states into intermediate reasoning steps, defining a new frontier for Vision-Language Models. Existing approaches diverge along two lines. Tool-assisted methods apply explicit visual operations but suffer from high latency and restricted manipulation types. Latent reasoning methods autoregressively produce implicit visual states, but underperform tool-assisted methods, and their latent tokens fail to capture effective visual information. In this work, we propose DeepLatent, a parallel framework for latent visual reasoning. First, we introduce LatentFormer. It uses learnable 2D tokens to generate context-conditioned latent states in parallel, anchoring every visual update directly in the original image features. Second, we design a continuous-space reinforcement learning algorithm. It optimizes latent modulation parameters directly in the embedding space, significantly improving latent representation quality. The framework is trained via knowledge distillation followed by this continuous-space RL algorithm. Furthermore, we contribute DeepLatent-180K, a large-scale dataset tailored for latent visual reasoning. Extensive evaluations across multiple benchmarks demonstrate that DeepLatent achieves state-of-the-art performance.