cs.LGOct 30, 2026

Latent-Lens: Visual Perception in Small Language Models Through Latent Communication

Authors: Mo Shahdloo, Faegheh Sardari, Mathew Salvaris

Organizations: Microsoft Applied Sciences Group (ASG)

Abstract

Bridging independent vision-language models (VLMs) and small language models (SLMs) in a modular multi-agent system typically forces them to exchange information as discrete text, which creates a lossy bottleneck for spatial and fine-grained details. This constraint is especially acute for SLMs, whose limited capacity makes them more reliant on the quality of the input signal. Recent latent-communication methods exchange states between language models. How a VLM can pass visual information to a separately pretrained text-only SLM though remains unexplored. We propose Latent-Lens, a lightweight codec that bridges the representation gap between a frozen VLM and a frozen SLM directly in latent space, enabling them to communicate purely through hidden states. The codec consists of a domain encoder that projects hidden states from the VLM into the SLM's embedding space, and an attention decoder formalised as a LoRA that teaches the SLM to attend to these visual prefixes, while training only ∼1.6{\sim}1.6M parameters (0.2% of the combined model size). On visual question answering (CLEVR, GQA), Latent-Lens achieves 81.0±7.981.0\pm7.9% and 65.4±5.465.4\pm5.4%, compared with 58.3±5.758.3\pm5.7% and 61.8±1.061.8\pm1.0% for text communication with the same LoRA budget (+30.2--69.8pp relative to frozen text communication), while on image captioning (Flickr8k) it improves over the same baseline by +16.5 (+18.8 frozen). We also evaluate Latent-Lens with TinyLlama-1.1B and InternVL2-1B, which use different architectures, vocabularies, and hidden dimensions.

Figures & tables

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Free the Language Model From the Vision Encoder: Semantic Serialization as a Perception Interface for Small Language Models

    Aug 29, 2026Cong Xu, Ravi SankarVision-Language ModelsEmbodied QA

  2. DeepLatent: Think with Images via Parallel Latent Visual Reasoning

    May 30, 2026Dongchen Lu, Zhimo Li, Mao Shu +1Latent Visual ReasoningVLM Reasoning