Free the Language Model From the Vision Encoder: Semantic Serialization as a Perception Interface for Small Language Models

Cong Xu; Ravi Sankar

cs.ROAug 29, 2026

Free the Language Model From the Vision Encoder: Semantic Serialization as a Perception Interface for Small Language Models

Authors: Cong Xu, Ravi Sankar

Organizations: iCONS Lab, Department of Electrical and Computer Engineering, University of South Florida, Tampa, FL 33620 USA

Abstract

End-to-end vision-language models (VLMs) bind visual competence to the scale of their language model: as the language model shrinks, perception and reasoning degrade together. We study an embodied scene question-answering (QA) interface in which vision never enters the language model. A frozen perception stack detects and ranges objects; a deterministic semantic serializer compiles the perceived state, errors included, into decision-aligned text; an unmodified text-only large language model (LLM) answers. On a visible-scope-matched, occlusion-audited campus-robot benchmark, under a prospectively frozen criterion, the serialized interface, using detectors fine-tuned in-domain within each fold, outperforms a zero-shot VLM whose language model has the same 7B scale (0.7892 vs 0.7462), with a larger margin at 3B (0.7673 vs 0.6913). Preregistered decoupling experiments show the gain survives paraphrase, attributing it to decision-aligned computation rather than answer-string leakage, while novel judgment vocabularies bound its scope. The advantage grows as the reader shrinks to 1.5B and reverses at 0.5B, and a ground-truth oracle locates the reader-capability floor. Under matched task supervision the interfaces converge: a VLM fine-tuned with low-rank adaptation (LoRA) overtakes the zero-shot system but only ties an equally supervised text reader (0.8441 vs 0.8396, no statistically resolved difference), and both routes remain perception-bound. Reported perception parameters are comparable to those of the VLM's vision tower, and total compute is not smaller.

Figures & tables

abbreviation	meaning
0.5B, 1.5B, 3B, 4B, 7B, 8B	model size in billions of trainable parameters (for text readers: the language model; for VLMs: the whole model)
∗	a difference whose 95% confidence interval excludes zero (“resolved”)
cf / dev	confirmatory bank (1328 questions, used once, for the final test) / development bank (1366 questions, used to build the system)
CI	confidence interval (always 95%, cluster bootstrap)
CODa	UT Campus Object Dataset [ 11 ]
D0–D3	serializer variants: D0 = Scoped (shipped); D1 = numeric ranges; D2 = numeric plus alternative bands; D3 = alternative bands only

TABLE I: Abbreviations used in this paper.

term	meaning
language model, decoder	a neural network that produces text one token at a time; the “decoder” is the part that generates the answer. Our reader is a text-only decoder
token, visual token	the unit a language model reads; text is split into word pieces, and a VLM converts image patches into “visual tokens” that occupy the same input positions
vision encoder	the network (a ViT) that turns an image into visual tokens; in a VLM it is wired to the language model
end-to-end VLM	a single network that takes pixels and text directly and answers; our baseline (the pixel arm)
reader	the text-only language model that reads the serialized state and answers; it never sees the image
serializer, serialized (compiled) state	a deterministic program, not a model, that turns the detected objects (class, bearing, range, confidence) into a short text; “compiled” stresses that closed-form quantities are computed before the reader sees them

TABLE II: Technical terms, in plain language, as used in this paper.

arm	acc	acc $\mid$ parsed	parse	longest unparsed run
Scoped 7B	0.7892	0.7951	0.9925	1
pixel 7B	0.7462	0.7462	1.000	0
Scoped 3B	0.7673	0.7714	0.9947	1
pixel 3B †	0.6913	0.6928	0.9977	1

TABLE III: Instrument health, confirmatory bank (1328 items per arm). Unparsed outputs count as incorrect. The two same-decoder VL-on-text auxiliary arms (parse

0.974

0.968

) are read on parsed items (

+0.058

+0.120

, same sign; supplement). † After the reload re-run.

Fig. 1: Top: the stack. Frozen vision modules perceive from the front stereo pair; a deterministic serializer compiles the perceived state into decision-aligned text; an unmodified text-only LLM answers; the baseline replaces everything left of the LLM with a ViT encoder over the monocular keyframe. Bottom: confirmatory-bank paired accuracy differences with cluster-bootstrap 95% CIs (out-of-fold; row labels give pixel

\to

ours accuracies; parse rates in Table III ).

reader	0.5B	1.5B	3B	7B
Scoped @ cf	0.5143	0.7357	0.7673	0.7892
GT @ cf	0.6152	0.8660	0.9247	0.9699
Scoped @ N2	0.4132	0.4591	0.5014	0.5860
D1 @ N2	0.4336	0.4666	0.4422	0.5963
	moondream (1.4B) / llava-phi3 (3.8B) / Qwen2.5-VL (3B / 7B)
pixel @ cf	0.6114 / 0.6468 / 0.6913 / 0.7462

TABLE IV: Reader-scale sweep (accuracy; cf = confirmatory bank). Pixel arms are cross-family, hence descriptive (A4). GT = ground-truth-state oracle. On N2, text arms cover 3971 of the 3974 items and pixel arms all 3974; contrasts are paired.

Fig. 2: Reader-scale sweep on the confirmatory bank. The serialized interface (solid) tracks its ground-truth-state ceiling (dashed) down to 1.5B and collapses at 0.5B, where even the oracle falls to

0.6152

: below 1.5B the regime is reader-bound; above 3B it is perception-bound (the gap to the ceiling, not reader capability, dominates). Pixel VLM arms (open markers) are shown at their own decoder scales; cross-family points are descriptive.

arm	grad. samples	interface	acc
pixel (zero-shot)	0	pixels	0.6913
Scoped (zero-shot)	0	serialized	0.7673
gb-tree scorer	dev (shallow)	compiled state	0.8245
FT-VLM (LoRA, OOF)	16k	pixels	0.8441
FT-Text (LoRA, OOF)	16k	serialized	0.8396
GT oracle (ceiling)	0	serialized (GT)	0.9247

TABLE V: Supervision ledger (3B readers and VLMs; confirmatory bank). For every arm: how many gradient samples it was trained on, which input form it reads, and its accuracy. Full ledger in the supplement.

scorer	cf	N1	N2
rule	0.8027	0.3479	0.3510
logistic regression	0.8080	0.3479	0.3332
gradient-boosted trees	0.8245	0.3479	0.3183
zero-shot 7B reader ( Scoped )	0.7892	0.7914	0.5860

TABLE VI: Option-aware learned baselines under the frozen probe banks (overall accuracy; fit on the development bank, evaluated as-is, no refit, no parser repair). Chance

=1/3

. cf = confirmatory bank.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

arm	definition
pixel 3B/7B	Qwen2.5-VL on the raw front-camera image plus the question
pixel @ N1/N2	the same protocol run on the frozen N1/N2 probe banks
D0 (= Scoped )	decision-scoped serializer: counts, corridor/overall arg-mins, per-class band and sector; per-object rows only for counting
D1 (numeric)	D0 with every distance as one-decimal meters, $\varepsilon$ flags dropped
D2 (numeric+alt)	D1 plus close/mid/far tags at 4/12 m (disjoint vocabulary)
D3 (alt-only)	tags only, no numerals (negative control)

Appendix

TABLE VII: Arm definitions. cf = confirmatory bank; dev = development bank.

family	scale	$n$	$\Delta$	95% CI
counting	3B	302	$+0.1358^{*}$	$[+0.0413,\,+0.2240]$
counting	7B	302	$+0.1358^{*}$	$[+0.0541,\,+0.2022]$
direction	3B	289	$-0.0381$	$[-0.1308,\,+0.0338]$
direction	7B	289	$-0.0242$	$[-0.0996,\,+0.0393]$
nearest-class	3B	243	$+0.0412$	$[-0.0265,\,+0.0940]$
nearest-class	7B	243	$-0.0617^{*}$	$[-0.1429,\,-0.0089]$

Appendix

TABLE VIII: Per-family paired accuracy differences, Scoped

-

pixel , confirmatory bank (cluster bootstrap, 95% CIs). ∗ = CI excludes zero.

arm	overall	counting	direction	near.-class	near.-dist	path-object
blind 3B	0.4142	0.374	0.439	0.342	0.492	0.425
blind 7B	0.4089	0.278	0.439	0.313	0.437	0.604

Appendix

TABLE IX: Blind-arm accuracy by family, confirmatory bank (chance

=1/3

module	parameters	mean latency
YOLO11l + YOLO11m, full frame	25.3 M + 20.1 M	62.5 ms
YOLO11l + YOLO11m, $2{\times}2$ tiling	(same weights)	221.3 ms
LLMDet (absence recovery only)	233.0 M	298.0 ms
UniDepthV2 (monocular depth)	353.8 M	98.7 ms
stereo semi-global block matching (SGBM, CPU)	0	267.7 ms
semantic serializer (code)	${\approx}0$	0.1 ms

Appendix

TABLE X: Perception-stack modules: parameters and measured mean latency per frame (single stream; 200 frames).

model	mean	median
Qwen2.5 0.5B / 1.5B (text)	0.151 / 0.181	0.151 / 0.175
Qwen2.5 3B / 7B (text)	0.196 / 0.241	0.195 / 0.230
Qwen2.5-VL 3B / 7B (VLM)	0.972 / 1.109	0.970 / 1.102

Appendix

TABLE XI: Per-question latency (mean / median, s).

quantity	value
counting / direction / nearest-class	302 / 289 / 243
nearest-distance / path-object	254 / 240
gold letters A / B / C / D (4-option bank)	337 / 356 / 316 / 319
cluster sizes (21 sequences), min / median / max	1 / 43 / 254
separation from dev bank, min / median / P95 / max	1.5 / 2.1 / 4.7 / 11.4 s

Appendix

TABLE XII: Confirmatory-bank composition (1328 questions, one per frame).

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 736 papers

Jul 10, 2026

Test-Time Scaling for Small VLMs on Multilingual Visual MCQ

Test-time scaling (TTS) reliably improves reasoning in large language models, but whether it transfers to small open vision-language models remains unclear. We examine this on EXAMS-V, a multilingual visual multiple-choice benchmark, comparing self-consistency, describe-then-reason with PRM-guided beam search, and two post-hoc selectors across Qwen2.5-VL-7B-Instruct and Qwen3.5-4B. What matters is the conditions under which TTS runs, not the search or verification machinery. The largest factor is parseability: an early prompt format left many chains reasoning correctly yet never committing to an answer letter, which a standard answer cue and a guided repair step largely remove. A larger decoding budget removes the rest: raising the per-chain token limit from 1k to 2k recovers 3.7 pp, whereas sampling more chains (8 to 16) adds only 0.15 pp. Once chains have room to finish, elaborate methods contribute little: PRM-guided beam search trails plain self-consistency by 0.39 pp at over eight times the cost, and neither a training-free generative critic nor a trained multimodal PRM beats majority vote across both policies. The largest gain comes instead from the policy model itself (+11.4 pp). Our best configuration reaches 84.1% on the held-out ImageCLEF 2026 test split, ranking first on the Visual MCQ leaderboard.

Spiros Baxevanakis, Peng-Jian Yang

University of Amsterdam, Science Park 904, 1098 XH Amsterdam, The Netherlands

Visual Question Answering Test-Time Scaling for VLMs Test-Time Scaling Multilingual VLM Evaluation VLM Reasoning Multimodal Foundation Models

Ask PDF

Sep 28, 2026

From Perception to Integration: Revisiting the Internal Dynamics of Reasoning in Vision-Language Models

Vision-language models (VLMs) can answer simple visual questions, but often struggle when one question requires several visual judgments. We study this gap with controlled tasks for feature binding, numerosity, spatial relations, and amodal completion, together with a Composite task that combines them. Matched counterfactual image pairs isolate changes in the visual evidence needed to answer. Across four models, direct answers, hidden-state readouts, and state interventions show that the individual judgments can be made without explicit reasoning and that intervening on the corresponding states can affect the answer. During reasoning, the Composite answer becomes decodable from hidden states and usable from shortened traces, often before the model stops on its own. We train a small detector to predict this readiness and stop reasoning at that point. On MMStar and RealWorldQA, this reduces mean reasoning tokens by 79.1% and 74.5%, while average accuracy rises by 3.13 and 3.30 percentage points, respectively. These findings connect the internal development of answer readiness to a practical rule for allocating reasoning computation.

Rong Yu Xu, Prayag Tiwari, Shaolei Zhang

Shenzhen College of International Education · Halmstad University, Sweden · Renmin University of China

Efficient VLM Inference VLM Interpretability Latent Visual Reasoning VLM Reasoning Reasoning for VLMs Compositional Reasoning in VLMs

Ask PDF

Oct 27, 2026

V-LUMEN: Visual Lookup Memory for Embedding Scaling in Vision-Language Models

Embedding-level scaling expands the capacity of language models through lookup-based memory rather than by increasing model depth, width, or dense computation. Extending this idea to vision-language models (VLMs) is challenging because visual features are continuous, spatially structured, and query-dependent, unlike discrete text tokens that naturally support n-gram lookup. We introduce V-LUMEN, a visual lookup memory for VLMs. V-LUMEN discretizes visual features with a frozen codebook, builds spatially aware memory keys through spatial aggregation-based hashing, and modulates them with text-conditioned relevance signals; all addressing is non-parametric, and the retrieved embeddings are fused into the backbone through a gated residual pathway. On five reasoning-intensive benchmarks, V-LUMEN substantially improves a frozen Qwen3-VL-2B backbone and transfers zero-shot to three benchmarks whose images are disjoint from the training data. At a matched parameter budget, moving part of a LoRA or MoE adapter's capacity into V-LUMEN preserves accuracy on two backbones, indicating that external visual memory is an independent capacity axis rather than a stronger adaptation method. Because lookup cost is constant in table size and incurred only once at prefill, this capacity grows in storage rather than in per-token compute. Code is available at https://github.com/miso-choi/V-Lumen.

Miso Choi, Daekeun Kim, Eunji Kim +1

Dept. of Computer Science and Engineering, Korea University · Amazon

Memory-Augmented VLMs Efficient VLM Inference Reasoning for VLMs

Ask PDF

Abstract

Vision-Language Models

Explore similar work

Test-Time Scaling for Small VLMs on Multilingual Visual MCQ

From Perception to Integration: Revisiting the Internal Dynamics of Reasoning in Vision-Language Models

V-LUMEN: Visual Lookup Memory for Embedding Scaling in Vision-Language Models