WIRE: Weight Space Implicit Representation Encoding of Frozen Video LLMs

Ameen Ali; Tamim Zoabi; Lior Wolf; Chaim Baskin

cs.CVOct 12, 2026

WIRE: Weight Space Implicit Representation Encoding of Frozen Video LLMs

Authors: Ameen Ali, Tamim Zoabi, Lior Wolf, Chaim Baskin

Organizations: Tel Aviv University · INSIGHT Lab, Ben-Gurion University of the Negev, Israel

Abstract

Video language models represent a video as a long sequence of visual embeddings in a decoder's input. Every visual embedding is processed by every decoder layer during prefill, stored in the key value cache, and attended by every generated token, so a long video raises prefill, memory, and generation cost at once. We present WIRE (Weight Space Implicit Representation Encoding), which reduces this recurring cost by moving part of the video conditioned computation of a frozen video language model from its input activations into a temporary low rank update of its weights. WIRE keeps a short explicit visual context, so the update acts as a correction rather than a replacement of the video. WIRE first compresses the full resolution embeddings of each frame into a small set of carrier tokens using a question conditioned bidirectional selective scan, and physically contracts the decoder input so every decoder layer, cache entry, and decoding step operates on the shorter sequence. Hypernetworks at selected decoder layers read the compressed forward pass together with a readout of the full resolution features removed by contraction and generate LoRA updates to the attention and feedforward projections. The updates are merged after prefill, so autoregressive decoding runs the unmodified frozen backbone. On Qwen3-VL-8B, WIRE with 16 carrier tokens per frame contracts the visual stream by $3.5\times$ , reduces decoder prefill time by $3.35\times$ at 256 frames and $4.14\times$ at 1024 frames, shrinks the key value cache proportionally, and keeps per token decoding latency approximately constant. Across nine benchmark settings at 256 frames, it obtains the highest average accuracy among efficient video language models we evaluate under a common 256 frame budget.

Figures & tables

Figure 1: Latency versus video length. Times are measured on one B200 GPU at batch size one in bfloat16, excluding video file decoding. Left: total model time to first token. Right: decoder prefill only. Wire contracts the sequence before the first decoder layer, so its advantage grows with frame count and it continues to run beyond input lengths where several baselines exhaust memory.

Figure 2: Wire overview. The frozen vision encoder produces $n_{t}$ full resolution visual tokens for frame $t$ . Bilinear resampling initializes $k_{t}$ carrier tokens, and a bidirectional Mamba2 scan refines the carrier tokens while reading the full resolution visual stream and the question. The carrier tokens replace the original frame tokens in the decoder prefix. At selected decoder layers, a hypernetwork uses the contracted decoder state and a full resolution per frame readout to generate LoRA updates for selected attention and MLP projections. The updates are merged after prefill. Decoding then uses the frozen backbone over the contracted prefix.

Open models
Model	NextQA test ( Xiao et al., 2021 )	PerceptionTest test ( Patraucean et al., 2023 )	MVBench test ( Li et al., 2024b )	Tomato test ( Shangguan et al., 2024 )	Short QA avg.	Video-MME test ( Fu et al., 2025 )	Video-MME-Sub test ( Fu et al., 2025 )	LongVideoBench val ( Wu et al., 2024 )	MLVU test MCQ ( Zhou et al., 2025 )	LVBench test ( Wang et al., 2024 )	Long QA avg.	Average
Qwen3-VL-8B ( Bai et al., 2025a )	83.4	72.7	68.7	35.7	65.1	71.4	75.2	62.4	57.6	58.0	64.9	65.0
Eagle2.5-8B ( Chen et al., 2025 )	85.0	81.0	74.8	31.0	68.0	72.4	75.7	66.4	60.4	50.9	65.2	66.4
PLM-8B ( Cho et al., 2026 )	84.1	82.7	77.1	33.2	69.3	58.3	65.4	56.9	52.6	44.5	55.5	61.6
LLaVA-Video-7B ( Zhang et al., 2025d )	83.2	68.8	58.6	24.9	58.9	63.3	69.7	58.2	52.8	44.2	57.6	58.2
Molmo2-8B ( Clark et al., 2026 )	86.2	82.1	75.9	39.6	71.0	69.9	75.8	67.5	60.2	52.8	65.2	67.8

Table 1: Video question answering accuracy , in percent. The short average covers the first four columns and the long average the next five. Rows labelled Wire give our compressor and hypernetwork on a frozen Qwen3-VL-8B-Instruct backbone at four values of the per frame token budget

k

, all at

256

sampled frames. Bold marks the best entry among the efficient models re-evaluated under our controlled

256

-frame protocol ( Wire , BIMBA, Vamba, VideoNSA). MLVU and LVBench entries are the test split. Table 4 in the appendix adds proprietary models, additional open baselines, and further efficient methods with the accuracies stated in their own papers.

Figure 3: Qualitative examples at 256 frames. On a question about a video’s global subject, Wire matches the uncompressed backbone while the efficient baselines pick locally plausible distractors. On a temporally localized question all systems answer correctly.

Subset	Disc.	Appl.	$\Delta$	Subset	Disc.	Appl.	$\Delta$
Video-MME				LongVideoBench
Counting	40.67	44.03	$+3.36$	Sub. $\to$ next	45.21	57.53	$+12.32$
Law	61.19	67.62	$+6.43$	Scene $\to$ sub. †	32.88	43.84	$+10.96$
Athletics	54.36	59.23	$+4.87$	Sub. $\to$ object †	55.41	66.22	$+10.81$
MVBench				MLVU
Action count	46.33	51.83	$+5.50$	Count	28.33	33.33	$+5.00$

Table 2: Paired update analysis at fixed compressed context. Disc. discards the per example LoRA update and Appl. applies it;

\Delta

is the paired difference in points at identical carrier context. † marks subsets not significant under the exact McNemar test.

Compressor variant	Video-MME (no sub.)	$\Delta$
Wire (full compressor)	66.2	—
$-$ question conditioning (test time)	64.3	$-1.9$
$-$ bidirectional scan (forward only, retrained)	65.1	$-1.1$
$-$ selective scan (bilinear pooling only, retrained)	64.7	$-1.5$

Table 3: Compressor ablations at

k{=}16

on Video-MME without subtitles (accuracy %,

\Delta

vs. the full compressor). Each variant changes only the compressor.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Proprietary models, API only
Model	NextQA test ( Xiao et al., 2021 )	PerceptionTest test ( Patraucean et al., 2023 )	MVBench test ( Li et al., 2024b )	Tomato test ( Shangguan et al., 2024 )	Short QA avg.	Video-MME test ( Fu et al., 2025 )	Video-MME-Sub test ( Fu et al., 2025 )	LongVideoBench val ( Wu et al., 2024 )	MLVU test MCQ ( Zhou et al., 2025 )	LVBench test ( Wang et al., 2024 )	Long QA avg.	Average
GPT-5 ( OpenAI, 2025 )	86.3	79.4	74.1	53.0	73.2	83.3	86.9	72.6	77.7	65.2	77.1	75.4
GPT-5 mini ( OpenAI, 2025 )	83.2	72.0	66.5	44.1	66.5	77.3	82.3	69.7	69.1	54.7	70.6	68.8
Gemini 3 Pro ( Google, 2025 )	84.3	77.6	70.4	48.3	70.2	88.6	87.5	75.9	75.7	77.0	80.9	76.1
Gemini 2.5 Pro ( Comanici et al., 2025 )	85.3	78.4	70.6	48.6	70.7	87.8	87.8	76.8	81.5	75.7	81.9	76.9
Gemini 2.5 Flash ( Comanici et al., 2025 )	81.8	74.7	67.0	39.1	65.7	84.2	84.2	73.1	75.1	64.9	76.3	71.6

Appendix

Table 4: Video question answering accuracy, full comparison , in percent. Table 1 in the main text is the subset of these rows most directly comparable to Wire . As in Table 1 , the final efficient block reports accuracies stated in each method’s own paper under different backbones, frame counts, and protocols, so these rows are partial and receive no aggregate.

$k$	frames	base	Wire	base	Wire	encoder	base	Wire
		prefill (ms)		TTFT (ms)			decode (ms/token)
16	128	180	54	274	162	94	18.8	19.7
16	256	369	110	550	323	181	18.9	18.8
16	512	851	228	1216	659	365	21.1	18.9
16	1024	2086	504	2819	1367	733	21.1	18.2
6	128	183	30	276	136	93	19.0	19.1

Appendix

Table 5: Latency of the uncompressed Qwen3-VL-8B backbone and Wire on one B200 GPU in bfloat16. All timings are batch size one: per example updates cannot be batched across examples. For Wire , prefill and time to first token include the compressor, the hypernetwork forward, and the cost of merging the generated update. Prefill excludes the vision encoder, while time to first token includes it. Decoding is measured after merging the generated update.

Figure 4: Video needle in a haystack under decoder input ablation. Retrieval accuracy (%) for a planted word card needle across five insertion depths (columns of each grid) and three haystack lengths (rows). Darker cells are higher. The uncompressed backbone and Wire at carrier budgets

k\in\{16,12,8,6\}

keep the video in the decoder input. The two

k{=}0

grids remove every visual token, so the generated update is the only path from video to answer, and differ only in whether that update is applied. Chance is

25\%

Efficient VLM Inference

Efficient vision-language model inference reduces the compute, memory, latency, and energy needed to process images, video, and text.

See all 487 papers

Video Encoders Built on Image Representations
Oct 5, 2026Jusheng Zhang, Wenhao Wang, Longqi Cai +3Video-Language Models Video Representation Learning
Stanford University · Vast Intelligence Lab · Google DeepMind
TTF: Temporal Token Fusion for Efficient Video-Language Model
May 8, 2026Simin Huo, Ning LIToken Merging Efficient VLM Inference
Shanghai Jiao Tong University
PREM: Prefix-Steered Recurrent Memory for Long-Video Understanding
Sep 20, 2026Siru Zhong, Qiongyan Wang, Xiaohui Lv +5Memory-Augmented VLMs Recurrent Neural Networks
The Hong Kong University of Science and Technology (Guangzhou) · Beta Infinity · Tsinghua University

Oct 5, 2026J/K move · Enter open · S save

Video Encoders Built on Image Representations

Jusheng Zhang, Wenhao Wang, Longqi Cai, Liangzhe Yuan+2

Stanford University · Vast Intelligence Lab · Google DeepMind

Video-Language Models Video Representation Learning Video Token Compression

Open paper AskPDF arXiv

The design of a video encoder determines when frames begin to interact and which frame-specific visual evidence remains accessible to the language model. Native video pathways couple neighboring frames during visual encoding, whereas image pathways preserve independently computed frame representations but incur a much larger visual-token cost when all image tokens are forwarded. We ask a basic question: whether a compact video encoder can instead be built on image representations. To answer this question, we separate three operations that are often coupled: per-frame representation, cross-frame token allocation, and temporal interaction. A frozen image encoder first produces frame-specific candidates. A question-aware selector then allocates a fixed token budget across frames using relevance, diversity, and cross-frame correspondence, after which a lightweight learned refiner reads neighboring-frame context and writes residual updates only to the retained anchors. This preserves source positions and keeps the visual output at the fixed budget. Across 13 benchmarks and three vision-language backbones, the resulting pathway matches full-image aggregate performance while using only about 28%-35% of its visual tokens. Specifically, on Qwen3-VL-8B, it achieves a 13-benchmark macro-average of 62.75 with 1,535 visual tokens, compared with 62.58 for the full Image pathway at 4,424 tokens and 59.49 for native Conv3D at 2,212 tokens. On Qwen3-VL-32B, it reaches a 13-benchmark macro-average of 66.28, compared with 66.09 for Image, while providing a 2.16x end-to-end speedup. These results show that compact video encoding does not require early temporal mixing: frame-specific evidence can be preserved first, allocated jointly, and temporally contextualized after selection.

Abstract

Efficient VLM Inference

Explore similar work

Video Encoders Built on Image Representations

TTF: Temporal Token Fusion for Efficient Video-Language Model

PREM: Prefix-Steered Recurrent Memory for Long-Video Understanding