Enhancing Long-Video VLM Embeddings with Query-Aware Streaming Latent Reasoning

Haozhe Chi; Song Jin; Yang Jin; Yadong Mu

cs.CVOct 4, 2026

Enhancing Long-Video VLM Embeddings with Query-Aware Streaming Latent Reasoning

Authors: Haozhe Chi, Song Jin, Yang Jin, Yadong Mu

Organizations: Peking University · Renmin University of China

Abstract

Long-video embedding requires capturing sparse query-relevant evidence under a limited visual-token budget. Uniform sampling can miss brief events in videos spanning minutes or hours, whereas encoding more frames in a single context increases memory and computation. We introduce Query-Aware Streaming Latent Reasoning (QASLR), a post-training framework that accumulates evidence across clips while keeping the embedding size fixed. QASLR selects a bounded set of frames, restores their temporal order, and processes them clip by clip with a vision-language backbone. A compact set of persistent think tokens cross-attends to each clip's features, while an embed token reads out a normalized representation after every update. This design integrates evidence across multiple backbone calls without requiring all selected frames to share a single context. Training combines final contrastive learning, step-wise contrastive supervision, and final-embedding self-distillation. Intermediate supervision trains partial-video readouts for retrieval, while self-distillation regularizes them toward the final representation. Query-aware selection produces query-conditioned representations for candidate-set scoring and reranking, whereas query-independent selection enables reusable corpus indexing. Under the full training recipe, HourVideo retrieval Hit@1 increases from 54.2 to 70.7 and from 57.4 to 72.8 for 2B and 8B Qwen3-VL-Embedding backbones, respectively. Gains extend to the evaluated moment-retrieval and video-QA tasks, and the streaming head transfers to a second Qwen-family embedding backbone. These results support streaming latent aggregation as an effective approach to integrating long-video evidence into fixed-dimensional representations.

Figures & tables

Figure 1: From one-shot video encoding to streaming evidence accumulation. The upper-left pathway illustrates a one-shot baseline that pools representations from sparsely sampled frames into a single video embedding. The lower-left pathway shows QASLR processing selected frames as ordered clips and producing embeddings throughout the sequence. The right-hand panels summarize example training signals and the intended capabilities of persistent memory, intermediate supervision, optional early exit, and parameter-efficient post-training.

Figure 2: QASLR architecture, training objectives, and optional early-exit inference. Selected frames are ordered and partitioned into clips. The illustrated 128-frame configuration contains eight clips of 16 frames each. Per-clip backbone features are compressed into evidence banks that update persistent think-token memory through cross-attention and feed-forward refinement. An embed token reads the memory to produce an L2-normalized embedding after each update. Training combines final contrastive learning, step-wise supervision, and final-embedding self-distillation across the supported video tasks. The optional stop gate terminates processing when its stopping criterion is met.

Retrieval-style tasks
Backbone	Method	HourVideo CLS	HourVideo Ret.	HourVideo MRet.	LVBench Ret.	LongVideoBench MRet.
Qwen3-VL 2B	Original	51.1 / 85.1	54.2 / 77.1	24.3 / 62.2	65.4 / 98.1	34.2
	Single-round PT	52.4 / 86.3	55.0 / 76.7	25.2 / 63.0	66.6 / 98.0	35.1
	QASLR	54.2 / 88.1	70.7 / 92.5	30.1 / 65.8	75.7 / 98.3	37.0
Qwen3-VL 8B	Original	52.2 / 83.9	57.4 / 87.5	24.9 / 63.1	68.1 / 98.4	33.2
	Single-round PT	53.0 / 84.3	57.8 / 88.4	25.6 / 63.8	69.2 / 98.1	34.5

Table 1: Long-video embedding performance on Qwen3-VL-Embedding backbones. The original checkpoints, single-round post-training (PT), and full QASLR recipe are compared at the 2B and 8B scales. CLS, Ret., and MRet. denote classification, video retrieval, and video moment retrieval, respectively. Retrieval-style entries report Hit@1/Hit@5, except LongVideoBench MRet., which reports Hit@1 only. Question-answering (QA) entries report accuracy. Avg. is the unweighted mean across the four QA tasks. All scores are percentages, with higher values indicating better performance. Bold numeric entries identify the best result for each metric within each backbone scale.

Retrieval-style tasks
Backbone	Method	HourVideo Ret.	LVBench Ret.	LongVideoBench MRet.	Avg. H@1
Qwen3-VL 2B	Original	54.2 / 77.1	65.4 / 98.1	34.2	51.3
	Single-round PT	55.0 / 76.7	66.6 / 98.0	35.1	52.2
	Single-round PT w/ sub-batching	66.1 / 84.2	67.5 / 98.2	36.7	56.8
	QASLR w/o sub-batching	55.4 / 78.8	73.3 / 98.0	34.8	54.5
	QASLR w/ sub-batching	70.7 / 92.5	75.7 / 98.3	37.0	61.1

Table 2: Effects of streaming aggregation and interleaved sub-batching. All configurations use query-aware selection. PT and QASLR are evaluated with or without sub-batching. Ret. and MRet. denote video retrieval and video moment retrieval. Retrieval entries report Hit@1/Hit@5, except LongVideoBench MRet., which reports Hit@1 only. Avg. H@1 averages Hit@1 across the three retrieval tasks. QA entries report accuracy. All scores are percentages, and bold numeric entries mark the best result within each backbone and metric. QASLR with sub-batching achieves the highest retrieval Hit@1, whereas the strongest QA configuration varies across tasks.

Figure 3: Within-family transfer of QASLR to RzenEmbed 7B. Grouped bars compare the original checkpoint, single-round post-training, and QASLR across seven tasks using the same RzenEmbed backbone. Classification (CLS), video retrieval (Ret.), and moment retrieval (MRet.) report Hit@1. Question-answering tasks report accuracy. All scores are percentages, and annotations above the QASLR bars give absolute improvements over the original checkpoint in percentage points. QASLR exceeds both baselines on every displayed task, with gains of 3.4–6.0 points over the original checkpoint.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Input: training pairs $\{(q_{i},p_{i})\}_{i=1}^{P}$ , selected frame indices, VLM backbone $F_{\theta}$ , streaming head $G_{\phi}$ , temperature $\tau$ , loss weights $\lambda_{\mathrm{step}},\lambda_{\mathrm{distill}}$ .
1. Sample a mini-batch of query-positive pairs. If interleaved sub-batching is enabled, group examples with similar task and input format into homogeneous sub-batches.
2. For each streaming video item, decode selected frame indices from the original video and restore chronological order.
3. Split selected frames into clips $[C_{1},\ldots,C_{T}]$ .
4. For each clip $C_{t}$ , encode $U_{t}=F_{\theta}(C_{t},c)$ , where $c$ is the permitted conditioning input, omitted for query-independent video encoding.
5. Compress valid hidden states into a fixed-size evidence bank $E_{t}$ .
6. Initialize think-token memory $H_{0}$ and update it sequentially: $H_{t}=\mathrm{Update}_{\phi}(H_{t-1},E_{t})$ .

Appendix

Table 3: QASLR post-training procedure for unified query–positive pairs. Selected frame indices are decoded on demand and restored to chronological order before clip-wise backbone encoding. A bounded evidence bank updates the persistent think-token memory, which is initialized once per video and read out after every clip. The optimization combines final contrastive learning with the enabled auxiliary objectives. Interleaved sub-batching groups compatible task and input formats before parameter updates. The gate objective applies only when training the optional early-exit module; it is not required for fixed-length streaming inference.

Mode	Query-aware	Pre-indexable	First-stage R@ $M$	Hit@1	Latency (s)
Uniform, single-round	No	Yes	–	55.0	0.18
Uniform, QASLR + SB	No	Yes	–	64.1	0.61
Two-stage, single-round + SB	Yes	First stage	95.8	66.1	1.26
Two-stage, QASLR + SB	Yes	First stage	95.8	70.7	1.72

Appendix

Table 4: Indexable and query-conditioned retrieval on HourVideo with Qwen3-VL-Embedding 2B. Uniform-selection methods produce one query-independent representation per video that can be reused across queries. SB denotes interleaved sub-batching; the uniform single-round row does not use SB. Two-stage methods apply query-aware selection and candidate scoring only to the top-

M

videos returned by a query-independent first stage. First-stage R@

M

measures positive-video recall before second-stage scoring, and Hit@1 measures final retrieval performance; both are percentages. First stage indicates that only the initial retrieval stage is pre-indexable, and a dash marks an inapplicable shortlist-recall entry. Latency is measured in seconds per query on NVIDIA H100 hardware with a warm cache.

Aggregator	Ordered	Persistent memory	Ret. Hit@1	QA Acc.
Mean pooling	Yes	No	58.6	60.2
Recurrent aggregator	Yes	Yes	63.5	61.8
Transformer aggregator	Yes	Yes	66.2	62.7
QASLR, shuffled clips	No	Yes	64.1	61.6
QASLR, reversed clips	No	Yes	65.0	61.9
QASLR, reset memory	Yes	No	60.3	60.8

Appendix

Table 5: Aggregation, clip-order, and memory controls with Qwen3-VL-Embedding 2B. Variants use the same selected frames and backbone features, except that the last-clip-only control retains only the final clip. Alternative aggregators test pooling and additional modeling capacity, while QASLR controls perturb clip order or remove cross-clip memory. Ordered indicates chronological clip presentation, not whether an aggregator is sensitive to order; Persistent memory indicates retention of information across clips. Ret. Hit@1 is HourVideo retrieval Hit@1, and QA Acc. is EgoSchema accuracy. Scores are percentages, and bold numeric entries mark the highest value in each metric.

Step loss	Distillation	25%	50%	75%	Final	Mean H@1
No	No	43.8	54.6	62.8	67.9	57.3
Yes	No	49.7	59.8	66.5	69.8	61.5
No	Yes	51.2	61.7	67.9	70.1	62.7
Yes	Yes	55.6	64.8	69.1	70.7	65.1

Appendix

Table 6: Auxiliary-objective ablation and intermediate retrieval quality on HourVideo. The four Qwen3-VL-Embedding 2B configurations independently enable step-wise contrastive supervision and final-embedding self-distillation, with the final contrastive loss active throughout. Entries report retrieval Hit@1 (%) after processing 25%, 50%, 75%, or all selected clips; Final denotes the complete sequence. Mean H@1 is the unweighted mean of the four checkpoint scores. Bold numeric entries mark the highest value in each column, including intermediate and final performance.

Multimodal Embedding

Multimodal embedding maps text, images, audio, video, and other data into representations that support comparison and retrieval across modalities.

See all 159 papers

FrameScout: Scouting Query-Relevant Frames for Long Video Understanding
Oct 13, 2026Haonan Hu, Shuhao Chen, Weisen Jiang +3Query-Aware Video Frame Selection Long-Video Understanding
Southern University of Science and Technology · The Hong Kong University of Science and Technology · South China University of Technology +1
MemoryCard: Topic-Aware Multi-Modal Clue Compression for Long-Video Question Answering
Jun 4, 2026Qing Yang, Pengcheng Huang, Xinze Li +6Video Summarization Video QA
School of Computer Science and Engineering, Northeastern University, Shenyang, China · Department of Computer Science and Technology, Tsinghua University, Beijing, China · Digital China Group, Beijing, China
Beyond Frame Selection: Generative Latent Evidence Aggregation for Long-Video Understanding
Jul 30, 2026Bowen Liu, Shuning Wang, Xinpeng Ding +3Video-Language Models Long-Video Understanding
The Hong Kong University of Science and Technology · Alibaba Group · Xidian University +1

Oct 13, 2026J/K move · Enter open · S save

FrameScout: Scouting Query-Relevant Frames for Long Video Understanding

Haonan Hu, Shuhao Chen, Weisen Jiang, Lizhao Gao+2

Southern University of Science and Technology · The Hong Kong University of Science and Technology · South China University of Technology+1

Query-Aware Video Frame Selection Long-Video Understanding Streaming Video Understanding

Open paper AskPDF arXiv

Long video understanding requires VideoLLMs to answer queries about videos spanning tens of minutes to hours, but such long videos produce far more visual tokens than current VideoLLMs can process. To address this issue, keyframe selectors are proposed to select the most query-relevant frames by computing the similarity between frame and query embeddings. Existing selectors either encode frames independently without temporal context or process chunks in isolation with ranking objectives that are difficult to optimize. To alleviate those limitations, we propose FrameScout with two core designs: (i) a streaming selector architecture that leverages a sliding-window KV cache and successor aggregation to produce temporally-contextualized frame embeddings; (ii) a frame-query contrastive objective that aligns frame and query embeddings in a shared space, directly training the selector to distinguish relevant frames from irrelevant ones. Extensive experiments on three long-video benchmarks and six base VideoLLMs demonstrate that FrameScout consistently achieves state-of-the-art performance. Code is available at https://github.com/Hubert-1102/FrameScout.

Abstract

Multimodal Embedding

Explore similar work

FrameScout: Scouting Query-Relevant Frames for Long Video Understanding

MemoryCard: Topic-Aware Multi-Modal Clue Compression for Long-Video Question Answering

Beyond Frame Selection: Generative Latent Evidence Aggregation for Long-Video Understanding