Storing Less, Finding More: How Novelty Filtering Improves Cross-Modal Retrieval on Edge Cameras

Sherif Abdelwahab

cs.CVMar 31, 2026

Storing Less, Finding More: How Novelty Filtering Improves Cross-Modal Retrieval on Edge Cameras

Authors: Sherif Abdelwahab

Abstract

Always-on edge cameras generate continuous video streams where redundant frames degrade cross-modal retrieval by crowding correct results out of top-k search. This paper presents a streaming retrieval architecture: an on-device epsilon-net filter retains only semantically novel frames, building a denoised embedding index; a cross-modal adapter and cloud re-ranker compensate for the compact encoder's weak alignment. A single-pass streaming filter outperforms offline alternatives (k-means, farthest-point, uniform, random) across eight vision-language models (8M-632M) on two egocentric datasets (AEA, EPIC-KITCHENS). Combined, the architecture reaches 45.6% Hit@5 on held-out data using an 8M on-device encoder at an estimated 2.7 mW.

Figures & tables

Figure 1: Architecture overview. On-device (blue): encode and filter frames. Query time (purple): text query projected into image space via LiT adapter, re-ranked in the cloud. Circled numbers match stages in the text.

Model	Params	Training	Dim
TinyCLIP [ 20 ]	8M	CLIP distillation	512
MobileCLIP-S0 [ 18 ]	11M	CLIP distillation	512
CLIP ViT-B/32 † [ 15 ]	88M	Contrastive	512
CLIP ViT-L/14 [ 15 ]	304M	Contrastive	768
SigLIP ViT-B/16 [ 21 ]	86M	Sigmoid	768
SigLIP 2 SO400M [ 17 ]	400M	Sigmoid+caption	1152

Table 1: Evaluation models (8M–632M params, four training paradigms). Each evaluates retrieval independently.

Figure 2: Top-5 retrieval on 19 AEA sequences (553 events). CLIP B/32 selects which frames to keep; each model on the x-axis re-embeds those frames in its own space and retrieves independently. Novelty outperforms all offline baselines in five of eight models. Dashed line: FT-TinyCLIP+LiT on held-out data (25.0%).

Model	Full	Nov	KM	FP	Uni	Rnd	$p$
SigLIP 2	45.2	54.6	50.8	52.8	50.5	44.6	.003
ImageBind	36.7	43.8	43.4	45.4	41.2	37.2	.007
CLIP L/14	36.5	41.4	40.3	41.6	40.3	35.9	.12
CLIP B/32 †	30.7	40.3	36.5	38.7	35.6	30.4	$<$ .001
SigLIP	34.0	39.2	37.4	37.6	39.6	32.8	.27
MobileCLIP	26.9	35.8	33.1	34.7	31.1	28.3	$<$ .001

Table 2: Filtering improves retrieval across all eight models. CLIP B/32 selects frames; each model re-embeds and retrieves independently. Top-5 (%), AEA, 553 events, matched frame count.

Image index	Text encoder	Top-5	$\Delta$
All frames	CLIP B/32 text	11.0%	—
$\varepsilon$ -net	CLIP B/32 text	14.3%	+3.3
All frames	+ LiT adapter	15.2%	+4.2
$\varepsilon$ -net	+ LiT adapter	21.3%	+10.3

Table 3: TinyCLIP 4-way: filtering (

\times 1.30

) and text adaptation (

\times 1.38

) multiply independently.

Split	Configuration	Top-5	$\Delta$
AEA test (unseen)	TinyCLIP	16.9%	—
AEA test (unseen)	FT-TinyCLIP+LiT	25.0%	+8.1
AEA train (seen)	TinyCLIP	13.5%	—
AEA train (seen)	FT-TinyCLIP+LiT	48.3%	+34.7
EPIC test (unseen)	TinyCLIP	1.6%	—
EPIC test (unseen)	FT-TinyCLIP+LiT	6.0%	+4.3

Table 4: Joint fine-tuning (top-5,

\varepsilon

-net index).

$\tau$	Comp	Novelty	Uniform	Full	Nov $>$ Full
0.90	35 $\times$	5.6%	5.4%	5.8%	3/7
0.92	16 $\times$	6.0%	5.6%	5.8%	4/7
0.94	7 $\times$	6.6%	6.4%	5.8%	6/7
0.95	4 $\times$	6.7%	6.4%	5.8%	6/7
0.96	3 $\times$	6.8%	6.0%	5.8%	6/7

Table 5: EPIC-KITCHENS threshold sweep (CLIP B/32, top-5).

Figure 3: Left: cumulative improvement on held-out data (dashed line: CLIP B/32 on all frames). Right: re-ranker candidates needed at equal hit rate (

5.7\times

fewer at 70%).

Cross-Modal Retrieval

Cross-modal retrieval matches queries and items across different data types, such as text, images, video, audio, and sensor signals.

See all 184 papers

OmniTide: Co-Designing Algorithms and Systems for Efficient On-Device Omni-LLM Streaming
Sep 28, 2026Zongshang Shen, Wangsong Yin, Daliang Xu +2Efficient Multimodal Inference On-Device Language Model Inference
Key Lab of High Confidence Software Technologies (Peking University), Beijing, China · State Key Laboratory of Networking and Switching Technology (BUPT), Beijing, China
Allocate Before You Embed: Adaptive Visual Input Allocation for Video Embeddings
Sep 1, 2026Song Jin, Zhongtao Jiang, Chenglei Shen +5Multimodal Embedding Information Retrieval
Gaoling School of Artificial Intelligence, Renmin University of China · Institute of Automation, Chinese Academy of Science · Peking University
To Be Multimodal or Not to Be: Query-Adaptive Audio-Visual Person Retrieval via Active Modality Detection
Jun 4, 2026Erfan Loweimi, Mengjie Qian, Kate Knill +7Cross-Modal Retrieval Missing-Modality Learning
University of Cambridge, UK · Southwest Jiaotong University, China · University of Surrey, UK +1

Sep 28, 2026J/K move · Enter open · S save

OmniTide: Co-Designing Algorithms and Systems for Efficient On-Device Omni-LLM Streaming

Zongshang Shen, Wangsong Yin, Daliang Xu, Mengwei Xu+1

Key Lab of High Confidence Software Technologies (Peking University), Beijing, China · State Key Laboratory of Networking and Switching Technology (BUPT), Beijing, China

Efficient Multimodal Inference On-Device Language Model Inference Multimodal Large Language Models KV-Cache Management

Open paper AskPDF arXiv

On-device streaming omni-modal inference safeguards user privacy and eliminates prohibitive per-token API costs, but faces a critical bottleneck: the continuous influx of multimodal data rapidly exhausts constrained memory and compute budgets via monotonic KV cache growth. Existing sparse attention methods fall short, either incurring prohibitive online estimation latency or destroying interleaved cross-modal context, while failing to resolve physical memory fragmentation. We present OmniTide, the first algorithm-system co-design tailored for efficient on-device streaming omni-modal inference. Driven by the observation of modality-aware structural sparsity, OmniTide adopts a unit-based abstraction with two components: (1) At the algorithm level, OmniPick logically retains critical multimodal context based on unit boundaries and modality importance to preserve task accuracy; (2) At the system level, OmniPage physically partitions the cache by retention likelihood and dynamically compacts surviving sparse tokens, minimizing both memory fragmentation and data-movement overhead. Extensive evaluations across three streaming benchmarks and two consumer-device architectures show that OmniTide achieves up to

12.72\times

kernel speedups and

2.40\times

lower stream-loop latency. On StreamingBench, it improves accuracy by up to 18.0 percentage points over sliding-window baselines at comparable session cost. OmniPage further reduces the physical KV span by up to 26.7% relative to native logical eviction, unlocking real-time, infinite-context streaming on edge devices.

Abstract

Cross-Modal Retrieval

Explore similar work

OmniTide: Co-Designing Algorithms and Systems for Efficient On-Device Omni-LLM Streaming

Allocate Before You Embed: Adaptive Visual Input Allocation for Video Embeddings

To Be Multimodal or Not to Be: Query-Adaptive Audio-Visual Person Retrieval via Active Modality Detection