cs.CVNov 17, 2026

Transfer Once, Ask Locally: Compact Audio-Visual Memory for On-Device Question Answering

Authors: Qitong Wang, Jihui Jin, Divya Kothandaraman, Vishnu Raj, Christopher Rasmussen

Organizations: Dolby Laboratories, Inc. · University of Delaware

Abstract

Audio-visual question answering on lightweight user devices requires balancing speed, memory usage, privacy, and communication. One approach is to run audio-visual encoders on a server and a lightweight language decoder on the user device, keeping questions local while offloading media encoding. However, transferring and processing the full multimodal representation remains costly. We propose a training-free, query-agnostic token pruning framework that constructs a compact, reusable memory before transmission. We define video spatio-temporal blocks and aligned audio windows as players and estimate their importance through Monte Carlo Shapley valuation. We use an embedding coverage objective that combines query-independent utility with cross-modal redundancy calibration and rewards unimodal diversity through marginal coverage gains. We retain the original token embeddings of high-scoring players under a limited token budget and transfer the resulting memory once for reuse across subsequent local questions. Experiments on Video-MME, Daily-Omni, and Video-Holmes using video-SALMONN 2+ (3B) and video-SALMONN 2 Pro (4B) show that our method achieves the best or tied-best QA accuracy among the compared compression methods across all evaluated settings. On Daily-Omni with video-SALMONN 2+ (3B) at 80% pruning, our method improves accuracy by 12.86--14.62 percentage points over these methods while achieving the lowest device-side LLM prefill and decoding latency, with up to 1.92×1.92\times decoding speedup at comparable embedding memory usage.

Figures & tables

Explore similar work

CardsList
  1. MemoryCard: Topic-Aware Multi-Modal Clue Compression for Long-Video Question Answering

    Jun 4, 2026Qing Yang, Pengcheng Huang, Xinze Li +6Video SummarizationVideo Question Answering

  2. O-MARC: Omni Memory-Augmented Compression Distillation for Efficient Video Understanding

    May 26, 2026Peiran Wu, Yunze Liu, Chi-Hao Wu +2Audio-Visual UnderstandingEfficient Multimodal Inference