Transfer Once, Ask Locally: Compact Audio-Visual Memory for On-Device Question Answering
Organizations: Dolby Laboratories, Inc. · University of Delaware
Abstract
Audio-visual question answering on lightweight user devices requires balancing speed, memory usage, privacy, and communication. One approach is to run audio-visual encoders on a server and a lightweight language decoder on the user device, keeping questions local while offloading media encoding. However, transferring and processing the full multimodal representation remains costly. We propose a training-free, query-agnostic token pruning framework that constructs a compact, reusable memory before transmission. We define video spatio-temporal blocks and aligned audio windows as players and estimate their importance through Monte Carlo Shapley valuation. We use an embedding coverage objective that combines query-independent utility with cross-modal redundancy calibration and rewards unimodal diversity through marginal coverage gains. We retain the original token embeddings of high-scoring players under a limited token budget and transfer the resulting memory once for reuse across subsequent local questions. Experiments on Video-MME, Daily-Omni, and Video-Holmes using video-SALMONN 2+ (3B) and video-SALMONN 2 Pro (4B) show that our method achieves the best or tied-best QA accuracy among the compared compression methods across all evaluated settings. On Daily-Omni with video-SALMONN 2+ (3B) at 80% pruning, our method improves accuracy by 12.86--14.62 percentage points over these methods while achieving the lowest device-side LLM prefill and decoding latency, with up to decoding speedup at comparable embedding memory usage.
Figures & tables
| Method | Pruning Ratio | Video-MME | Daily-Omni | Video-Holmes |
|---|---|---|---|---|
| video-SALMONN 2+ (3B) | ||||
| Full Tokens | 0% | 64.59 | 65.07 | 43.06 |
| DivPrune [ 1 ] | 61.70 | 52.30 | 39.63 | |
| VisionZip [ 27 ] | 59.04 | 51.04 | 38.54 | |
| OmniZip [ 22 ] | 61.62 | 52.80 | 39.51 | |
| Ours | 80% | 63.92 | 65.66 | 41.10 |
Audio-Visual QA
Audio-visual question answering (AVQA) combines sound and video to answer questions about events, speech, music, space, and intent.
See all 31 papers