Modality-Aware Zero-Shot Pruning and Sparse Attention for Efficient Multimodal Edge Inference

Yueyuan Sui; Payal Mohapatra; Doğaç Eldenk; Haodong Yang; Yiting Zhang; Haoyan Zhang; Qi Zhu; Stephen Xia

cs.LGApr 10, 2026

Modality-Aware Zero-Shot Pruning and Sparse Attention for Efficient Multimodal Edge Inference

Authors: Yueyuan Sui, Payal Mohapatra, Doğaç Eldenk, Haodong Yang, Yiting Zhang, Haoyan Zhang, Qi Zhu, Stephen Xia

Organizations: Northwestern University, Evanston, Illinois, USA

Abstract

Edge devices increasingly run multimodal sensing pipelines that must remain accurate despite fluctuating power budgets and unpredictable sensor dropout. Existing pruning methods fail under these conditions: they generally require fine-tuning after compression, consuming over $10\times$ the deployment energy, and they assign static importance scores that are blind to which sensors are present. We present the SentryFuse framework, which addresses both challenges jointly through two key components. First, SentryGate learns modality-conditioned importance scores during training via first-order saliency supervision and then prunes attention heads and feed-forward channels at deployment without fine-tuning. Second, SentryAttend replaces dense self-attention, a key bottleneck in contemporary multimodal architectures, with sparse grouped-query attention, yielding a net 15% reduction in GFLOPs across three different multimodal architectures. Across three applications and multimodal backbones, SentryGate achieves a 12.7% average accuracy improvement over the strongest pruning baseline, and upto to 18% under modality dropout conditions. Together, SentryFuse reduces memory by 28.2% and lowers latency by up to $1.63\times$ without further fine-tuning, establishing modality-aware zero-shot compression as a practical path to multimodal intelligence on heterogeneous edge hardware.

Figures & tables

Figure 1. Multimodal ( Mohapatra et al., 2025b ) vs. Multivariate ( Vaswani et al., 2017 ) modeling of heterogeneous sensor data under missingness.

Figure 2. Time-spent profile during the Transformer forward pass in various multimodal architectures.

Figure 3. Performance of modality-aware pruning ( SentryGate ) and modality-agnostic pruning (SynFlow ( Tanaka et al., 2020 ) ) applied to transformer heads in a multimodal baseline (MAESTRO ( Mohapatra et al., 2025b ) ) (a) with all modalities present and (b) with 40% modality dropout.

Figure 4. Total energy utilization in deploying a pruned model (23%) with a fine-tuning (3 epochs) and in zero-shot across (a) Jetson TX2 and (b) an NVIDIA L40 GPU.

Figure 5. Overview of the SentryFuse framework. I. During training (left), the multimodal backbone integrates SentryAttend , replacing dense self-attention with sparse grouped-query attention for an average 13% FLOP reduction. Simultaneously, SentryGate runs in observer mode, learning modality-conditioned structural importance under a curriculum of increasing modality dropout, guided by a saliency alignment loss. The backbone is optimized via the task loss $\mathcal{L}_{\text{cls}}$ , while gate parameters receive gradients only from the alignment objective. II. At inference (right), SentryGate is queried once offline to produce a subnetwork given available modalities and a target compute budget, enabling a train-once, deploy-many paradigm that adapts to (i) varying compute constraints and (ii) arbitrary sensor availability across heterogeneous edge devices.

Figure 6. Empirical long-tail self-attention patterns from two representative heads show that only a small fraction carry significant probability mass while the majority are nearly zero, motivating SentryAttend .

Figure 7. Attention weight heatmaps (head index vs. time/token) for a a modality-encoder (wrist BVP from WESAD dataset ( Schmidt et al., 2018 ) ) at three stages. (a) Dense baseline (6.175 GFLOPs): attention is distributed uniformly across all 8 heads with low per-head magnitude. (b) Sparse attention prior to SentryGate pruning (5.23 GFLOPs): inter-head variation emerges, with select heads becoming dominant, but all heads remain active. (c) SentryGate -pruned attention (3.81 GFLOPs, 28% head pruning): redundant heads are fully gated out (white columns), concentrating computation on salient heads and timesteps while preserving discriminative temporal structure.

Figure 8. Accuracy of FlexMoE+ SentryGate , FuseMoE+ SentryGate , and MAESTRO+ SentryGate under different numbers of missing modalities (0, 1, 2, and 4), on (a) WESAD, (b) DaliaHAR, and (c) DSADS.

Backbone	Strategy	Full				Drop 1				Drop 2				Drop 4
Backbone	Strategy	6	12	17	23	6	12	17	23	6	12	17	23	6	12	17	23
FlexMoE	Random	0.62	0.62	0.59	0.63	0.62	0.62	0.58	0.59	0.61	0.61	0.56	0.55	0.53	0.49	0.43	0.37
	Magnitude	0.61	0.61	0.56	0.55	0.61	0.61	0.56	0.55	0.59	0.59	0.56	0.55	0.51	0.51	0.46	0.46
	Synflow	0.56	0.56	0.54	0.54	0.56	0.56	0.54	0.55	0.53	0.54	0.38	0.38	0.55	0.52	0.49	0.49
	SentryGate	0.61	0.61	0.62	0.65	0.61	0.61	0.62	0.65	0.59	0.59	0.60	0.63	0.52	0.52	0.52	0.54
FuseMoE	Random	0.59	0.59	0.57	0.57	0.59	0.59	0.57	0.57	0.51	0.48	0.49	0.46	0.61	0.58	0.59	0.53

Table 1. WESAD accuracy under modality dropout and pruning ratios of 6%, 12%, 17%, and 23%. Orange marks backbone-wise local best; bold orange marks global best.

Backbone	Strategy	Full				Drop 1				Drop 2				Drop 4
Backbone	Strategy	6	12	17	23	6	12	17	23	6	12	17	23	6	12	17	23
FlexMoE	Random	0.58	0.47	0.43	0.22	0.57	0.49	0.47	0.23	0.57	0.48	0.46	0.22	0.39	0.30	0.30	0.40
	Magnitude	0.47	0.36	0.28	0.18	0.48	0.38	0.31	0.19	0.47	0.36	0.30	0.19	0.41	0.38	0.28	0.17
	Synflow	0.61	0.57	0.35	0.17	0.62	0.58	0.37	0.17	0.61	0.58	0.38	0.16	0.58	0.53	0.32	0.16
	SentryGate	0.72	0.68	0.57	0.24	0.71	0.68	0.56	0.23	0.71	0.66	0.56	0.34	0.61	0.55	0.44	0.36
FuseMoE	Random	0.49	0.60	0.47	0.44	0.50	0.61	0.46	0.43	0.47	0.56	0.42	0.39	0.40	0.22	0.34	0.31

Table 2. DaliaHAR accuracy under modality dropout and pruning ratios of 6%, 12%, 17%, and 23%. Orange marks backbone-wise local best; bold orange marks global best.

Backbone	Strategy	Full				Drop 1				Drop 2				Drop 4
Backbone	Strategy	6	12	17	23	6	12	17	23	6	12	17	23	6	12	17	23
FlexMoE	Random	0.81	0.13	0.09	0.07	0.78	0.12	0.11	0.09	0.64	0.14	0.11	0.10	0.41	0.08	0.08	0.08
	Magnitude	0.84	0.55	0.16	0.11	0.82	0.52	0.19	0.14	0.73	0.34	0.14	0.12	0.46	0.22	0.16	0.10
	Synflow	0.84	0.52	0.18	0.13	0.80	0.60	0.14	0.12	0.71	0.44	0.16	0.14	0.40	0.30	0.11	0.09
	SentryGate	0.86	0.86	0.57	0.23	0.83	0.81	0.49	0.19	0.78	0.75	0.42	0.12	0.49	0.45	0.21	0.16
FuseMoE	Random	0.81	0.49	0.77	0.74	0.75	0.73	0.71	0.70	0.70	0.66	0.63	0.55	0.46	0.39	0.36	0.31

Table 3. DSADS accuracy under modality dropout and pruning ratios of 6%, 12%, 17%, and 23%. Orange marks backbone-wise local best; bold orange marks global best.

Dataset	Backbone	Orig.		+ SentryAttend
Dataset	Backbone	Acc $\uparrow$	GFLOPs $\downarrow$	Acc $\uparrow$	GFLOPs $\downarrow$
WESAD	MAESTRO	0.75	6.15	0.76	5.23
	Transformer	0.67	22.25	0.65	15.81
	iTransformer	0.61	5.76	0.62	5.06
	FlexMoE	0.65	16.68	0.67	16.35
	FuseMoE	0.68	12.28	0.65	11.47

Table 4. Accuracy and GFLOPs before and after SentryAttend on WESAD, DaliaHAR, and DSADS. Higher accuracy and lower GFLOPs are better; best and second-best values are shown in bold orange and underlined, respectively.

Metric	MAESTRO	+ SentryGate	+ SentryFuse
Mem. (MB) $\downarrow$	6.07	4.80	4.36
GFLOPs $\downarrow$	6.83	5.07	4.41
Accuracy / Latency (ms) $\downarrow$
GPU	0.75/24.50	0.75/24.35	0.77/24.32
CPU	0.74/60.19	0.74/43.96	0.76/42.72
Jetson	0.74/295.87	0.74/257.82	0.76/254.77

Table 5. WESAD hardware deployment (MAESTRO; latter two use 23% pruning; SentryFuse = SentryGate + SentryAttend ).

Figure 9. Accuracy–storage Pareto trade-offs for SentryFuse under post-training quantization on (a) WESAD, (b) DSADS and (c) DaliaHAR. Each point represents a model with a specific pruning ratio and numerical precision (FP32, FP16, FP8). The dashed line indicates the Pareto frontier across all configurations.

Figure 10. Latency, average power draw, and total energy consumption of SentryFuse under FP32 and FP16 quantization across pruning ratios.

Figure 11. Accuracy of random pruning, our SentryFuse , and a Taylor-based saliency teacher across pruning ratios on DaliaHAR under (a) full modalities and (b) 4-modality dropout.

Figure 12. Accuracy of SentryFuse under different grouped-query attention configurations (1, 2, and 8 key-value groups) across pruning ratios on DaliaHAR under (a) full modalities and (b) 4-modality dropout.

Explore similar work

FusionSense: Tri-Stage Near-Sensor Learning for Runtime-Adaptive Multimodal Edge Intelligence
May 19, 2026Sanggeon Yun, Ryozo Masukawa, Minhyoung Na +5Efficient Multimodal Inference Edge Computing
University of California, Irvine Irvine, CA, USA · Kookmin University Seoul, South Korea · Shibaura Institute of Technology Saitama, Japan
EMMI: Edge Multi-Modal Intelligence for Communication-Efficient MLLM Inference via Fused Representation Compression
Sep 10, 2026Motahare Mounesan, Irfan KhanEfficient Multimodal Inference Edge Computing
Texas A&M University
Beyond Independent Optimization: Compression, MoE Routing, and Quantization Interactions in Multimodal Edge Intelligence
Jul 23, 2026Jay Gor, Karm Dave, Akshita Abrol +3Efficient Multimodal Inference VLM Quantization
Nirma University · Singapore Institute of Technology, Singapore · Department of CE-AI and Big Data, Marwadi University

May 19, 2026J/K move · Enter open · S save

FusionSense: Tri-Stage Near-Sensor Learning for Runtime-Adaptive Multimodal Edge Intelligence

Sanggeon Yun, Ryozo Masukawa, Minhyoung Na, Hyunwoo Oh+4

University of California, Irvine Irvine, CA, USA · Kookmin University Seoul, South Korea · Shibaura Institute of Technology Saitama, Japan

Efficient Multimodal Inference Edge Computing Multimodal Sensor Fusion

Open paper AskPDF arXiv

Autonomous systems and smart-industry deployments increasingly split computation across near-sensor, edge, and cloud resources, where tight energy, latency, and reliability budgets demand run-time adaptivity. In practice, deciding what to compute and transmit at each point is pivotal; yet as multimodal sensor suites (cameras, LiDAR/depth, etc.) proliferate at the edge, most prior approaches either (i) fuse modalities on powerful servers or (ii) apply uni-modal near-sensor filters that ignore cross-modal dependencies, leading to redundant transmissions or missed events. We present FusionSense, a fusion-aware intelligent sensing framework for energy-constrained autonomous edge systems. Lightweight near-sensor classifiers are trained via a three-step procedure: (i) a server-side fusion model learns the downstream task, (ii) filter-out-safe (FoS) labels quantify each modality's necessity relative to the fused decision, and (iii) an edge-side fusion model is compacted by injecting near-sensor predictions as auxiliary signals. The result is a run-time decision layer that jointly reduces compute and communication while scaling linearly with sensor count. On a dual-modality (RGB+Depth/LiDAR) setup with SynDrone, FusionSense sustains task quality at substantially higher data-reduction rates than uni-modal filters and delivers large end-to-end gains: up to 33x lower energy at 1% FoI prevalence, 11x at 10%, a 92.3% reduction in quality loss at a fixed 30% data reduction, and roughly 1.5x higher energy savings than the best prior filtering baseline.

Abstract

Explore similar work

FusionSense: Tri-Stage Near-Sensor Learning for Runtime-Adaptive Multimodal Edge Intelligence

EMMI: Edge Multi-Modal Intelligence for Communication-Efficient MLLM Inference via Fused Representation Compression

Beyond Independent Optimization: Compression, MoE Routing, and Quantization Interactions in Multimodal Edge Intelligence