CST: Collaborative Selective Transmission for Communication-Efficient Multimodal Edge Inference
Authors: Hai Chi, Junrui Zhang, Rui Ning, Chonggang Wang, Robert Gazda, Huanrui Yang, Hongyi Wu
Organizations: Department of Electrical and Computer Engineering, University of Arizona, Tucson, AZ, USA · Department of Computer Science, Old Dominion University, Norfolk, VA, USA · InterDigital, Conshohocken, PA, USA
Collaborative multimodal inference improves edge perception by combining observations from distributed sensing devices, but transmitting high-dimensional helper representations incurs substantial communication overhead and can lead to high end-to-end latency. Existing communication-efficient methods reduce payloads through compression, semantic coding, or feature selection, yet typically optimize compactness or task relevance without explicitly accounting for information already represented at the main device. Consequently, task-relevant but redundant helper features may still consume bandwidth. We present Collaborative Selective Transmission (CST), a main-directed query--response framework that retrieves only helper information complementary to the current main representation. Inspired by Partial Information Decomposition and the Multiview Redundancy Assumption, CST learns sample-adaptive, helper-specific sparse retrieval supports while discouraging retrieval of semantics already covered by the main device or duplicated across helpers. During inference, the main device transmits only support indices, and each helper returns the corresponding latent values, avoiding dense helper-feature exchange. Across three real-world multimodal sensing benchmarks, CST transmits no more than 14.18% of helper feature values while achieving best or near-best task performance among the evaluated methods. Experiments on a five-node NVIDIA Jetson Orin Nano testbed across 5--100 Mbps demonstrate up to a 4.27× speedup over Transmit-All in end-to-end inference, confirming practical end-to-end latency reductions.
Figures & tables
Fig. 1: Collaborative multimodal transmission with three representative MM-Fi [ 36 ] helpers in a 3D pose estimation example. (a) Transmit-All sends complete helper representations. (b) Representative methods use local selection, learned compression, or request-guided helper-side selection; the paths denote distinct method families. (c) CST generates helper-specific sparse supports from the current main representation and retrieves the corresponding values.
Fig. 2: Latency profiling on MM-Fi [ 36 ] in a network under a shared aggregate bandwidth of 5 Mbps, with RTT and jitter configured to 30 ms and 5 ms, respectively, and a batch size of 32. Communication dominates the end-to-end latency in all shown baseline methods, including Transmit-All, VDDIB-SR [ 26 ] , SINFONY [ 1 ] , and How2Comm [ 35 ] , whereas CST achieves a total latency of 7.71 s.
Fig. 3: Overview of CST, illustrated with an MM-Fi 3D pose estimation example using RGB as the main modality and LiDAR and WiFi-CSI as two representative helpers. The main representation generates a shared training-time redundancy anchor and helper-specific complementary anchors. During inference, the main device sends only the support indices of each complementary anchor, and the helpers return the requested latent values for residual fusion and task prediction. Additional helpers follow the same query–response path.
Fig. 4: Qualitative RGB-helper selection on DarkAct with Thermal as the main modality. Highlighted locations denote retained features. CST uses a smaller support concentrated on informative human and scene regions.
Dataset
Task
Modalities
Seq. Length ( Gi )
Metrics
MM-Fi [ 36 ]
3D Pose Estimation
RGB , Depth, LiDAR, mmWave, WiFi-CSI
32, 180, 32, 32, 32
MPJPE, PA-MPJPE
CUHK-S [ 12 ]
Action Recognition
Skeleton , Depth/Color, IR, IMU, Radar
32, 196, 196, 128, 20
Acc., F1, Prec., Rec.
DarkAct [ 29 ]
Action Recognition
Thermal , RGB
32, 32
Accuracy
TABLE I: Datasets, modality assignments, sequence lengths, and task-specific performance metrics. Main modalities are bold. Metric definitions follow the benchmark papers.
Method
MM-Fi
CUHK-S
DarkAct
MPJPE ↓
PA-MPJPE ↓
Tx (%) ↓
Acc. (%) ↑
F1 (%) ↑
Prec. (%) ↑
Rec. (%) ↑
Tx (%) ↓
Acc. (%) ↑
Tx (%) ↓
Main-Only
119.20
68.94
–
66.28
63.43
65.50
63.90
–
64.55
–
Transmit-All
87.35
37.06
100.00
67.11
63.99
65.18
64.83
100.00
67.84
100.00
BottleNet++ [ 27 ]
86.13
38.71
50.00
68.29
66.78
69.96
67.74
50.00
66.45
50.00
VIB-Dyn [ 25 ]
87.72
44.28
30.66
72.82
68.52
69.69
69.90
30.86
66.35
30.66
VDDIB-SR [ 26 ]
86.39
38.76
31.21
70.81
67.29
68.64
68.89
30.82
66.45
45.34
TABLE II: Task performance and transmission rate on three benchmarks. Tx reports the transmitted representation size as a percentage of the complete helper representations. MM-Fi errors are in millimeters; CUHK-S F1, precision, and recall are macro-averaged. Actual bidirectional traffic for representative methods measured on the testbed is reported separately in Table III . ↑ and ↓ indicate whether higher or lower values are preferred, respectively. Best observed values are bold.
Method
MM-Fi
CUHK-S
DarkAct
Transmit-All
565.59
1106.30
65.65
BottleNet++
282.97
553.33
32.84
VIB-Dyn
173.67
341.65
20.17
VDDIB-SR
172.68
341.49
29.97
SINFONY
282.97
553.33
32.84
DynamicViT
189.91
482.26
36.05
TABLE III: Mean bidirectional traffic (kilobytes per sample) on the testbed, including requests, responses, framing, and metadata.
Fig. 5: Mean per-sample end-to-end latency, computed as the mean complete-batch latency divided by the batch size of 32, versus shared aggregate bandwidth on MM-Fi and CUHK-S. RTT and jitter are configured to 30 ms and 5 ms, respectively.
Method
10 ms
30 ms
50 ms
100 ms
Mean
p95
Mean
p95
Mean
p95
Mean
p95
Transmit-All
8.95
9.31
8.76
8.78
8.80
8.83
8.88
8.92
SINFONY
5.03
5.37
4.99
4.99
4.99
5.01
5.12
5.20
DynamicViT
3.78
4.13
3.89
4.27
3.87
4.31
3.99
4.39
How2Comm
5.31
5.71
5.36
5.75
5.41
5.78
5.62
6.01
CST
2.86
2.89
3.05
3.10
2.97
3.00
3.26
3.29
TABLE IV: Mean and p95 complete-batch end-to-end latency on MM-Fi under varying configured RTTs. The bandwidth is 10 Mbps, the jitter target is 5 ms, and the batch size is 16. Values are in seconds and computed over 20 measured batches.
Fig. 6: Mean complete-batch end-to-end latency versus batch size on MM-Fi and CUHK-S. The shared aggregate bandwidth is 10 Mbps, with configured RTT and jitter targets of 30 ms and 5 ms, respectively.
Variant
Acc. ↑
F1 ↑
Prec. ↑
Rec. ↑
Tx (%) ↓
Full CST
72.99
68.59
70.81
69.10
8.66
Direct- Ac
72.32
67.90
70.79
68.59
51.82
Direct- Ac + Laux
72.48
68.27
70.72
68.90
50.79
w/o Lsim
72.31
67.57
70.42
67.85
42.34
w/o Lortho
72.32
68.03
70.72
69.04
36.62
w/o Laux
72.82
68.30
70.17
68.91
37.26
TABLE V: CUHK-S ablation results. Direct- Ac uses only Ltask and Lsparsity ; the second variant retains Laux while removing the redundancy branch. All values are percentages, and best values are bold.
Method
UH
S
C=UH+S↑
Tx (%) ↓
Transmit-All
1.783±0.078
1.464±0.057
3.247±0.125
100.00
BottleNet++
1.602±0.044
1.406±0.046
3.008±0.078
50.00
VIB-Dyn
1.576±0.047
1.403±0.047
2.979±0.078
30.66
SINFONY
1.651±0.111
1.375±0.078
3.026±0.166
50.00
DT-JSCC
1.659±0.133
1.363±0.077
3.023±0.197
50.00
DynamicViT
1.712±0.154
1.290±0.088
3.002±0.216
56.03
TABLE VI: Post-hoc Flow-PID results on DarkAct, reported in bits as mean ± standard deviation over three fixed Flow initializations. Upward and downward arrows indicate that higher and lower values are preferred, respectively. The largest mean C and lowest Tx among communication-efficient methods are bold.
Communication-efficient collaborative perception studies how networked agents share useful sensor information with minimal bandwidth while improving their joint view of the scene.
Recent advances in multimodal large language mod- els (MLLMs) have opened new opportunities for edge intelligence by enabling reasoning across heterogeneous sensor modalities, such as vision, text, and telemetry data. However, deploying these capabilities on resource-constrained edge platforms remains challenging due to the substantial computational, memory, and communication demands of modern MLLMs. Rather than transmitting raw sensor observations or partitioning neural networks at intermediate layers, Edge Multi-Modal Intelligence (EMMI) communicates a compact representation between edge devices and server resources, enabling communication-efficient edge MLLM inference. To achieve this, EMMI performs modality-specific encoding, cross-modal representation fusion, and learned compression at the edge, transmitting only a compact latent representation to server-side resources for high-capacity MLLM reasoning. This representation-centric design reduces communication overhead, preserves local data privacy, and provides a fixed-size interface between heterogeneous edge devices and server-side MLLMs. Evaluation on a representative multimodal benchmark demonstrates that EMMI can reduce the communication payload by 32x while maintaining comparable downstream accuracy, resulting in up to a 3.4x reduction in estimated end-to-end inference latency under bandwidth-constrained edge conditions.
Large multimodal models (LMMs) support diverse visual understanding and reasoning tasks but are often impractical to run entirely on resource-constrained devices. Device-edge co-inference reduces device computation, yet transmitting visual data over bandwidth-limited uplinks can introduce substantial delay. Task-oriented feature compression (TOFC) reduces the payload through feature aggregation and entropy coding. However, continuous-feature coding remains costly, and query-agnostic aggregation may discard task-relevant local evidence. We propose query-guided task-oriented feature compression (Q-TOFC) for device-edge multimodal inference. Q-TOFC employs residual vector quantization (RVQ) to encode each merged feature as a compact sequence of codebook indices, reducing its representation cost and allowing more features to be transmitted. It further incorporates query relevance into feature aggregation and uses a quantization error compensation adapter to mitigate the distortion introduced by discrete quantization. Experiments on seven multimodal benchmarks show that Q-TOFC reduces the visual payload by 53.6% relative to TOFC while maintaining comparable average normalized task performance. End-to-end latency evaluations further demonstrate lower latency under bandwidth-constrained uplinks.
Luning Pang, Cheng Yuan, Jiawei Shao +2
Department of Electronic Engineering, and Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China · Institute of Artificial Intelligence (TeleAI), China Telecom, Beijing 100033, China
Edge devices increasingly run multimodal sensing pipelines that must remain accurate despite fluctuating power budgets and unpredictable sensor dropout. Existing pruning methods fail under these conditions: they generally require fine-tuning after compression, consuming over 10× the deployment energy, and they assign static importance scores that are blind to which sensors are present. We present the SentryFuse framework, which addresses both challenges jointly through two key components. First, SentryGate learns modality-conditioned importance scores during training via first-order saliency supervision and then prunes attention heads and feed-forward channels at deployment without fine-tuning. Second, SentryAttend replaces dense self-attention, a key bottleneck in contemporary multimodal architectures, with sparse grouped-query attention, yielding a net 15% reduction in GFLOPs across three different multimodal architectures. Across three applications and multimodal backbones, SentryGate achieves a 12.7% average accuracy improvement over the strongest pruning baseline, and upto to 18% under modality dropout conditions. Together, SentryFuse reduces memory by 28.2% and lowers latency by up to 1.63× without further fine-tuning, establishing modality-aware zero-shot compression as a practical path to multimodal intelligence on heterogeneous edge hardware.