Organizations: School of Computing and Communications, Lancaster University, LA1 4YW, U.K. · School of Communication and Information Engineering, Shanghai University, 200444, China · Department of Computer Science, University of Exeter, EX4 4QF, U.K. · KU 6G Research Center, Khalifa University of Science and Technology, 127788, Abu Dhabi, UAE
Multi-source semantic information fusion enables edge networks to support global inference using observations collected by distributed terminals. However, limited communication and computation resources restrict semantic uploading, leaving the edge server with incomplete tokens that differ in modality, semantic granularity, arrival time, and channel condition. The value of these tokens also depends on the complementary or redundant information provided by other terminals. In this paper, we propose a multi-source and multi-dimensional semantic information fusion framework that coordinates terminal uploading and edge-side fusion under resource constraints. We develop Hyper-Slot to combine semantic content with available token attributes through iterative token-to-slot assignment and slot-to-token feedback, producing a compact representation for global inference. To encourage useful semantic uploading, we further design a contribution-based incentive mechanism that uses Hyper-Slot assignments to evaluate information coverage and cross-type complementarity. A bi-level Stackelberg-Nash game determines the server reward pool and terminal transmission-quality strategies while accounting for computation and communication energy costs. Experiments on a tri-modal extension of COCO2014 and image-text retrieval on Flickr30K show that Hyper-Slot improves mAP by 3.66 percentage points and RSum by 15.9 points over the strongest evaluated baselines, respectively, while reducing the number of parameters by more than 80% compared with a naive concatenation-based fusion baseline.
Figures & tables
Fig. 1: Overview of the proposed multi-source and multi-dimensional semantic information fusion framework.
Fig. 2: Structure of the proposed Hyper-Slot module. Semantic tokens are assigned to latent slots using semantic content and token attributes, and iteratively aggregated into a compact representation for global inference.
Method
mAP over random seeds
Efficiency
Seed 42
Seed 123
Seed 456
Seed 789
Seed 1024
Mean ± Std
Params (M)
GFLOPs
Time (ms)
Memory (MB)
Mean Pool [ 34 ]
0.5431
0.5349
0.6876
0.6964
0.6920
0.6308 ± 0.0839
1.193
0.0582
0.0300
132.62
Concat MLP [ 33 ]
0.6345
0.6309
0.6386
0.6536
0.6568
0.6429 ± 0.0116
22.034
0.0999
0.0274
450.40
SCAN [ 17 ]
0.6564
0.6706
0.7599
0.7702
0.7651
0.7244 ± 0.0560
3.365
0.3364
0.0880
165.53
Slot Attention [ 22 ]
0.7009
0.6969
0.7312
0.7502
0.7407
0.7240 ± 0.0239
1.860
0.1526
0.0565
143.38
HGNN [ 21 ]
0.6548
0.6469
0.7021
0.7195
0.7108
0.6868 ± 0.0336
2.183
0.2263
0.0720
147.94
TABLE I: Comparison of mAP and computational efficiency over five random seeds.
Fig. 3: Comparison of per-class mAP across five categories on COCO2014 dataset.
Fig. 4: Visualization of token-to-slot assignments in Hyper-Slot. Colors indicate the assigned slots, and star markers denote the slot anchors.
Fig. 5: Effect of heterogeneity descriptor configurations on mAP.
Fig. 6: Comparison of token selection strategies under different token transmission budgets. The dashed line indicates the full-transmission performance.
Fig. 7: Structural ablation of Hyper-Slot: mAP over training epochs for the compact-only variant, the dependency-only variant, and the full model.
Method
RSum over random seeds
Efficiency
Seed 42
Seed 123
Seed 456
Seed 789
Seed 1024
Mean ± Std
Params (M)
GFLOPs
Time (ms)
Memory (MB)
VSE++ [ 36 ]
411.2
417.5
412.9
419.0
415.1
415.1 ± 3.2
2.886
0.1825
0.0330
31.39
Concat MLP [ 34 ]
428.6
433.1
430.5
436.2
432.4
432.2 ± 2.9
80.484
0.3377
0.0632
330.15
GPO [ 34 ]
446.8
451.5
454.2
456.0
450.9
451.9 ± 3.5
2.911
0.1826
0.1267
50.52
HGNN [ 21 ]
436.4
443.1
439.8
446.2
441.5
441.4 ± 3.7
5.018
0.3056
0.1025
49.13
Slot Attention [ 22 ]
458.7
463.4
461.2
467.0
464.8
463.0 ± 3.2
26.006
1.3289
0.3087
132.91
TABLE II: Comparison of RSum and computational efficiency on Flickr30K over five random seeds.
Fig. 8: Effect of the reward pool on server utility, fusion benefit, total system cost, and equilibrium resource utilization.
Fig. 9: Terminal contributions and reward allocation under the uploading strategies
Edge-cloud hybrid inference offloads difficult inputs to a powerful remote model, but the uplink channel imposes hard per-request constraints on the number of bits that can be transmitted. We show that selecting transmitted content based solely on attention-based importance, the standard approach in collaborative inference, is inherently limited under hard budgets. Two findings support this claim. First, replacing high-importance units with low-importance but complementary ones improves server accuracy. This shows that what matters is not individual importance but how well the transmitted set covers diverse aspects of the input. Second, spatially uniform selection without any content information achieves competitive accuracy at moderate budgets. This confirms that spatial coverage alone carries independent value. Based on this analysis, we propose SAGE (Semantic Attention-Guided Evidence), a principled, training-free method that combines importance filtering with embedding-diversity sampling. SAGE achieves 93% of the server ceiling in offloaded accuracy while transmitting fewer than half of the available evidence units on ImageNet-1K, substantially outperforming importance-only composition.
Inhyeok Choi, Hyuncheol Park
School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea
Autonomous systems and smart-industry deployments increasingly split computation across near-sensor, edge, and cloud resources, where tight energy, latency, and reliability budgets demand run-time adaptivity. In practice, deciding what to compute and transmit at each point is pivotal; yet as multimodal sensor suites (cameras, LiDAR/depth, etc.) proliferate at the edge, most prior approaches either (i) fuse modalities on powerful servers or (ii) apply uni-modal near-sensor filters that ignore cross-modal dependencies, leading to redundant transmissions or missed events. We present FusionSense, a fusion-aware intelligent sensing framework for energy-constrained autonomous edge systems. Lightweight near-sensor classifiers are trained via a three-step procedure: (i) a server-side fusion model learns the downstream task, (ii) filter-out-safe (FoS) labels quantify each modality's necessity relative to the fused decision, and (iii) an edge-side fusion model is compacted by injecting near-sensor predictions as auxiliary signals. The result is a run-time decision layer that jointly reduces compute and communication while scaling linearly with sensor count. On a dual-modality (RGB+Depth/LiDAR) setup with SynDrone, FusionSense sustains task quality at substantially higher data-reduction rates than uni-modal filters and delivers large end-to-end gains: up to 33x lower energy at 1% FoI prevalence, 11x at 10%, a 92.3% reduction in quality loss at a fixed 30% data reduction, and roughly 1.5x higher energy savings than the best prior filtering baseline.
Sanggeon Yun, Ryozo Masukawa, Minhyoung Na +5
University of California, Irvine Irvine, CA, USA · Kookmin University Seoul, South Korea · Shibaura Institute of Technology Saitama, Japan
Collaborative multimodal inference improves edge perception by combining observations from distributed sensing devices, but transmitting high-dimensional helper representations incurs substantial communication overhead and can lead to high end-to-end latency. Existing communication-efficient methods reduce payloads through compression, semantic coding, or feature selection, yet typically optimize compactness or task relevance without explicitly accounting for information already represented at the main device. Consequently, task-relevant but redundant helper features may still consume bandwidth. We present Collaborative Selective Transmission (CST), a main-directed query--response framework that retrieves only helper information complementary to the current main representation. Inspired by Partial Information Decomposition and the Multiview Redundancy Assumption, CST learns sample-adaptive, helper-specific sparse retrieval supports while discouraging retrieval of semantics already covered by the main device or duplicated across helpers. During inference, the main device transmits only support indices, and each helper returns the corresponding latent values, avoiding dense helper-feature exchange. Across three real-world multimodal sensing benchmarks, CST transmits no more than 14.18% of helper feature values while achieving best or near-best task performance among the evaluated methods. Experiments on a five-node NVIDIA Jetson Orin Nano testbed across 5--100 Mbps demonstrate up to a 4.27× speedup over Transmit-All in end-to-end inference, confirming practical end-to-end latency reductions.
Hai Chi, Junrui Zhang, Rui Ning +4
Department of Electrical and Computer Engineering, University of Arizona, Tucson, AZ, USA · Department of Computer Science, Old Dominion University, Norfolk, VA, USA · InterDigital, Conshohocken, PA, USA