Organizations: School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan 430074, China · Peng Cheng Laboratory, Shenzhen, China · Pazhou Laboratory (Huangpu), Guangzhou, China · School of Mechanical Engineering and Electronic Information, China University of Geosciences, Wuhan 430074, China · Peng Cheng Laboratory, Shenzhen 518055, China · State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China
Semantic communication shifts the focus from bit-level accuracy to task-relevant semantic delivery, enabling efficient and intelligent communication for next-generation networks. However, existing multi-modal solutions often process all available data modalities indiscriminately, ignoring that their contributions to downstream tasks are often unequal. This not only leads to severe resource inefficiency but also degrades task inference performance due to irrelevant or redundant information. To tackle this issue, we propose a novel task-aware distributed information bottleneck (TADIB) framework, which quantifies the contribution of any set of modalities to given tasks. Based on this theoretical framework, we design a practical coding scheme that intelligently selects and compresses only the most task-relevant modalities at the transmitter. To find the optimal selection and the codecs in the network, we adopt the probabilistic relaxation of discrete selection, enabling distributed encoders to make coordinated decisions with score function estimation and common randomness. Extensive experiments on public datasets demonstrate that our solution matches or surpasses the inference quality of full-modal baselines while significantly reducing communication and computational costs.
Figures & tables
Fig. 1: Performance improvement and computational overhead with respect to the number of modalities on MM-Fi. The more modalities involved (from left to right), the higher the computational cost. However, inference performance does not increase monotonically when redundant, difficult-to-learn, or irrelevant modalities (e.g., RGB images) are added.
Image recovering/classification + machine translation + video sentiment analysis
No
DL-based
CCMT [ 13 ]
Multi-view images
Multi-task image classification
No
DL-based + Infomax
TABLE I: Comparative Analysis of Representative Related Works in Semantic Communication
Symbol
Description
X,Y,Z
Generic source observation, task target variable, and semantic representation.
H(⋅∣⋅),I(⋅;⋅)
Conditional entropy and mutual information.
DKL(P∥Q)
Kullback-Leibler divergence from Q to P .
K,K,k
Set, number, and index of distributed transmitters (encoders).
M,M,m
Modality set, number of local modalities, and modality index.
T,T,t
Task/receiver set, number, and index.
TABLE II: Main Notations
Fig. 2: System model of a multi-modal task-aware semantic communication.
Fig. 3: Task-aware modality selection in general distributed networks. Modalities (tasks) are distributed at different transmitters (receivers).
(a) AV-MNIST
Method
Under limits
R
Rate
N-CE
Acc 1
Acc 2
Acc 3
p TADIB (CR)
Yes
0.56 ± 0.05
166.78 ± 0.96
-0.39 ± 0.05
98.60 ± 1.89
99.39 ± 2.17
96.50 ± 1.59
p TADIB (w/o CR)
Yes
0.91 ± 0.05
185.42 ± 1.54
-0.72 ± 0.05
97.50 ± 1.61
93.50 ± 3.64
87.75 ± 4.52
RS-DIB
Yes
0.94 ± 0.03
141.76 ± 1.08
-0.70 ± 0.03
87.20 ± 2.49
86.20 ± 3.95
92.60 ± 1.70
Greedy-DIB
Yes
0.66 ± 0.03
186.05 ± 1.23
-0.47 ± 0.03
98.25 ± 1.39
97.81 ± 2.45
96.91 ± 1.50
MI-DIB-C
Yes
0.77 ± 0.03
219.11 ± 1.02
-0.55 ± 0.03
89.99 ± 2.68
95.19 ± 3.17
95.68 ± 2.50
TABLE III: Performance comparison on (a) AV-MNIST and (b) MM-Fi (95% confidence intervals with 10 independent runs). “Under limits” indicates satisfaction of both the transmitter-side and receiver-side access constraints. Lower values are better for R , sum-rate, MPJPE, and PA-MPJPE, whereas higher values are better for N-CE and accuracy. Within the access-feasible group, the best value in each metric is shown in bold, all ties are bolded.
method
(a) AV-MNIST
(b) MM-Fi
time
pairs
cost
time
pairs
cost
p TADIB
1.64
8.00
1.81 × 10 7
5.12
5.00
6.20 × 10 7
∼ w/o CR
2.20
7.10
1.66 × 10 7
5.10
4.86
4.74 × 10 7
MI-DIB
11.69
18.00
3.47 × 10 7
22.15
24.00
1.24 × 10 8
VDDIB
17.52
27.00
4.95 × 10 7
24.87
48.00
2.40 × 10 8
TABLE IV: Physical training time, the number of active pairs, and computational cost on (a) AV-MNIST and (b) MM-Fi datasets. Training time is measured in seconds per epoch (s/epoch). The number of active modality-task matching pairs is averaged over 10 independent runs. The average computational cost is measured by FLOPs. The lowest value in each dataset-metric column is shown in bold.
K
candidate pairs
parameters
time
peak memory
cost
3
27
16.30
1.53
318.94
1.87 ×107
6
54
27.03
1.28
436.55
2.26 ×107
9
81
37.76
1.48
600.99
2.46 ×107
15
135
59.21
1.32
927.93
2.87 ×107
TABLE V: Controlled network-size scaling of p TADIB on AV-MNIST: Physical training time is measured in seconds per epoch (s/epoch), number of parameters is measured in millions (M), peak GPU memory is measured in MiB, and computational cost is measured in FLOPs.
Fig. 4: Rate-relevance tradeoffs on (a) AV-MNIST and (b) MM-Fi. Dashed curves marked by † emphasize unconstrained baselines.
Fig. 5: Training dynamics on information plane (sum-rate vs N-CE).
metric/ratio
1/16
1/8
1/4
1/2
1/1
2/1
4/1
(a)
sum-rate
174.43
187.97
183.25
165.08
140.42
142.47
143.13
N-CE
-0.72
-0.67
-0.61
-0.45
-0.92
-0.96
-0.95
(b)
sum-rate
93.92
72.71
47.60
49.31
52.50
53.12
76.75
N-CE
-0.40
-0.26
-1.81
-1.88
-1.98
-1.87
-0.31
TABLE VI: Overall performance of p TADIB varying with learning rate ratio r from 1/16 to 4/1 on (a) AV-MNIST and (b) MM-Fi datasets.
Fig. 6: Convergence of log-likelihood ( 19 ) (log-likelihood vs iteration round).
Fig. 7: Comparison of converged A with and without CR
Fig. 8: Convergence of randomized selection A on different datasets
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Fig. 9: Converged A without access limits at β =1e-3. For AV-MNIST, rate: 172.27, N-CE: -0.15. For MM-Fi, rate: 78.17, N-CE: -0.36.
Fig. 10: Converged A without access limits at β→ 0. For AV-MNIST, rate: 1690.59, N-CE: -0.08. For MM-Fi, rate: 4592.59, N-CE: -0.14.
Fig. 11: Capacity dependence of empirical TADIB score fidelity on AV-MNIST. (a) Spearman ordering correlation of candidate scores. Higher correlation indicates closer score ordering to the Base proxy. (b) Simple regret evaluated under the Base proxy.
Fig. 12: Channel robustness on AV-MNIST. † emphasizes the unconstrained VDDIB reference.
Fig. 13: Impact of β on the empirical TADIB matching score Rβ on AV-MNIST. Each group compares the four access-feasible methods.
School of Electronics and Information Technology, Sun Yat-sen University, China · EURECOM, Sophia Antipolis, France · Singapore University of Technology and Design, Singapore
the School of Artificial Intelligence at the Xidian University, Shaanxi 710071, China · the School of Electronic Information and Communications at the Huazhong University of Science and Technology, Wuhan 430074, China · the Pengcheng Laboratory, Shenzhen, Guangdong 518055, China +2