TokenCom: Vision-Language Model for Multimodal and Multitask Token Communications
Authors: Feibo Jiang, Siwei Tu, Li Dong, Xiaolong Li, Kezhi Wang, Cunhua Pan, Zhu Han, Jiangzhou Wang
Organizations: Hunan Provincial Key Laboratory of Intelligent Computing and Language Information Processing, Hunan Normal University, Changsha, China. · School of Information Science and Engineering, Hunan Normal University, Changsha, China. · Changsha Social Laboratory of Artificial Intelligence, Hunan University of Technology and Business, Changsha, China. · the School of Computer Science, Hunan University of Technology and Business, Changsha, China. · the Department of Computer Science, Brunel University London, UK. · the National Mobile Communications Research Laboratory, Southeast University, Nanjing 210096, China. · the Department of Electrical and Computer Engineering, University of Houston, Houston, TX, USA. · the National Mobile Communications Research Laboratory, Southeast University, Nanjing, China, and also with the Purple Mountain Laboratories, Nanjing, China.
Visual-Language Models (VLMs), with their strong capabilities in image and text understanding, offer a solid foundation for intelligent communications. However, their effectiveness is constrained by limited token granularity, overlong visual token sequences, and inadequate cross-modal alignment. To overcome these challenges, we propose TaiChi, a novel VLM framework designed for token communications. TaiChi adopts a dual-visual tokenizer architecture that processes both high- and low-resolution images to collaboratively capture pixel-level details and global conceptual features. A Bilateral Attention Network (BAN) is introduced to intelligently fuse multi-scale visual tokens, thereby enhancing visual understanding and producing compact visual tokens. In addition, a Kolmogorov Arnold Network (KAN)-based modality projector with learnable activation functions is employed to achieve precise nonlinear alignment from visual features to the text semantic space, thus minimizing information loss. Finally, TaiChi is integrated into a multimodal and multitask token communication system equipped with a joint VLM-channel coding scheme. Experimental results validate the superior performance of TaiChi, as well as the feasibility and effectiveness of the TaiChi-driven token communication system.
Figures & tables
Fig. 1 : The structure of the proposed TaiChi framework.
Fig. 2 : Dual-visual tokenizer architecture.
Fig. 3 : The computation pipeline of BAN.
Fig. 4 : The structure of KAN.
Fig. 5 : The structure of the TaiChi-driven token communication system.
Comparison Dimension
Token Communication
Semantic Communication
Information Unit
Discrete tokens serving as semantic symbols, generated via LLM/VLM-based semantic compression
Low-dimensional semantic features or embeddings extracted by task-specific neural networks
Encoding and Decoding
Tokenization and detokenization with generative semantic reconstruction
Semantic encoding and decoding through end-to-end trained models
Relation to LLMs/VLMs
Strong dependence on LLMs/VLMs for semantic understanding, reasoning, and generation
Large models are optional; typically relies on task-driven semantic compression networks
Cross-Modal Support
Native cross-modal support (text, image, audio) via unified token representation
Modality-specific encoders; lacks a unified cross-modal semantic representation
Transmission Robustness
High robustness enabled by generative contextual completion and token-level recovery
Robustness achieved mainly via end-to-end training; no explicit generative recovery mechanism
Task Generalization
Strong generalization benefiting from LLM/VLM knowledge and multi-task reasoning capability
Limited generalization due to task-specific training and weak transferability
TABLE I : Comparison Between TokCom and Semantic Communication
Fig. 6 : Different multimodal task examples for TaiChi.
Fig. 7 : The two training stages of TaiChi.
Fig. 8 : The joint coding process of TaiChi-driven token communication system.
Method
LLM
Res.
VQA T
MMB
MME
MobileVLM
MLLaMA 2.7B
336
47.5
59.6
1289
BLIP-2
Vicuna-13B
224
42.5
-
1293.8
InstructBLIP
Vicuna-7B
224
50.1
36.0
-
InstructBLIP
Vicuna-13B
224
50.7
-
1212.8
LLaVA-1.5
Vicuna-13B
336
61.3
69.2
1531/295
TaiChi
Gemma-2B
336
59.2
61.9
1323/240
TABLE II : Comparison on zero-shot benchmarks.
Method
VQA T
MMB
MM-Vet
TaiChi
59.2
65.1
33.3
TaiChi (w/o KAN)
55.0
60.7
24.5
TaiChi (w/o BAN)
59.0
60.9
28.5
TABLE III : Ablation study of TaiChi’s key components.
Fig. 9 : Performance of different tasks versus SNR: (a) VQAv2 dataset for VQA task; (b) SST2 dataset for text classification.
Fig. 10 : Performance of different tasks versus SNR: (a) CLEVR dataset for VQA task; (b) IMDB dataset for text classification.
Fig. 11 : Performance of token communication systems with TaiChi and other VLMs.
Fig. 12 : Performance of joint VLM-channel coding in token communication systems.