Privacy-preserving semantic understanding of human activities is important for indoor sensing, yet existing Wi-Fi CSI-based systems mainly focus on pose estimation or predefined action classification rather than fine-grained language generation. Mapping CSI to natural-language descriptions remains challenging because of the semantic gap between wireless signals and language and direction-sensitive ambiguities such as left/right limb confusion. We propose WiFi2Cap, a three-stage framework for generating action captions directly from Wi-Fi CSI. A vision-language teacher learns transferable supervision from synchronized video-text pairs, and a CSI student is aligned to the teacher's visual space and text embeddings. To improve direction-sensitive captioning, we introduce a Mirror-Consistency Loss that reduces mirrored-action and left-right ambiguities during cross-modal alignment. A prefix-tuned language model then generates action descriptions from CSI embeddings. We also introduce the WiFi2Cap Dataset, a synchronized CSI-RGB-sentence benchmark for semantic captioning from Wi-Fi signals. Experimental results show that WiFi2Cap consistently outperforms baseline methods on BLEU-4, METEOR, ROUGE-L, CIDEr, and SPICE, demonstrating effective privacy-friendly semantic sensing.
Figures & tables
Fig. 1: WiFi2Cap framework. (a) Stage 1: Vision–language teacher trained with contrastive learning and Mirror-Consistency. (b) Stage 2: CSI–text student trained via teacher-guided visual alignment and CSI–text contrastive learning with Mirror-Consistency. (c) Stage 3: Prefix-guided language generation.
Fig. 2: Data acquisition setup and multimodal samples. Left: the physical collection setup, including one transmitter (Tx), three receivers (Rx1–Rx3), the RGB camera, and the 4×6 grid of participant positions. Right: synchronized CSI heatmaps, RGB frames, and sentence-level action descriptions.
Fig. 3: CSI encoder. Dual ResNet-18 backbones encode amplitude and phase inputs, followed by gated fusion and projection to a CSI embedding.
Dataset
Method
BLEU-4
METEOR
ROUGE-L
CIDEr
SPICE
WiFi2Cap
Baseline (CSI → LM)
14.85
25.86
32.38
0.12
0.28
Full WiFi2Cap
51.78
57.48
64.32
0.52
0.63
Person-in-WiFi 3D
Baseline (CSI → LM)
12.15
20.38
26.12
0.10
0.23
Full WiFi2Cap
47.07
58.80
57.26
0.43
0.51
TABLE I: Captioning results on the WiFi2Cap and Person-in-WiFi 3D datasets. Best scores within each dataset block are in bold .
Ground Truth
Prediction
Sample 1 (correct)
The person raises both arms and moves them back and forth in a synchronized waving motion while staying in a squat.
The person raises both arms and swings them back and forth in a synchronized waving motion while staying in a squat.
Sample 2 (partially correct)
A man stands upright on his left foot, facing forward , and maintains his balance with a steady posture.
A man stands upright on his left foot, facing away , while maintaining his balance and keeping his body steady.
TABLE II: Qualitative examples of generated captions.
Stages
Metrics
S1
S2-1
S2-2
S3
BLEU-4
METEOR
ROUGE-L
CIDEr
SPICE
–
–
✓
✓
22.57
35.02
40.39
0.21
0.38
✓
✓
–
✓
47.10
51.08
57.74
0.47
0.61
✓
✓
✓
✓
51.78
57.48
64.32
0.52
0.63
TABLE III: Ablation on training stages. A check mark indicates the stage is enabled.
Setting
BLEU-4
METEOR
ROUGE-L
CIDEr
SPICE
w/o Mirror-Consistency
35.25
44.53
51.04
0.36
0.46
Teacher-only Mirror
44.38
52.29
58.75
0.47
0.59
Full Mirror-Consistency
51.78
57.48
64.32
0.52
0.63
TABLE IV: Effect of the mirror-consistency loss.
Language Model
BLEU-4
METEOR
ROUGE-L
CIDEr
SPICE
GPT-2
51.78
57.48
64.32
0.52
0.63
Microsoft/phi-2
47.97
52.75
59.55
0.46
0.62
Qwen
55.11
60.13
63.83
0.53
0.66
TABLE V: Effect of the language model in Stage 3 (CSI-conditioned generation).
Text Backbone
BLEU-4
METEOR
ROUGE-L
CIDEr
SPICE
CLIP B/32
51.78
57.48
64.32
0.52
0.63
CLIP L/14
53.76
58.37
65.21
0.53
0.67
TABLE VI: Choice of backbone for the text encoder.
Fig. 4: Visualization of modality alignment between CSI and text embeddings. Each row denotes a CSI embedding and each column denotes a text embedding; brighter values indicate higher cosine similarity. A clearer diagonal pattern after training indicates stronger CSI–text correspondence.
Department of Intelligent Semiconductors, Soongsil University, Republic of Korea · Department of AI Convergence Security, Soongsil University, Republic of Korea · School of AI Software, Soongsil University, Republic of Korea