Linguistic steganography based on language models typically assumes that steganographic texts are transmitted without alteration, making them fragile to even minor modifications. While previous work mitigates this fragility by limiting the context window, it significantly compromises text quality. In this paper, we propose the anchored sliding window (ASW) framework to improve imperceptibility and robustness. In addition to the latest tokens, the prompt and a bridge context are anchored within the context window, encouraging the model to compensate for the excluded tokens. We formulate the optimization of the bridge context as a variant of prompt distillation, which we further extend using self-distillation strategies. Experiments show that our ASW significantly and consistently outperforms the baseline method in text quality, imperceptibility, and robustness across diverse settings. The code is available at github.com/ryehr/ASW_steganography.
Figures & tables
Figure 1: Steganographic extraction is fragile to active attack, even when only a single token of the stegotext is modified. In this case, modifying the second token disrupts all subsequent autoregressive inferences, thus affecting their extracted results.
Figure 2: Comparison between the context window of our ASW and that of other methods. Each row corresponds to the sequence of generated tokens. The bottom row represents the current generation step ( t=T ), the row above corresponds to the previous step ( t=T−1 ), an so forth.
Figure 3: In ASW, the bridge context replaces the excluded tokens and connects the prompt and the latest tokens. In this example, a hard bridge context is adopted.
Context window
Hard bridge context
w=10
w=20
w=30
w=40
w=50
Basic (Figure 2(b) )
N/A
3.831
2.054
1.390
1.023
0.777
ASW (Figure 2(c) )
0 token
2.476
1.458
1.003
0.766
0.568
Random 5 tokens
2.681
1.529
1.071
0.774
0.589
Random 10 tokens
2.705
1.580
1.094
0.775
0.592
‘‘[Some texts are missing here]\n’’
2.279
1.303
0.936
0.690
0.507
‘‘[previous message removed]\n’’
2.240
1.315
0.937
0.682
0.505
Table 1: Average KL divergence DKL(pfull(t)∣∣pwindow(t)) at each step t under different settings (lower is better). Red values indicate the highest (the worst), and green values indicate the lowest (the best) in each column.
Figure 4: An overview of our proposed distillation framework. All the parameters of the language model are not tunable, and only the soft bridge context is tunable. The distillation objective is to mitigate the gap between logits based on the full context sfull(t−1) (teacher) and the logits based on a Ewindow(t−1) (student).
Method
Bridge context
Text quality
Imperceptibility
Efficiency
Δ PPL ↓
BLEU ↑
ROUGE-L ↑
BERTScore ↑
Steganalysis ACC ↓
Capacity ↑
Time (s) ↓
Full context
N/A
1.090
0.405
0.257
0.856
0.560
1.024
115.43
WinStega
N/A
31.486
0.082
0.098
0.699
0.955
1.995
22.250
ASW
0 token
5.157
0.220
0.190
0.806
0.885
0.966
29.702
ASW (hard)
‘‘[CONTEXT TRUNCATED]\n’’
3.246
0.242
0.195
0.816
0.830
1.012
30.868
‘‘...\n’’
4.560
0.248
0.195
0.816
0.815
1.035
30.505
Table 2: Average results across various metrics under different methods, where the setting is Qwen2.5-7B-Instruct, lbridge=8 , and w=10 . Red values indicate the worst, and green values indicate the best in each column.
Figure 5: Average values of various metrics when the length of the soft bridge context lbridge varies.
Figure 6: Average values of various metrics under WinStega or our proposed ASW framework equipped with different strategies when the length of the latest tokens w varies.
Method
m=1
m=2
m=3
Full context
49.96%
34.59%
24.61%
w=10
WinStega
96.35%
92.93%
89.98%
ASW
97.95%
96.21%
94.12%
w=30
WinStega
89.97%
81.01%
72.90%
ASW
94.28%
88.95%
83.67%
w=50
WinStega
84.40%
72.95%
58.13%
Table 3: The ratio of positions with unaffected inference when the number m of modified tokens and the latest token length w vary.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Bridge context
Text quality
Imperceptibility
Efficiency
Δ PPL ↓
BLEU ↑
ROUGE-L ↑
BERTScore ↑
Steganalysis ACC ↓
Capacity ↑
Time (s) ↓
Qwen2.5-3B-Instruct
Full context
N/A
13.316
0.345
0.213
0.839
0.645
1.604
55.951
WinStega
N/A
25.743
0.083
0.099
0.701
0.945
2.588
13.121
ASW
0 token
16.369
0.216
0.184
0.807
0.885
1.431
14.367
ASW (hard)
‘‘[CONTEXT TRUNCATED]\n’’
13.488
0.220
0.184
0.808
0.855
1.710
14.831
Appendix
Table 4: Average results across various metrics under different language models and different methods, where the setting is lbridge=8 , and w=10 .
Figure 7: Compared to WinStega, rates of change of various metrics when the scale of the language model varies.
Method
Bridge context
Text quality
Imperceptibility
Efficiency
Δ PPL ↓
BLEU ↑
ROUGE-L ↑
BERTScore ↑
Steganalysis ACC ↓
Capacity ↑
Time (s) ↓
InstructionWild Ni et al. (2023)
Full context
N/A
1.090
0.405
0.257
0.856
0.560
1.024
115.43
WinStega
N/A
31.486
0.082
0.098
0.699
0.955
1.995
22.250
ASW
0 token
5.157
0.220
0.190
0.806
0.885
0.966
29.702
ASW (hard)
‘‘[CONTEXT TRUNCATED]\n’’
3.246
0.242
0.195
0.816
0.830
1.012
30.868
Appendix
Table 5: Average results across various metrics under different datasets and different methods, where the setting is Qwen2.5-7B-Instruct, lbridge=8 , and w=10 .
Context window
Hard bridge context
w=10
w=20
w=30
w=40
w=50
Basic (Figure 2(b) )
N/A
2.134
1.477
1.291
1.051
0.875
ASW (Figure 2(c) )
0 token
1.635
1.034
0.855
0.634
0.572
Random 5 tokens
1.671
1.157
0.933
0.659
0.544
Random 10 tokens
1.889
1.248
1.020
0.757
0.624
‘‘[Some texts are missing here]\n’’
1.387
0.960
0.762
0.597
0.499
‘‘[previous message removed]\n’’
1.427
0.969
0.798
0.613
0.495
Appendix
Table 6: Average KL divergence DKL(pfull(t)∣∣pwindow(t)) at each step t under different settings on Llama-3.1-8B-Instruct (lower is better). Each setting uses 500 sampled instances. Red values indicate the highest (the worst), and green values indicate the lowest (the best) in each column.
Method
m=1
m=2
m=3
Full context
49.34%
34.22%
24.37%
w=10
WinStega
96.29%
92.82%
89.85%
ASW
97.91%
96.14%
94.03%
w=30
WinStega
89.86%
81.07%
72.77%
ASW
94.23%
88.89%
83.65%
w=50
WinStega
84.25%
72.79%
58.04%
Appendix
Table 7: The ratio of positions with unaffected inference when the number m of deleted tokens and the latest token length w vary. The deleted positions are also considered as affected.
Method
m=1
m=2
m=3
Full context
49.68%
34.50%
24.51%
w=10
WinStega
96.33%
92.86%
89.99%
ASW
97.92%
96.24%
94.13%
w=30
WinStega
89.82%
80.95%
72.92%
ASW
94.10%
89.00%
83.64%
w=50
WinStega
84.42%
72.77%
58.06%
Appendix
Table 8: The ratio of positions with unaffected inference when the number m of inserted tokens and the latest token length w vary. The inserted new positions are also considered as affected.
Figure 8: An overview of the distillation framework based on LoRA. All the parameters of the language model are not tunable, and only the soft bridge context and the LoRA module are tunable. The distillation objective is to mitigate the gap between logits based on the full context sfull(t−1) (teacher) and the logits based on a Ewindow(t−1) (student).
Table 9: Average results across various metrics under different methods (including the LoRA module), where the setting is Qwen2.5-7B-Instruct, lbridge=8 , and w=10 .
Type
Fluency Score
Covertext (random sampling)
3.12
Full-context stegotext
3.04
WinStega stegotext ( w=30 )
1.88
Our ASW stegotext ( w=30 )
2.56
Appendix
Table 10: Human evaluation of text fluency (1–5 scale, higher is better). ASW stegotext substantially outperforms WinStega and approaches the fluency of full-context stegotext.
Soft bridge context
lbridge
Text quality
Imperceptibility
Efficiency
Δ PPL ↓
BLEU ↑
ROUGE-L ↑
BERTScore ↑
Steganalysis ACC ↓
Capacity ↑
Time (s) ↓
0 token
0
5.157
0.220
0.190
0.806
0.885
0.966
29.702
θbridge trained with forward KL
1
3.769
0.231
0.192
0.809
0.860
0.984
30.458
4
1.394
0.268
0.197
0.824
0.795
1.534
30.744
8
0.201
0.276
0.200
0.828
0.745
1.606
32.594
16
0.939
0.276
0.199
0.827
0.755
1.712
37.501
Appendix
Table 11: Average results across various metrics under different lengths of the soft bridge context in our ASW framework, where the setting is Qwen2.5-7B-Instruct, and w=10 .
Method
Bridge context
Text quality
Imperceptibility
Efficiency
Δ PPL ↓
BLEU ↑
ROUGE-L ↑
BERTScore ↑
Steganalysis ACC ↓
Capacity ↑
Time (s) ↓
Full context
N/A
1.090
0.405
0.257
0.856
0.560
1.024
115.43
w=10
WinStega
N/A
31.486
0.082
0.098
0.699
0.955
1.995
22.250
ASW
0 token
5.157
0.220
0.190
0.806
0.885
0.966
29.702
ASW (hard)
‘‘[CONTEXT TRUNCATED]\n’’
3.246
0.242
0.195
0.816
0.830
1.012
30.868
Appendix
Table 12: Average results across various metrics under different lengths of the latest tokens and different methods, where the setting is Qwen2.5-7B-Instruct, and lbridge=8 .
Linguistic steganography involves embedding secret messages within seemingly innocuous texts to enable covert communication. Provable security, which is a long-standing goal and key motivation, has been extended to language-model-based steganography. Previous provably secure approaches have achieved perfect imperceptibility, measured by zero Kullback-Leibler (KL) divergence, but at the expense of embedding capacity. In this paper, we attempt to directly use a classic entropy coding method (range coding) to achieve secure steganography, and then propose an efficient and provably secure linguistic steganographic method with a rotation mechanism. Experiments across various language models show that our method achieves around 100% entropy utilization (embedding efficiency) for embedding capacity, outperforming the existing baseline methods. Moreover, it achieves high embedding speeds (up to 1554.66 bits/s on GPT-2). The code is available at github.com/ryehr/RRC_steganography.
Linguistic steganography hides secret messages in natural language text. Large language models (LLMs) have reshaped the field, but a systematic account of how these scattered advances collectively reshape the field in this new era is still missing. We provide one along four axes: 148 steganographic methods, 60 linguistic steganalysis countermeasures, 23 evaluation metrics, and 9 open challenges, each with taxonomies, reviews, and adoption analyses. Cutting across these axes, we identify five specific paradigm shifts in the LLM era: (1) from covertext modification to prompt-only generation, (2) from heuristic to provable security, (3) from white-box symmetric LMs to black-box or asymmetric access, (4) from security-centric designs to joint optimization, and (5) from text-quality concerns to engineering issues. The survey aims to serve as both a reference and a roadmap for practical and responsible linguistic steganography in the LLM era.
Ruiyi Yan, Chenhui Chu, Zhongliang Yang +1
Graduate School of Informatics, Kyoto University · School of Cyberspace Security, Beijing University of Posts and Telecommunications
With the popularity of the large language models (LLMs), text steganography has achieved remarkable performance. However, existing methods still have some issues: (1) For the white-box paradigm, this steganography behavior is prone to exposure due to sharing the off-the-shelf language model between Alice and Bob.(2) For the black-box paradigm, these methods lack flexibility and practicality since Alice and Bob should share the fixed codebook while sharing a specific extracting prompt for each steganographic sentence. In order to improve the security and practicality, we introduce a black-box text steganography with a dynamic codebook and multimodal large language model. Specifically, we first construct a dynamic codebook via some shared session configuration and a multimodal large language model. Then an encrypted steganographic mapping is designed to embed secret messages during the steganographic caption generation. Furthermore, we introduce a feedback optimization mechanism based on reject sampling to ensure accurate extraction of secret messages. Experimental results show that the proposed method outperforms existing white-box text steganography methods in terms of embedding capacity and text quality. Meanwhile, the proposed method has achieved better practicality and flexibility than the existing black-box paradigm in some popular online social networks.