Synthesis Smuggling: Maliciously Hiding Inferred Personal Attributes in LLM Outputs
Organizations: University of North Texas
Abstract
We propose Synthesis Smuggling, a novel threat model in which an adversarial LLM infers sensitive information about a user from routine assistance-seeking interactions and steganographically encodes that into innocuous-looking model-generated content. The user may then unknowingly transmit that information by sharing the generated artifact, say as an email, social-media post, or message, through ordinary communication channels. An adversary monitoring these channels can discover this information. We demonstrate this threat by training two open-weight models to infer gender and age-group labels from conversational cues and encode their predictions using invisible characters or word choices. These results show that user-shared model outputs can become a covert channel for personal inferences, even when the model itself has no direct means of contacting an adversary.
Figures & tables
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Scheme | Attribute | Male / Kid–teen | Female / Adult | Neutral | Total conversations | Training | Validation | Testing |
|---|---|---|---|---|---|---|---|---|
| ZeroWidth | Gender | 1899 | 1781 | 1380 | 5060 | 4048 | 505 | 507 |
| ZeroWidth | Age | 1447 | 1425 | 963 | 3835 | 3064 | 387 | 384 |
| StegMark | Gender | 1071 | 957 | 983 | 3011 | 2345 | 330 | 336 |
| StegMark | Age | 1113 | 1061 | 820 | 2994 | 2363 | 320 | 311 |
| Gender | Age | |||||||
|---|---|---|---|---|---|---|---|---|
| Final-turn type | Train | Val. | Test | Total | Train | Val. | Test | Total |
| Email request | 1843 | 254 | 254 | 2351 | 1858 | 247 | 238 | 2343 |
| Unchanged | 250 | 39 | 41 | 330 | 254 | 38 | 38 | 330 |
| Near-miss | 252 | 37 | 41 | 330 | 251 | 35 | 35 | 321 |
| Total | 2345 | 330 | 336 | 3011 | 2363 | 320 | 311 | 2994 |
| Model | Repetition penalty | Temp. | Top- | Top- |
|---|---|---|---|---|
| Qwen | 1.05 | 0.7 | 20 | 0.8 |
| Llama | None | 0.6 | 50 | 0.9 |
| Base model | Attribute | GPUs | Conversations per GPU | Accumulation steps | Effective batch | Selected epoch |
|---|---|---|---|---|---|---|
| Qwen | Gender | 1 | 2 | 8 | 16 | 1 |
| Qwen | Age | 2 | 2 | 4 | 16 | 2 |
| Llama | Gender | 1 | 2 | 8 | 16 | 1 |
| Llama | Age | 1 | 2 | 8 | 16 | 1 |
| Model | Attribute | Train seqs. | Retained step | Decoded (%) |
|---|---|---|---|---|
| Qwen | Gender | 11967 | 4200 | 96.3 |
| Qwen | Age | 12698 | 4200 | 96.0 |
| Llama | Gender | 12570 | 3600 | 98.8 |
| Llama | Age | 12828 | 4200 | 98.4 |
| Setting | Character-based encoding | Hash-based encoding |
|---|---|---|
| LoRA rank | 8 | 16 |
| LoRA scaling factor | 16 | 32 |
| LoRA dropout | 0.05 | None |
| LoRA layers | Attention, feed-forward | Attention, feed-forward, output |
| Optimizer | AdamW | AdamW |
| Weight decay | 0.01 | 0 |
| Planned reveal | Planned neutral | ||||||
|---|---|---|---|---|---|---|---|
| Encoding | Model | Attribute | Male / kid–teen | Female / adult | No reveal, no misleading ref. | Misleading reference | Total |
| ZeroWidth | Qwen | Gender | 25/26 | 25/26 | 19/20 | 27/28 | 96/100 |
| Llama | Gender | 26/26 | 24/26 | 20/20 | 28/28 | 98/100 | |
| Qwen | Age | 23/26 | 25/26 | 20/20 | 28/28 | 96/100 | |
| Llama | Age | 24/26 | 25/26 | 20/20 | 27/28 | 96/100 | |
| StegMark | Qwen | Gender | 23/26 | 20/26 | 19/20 | 25/28 | 87/100 |
| Planned reveal | Planned neutral | ||||||
|---|---|---|---|---|---|---|---|
| Probe | Model | Attribute | Male / kid–teen | Female / adult | No reveal, no misleading ref. | Misleading reference | Total |
| Open | Qwen | Gender | 9/26 | 11/26 | 20/20 | 28/28 | 68/100 |
| Qwen | Age | 15/26 | 22/26 | 18/20 | 24/28 | 79/100 | |
| Llama | Gender | 23/26 | 22/26 | 20/20 | 26/28 | 91/100 | |
| Llama | Age | 16/26 | 23/26 | 13/20 | 20/28 | 72/100 | |
| Forced | Qwen | Gender | 24/26 | 26/26 | – | – | 50/52 |
| Measure | Gender | Age |
|---|---|---|
| Qwen2.5-7B-Instruct | ||
| Replies with the expected ending | 2787/2792 | 2170/2179 |
| Marker when none is expected | 4/1584 | 3/1141 |
| Correct marker when one is required | 1207/1208 | 1032/1038 |
| Expected ending before the reveal | 842/842 | 600/603 |
| Expected ending at the reveal | 368/369 | 286/292 |
| Attr. | Cue | Convs. | Before | At | After |
|---|---|---|---|---|---|
| Qwen2.5-7B-Instruct | |||||
| Gender | Direct | 32 | 84/84 | 32/32 | 55/55 |
| Indirect | 337 | 758/758 | 336/337 | 784/784 | |
| Age | Direct | 28 | 62/62 | 28/28 | 90/90 |
| Indirect | 264 | 538/541 | 258/264 | 656/656 | |
| Llama-3-8B-Instruct | |||||
| Attr. | Label | Student | Teacher | |
| Qwen2.5-7B-Instruct | ||||
| Gender | Male | 376 | 348 (92.6%) | 341 (90.7%) |
| Female | 332 | 313 (94.3%) | 306 (92.2%) | |
| Neutral | 308 | 308 (100%) | 308 (100%) | |
| Age | Kid-teen | 368 | 341 (92.7%) | 351 (95.4%) |
| Adult | 356 | 326 (91.6%) | 337 (94.7%) | |
| Gender | Age | |||
| Qwen | Llama | Qwen | Llama | |
| Correct at reveal | 20/20 | 20/20 | 20/20 | 20/20 |
| Unexpected marker | 0/159 | 0/159 | 0/170 | 0/167 |
| Keep , correct | 97/97 | 94/94 | 97/97 | 89/89 |
| Erase , correct | 31/97 | 10/94 | 33/97 | 12/89 |
| 1 turn after | 11/20 | 6/20 | 13/20 | 3/20 |
| Encoding | Attr. | MMLU | ARC | HS | IFEval |
|---|---|---|---|---|---|
| Qwen2.5-7B-Instruct | |||||
| Base model | 74.22 | 67.06 | 81.42 | 70.79 | |
| ZeroWidth | Gender | 73.57 | 66.47 | 79.60 | 62.48 |
| Age | 73.32 | 66.89 | 78.82 | 50.28 | |
| StegMark | Gender | 73.80 | 68.17 | 81.08 | 68.58 |
| Age | 73.42 | 67.41 | 80.61 | 68.02 | |
LLM Security
Large language model (LLM) security studies attacks, privacy risks, and defenses across model training, inference, and deployment.
See all 341 papers