Social media enables cross-lingual communication, yet the meaning of user-generated content (UGC) is often distributed across text, images, and cultural knowledge, making translation challenging. Existing evaluations typically treat images as optional and assume textual information is sufficient, obscuring whether errors stem from missing visual evidence or inadequate cultural adaptation. We introduce Culture-MMT, a diagnostic benchmark for distributed-evidence pragmatic reconstruction in social-media translation. It contains 498 curated, de-identified UGC instances across 20 domains, categorized into Symbol, Visual, and Hybrid by their required evidence. We also develop a human-aligned domain-specific judge, Judger, and a controlled evaluation framework that systematically varies the evidence available to translators. Across 17 model configurations, real images improve performance by +0.280 on Visual and +0.197 on Hybrid, but only +0.050 on Symbol, while random images provide no comparable gains. Human-authored descriptions further show that jointly dependent cases remain more challenging even when visual facts are explicit. These results characterize culturally effective social-media translation as {meaning reconstruction from distributed evidence}, highlighting visual grounding and cultural-pragmatic adaptation as distinct model bottlenecks.
Figures & tables
图 1: Evidence dependence in UGC translation. Left: text-sufficient cultural slang. Right: visually dependent, multimodal input recovers missing translation evidence.
Type
Meaning
Example
Symbol
Ambiguity comes from internet memes and in-group jargon; cultural knowledge alone suffices and the image is not needed.
“避雷” is a warning to steer clear of a bad product, not about lightning; “爸爸” refers to a (male) idol, not a father.
Visual
The key sense is polysemous and can only be fixed by the image; a dictionary does not help resolve it.
“单反” with a tennis photo means a one-handed backhand, not the default DSLR camera.
Hybrid
Both are required at once: the jargon needs cultural knowledge and the image must disambiguate it.
“肥宅复健 Day 1” is self-mocking jargon; only the photo says if “复健” means exercise or injury rehab.
表 1: The three evidence-dependency types (166 notes each).
图 2: The data-construction pipeline of Culture-MMT. Models are used for candidate enrichment; the evidence-dependency type and final inclusion are decided by humans.
图 3: Dataset statistics. (Left) Notes are balanced across the three evidence-dependency types (166 each) over 20 domains. (Right) Source length (title+body): mean 66.9, median 55, and p90 117 characters (dashed line marks the median).
Category
Model configuration
Commercial (6)
Gemini-3.1-pro (high)
Gemini-3.1-pro (low)
GPT-5.5, Kimi-K3
GLM-5.3-Flash, Qwen3.7-Flash
Open-weight (4)
Qwen3.5-397B-A17B
Qwen3.5-122B-A10B
表 2: Models used for judge validation.
Judge
ACC
Cohen’s κ
P-01
R-01
F1-01
P-23
R-23
F1-23
Judger (ours)
0.781
0.562
1.000
0.562
0.720
0.696
1.000
0.820
Gemini-3.1-pro
0.767
0.534
0.809
0.699
0.750
0.735
0.835
0.782
Qwen3.5-397B
0.733
0.466
0.736
0.727
0.731
0.730
0.739
0.735
Qwen3-32B
0.552
0.104
1.000
0.104
0.189
0.528
1.000
0.691
Qwen3.5-27B
0.773
0.546
0.809
0.715
0.759
0.745
0.831
0.786
表 3: Evaluator agreement with human experts on the balanced test set. -01 and -23 denote the invalid (scores 0–1) and valid (scores 2–3) classes, respectively.
Text-only
Multimodal
Model
Valid
Inval.
0
1
2
3
Valid
Inval.
0
1
2
3
Δ Valid
Δ Mean
Proprietary Models
Gemini-3.1-Pro (high)
78.1
21.9
3.8
18.1
20.9
57.2
82.7
17.3
2.2
15.1
20.9
61.9
+4.62
+0.108
Gemini-3.1-Pro (low)
79.1
20.9
3.6
17.3
22.9
56.2
82.7
17.3
1.6
15.7
24.1
58.6
+3.61
+0.080
GPT-5.6-Sol
72.1
27.9
5.2
22.7
16.9
55.2
81.1
18.9
2.2
16.7
16.9
64.3
+9.03
+0.211
GPT-5.5
73.9
26.1
3.8
22.3
20.5
53.4
79.1
20.9
2.4
18.5
21.1
58.0
+5.22
+0.112
表 4: Cultural effectiveness and score distribution across 17 models. Reporting rates (%), score breakdowns, and multimodal gains ( Δ ). Bold marks group best.
图 4: Visual gains across evidence types and configurations. (a) Mean gain Δ : Symbol remains unchanged, while Visual and Hybrid gain significantly. (b) Per-configuration breakdown. (c) Text-only vs. multimodal scores ( r=0.989 ).
Model
Text-only
Multimodal
Δ Overall
Δ Symbol
Δ Visual
Δ Hybrid
Proprietary Models
Gemini-3.1-Pro (high)
2.315
2.424
+0.108
+0.012
+0.175
+0.139
Gemini-3.1-Pro (low)
2.317
2.398
+0.080
− 0.078
+0.145
+0.175
GPT-5.6-Sol
2.221
2.432
+0.211
+0.048
+0.325
+0.259
GPT-5.5
2.235
2.347
+0.112
+0.000
+0.181
+0.157
Qwen3.7-Flash
2.207
2.337
+0.131
+0.006
+0.343
+0.042
表 5: Visual gains across 17 models. Reporting mean scores under Text-only and Multimodal inputs, and their differences ( Δ=Multimodal−Text-only ) overall and per dependency type.
Model configuration
Condition
Symbol
Visual
Hybrid
Overall
Gemini-3.1-pro (high)
Text-only
2.530
2.277
2.139
2.315
Random image
2.458
2.307
2.115
2.293
Human caption
2.512
2.440
2.283
2.412
Real multimodal
2.542
2.452
2.277
2.424
Qwen3.5-397B-A17B
Text-only
2.217
1.879
1.765
1.954
Random image
2.120
1.873
1.560
1.851
表 6: Evidence-attribution ablations across four representative configurations. Mean scores ( 0 – 3 ) across dependency types under four translator inputs: Text-only , Random image (placebo control), Human caption (oracle perception), and Real multimodal .
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
图 5: Three representative failure modes of multimodal cultural translation: (a) visual-evidence recognition, (b) cross-modal evidence integration, and (c) cultural transfer and target-language expression. Columns marked “*” are benchmark information; unmarked columns show Gemini-3.1-pro outputs under different translator input conditions, with the upper-right number indicating the score.
Strategy
Mode
Rule
Real Scene Grounded
image-to-image
Keep the UI/elements recognizable; change only the composition. Rewrite in-image text that is a translation point.
Celebrity
image-to-image
Preserve the likeness; change viewpoint, pose and composition. Keep the nickname.
Text in Image
text-to-image
Change the layout; rewrite in-image text into a semantically equivalent synthetic version.
Generic
text-to-image
Rebuild fully with unidentifiable fictional people or objects.
Appendix
表 7: Four image-rewriting strategies.
Metric
Mean
Median
Worst case
Text 3-gram repetition rate ( ↓ )
0.034
0.024
max 0.191
SequenceMatcher similarity ( ↓ )
0.216
0.205
max 0.590
Image pHash Hamming distance / 64 ( ↑ )
30.5
—
min 12
Image SSIM ( ↓ )
0.192
—
max 0.680
Appendix
表 8: Sealing-stage rewrite-quality metrics (full set of 498 notes).
Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.