Social-media comments are interactions, yet they are often translated and evaluated as isolated text. This mismatch is especially problematic when meaning depends on surrounding discussion, community-specific slang, or culturally situated expressions. We introduce CULTURE-CMT, a Chinese-English benchmark for contextualized social-media comment translation that restores the post and conversational context naturally available to users. Its 572 manually verified examples are organized into Slang, Context, and Both, enabling diagnostic analysis of distinct and interacting sources of translation difficulty. To support scalable evaluation, we further develop a retrieval-augmented cultural-effectiveness Judger that combines a calibrated scoring model with dynamically retrieved cultural knowledge. CULTURE-CMT reveals several non-trivial patterns. Providing interaction context improves most model configurations, but does not eliminate culturally ineffective translations. Examples requiring both contextual grounding and culturally situated language remain substantially more difficult than either source of difficulty alone. Moreover, larger models are not consistently better, and thinking produces highly model-dependent gains, including occasional degradation. In contrast, a 27B model adapted on benchmark-disjoint social-media data achieves the lowest observed invalidity rate among the evaluated configurations under thinking inference with context. Together, these results suggest that realistic social-media translation is not simply a matter of scaling models or reasoning longer, but of learning to translate meaning as it is constructed within evolving online interactions.
Figures & tables
Figure 1: Interaction context can be necessary to resolve ambiguous social-media expressions and recover the intended translation. The expression “Ziwei” is ambiguous in isolation but is correctly resolved as Alexander Zverev when the original post context is available.
Figure 2: Construction pipeline of CULTURE-CMT, from 40K contextualized social-media comments to 572 manually verified benchmark examples.
Figure 3: Dataset composition by difficulty type, interaction structure, comment length, and topical domain.
Figure 4: Overview of the CULTURE-CMT evaluation framework. The cultural-effectiveness Judger is trained through two-stage SFT and GRPO, and is augmented with retrieved cultural knowledge at inference time.
Figure 5: Agreement with human judgments for static and agent-enhanced Judgers ( N=432 ). (a) Binary accuracy with Cohen’s κ in parentheses; (b) four-way accuracy with Spearman’s ρ in parentheses. Gains are computed from unrounded accuracies.
With Context
Without Context
Context Gain
Model
0
1
2
3
Invalid ↓
Mean ↑
0
1
2
3
Invalid ↓
Mean ↑
ΔInv↑
ΔMean↑
Non-thinking inference
Qwen3.8-Max
0.2
20.1
25.5
54.2
20.3
2.337
0.3
23.6
27.6
48.4
24.0
2.241
+3.7
+0.096
GLM-5.2
1.6
20.1
30.6
47.7
21.7
2.245
1.0
29.0
29.7
40.2
30.1
2.091
+8.4
+0.154
Qwen3.8-27B
0.3
21.3
29.9
48.4
21.7
2.264
0.7
24.7
26.6
48.1
25.3
2.220
+3.7
+0.044
Claude Opus 5
0.7
22.2
28.8
48.3
22.9
2.247
0.5
27.3
27.8
44.4
27.8
2.161
+4.9
+0.086
Table 1: Overall CULTURE-CMT results with and without interaction context ( N=572 in each condition). Missing outputs receive score 0; gains use unrounded values. Lower Invalid is better; higher Mean is better.
Figure 6: Invalid rates (%) by difficulty under thinking inference with context. Subset sizes are 207 (Slang), 204 (Context), and 161 (Both).
Figure 7: Scale, reasoning, and adaptation under the contextualized setting. (a) Scale: Invalid Rates across the evaluated Qwen3.5 sizes are non-monotonic. (b) Reasoning: thinking/non-thinking differences vary across models. (c) Adaptation: SFT of the same 27B backbone lowers observed Invalid Rates in both modes.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Collection
Size
Role and stated separation
Benchmark source pool
40K comments
Source of the 2,205 candidates and final 572 examples; disjoint from the Judger collection.
Judger collection
200K pairs
160K Gemini-labeled and 40K human-labeled pairs for staged training and held-out development validation.
Judger GRPO subset
20K pairs
Subset of the human-labeled Judger collection, not an additional independent collection.
Translation collection
1.5M collected; 1.2M retained
Translation adaptation; retained examples are disjoint from CULTURE-CMT.
Development validation
1K–2K per set
Held out from training collections; used separately from final evaluation.
Final Judger evaluation
825 candidates; 432 retained
Multi-model benchmark translations, human-labeled and subsampled by score; evaluation only.
Appendix
Table 2: Roles of the data collections. Collection sizes describe the data pools; development validation examples are held out from training.
Setting
Value
Initialization / reference
Qwen3.5-27B Judger SFT checkpoint
Rollout iterations / prompts per batch
120 / 64
Samples per prompt / global batch size
4 / 64
Training rollout temperature
0.9
Maximum generated response length
2,048 tokens (training and evaluation)
Evaluation samples / temperature
1 per prompt / 0 (greedy)
Appendix
Table 3: Judger GRPO launch configuration. Training sampling and deterministic evaluation use different temperatures.
Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.
Social media platforms enable large-scale cross-lingual communication, but translating user-generated content (UGC) remains challenging due to its informal style, cultural references, and interaction-based expressions. While recent LLMs have improved translation quality, existing benchmarks and metrics often fail to capture whether translations convey intended meaning and cultural resonance in real-world settings. In this work, we introduce CULTURE-MT, a benchmark for social media translation that focuses on both CULtural Transmission and UGC-specific emotion REsonance. CULTURE-MT consists of 1,002 UGC notes across 14 domains, categorized into four types based on culture-loaded symbols and linguistic style features. We also construct UGC-oriented training data to fine-tune Qwen3-8B and Qwen3-32B as baselines. We propose cultural effectiveness as a new evaluation criterion, focusing on expression accuracy and cultural adaptability. Testing 15 models, including the baselines, we find that traditional metrics fail to capture cultural effectiveness. We also observe that cultural effectiveness on base LLMs correlates with model size. Our work provides a comprehensive evaluation system for UGC translation models and will offer an open evaluation platform to advance research in this area. We release the CULTURE-MT benchmark and provide an online leaderboard where submitted translation results can be evaluated by our trained JUDGER.
Linjuan Wu, Ruiqi Zhang, Xinze Lyu +7
Zhejiang University · Xiaohongshu Inc. · Fudan University.
Social media translation requires preserving meaning while capturing the cultural nuances of user-generated content (UGC). However, gaps in the cultural knowledge of LLM-based evaluators can lead to inaccurate assessments and unreliable training rewards. To improve these signals, we use retrieval-enhanced cultural evaluation to train CUE-MT, a family of social media translation models. A calibrated, domain-specific Judger guides supervised data refinement and is augmented by an Agent for hard-example mining and reinforcement learning. The Agent retrieves relevant external evidence as needed and distills it into concise hints that the Judger uses to score translations and assign rewards. Across 9B, 27B, and 35B-A3B backbones, CUE-MT outperforms the evaluated open-source baselines of comparable size on CUE-MT-Bench and three public benchmarks. Human evaluation on CUE-MT-Bench shows cultural effectiveness gains of 7.44-15.12 percentage points over the strongest external baseline in each size group. Retrieval improves the Judger's four-way classification accuracy by 8.07 percentage points over a matched no-retrieval baseline that uses LLM-generated hints alone. Training ablations further support the combination of hard-example mining and Agent-enhanced rewards.
Automatic translation quality metrics trained on general-domain corpora systematically fail on social media content, where communicative intent is encoded in culturally loaded expressions (internet slang, homophonic ciphers, and platform-specific idioms) rather than surface token patterns. We conduct a systematic empirical analysis demonstrating that standard metrics including COMET, XCOMET, and BERTScore exhibit near-zero or negative correlation with human cultural judgments, and even display a severity inversion in which scores increase as translation quality deteriorates. We further show that this failure extends to large language model judges: Qwen3-235B achieves Cohen's kappa of only 0.162, revealing that the bottleneck is not reasoning capacity but cultural grounding: models lack the domain-specific cultural knowledge needed to identify which aspects of a translation require scrutiny. To address this, we propose CuRIL, a reinforcement learning framework that internalizes cultural reasoning: cultural annotations are prepended inside the model's reasoning, excluded from policy gradients via a token-level loss mask, and injected with a probability that decays to zero over training, progressively forcing autonomous cultural judgment. On a 1,444-sample human-annotated social media translation benchmark, Qwen3-8B trained with CuRIL achieves Cohen's kappa 0.370 and Exact Match accuracy of 45.22%, approaching Gemini-3.1-Pro with 30x fewer parameters and surpassing models up to 235B in scale. We further demonstrate that our judge produces reliable reward signals for downstream translation optimization, reducing the low-quality translation rate by over 20 percentage points under independent human evaluation.