Organizations: Xinjiang Technical Institute of Physics & Chemistry, Chinese Academy of Sciences, Urumqi, China · University of Chinese Academy of Sciences, Beijing, China (second affiliation) · Xinjiang Laboratory of Minority Speech and Language Information Processing, Urumqi, China (third affiliation)
Culture-expressions, such as idioms, slang, and culture-specific items (CSIs), are pervasive in natural language and encode meanings that go beyond literal linguistic form. Accurately translating such expressions remains challenging for machine translation systems. Despite this, existing benchmarks remain fragmented and do not provide a systematic framework for evaluating translation performance on culture-loaded expressions. To address this gap, we introduce CulT-Eval, a benchmark designed to evaluate how models handle different types of culturally grounded expressions. CulT-Eval comprises over 7,959 carefully curated instances spanning multiple types of culturally grounded expressions, with a comprehensive error taxonomy covering culturally grounded expressions. Through extensive evaluation of large language models and detailed analysis, we identify recurring and systematic failure modes that are not adequately captured by existing automatic metrics. Accordingly, we propose a complementary evaluation metric that targets culturally induced meaning deviations overlooked by standard MT metrics. The results indicate that current models struggle to preserve culturally grounded meaning and to capture the cultural and contextual nuances essential for accurate translation. Our benchmark and code are available at https://anonymous.4open.science/r/CulT-Eval-E75D/.
Figures & tables
Figure 1. Representative CulT-Eval instances.
Category
Count
Ratio
Avg. Term Len. (Char)
Avg. Context
(%)
Source (Zh)
Target (En)
Source (Zh)
Linguistic
2,512
31.6
5.13
29.43
22.29
Social
2,399
30.1
3.21
24.22
23.72
Material
1,594
20.0
3.06
22.97
24.39
Ecological
833
10.5
2.82
20.13
22.66
Religious
621
7.8
3.13
23.01
24.22
Table 1. Statistics of the CulT-Eval benchmark.
Metric
Pearson r
Spearman ρ
BLEU
30.2
28.4
ChrF++
22.4
20.1
BERTScore
27.5
25.3
COMET
44.5
39.0
MetricX-QE
24.6
22.8
ACRE (Ours)
68.4
65.1
Table 2. Pearson ( r ) and Spearman ( ρ ) correlation coefficients between metrics and human annotations on CulT-Eval.
Figure 2. Overview of the CulT-Eval benchmark.
Model
BLEU
CHRF++
BERTScore
COMET
MetricX-QE
0-shot
1-shot
0-shot
1-shot
0-shot
1-shot
0-shot
1-shot
0-shot
1-shot
Machine Translation Models
Hunyuan-MT-7B
–
–
–
–
–
–
–
–
–
–
Madlad400-10B-MT
–
–
–
–
–
–
–
–
–
–
NLLB200-3-3B
–
–
–
–
–
–
–
–
–
–
Open-Sourced and Proprietary LLMs
Table 3. Comprehensive evaluation results across five metrics.
Type
Content Analysis
Score
Valid
High Error
SRC …强调自己 吃软不吃硬 ，如果… REF …amenable to coaxing but not coercion… SYS …better to persuasion than force…
32.1 0.83
✗
Low Good
SRC …典型的 关系户 问题，在一些… REF …typical issue of nepotism, which… SYS …typical case of nepotism driven by…
20.3 0.64
✓
Table 4. Performance Analysis.
Figure 3. Performance analysis of six selected models. The left chart displays the overall Cultural Correctness score. The right chart visualizes the distribution of seven specific error types within the incorrect samples.
Table 6. Comprehensive ACRE evaluation results across proprietary and open-source models.
Configuration
Correlation ( r )
Δr
Full ACRE
0.88
–
w/o Validity Gate ( Ivalid )
0.76
-0.12
w/o Adaptive Routing (Protocols)
0.81
-0.07
w/o Explication Anchor ( E )
0.62
-0.26
w/o Reference Anchor ( R )
0.85
-0.03
Table 7. Ablation study of ACRE. We compare the semantic validity (Stage I) and quality profiling (Stage II) across zero-shot and one-shot settings.
Figure 5. Sensitivity of evaluation metrics to cultural translation errors.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6. The prompt template for the Culture-Mining.
Figure 7. Prompt template for fine-grained cultural classification.
Figure 8. The Core Dispatcher prompt. It routes instances to the correct evaluation protocol (Fact-Centric or Style-Centric) based on linguistic features and functional context.
Figure 9. The Semantic Validator prompt. It acts as a strict "Validity Gate" ( Ivalid ), filtering out hallucinations by anchoring evaluation to the Cultural Explication.
Figure 10. The Fidelity Critic prompt. It dynamically adjusts criteria (e.g., penalizing literalism in Protocol B vs. generalization in Protocol A) to assess translation nuance.
Figure 11. The Communicator prompt. It rewards explicitation strategies consistent with Appiah’s "Thick Translation" theory.
Cross-cultural language model evaluation tests whether AI systems understand and respond appropriately across languages, communities, and cultural contexts.