Pragmatics Meets Culture: Culturally-adapted Artwork Description Generation and Evaluation

Lingjun Zhao; Dayeon Ki; Marine Carpuat; Hal Daumé

cs.CLApr 2, 2026

Pragmatics Meets Culture: Culturally-adapted Artwork Description Generation and Evaluation

Authors: Lingjun Zhao, Dayeon Ki, Marine Carpuat, Hal Daumé

Organizations: Department of Computer Science, University of Maryland, College Park, Maryland, USA

Abstract

Language models are known to exhibit various forms of cultural bias in decision-making tasks, yet much less is known about their degree of cultural familiarity in open-ended text generation tasks. In this paper, we introduce the task of culturally-adapted art description generation, where models describe artworks for audiences from different cultural groups who vary in their familiarity with the cultural symbols and narratives embedded in the artwork. To evaluate cultural competence in this pragmatic generation task, we propose a framework based on culturally grounded question answering. We find that base models are only marginally adequate for this task, but, through a pragmatic speaker model, we can improve simulated listener comprehension by up to 8.2%. A human study further confirms that the model with higher pragmatic competence is rated as more helpful for comprehension by 8.0%.

Figures & tables

Figure 1: Example of a Gemma3-generated description of artwork for audiences unfamiliar with the cultural context. The description fails to explain the symbolism of the lotus leaf and therefore lacks evidence to help them answer the culturally-attuned question.

Figure 2: Our approach uses a self-improving speaker model to generate pragmatic artwork descriptions. Given an artwork and the listener’s cultural group, the pragmatic speaker first samples multiple descriptions, then ranks them by simulating how the listener would answer culturally-attuned questions when provided with each description. The model then selects the description with the highest self‑evaluation score and presents it to the listener, either an external simulated listener or human listener.

Panel of Table 1 — Table 1: Question-answering accuracy for culturally-agnostic and culturally-attuned artwork‑related questions, using descriptions generated by different speaker models. An external simulated listener (Qwen2.5‑VL), either familiar or unfamiliar with the artwork’s culture, answers the questions. ${\dagger}$ denotes when the pragmatic model has significantly higher accuracy than the base model ( $p<0.05$ ) as determined by a two-sample $t$ -test. Takeaway: VLMs can self-improve their pragmatic competence when used as speaker models.

Figure 3: Human users preference rates for descriptions generated by the base speaker and the pragmatic speaker. Takeaway: The pragmatic speaker improves user comprehension and introduces new information, but insufficiently accounts for users’ existing knowledge.

	Comprehension	Knowledge Gain	Prior Knowledge	QA Accuracy Gain
Self-eval. Score	0.08 (-0.06, 0.22)	0.04 (-0.10, 0.18)	0.06 (-0.08, 0.20)	0.15 (0.05, 0.25)

Table 2: Kendall’s

\tau

correlation between model self-evaluation score ( Eq 5 ), human preference over a pair of descriptions, and human QA accuracy gain from using the description versus not using it. Ranges are 95% confidence intervals. Takeaway: Model self‑evaluation score has stronger correlation with human QA accuracy gain than with human preferences.

Panel of Figure 4 — Figure 4: The pragmatic speaker outperforms the base speaker in question-answering under an external simulated listener, both when (a) the listener aligns with human understanding and when (b) it does not. Colors denote semantically similar sentences.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

QA Accuracy	Familiar		Unfamiliar
Speaker	W.o. Description	W. Description	W.o. Description	W. Description
Base Gemma	35.9 (24.0–47.7)	39.9 (27.7–52.1)	29.8 (18.1–41.6)	36.4 (23.4–49.4)
Pragmatic Gemma	47.6 (37.3–57.9)	47.0 (35.5–58.5)	34.5 (24.1–45.0)	40.0 (28.6–51.4)

Appendix

Table 3: Human user QA accuracy, ranges are 95% confidence intervals. Takeaway: Users perform better on culturally familiar artworks than on unfamiliar ones, consistent with simulated evaluation. Descriptions help understanding of culturally unfamiliar artworks, while differences between models are not significant.

Figure 5: Screenshot of the human evaluation pipeline. Each participant first reads the task instructions and important notes.

Figure 6: Participants select culture symbols from a list of visual elements, without any descriptions.

Figure 7: Participants answer three questions about each symbol, without being provided any artwork description.

Figure 8: Participants are then shown a randomly sampled artwork description, either from the base or the pragmatic speaker model, and asked to complete the same tasks.

Figure 9: Participants make subjective judgments given two descriptions generated by the base and pragmatic models.

Cultural Knowledge in Language Models

Cultural knowledge in language models concerns how systems acquire, access, and apply culture-specific facts, norms, meanings, and practices.

See all 132 papers

LLMs Infer Cultural Context but Fail to Apply It When Responding
Jun 16, 2026Yisong Miao, Jian Zhu, Vered ShwartzPragmatic Reasoning in Language Models Cultural Bias in Language Models
National University of Singapore · University of British Columbia · Canada CIFAR AI Chair, Vector Institute
Do LLMs Use Cultural Knowledge Without Being Told? A Multilingual Evaluation of Implicit Pragmatic Adaptation
Apr 20, 2026Mehwish Nasim, Sanjeevan Selvaganapathy, Neel Ganapathi Sabhahit +6Multilingual Language Models Multilingual Language Model Evaluation
1Network Analysis and Social Influence Modelling (NASIM) Lab · 3Computational Social Science & Systems Analysis University of Münster · School of Computing Macquarie University
Calibrated Ambiguity in Multimodal Language Models: Humans reach for cultural references, while models describe the picture
Sep 11, 2026Cody Kommers, Mingrui Ye, Evelyn Gius +4Pragmatic Reasoning in Language Models Multimodal Large Language Models
King’s College London London, United Kingdom · University of Edinburgh Edinburgh, United Kingdom

Jun 16, 2026J/K move · Enter open · S save

LLMs Infer Cultural Context but Fail to Apply It When Responding

Yisong Miao, Jian Zhu, Vered Shwartz

National University of Singapore · University of British Columbia · Canada CIFAR AI Chair, Vector Institute

Pragmatic Reasoning in Language Models Cultural Bias in Language Models Language Model Bias Evaluation Cultural Knowledge in Language Models

Open paper AskPDF arXiv

Recent work has shown that LLMs overrepresent dominant cultures, particularly Western ones, while marginalizing others. We investigate whether this affects models' ability to generate culturally adapted responses by evaluating their use of local measurement units based on the user's perceived cultural background. We introduce Cultural and Pragmatic Response Inference (CAPRI), a dataset of conversations with varying levels of cultural cues. Experiments with state-of-the-art LLMs show that models can infer cultural background and recall relevant conventions, but often fail to utilize the information to adapt their answers to the relevant cultural conventions, unless explicitly prompted to perform the tasks sequentially. We further evaluate adaptation to the interpretation of time and quantity expressions, two subjective language grounding dimensions that are affected by culture. We find that models increasingly adapt their answers as cultural cues accumulate, but their priors are not culture-neutral, sometimes aligning with the model's country of origin. Overall, CAPRI provides a resource for future research aimed at narrowing the gap between cultural knowledge and culturally adaptive language generation.

Abstract

Cultural Knowledge in Language Models

Explore similar work

LLMs Infer Cultural Context but Fail to Apply It When Responding

Do LLMs Use Cultural Knowledge Without Being Told? A Multilingual Evaluation of Implicit Pragmatic Adaptation

Calibrated Ambiguity in Multimodal Language Models: Humans reach for cultural references, while models describe the picture