Evaluating Cultural Reasoning and Robustness in LLMs for Spanish Political Discourse
Organizations: School of Engineering and Sciences Tecnologico de Monterrey Mexico
Abstract
Large language models (LLMs) are increasingly used to interpret, generate, and mediate public-facing language, which makes their reliability especially consequential in politically sensitive settings. Although sentiment analysis is a common task in this area, affective meaning in social media discourse is often conveyed through culturally situated and pragmatically indirect cues that are not well captured by label-centric evaluation alone. In this paper, we evaluate the robustness of LLMs for sentiment analysis of Spanish political discourse from social media, with emphasis on culturally situated affective interpretation. We introduce a diagnostic protocol that combines three independent human annotation views into Gold and Ambiguous subsets, and extends the corpus with a Synthetic subset built from sentiment-flipped counterfactuals selected through model-conditioned perplexity and LLM-as-a-judge criteria. Using this framework, we compare proprietary and open-weight models under zero-shot and few-shot prompting, focusing on predictive robustness across human agreement conditions and counterfactual shift. Our results show that annotation ambiguity and counterfactual perturbation reveal failure modes that conventional evaluation on the original corpus tends to obscure. GPT-4.1-mini remains the most stable among the evaluated models, whereas the two smaller open-weight models degrade more markedly under synthetic perturbations, and few-shot prompting provides only limited gains. Overall, the paper contributes a reproducible protocol for robustness-oriented evaluation in political sentiment analysis, an empirical characterization of model behavior under ambiguity and sentiment-flipped counterfactuals, and a publicly available Spanish social media dataset with three-way human sentiment annotations.
Figures & tables
| Platform | POS | NEG | NEU | GOLD | AMBIGUOUS | SYNTHETIC | TOTAL |
|---|---|---|---|---|---|---|---|
| 144 | 99 | 95 | 234 | 104 | 243 | 338 | |
| 193 | 52 | 97 | 220 | 122 | 245 | 342 | |
| X | 72 | 276 | 31 | 253 | 126 | 348 | 379 |
| YouTube | 14 | 67 | 214 | 142 | 153 | 81 | 295 |
| Total | 423 | 494 | 437 | 849 | 505 | 917 | 1,354 |
| Prompt | Core instruction and expected output |
|---|---|
| Sentiment classification | Classify the input post as negative , neutral , or positive . Return a JSON object containing the predicted label, supporting keywords, a brief explanation, and a confidence score from to . |
| Few-shot classification | Apply the same classification schema using labeled examples sampled from the annotated corpus. Include up to examples per sentiment class when available. |
| Counterfactual generation | Generate three alternative rewrites of a non-neutral post that invert its sentiment while preserving language, topic, political context, informal tone, and plausibility. Return each candidate as a JSON object. |
| Counterfactual selection | Select the candidate that most effectively inverts sentiment while preserving fluency, realism, and contextual consistency. Return the selected counterfactual, its predicted sentiment, and a brief justification. |
| Condition | Prompt | GPT Gold | GPT Amb. | DeepSeek Gold | DeepSeek Amb. | Llama Gold | Llama Amb. |
|---|---|---|---|---|---|---|---|
| Original | Zero-shot | 0.88 | 0.44 | 0.76 | 0.39 | 0.76 | 0.39 |
| Original | Few-shot | 0.89 | 0.47 | 0.77 | 0.41 | 0.72 | 0.38 |
| Synthetic-PPL | Zero-shot | 0.65 | 0.55 | 0.27 | 0.32 | 0.30 | 0.34 |
| Synthetic-PPL | Few-shot | 0.66 | 0.55 | 0.28 | 0.30 | 0.28 | 0.30 |
| Synthetic-Judge | Zero-shot | 0.64 | 0.55 | 0.29 | 0.31 | 0.31 | 0.30 |
| Synthetic-Judge | Few-shot | 0.65 | 0.54 | 0.31 | 0.31 | 0.29 | 0.28 |
Sentiment Analysis
Sentiment analysis detects and measures opinions, emotions, and evaluative stance in text and multimodal data.
See all 136 papers