Metamorphic Testing for Evaluating Robustness of Sentiment Analysis Models
Abstract
Machine learning systems are challenging to test using traditional software testing methods because their behavior is learned from data rather than explicitly programmed. This challenge is particularly significant in natural language processing (NLP), where small input variations can lead to unexpected model behavior. Conventional evaluation metrics, such as accuracy and F1-score, often fail to capture such inconsistencies. Metamorphic testing addresses this limitation by evaluating expected relationships between original and transformed inputs. With the increasing prevalence of generative artificial intelligence (GenAI) and language-based AI systems, the need for systematic robustness testing has become even more pronounced. Motivated by this growing need, this paper presents a lightweight, batch-based metamorphic testing approach for evaluating the robustness of such systems. In particular, we consider sentiment analysis as a representative NLP task and evaluate a pretrained classifier under four transformations: synonym replacement, typo injection, word-order variation, and negation. Using consistency score, flip accuracy, and confidence change, we identify strengths in handling semantic variations and vulnerabilities to textual noise and logical transformations. These findings highlight the value of metamorphic testing in identifying behavioral inconsistencies beyond conventional performance metrics and demonstrate its potential as a systematic framework for evaluating the robustness of language-based AI systems and guiding performance improvements before deployment.
Figures & tables
| Transformation | Metric | Score (%) |
|---|---|---|
| Synonym Replacement | Consistency Score | 100.0 |
| Typo Injection | Consistency Score | 98.5 |
| Word Order Change | Consistency Score | 96.5 |
| Negation | Flip Accuracy | 28.5 |
| Transformation | Avg Confidence Change |
|---|---|
| Synonym Replacement | 0.0011 |
| Typo Injection | 0.0045 |
| Word Order Change | 0.0005 |
| Negation | 0.0145 |
| Transformation | Original Text | Transformed Text | Orig. Label | Trans. Label |
|---|---|---|---|---|
| Negation | it ’s a bad thing when a movie has about as much substance as its end credits blooper reel . | it ’s a not bad thing when a movie has about as much substance as its end credits blooper reel . | NEG | POS |
| Negation | the heavy-handed film is almost laughable as a consequence . | not the heavy-handed film is almost laughable as a consequence . | NEG | POS |
| Typo | pumpkin means to be an outrageous dark satire on fraternity life , but its ambitions far exceed the abilities … | pumpkin means to be an outrageous dark satire on fraternity life , but its ambitions far exced the abilities … | POS | NEG |
| Word Order | it seems like i have been waiting my whole life for this movie and now i ca n’t wait for the sequel . | seems it like i have been waiting my whole life for this movie and now i ca n’t wait for the sequel . | POS | NEG |
| Word Order | the script kicks in , and mr. hartley ’s distended pace and foot-dragging rhythms follow . | script the kicks in , and mr. hartley ’s distended pace and foot-dragging rhythms follow . | POS | NEG |
| Typo | though perry and hurley make inspiring efforts to breathe life into the disjointed , haphazard script … | though perry and hurley make inspiring efforts to brethe life into the disjointed , haphazard script … | POS | NEG |
Language Model Robustness
Language model robustness studies how reliably language models preserve correct, safe behavior under input changes, distribution shifts, and adversarial pressure.
See all 519 papers