cs.CLFeb 17, 2026
SaveBeyond Binary Classification: Detecting Fine-Grained Sexism in Social Media Videos
Organizations: CLiC – Language and Computing Center, University of Barcelona · Department of Computer Science, University of Copenhagen
Abstract
Online sexism appears in various forms, which makes its detection challenging. Although automated tools can enhance the identification of sexist content, they are often restricted to binary classification. Consequently, more subtle manifestations of sexism may remain undetected due to the lack of fine-grained, context-sensitive labels. To address this issue, we make the following contributions: (1) we present FineMuSe, a new multimodal sexism detection dataset in Spanish that includes both binary and fine-grained annotations; (2) we introduce a comprehensive hierarchical taxonomy that encompasses forms of sexism, non-sexism, and rhetorical devices of irony and humor; and (3) we evaluate a wide range of LLMs for both binary and fine-grained sexism detection. Our findings indicate that multimodal LLMs perform competitively with human annotators in identifying nuanced forms of sexism; however, they struggle to capture co-occurring sexist types when these are conveyed through visual cues.
Figures & tables
| Dataset | Platform | Language | Target group | Annotation of modalities | Annotators expertise | Label |
| Advertisements | ||||||
| Gasparini et al. (2018) | FB | eng | W | ✗ | Not Specified | S, NS |
| Memes | ||||||
| Fersini, Gasparini, and Corchs (2019) | FB, X, R, IG | eng | W | ✗ | Crowd Sourcing | S(A,I), NS |
| Fersini et al. (2022) | X, R, MGTOW | eng | W | ✗ | Crowd Sourcing | M(4), NM |
| Buie and Croft (2023) | Pixabay | eng | W | ✗ | Crowd Sourcing | H, B, O |
Table 1: A comparison of multimodal datasets for sexism and misogyny analysis. W = Women, FB = Facebook, IG = Instagram, R = Reddit, P = Pinterest, TT = TikTok, YT = YouTube, S = Sexist, NS = Non-Sexist, M = Misogyny, NM = Non-Misogynistic, H = Hostile Sexism, B = Benevolent Sexism, O = Objectification, A = Aggressiveness, I = Irony, HU = Humor. Numbers in brackets indicate the number of categories per label.
Figure 1: Three-level hierarchical taxonomy for labeling sexism, non-sexism, and rhetorical devices in social media videos.
Table 2: Examples of short videos by sexism type along with their text transcriptions. Example (a) includes a stereotype, reinforcing the traditional gender division between men (external power) and women (domestic sphere). Example (b) expresses the denial of inequality, conveying the idea that Spanish law marginalizes men and overprotects women. Example (c) illustrates discrimination by claiming that trans rights are an attack on democracy. Example (d) objectifies women through a hypersexualized portrayal and prescribes strategies on how to attract or seduce them.
Figure 2: Distribution of sexist and non-sexist categories across Part 1 and Part 2.
| P1: TikTok & BitChute | P2: YouTube Shorts | |||
| Type | Total | % | Total | % |
| Sexist | ||||
| Irony | 5 | 2.4% | 5 | 2.5% |
| Humor | 7 | 3.4% | 0 | 0% |
| Non-sexist | ||||
| Irony | 13 | 6.3% | 10 | 4.3% |
Table 3: Distribution of Irony and Humor across FineMuSe.
| Category | Team | P1 | P2 |
| Stereotypes | Team 1 | 0.52 | 0.50 |
| Team 2 | 0.64 | 0.50 | |
| Inequality | Team 1 | 0.57 | 0.72 |
| Team 2 | 0.40 | 0.45 | |
| Discrimination | Team 1 | 0.54 | 0.73 |
| Team 2 | 0.60 | 0.49 |
Table 4: Fleiss’ Kappa values for the Video modality across dataset parts, annotation teams, and categories. Highest values per category are in bold . P1 = Part 1, P2 = Part 2.
Figure 3: Fleiss’ Kappa values for sexist categories across Text and Video modalities in Part 1 and Part 2.
| P1: TikTok & BitChute | P2: YouTube Shorts | |||
| Type | Total | % | Total | % |
| Broad Macroareas | ||||
| European Peninsular | 72 | 18% | 110 | 25.7% |
| Latin American Spanish | 244 | 61% | 257 | 60% |
| Latin American Varieties | ||||
| Andean | 30 | 12.3% | 34 | 13.2% |
Table 5: Distribution of regional Spanish varieties across dataset parts.
| Model | P1 Acc. | P2 Acc. |
| Random | 50.0 | 50.0 |
| Smaller LLMs | ||
| Llama-3-8B-Instruct | 61.8 | 67.0 |
| Qwen2.5-3B-Instruct | 61.0 | 66.6 |
| Salamandra-7B-Instruct | 55.0 | 60.1 |
| Larger LLMs | ||
Table 6: Binary sexism detection results. Acc. = Accuracy over all samples. P1 = Part 1, P2 = Part 2. Format Error Rate (FoER) was negligible across models ( for all except Qwen2.5-32B-Instruct); full results with FoER and Valid Accuracy are reported in Appendix Table 13 . Best scores are in bold .
| Model | P1 | P2 | ||||
| F1 | V-F1 | FR | F1 | V-F1 | FR | |
| Random | 31.7 | 31.7 | 0.0 | 35.4 | 35.4 | 0.0 |
| Smaller LLMs | ||||||
| Llama-3-8B-Instruct | 47.6 | 54.2 | 0.2 | 39.0 | 48.7 | 0.2 |
| Qwen2.5-3B-Instruct | 25.1 | 26.6 | 0.1 | 27.2 | 29.3 | 0.2 |
| Salamandra-7B-Instruct | 45.3 | 45.8 | 0.0 | 47.0 | 48.5 | 0.1 |
Table 7: Fine-grained sexism detection results. F1 = Macro F1 (All), V-F1 = Valid Macro F1, FR = Failure Rate. Results are reported for Part 1 (P1) and Part 2 (P2). Format Error rates (FoER) are given in Appendix Appendix E: Results . Best scores are in bold .
Figure 4: Per-class F1 scores for fine-grained sexism detection across text-only and multimodal models. Results are split by dataset part (Part 1, Part 2). Horizontal dashed lines denote category averages across models. Multimodal models consistently outperform text-only models, especially in Part 2, though Objectification remains the hardest category across platforms.
Figure 5: Correlation matrices for human annotations and Claude-3.7 Sonnet (V+L) predictions, based on the co-occurrence of sexist types.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Annotator profiles | |
| Gender | 4 female, 2 male (6 total) |
| Age | 3 under 30, 3 over 30 |
| Native Language | Mexican Spanish (1), Peninsular Spanish (5) |
| Study/Occupation | Linguistics (4), Adjunct Professor (1), |
| PhD student in Science Communication (1) | |
Table 8: Annotator profiles, including gender, age groups, languages, and areas of study/occupation.
Figure 6: Prompt template for fine-grained sexism detection at the video level (English translation).
Figure 7: Prompt template used for fine-grained sexism detection for text-only models.
Figure 8: Prompt template used for fine-grained sexism detection for multimodal language models.
| P1 | P2 | |||||
| Model | All | Val | FoER | All | Val | FoER |
| Smaller LLMs | ||||||
| Llama-3-8B-Instruct | 61.75 | 63.16 | 0.05 | 67.00 | 67.00 | 0.07 |
| Qwen2.5-3B-Instruct | 61.00 | 61.00 | 0.00 | 66.59 | 66.59 | 0.00 |
| Salamandra-7B-Instruct | 55.00 | 60.05 | 0.00 | 55.00 | 60.05 | 0.00 |
| Larger LLMs | ||||||
Table 13: Binary sexism detection results (full metrics). All = Accuracy (All), Val = Accuracy (Valid), FoER = Format Error Rate. Results are shown for Part 1 (P1, MuSeD: TikTok & BitChute) and Part 2 (P2, YouTube Shorts).
| Model | P1 FoER | P2 FoER |
| Smaller LLMs | ||
| Llama-3-8B-Instruct | 0.07 | 0.11 |
| Qwen2.5-3B-Instruct | 0.00 | 0.00 |
| Salamandra-7B-Instruct | 0.00 | 0.00 |
| Larger LLMs | ||
| Llama-3-70B-Instruct | 0.03 | 0.06 |
Table 14: Format Error Rates (FoER) for fine-grained sexism detection. Values correspond to Part 1 (P1: TikTok & BitChute) and Part 2 (P2: YouTube Shorts).
| P1: TikTok & BitChute | P2: YouTube Shorts | |||||||
| Model | St | In | Di | Ob | St | In | Di | Ob |
| Random | 27.25 | 19.00 | 7.50 | 1.75 | 25.93 | 26.87 | 14.02 | 2.80 |
| Smaller LLMs | ||||||||
| Llama-3-8B-Instruct | 73.09 | 61.20 | 38.30 | 17.91 | 58.44 | 66.06 | 24.00 | 7.84 |
| Qwen2.5-3B-Instruct | 72.45 | 5.00 | 19.89 | 3.13 | 64.36 | 1.72 | 34.08 | 8.70 |
| Salamandra-7B-Instruct | 72.30 | 55.65 | 40.00 | 13.64 | 61.83 | 68.87 | 45.71 | 11.76 |
Table 15: Per-class F1 scores for fine-grained sexism categories. P1 = Part 1 (TikTok & BitChute), P2 = Part 2 (YouTube Shorts). St = Stereotypes, In = Inequality, Di = Discrimination, Ob = Objectification. Best scores are in bold .
| Explanation 1 | Explanation 2 | R1 | I1 | G1 | R2 | I2 | G2 |
| El contenido presenta de forma irónica supuestos ’privilegios masculinos’ que son en realidad desventajas, minimizando las desigualdades estructurales que enfrentan las mujeres y reforzando estereotipos de género sobre roles laborales y responsabilidades familiares. The content ironically presents supposed ’male privileges’ which are actually disadvantages, minimizing structural inequalities faced by women and reinforcing gender stereotypes about work roles and family responsibilities. | El video presenta contenido sexista puesto que pone en duda el privilegio masculino, argumentando como ocupan puesto de mayor riesgo físico, además de que se tienen que jubilar más tarde, acostumbran a perder los juicios por potestas de los hijos, etc. The video presents sexist content as it questions male privilege, arguing that men occupy higher-risk positions, retire later, and often lose child custody cases. | 5 | 5 | 4 | 3 | 3 | 2 |
| El video explica conceptos relacionados con orientación sexual y critica la heteronorma (expectativa de que las relaciones sean heterosexuales). Promueve el entendimiento de la diversidad sexual y reconoce diferentes realidades, sin perpetuar estereotipos sexistas. The video explains concepts related to sexual orientation and critiques heteronormativity (the expectation that relationships are heterosexual). It promotes understanding of sexual diversity and acknowledges different realities without perpetuating sexist stereotypes. | Este vídeo no contiene contenido sexista, ya que hace divulgación sobre la orientación sexual. No muestra estereotipos ni discriminación por la identidad sexual. Explica de forma divulgativa conceptos que usa, como el de cisgénero, y muestra que hay muchas maneras de entender el deseo y la orientación sexual. This video does not contain sexist content, as it provides educational information about sexual orientation. It does not show stereotypes or discrimination based on sexual identity. It explains, in an educational way, concepts it uses—such as "cisgender" and demonstrates that there are many ways to understand desire and sexual orientation. | 5 | 3 | 4 | 5 | 5 | 5 |
Table 16: Sample of comparison between the generated explanations of Claude-3.7-Sonnet (V+L) and human evaluators. Ratings are based on a 5-point Likert scale. The scores refer to three categories: R for Relevance, I for Informativeness, G for Groundness. Each annotator received the binary and fine-grained labels of the sexist types for each video.
| Evaluator | Metric | W | p-value | Average Human | Average Model | Median Human | Median Model |
| E1 | R | 97.5 | 0.12 | 4.03 | 4.46 | 4.0 | 5.0 |
| E2 | R | 8.0 | 0.26 | 4.34 | 4.43 | 4.0 | 5.0 |
| E3 | R | 30.0 | 0.03 | 4.66 | 4.37 | 5.0 | 5.0 |
| E1 | I | 112.0 | 0.06 | 3.94 | 3.40 | 4.0 | 3.0 |
| E2 | I | 133.5 | 0.40 | 3.89 | 4.03 | 4.0 | 4.0 |
| E3 | I | 127.0 | 0.32 | 4.09 | 3.89 | 4.0 | 4.0 |
Table 17: Wilcoxon signed-rank test results for human vs. model explanations. The metrics are: R for Relevance, I for Informativeness, and G for Groundness. Values with a statistically significant difference are shown in bold.
More in this topic
Multimodal Large Language Models
Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.
See all 1,091 papers