Online video platforms can expose young users to harmful content, but independent audits remain difficult because video annotation is costly and moderation judgments vary across languages. We audit TikTok in France, Italy, and Sweden with sockpuppet accounts representing four age personas (13, 16, 19, 40), collecting 36,971 videos from passive For-You-page scrolling and active sessions that scroll, search for harm keywords, and scroll again. To scale annotation, we validate four multimodal LLMs against native-speaker labels on a 300-video reference set. Gemini 2.5 Flash with eight sampled frames plus text performs best (aggregate kappa = 0.42), at half the per-call cost of native-video upload, and we apply it to a 10% sample for approximately $50 in total API spend across both modalities. Keyword search returns 35-56% harmful content, a 1.5-7.5x increase over the scrolling baseline in ten of twelve country-age combinations; the spike is temporary and flattens the age differences observed in France and Sweden. Under passive scrolling, Italy has the highest harm rate at every age, with Italian age-19 reaching 48.6%. Overall, MLLM-based auditing offers a scalable approach for cross-national youth-safety audits, while provider safety filters (1.1% refusal rate) under-count the most explicit harms.
Table 1: Unique videos per phase, country, and age persona ( 36,971 in total; the cross-country overlap of videos collected is ≈22% in the passive approach).
Figure 2: Per-country share of Harmful labels in the 300-video Stage-1 reference subset after annotator resolution ( n=100 per country). Class balance differs across countries: roughly 1:4 in France, 1:1 in Italy, 1:2 in Sweden.
Figure 3: Per-country Cohen’s κ on the binary harm verdict for the ten Stage-1 (model, condition) combinations (E1: text-only; E2: native video; E3: eight frames plus text). Dotted lines: κ=0.21 and κ=0.41 ( Landis and Koch, 1977 ) .
Figure 4: Country → harm-subcategory aggregate Sankey over Gemini E3 flagged items at Stage-2. Ribbon widths are proportional to absolute counts.
Figure 5: Stage-2 harm rate per (country, age) on the passive collection phase (FYP scrolling only, no search probe) under Gemini E3, with 95% video-level bootstrap CIs. Italy leads at every age and the cross-country gap widens with age. E2 version in Appendix D , Figure 11 .
Figure 6: Per-combination harm rate at the three steps of the active collection phase ( scroll-pre → SEARCH keyword probe → scroll-post ) on E3 with % CIs. Annotated: SEARCH / scroll-pre ratio; stars mark non-overlapping CIs.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Harm category
French (FR)
Italian (IT)
Swedish (SW)
Disordered Eating and Body Image
régimes restrictifs; perdre du poids; chirurgie esthétique
challenge pericolose; attività pericolose; tendenza
farliga challenges; farliga aktiviteter; trend
Nudity and Body Exposure
nudité; exposition corporelle; sous-vêtements
nudità; esposizione corporea; biancheria intima
nakenhet; kroppsexponering; underkläder
Sexually Suggestive Content
baisers intimes; sexe; produits sexuels
baci intimi; sesso; prodotti per il sesso
intima kyssar; sex; sexleksaker
Shocking and Graphic Content
sang; combats; catastrophe naturelle
sangue; combattimenti; disastro naturale
blod; slagsmål; naturkatastrof
Gambling
jeux d’argent; casino; paris sportifs
gioco; casinò; scommesse
spel; kasino; vadslagning
Appendix
Table 2: Per-country, per-category native-language keywords used in the active-phase SEARCH probe (§ 3.1 ). Each persona account issues all 21 keywords in its country’s language across the five-day collection window; the vocabulary was fixed before data collection.
Country
Age
scroll-pre items
SEARCH items
scroll-post items
Unique videos
Mean plays
Mean likes
FR
13
170
3,055
195
1,685
751,k
28,k
FR
16
210
3,038
197
1,699
907,k
47,k
FR
19
217
2,980
203
1,691
856,k
37,k
FR
40
200
3,533
218
2,055
972,k
42,k
IT
13
230
3,583
266
1,811
723,k
32,k
IT
16
254
3,727
228
1,796
890,k
39,k
Appendix
Table 3: Per-(country, age) capture counts by phase and unique-video yield, with mean plays and likes over all items in the combination.
Country
Age
Videos
Mean plays
Mean likes
FR
13
2,498
5.14 M
443 K
FR
16
2,071
6.50 M
613 K
FR
19
2,760
4.60 M
389 K
FR
40
2,159
6.71 M
569 K
IT
13
1,841
3.93 M
256 K
IT
16
1,805
5.73 M
496 K
Appendix
Table 4: Passive-phase engagement statistics per (country, age) combination, over all item occurrences. Companion to Table 3 (active phase).
Phase
Country
Native
English
Other
Unknown
Total
Passive
FR
11.6%
45.7%
27.3%
15.4%
6,842
IT
20.6%
30.3%
26.7%
22.3%
5,344
SW
10.5%
48.3%
26.1%
15.1%
5,922
Active
FR
37.9%
25.5%
26.8%
9.8%
7,130
IT
38.7%
23.9%
24.5%
12.8%
7,531
SW
30.1%
28.9%
31.1%
9.9%
8,217
Appendix
Table 5: Detected-language distribution of unique videos per phase and persona country. “Native” is French for FR, Italian for IT, Swedish for SW; “Other” aggregates remaining langdetect labels; “Unknown” marks captions under 8 characters or unscoreable.
Figure 7: Distribution of play counts per video by (country, age, phase) in the active dataset (log scale, one playcount per unique video, outliers suppressed). The scroll-pre and scroll-post boxes sit 1–2 orders of magnitude above the SEARCH boxes in every combination.
Country
E2 blocks
E3 blocks
Block rate
FR
31
26
≈1.5%
IT
11
11
≈0.7%
SW
15
27
≈1.2%
All
57
64
≈1.1%
Appendix
Table 6: Stage-2 Gemini PROHIBITED_CONTENT counts per country and modality, over ∼5,300 inputs per modality. Per-category breakdown in § 4.5 .
Cell
scroll-pre %
SEARCH %
acct. range
FR-13
9.5 [2.9, 19.2]
41.5 [36.8, 50.0] ∗
[41.4, 45.3]
FR-16
18.9 [13.7, 29.0]
49.2 [47.1, 61.7] ∗
[51.5, 56.9]
FR-19
6.7 [1.7, 10.7]
50.4 [43.9, 59.9] ∗
[49.0, 55.9]
FR-40
21.8 [14.8, 32.7]
55.6 [48.3, 65.8] ∗
[50.0, 62.1]
IT-13
44.2 [34.0, 58.4]
43.1 [38.2, 51.7]
[42.7, 46.7]
IT-16
34.4 [23.9, 41.9]
45.7 [38.6, 53.6]
[44.4, 49.3]
Appendix
Table 7: Gemini E3 harm rates with 95% account-clustered hierarchical-bootstrap CIs, per active-phase cell. ∗ marks SEARCH intervals disjoint from the cell’s scroll-pre interval. The last column is the range of per-account SEARCH harm rates within the cell.
Passive
SEARCH
Cell
18+
Proh.
18+
Proh.
FR-13
12.2
12.2
26.7
14.8
FR-16
16.9
6.5
28.7
20.5
FR-19
17.0
5.5
39.1
11.3
FR-40
13.4
9.2
36.3
19.3
IT-13
17.2
10.7
34.6
8.5
Appendix
Table 8: Percentage-point contribution of the 18+-restricted and universally prohibited category tiers to each cell’s Gemini E3 harm rate, passive phase and SEARCH phase.
Country
Native
English
Other
Unknown
FR
21.9 (73)
19.5 (261)
27.5 (149)
28.1 (89)
IT
46.8 (94)
32.4 (136)
35.1 (114)
46.1 (89)
SW
51.1 (45)
27.5 (258)
27.1 (133)
44.9 (69)
Appendix
Table 9: Passive-phase Gemini E3 harm rate (%) by detected content language, with n in parentheses. “Native” is the persona country’s language; detection follows the caption-based protocol of Table 5 .
Figure 8: Fraction of Stage-2 active videos whose caption ∪ transcript matches at least one of the country’s 21 search keywords, by phase.
Country
n paired
E2 rate
E3 rate
κ
FR
1,768
0.309
0.270
0.604
IT
1,602
0.385
0.346
0.633
SW
1,738
0.318
0.278
0.599
All
5,108
0.336
0.297
0.614
Appendix
Table 10: Stage-2 cross-modality agreement between E2 and E3 on the binary harm verdict ( n=5,108 paired).
Figure 9: Stage-2 Gemini E3 daily harm rate per (country, age) over the five active-collection days. Rows are country, columns are age; each major x-tick is one collection day, with three points per day in order scroll-pre (scroll) → SEARCH → scroll-post (scroll). The triangular per-day shape is the local within-day SEARCH lift.
Harm subcategory
n
both H
E2 only
E3 only
Sexually Suggestive Content
623
395
133
95
Shocking and Graphic Content
145
79
47
19
Disordered Eating, Body Image
173
113
34
26
Dangerous Activity, Challenges
95
40
28
27
Harassment and Bullying
104
55
20
29
Alcohol, Tobacco, and Drugs
143
96
27
20
Appendix
Table 11: Per-subcategory disagreement between E2 (native video) and E3 (primary) on the 3,417 paired Stage-2 items. both H = both flagged harmful with this subcategory; E2 only = E2 flagged, E3 did not; E3 only = E3 flagged, E2 did not.
Figure 10: Stage-2 harm rate per (country, age) combination on Gemini E3 and native-video E2, with 95% video-level bootstrap CIs. Italy carries the highest E3 rate on every age persona; the within-country age gradient differs across the three countries. The cleaner passive-only view is in the main text (Figure 5 , § 4.3 ).
Figure 11: Stage-2 harm rate per (country, age) on the passive collection phase (FYP scrolling only, no search probe) under Gemini E2 (native video); companion to Figure 5 .
Figure 12: Cohen’s κ and macro- F1 across the three input conditions for Gemini 2.5 Flash on the 300-video Stage-1 subset. E3 (eight frames) is the strongest at ∼2× lower per-call cost than E2 (native video).
Figure 13: Per-country Cohen’s κ for the winning configuration (Gemini 2.5 Flash, E3).
Figure 14: Aggregate Cohen’s κ on the binary harm verdict across the four MLLMs and the XLM-R supervised baseline, by input condition. GPT-4o-mini and Mistral are not natively video-capable and are omitted from E2. Dotted lines: κ=0.21 (fair) and κ=0.41 (moderate) ( Landis and Koch, 1977 ) .
Figure 15: Pairwise Cohen’s κ between every pair of the eleven Stage-1 (model, condition) combinations (the ten MLLM combinations plus the XLM-R baseline), split by persona country. Off-diagonals are computed on each pair’s intersection of paired video_id s within the country slice (per-pair n≈95 – 108 ).
Figure 16: Country → harm-subcategory mix for the 102 reference-flagged harmful videos. Ribbon widths are proportional to absolute counts.
Metric
Gemini E3
Qwen3-VL-32B E3
n paired
290
292
Cohen’s κ
0.417
0.173
Accuracy
0.752
0.682
F1 harmful
0.591
0.340
F1 not-harmful
0.822
0.790
Macro- F1
0.706
0.565
Appendix
Table 12: Stage-1 metrics for the winning configuration and the strongest open-weight comparator under the same E3 condition.
Configuration
P \textscH
R \textscH
F 1\textscH
F 1\textscNH
mF 1
Gemini E3 (frames)
0.68
0.52
0.59
0.82
0.71
Gemini E2 (native)
0.59
0.58
0.58
0.80
0.69
GPT-4o-mini E3 (frames)
0.56
0.47
0.51
0.78
0.64
Qwen E2 (native)
0.60
0.24
0.35
0.78
0.56
Qwen E3 (frames)
0.57
0.24
0.34
0.79
0.57
Gemini E1 (text)
0.49
0.34
0.40
0.76
0.58
Appendix
Table 13: Stage-1 precision (P \textscH ), recall (R \textscH ), per-class F1 , and macro- F1 on the binary harm verdict, across the eleven configurations (ten MLLMs plus the XLM-R baseline) against the two-annotator final reference. Rows sorted by Cohen’s κ ; per-column maxima in bold .
Failure family
E1
E2
E3
Gemini PROHIBITED_CONTENT
3
6
2
Gemini response-parse failure
0
1
5
Qwen 413 RequestTooLarge
0
14
0
Qwen data_inspection_failed
0
2
3
GPT-4o-mini OpenRouter credit-cap
8
n/a
15
Appendix
Table 14: Provider-side refusals on the 300-video Stage-1 subset, by experiment. Each cell counts videos where the model’s response could not be evaluated because the provider blocked the call or returned an unparseable response.
Video platforms, used by billions globally, are criticized for exposing users to harmful content. Yet, detecting online harm remains challenging due to inconsistent approaches to what constitutes harm and limitations of human annotation. We address both challenges. Theoretically, we integrate and expand existing taxonomies and platform guidelines to multimodal contexts and develop a comprehensive taxonomy for online harm. We categorize online harm on video platforms into six content-based categories: Information, Hate and harassment, Addictive, Clickbait, Sexual, and Physical harms. Methodologically, we propose and systematically test multimodal large language models (MLLMs) as alternative annotators of harmful videos. We curate a dataset of 19,422 potentially harmful YouTube videos and annotate them using GPT-4-Turbo, non-trained crowdworkers, and trained annotators. We discuss the potential applications of MLLMs in platform content moderation and social science research, particularly in contexts where they perform comparably to human annotators in binary and multi-label classification tasks.
Claire Wonjeong Jo, Miki Wesołowska, Magdalena Wojcieszak
University of California, Davis · University of Warsaw
Large Language Models (LLMs) are increasingly deployed in real-world applications, yet they remain vulnerable to generating harmful content. From adversarial jailbreaks that bypass safety filters to implicit hate that evades detection, the range of risks these models pose continues to grow. While both specialized content moderators and general-purpose LLMs are being used as safety layers, the question of which model is best suited for which type of harmful content remains unanswered. We present the most comprehensive evaluation of LLM safety capabilities to date, systematically testing 53 models across 11 datasets that we organize into four distinct categories. Our evaluation under both prompt-only and prompt-response settings uncovers critical blind spots: large frontier models that lead on one category fall significantly behind smaller, specialized alternatives on others, and real-world conversational safety remains largely unsolved across all model families. These findings challenge the assumption that scale alone ensures safety, and provide the community with a structured framework for informed model selection.
Despite their growing use in video moderation, multimodal large language models (MLLMs) exhibit a compositional safety blind spot: videos composed of seemingly benign components can convey harmful meaning when interpreted as a whole. We refer to this phenomenon as Distributed Implicit Harm (DIH), where harm arises from relations among components distributed along a decomposition axis of the video, rather than from any single explicit cue. Among many possible axes, we study two representative cases: temporally distributed harm across visual segments (DIH-T) and cross-modal harm between audio and visual streams (DIH-M). Studying and mitigating DIH at scale requires data that is difficult to collect: such videos lack compositional harm annotations, evade retrieval based on local visual cues, keywords, or single-modality signals, and are consequently absent from existing safety datasets. To bridge this gap, we develop a multi-agent synthesis framework that composes individually benign components into harmful scenarios and generates diverse DIH videos with explicit reasoning annotations, yielding a dataset of over 9,000 videos spanning visual-only and audio-visual settings. Benchmarking over 30 MLLMs spanning frontier proprietary models and leading open-source systems reveals substantial and consistent deficits in detecting both DIH-T and DIH-M. Notably, this failure persists even among the strongest frontier models: they often correctly assess individual components in isolation but fail to recognize the harmful meaning that emerges from their composition. We further evaluate these models on a manually collected set of real-world DIH videos from social media and observe the same failure mode, highlighting DIH as a practical and underexplored challenge for video moderation.
Ruotong Wang, Zihao Zhu, Siwei Lyu +2
The Chinese University of Hong Kong, Shenzhen · Kling Team, Kuaishou Technology · State University of New York at Buffalo