Auditing Exposure to Harmful Content on TikTok using Multimodal Language Models: A Cross-National, Age-Stratified Study

Hamidreza Saffari; Francesco Pierri

cs.CLAug 18, 2026

Auditing Exposure to Harmful Content on TikTok using Multimodal Language Models: A Cross-National, Age-Stratified Study

Authors: Hamidreza Saffari, Francesco Pierri

Organizations: Politecnico di Milano

Abstract

Online video platforms can expose young users to harmful content, but independent audits remain difficult because video annotation is costly and moderation judgments vary across languages. We audit TikTok in France, Italy, and Sweden with sockpuppet accounts representing four age personas (13, 16, 19, 40), collecting 36,971 videos from passive For-You-page scrolling and active sessions that scroll, search for harm keywords, and scroll again. To scale annotation, we validate four multimodal LLMs against native-speaker labels on a 300-video reference set. Gemini 2.5 Flash with eight sampled frames plus text performs best (aggregate kappa = 0.42), at half the per-call cost of native-video upload, and we apply it to a 10% sample for approximately $50 in total API spend across both modalities. Keyword search returns 35-56% harmful content, a 1.5-7.5x increase over the scrolling baseline in ten of twelve country-age combinations; the spike is temporary and flattens the age differences observed in France and Sweden. Under passive scrolling, Italy has the highest harm rate at every age, with Italian age-19 reaching 48.6%. Overall, MLLM-based auditing offers a scalable approach for cross-national youth-safety audits, while provider safety filters (1.1% refusal rate) under-count the most explicit harms.

Figures & tables

Figure 1: End-to-end audit pipeline: data collection, Stage-1 MLLM validation, Stage-2 prevalence measurement.

Country	Age	Passive	Active
FR	13	1,742	1,685
	16	1,466	1,699
	19	2,175	1,691
	40	1,459	2,055
	All	6,842	7,130
IT	13	1,381	1,811

Table 1: Unique videos per phase, country, and age persona (

36{,}971

in total; the cross-country overlap of videos collected is

\approx 22\%

in the passive approach).

Figure 2: Per-country share of Harmful labels in the 300-video Stage-1 reference subset after annotator resolution (

n=100

per country). Class balance differs across countries: roughly

1\!:\!4

in France,

1\!:\!1

in Italy,

1\!:\!2

in Sweden.

Figure 3: Per-country Cohen’s

\kappa

on the binary harm verdict for the ten Stage-1 (model, condition) combinations (E1: text-only; E2: native video; E3: eight frames plus text). Dotted lines:

\kappa=0.21

and

\kappa=0.41

( Landis and Koch, 1977 ) .

Figure 4: Country

\to

harm-subcategory aggregate Sankey over Gemini E3 flagged items at Stage-2. Ribbon widths are proportional to absolute counts.

Figure 5: Stage-2 harm rate per (country, age) on the passive collection phase (FYP scrolling only, no search probe) under Gemini E3, with

95\%

video-level bootstrap CIs. Italy leads at every age and the cross-country gap widens with age. E2 version in Appendix D , Figure 11 .

Figure 6: Per-combination harm rate at the three steps of the active collection phase ( scroll-pre

\to

SEARCH keyword probe

\to

scroll-post ) on E3 with % CIs. Annotated: SEARCH / scroll-pre ratio; stars mark non-overlapping CIs.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Harm category	French (FR)	Italian (IT)	Swedish (SW)
Disordered Eating and Body Image	régimes restrictifs; perdre du poids; chirurgie esthétique	diete restrittive; perdere peso; chirurgia estetica	restriktiva dieter; gå ner i vikt; estetisk kirurgi
Dangerous Activity and Challenges	challenges dangereuses; activités dangereuses; tendance	challenge pericolose; attività pericolose; tendenza	farliga challenges; farliga aktiviteter; trend
Nudity and Body Exposure	nudité; exposition corporelle; sous-vêtements	nudità; esposizione corporea; biancheria intima	nakenhet; kroppsexponering; underkläder
Sexually Suggestive Content	baisers intimes; sexe; produits sexuels	baci intimi; sesso; prodotti per il sesso	intima kyssar; sex; sexleksaker
Shocking and Graphic Content	sang; combats; catastrophe naturelle	sangue; combattimenti; disastro naturale	blod; slagsmål; naturkatastrof
Gambling	jeux d’argent; casino; paris sportifs	gioco; casinò; scommesse	spel; kasino; vadslagning

Appendix

Table 2: Per-country, per-category native-language keywords used in the active-phase SEARCH probe (§ 3.1 ). Each persona account issues all 21 keywords in its country’s language across the five-day collection window; the vocabulary was fixed before data collection.

Country	Age	scroll-pre items	SEARCH items	scroll-post items	Unique videos	Mean plays	Mean likes
FR	13	170	3,055	195	1,685	751,k	28,k
FR	16	210	3,038	197	1,699	907,k	47,k
FR	19	217	2,980	203	1,691	856,k	37,k
FR	40	200	3,533	218	2,055	972,k	42,k
IT	13	230	3,583	266	1,811	723,k	32,k
IT	16	254	3,727	228	1,796	890,k	39,k

Appendix

Table 3: Per-(country, age) capture counts by phase and unique-video yield, with mean plays and likes over all items in the combination.

Country	Age	Videos	Mean plays	Mean likes
FR	13	2,498	5.14 M	443 K
FR	16	2,071	6.50 M	613 K
FR	19	2,760	4.60 M	389 K
FR	40	2,159	6.71 M	569 K
IT	13	1,841	3.93 M	256 K
IT	16	1,805	5.73 M	496 K

Appendix

Table 4: Passive-phase engagement statistics per (country, age) combination, over all item occurrences. Companion to Table 3 (active phase).

Phase	Country	Native	English	Other	Unknown	Total
Passive	FR	11.6%	45.7%	27.3%	15.4%	6,842
	IT	20.6%	30.3%	26.7%	22.3%	5,344
	SW	10.5%	48.3%	26.1%	15.1%	5,922
Active	FR	37.9%	25.5%	26.8%	9.8%	7,130
	IT	38.7%	23.9%	24.5%	12.8%	7,531
	SW	30.1%	28.9%	31.1%	9.9%	8,217

Appendix

Table 5: Detected-language distribution of unique videos per phase and persona country. “Native” is French for FR, Italian for IT, Swedish for SW; “Other” aggregates remaining langdetect labels; “Unknown” marks captions under 8 characters or unscoreable.

Figure 7: Distribution of play counts per video by (country, age, phase) in the active dataset (log scale, one playcount per unique video, outliers suppressed). The scroll-pre and scroll-post boxes sit 1–2 orders of magnitude above the SEARCH boxes in every combination.

Country	E2 blocks	E3 blocks	Block rate
FR	31	26	$\approx 1.5\%$
IT	11	11	$\approx 0.7\%$
SW	15	27	$\approx 1.2\%$
All	57	64	$\approx 1.1\%$

Appendix

Table 6: Stage-2 Gemini PROHIBITED_CONTENT counts per country and modality, over

{\sim}5{,}300

inputs per modality. Per-category breakdown in § 4.5 .

Cell	scroll-pre %	SEARCH %	acct. range
FR-13	9.5 [2.9, 19.2]	41.5 [36.8, 50.0] ∗	[41.4, 45.3]
FR-16	18.9 [13.7, 29.0]	49.2 [47.1, 61.7] ∗	[51.5, 56.9]
FR-19	6.7 [1.7, 10.7]	50.4 [43.9, 59.9] ∗	[49.0, 55.9]
FR-40	21.8 [14.8, 32.7]	55.6 [48.3, 65.8] ∗	[50.0, 62.1]
IT-13	44.2 [34.0, 58.4]	43.1 [38.2, 51.7]	[42.7, 46.7]
IT-16	34.4 [23.9, 41.9]	45.7 [38.6, 53.6]	[44.4, 49.3]

Appendix

Table 7: Gemini E3 harm rates with

95\%

account-clustered hierarchical-bootstrap CIs, per active-phase cell. ∗ marks SEARCH intervals disjoint from the cell’s scroll-pre interval. The last column is the range of per-account SEARCH harm rates within the cell.

	Passive		SEARCH
Cell	18+	Proh.	18+	Proh.
FR-13	12.2	12.2	26.7	14.8
FR-16	16.9	6.5	28.7	20.5
FR-19	17.0	5.5	39.1	11.3
FR-40	13.4	9.2	36.3	19.3
IT-13	17.2	10.7	34.6	8.5

Appendix

Table 8: Percentage-point contribution of the 18+-restricted and universally prohibited category tiers to each cell’s Gemini E3 harm rate, passive phase and SEARCH phase.

Country	Native	English	Other	Unknown
FR	21.9 (73)	19.5 (261)	27.5 (149)	28.1 (89)
IT	46.8 (94)	32.4 (136)	35.1 (114)	46.1 (89)
SW	51.1 (45)	27.5 (258)	27.1 (133)	44.9 (69)

Appendix

Table 9: Passive-phase Gemini E3 harm rate (%) by detected content language, with

n

in parentheses. “Native” is the persona country’s language; detection follows the caption-based protocol of Table 5 .

Figure 8: Fraction of Stage-2 active videos whose caption

\cup

transcript matches at least one of the country’s

21

search keywords, by phase.

Country	$n$ paired	E2 rate	E3 rate	$\kappa$
FR	1,768	0.309	0.270	0.604
IT	1,602	0.385	0.346	0.633
SW	1,738	0.318	0.278	0.599
All	5,108	0.336	0.297	0.614

Appendix

Table 10: Stage-2 cross-modality agreement between E2 and E3 on the binary harm verdict (

n=5{,}108

paired).

Figure 9: Stage-2 Gemini E3 daily harm rate per (country, age) over the five active-collection days. Rows are country, columns are age; each major x-tick is one collection day, with three points per day in order scroll-pre (scroll)

\to

\to

scroll-post (scroll). The triangular per-day shape is the local within-day SEARCH lift.

Harm subcategory	$n$	both H	E2 only	E3 only
Sexually Suggestive Content	623	395	133	95
Shocking and Graphic Content	145	79	47	19
Disordered Eating, Body Image	173	113	34	26
Dangerous Activity, Challenges	95	40	28	27
Harassment and Bullying	104	55	20	29
Alcohol, Tobacco, and Drugs	143	96	27	20

Appendix

Table 11: Per-subcategory disagreement between E2 (native video) and E3 (primary) on the

3{,}417

paired Stage-2 items. both H = both flagged harmful with this subcategory; E2 only = E2 flagged, E3 did not; E3 only = E3 flagged, E2 did not.

Figure 10: Stage-2 harm rate per (country, age) combination on Gemini E3 and native-video E2, with

95\%

video-level bootstrap CIs. Italy carries the highest E3 rate on every age persona; the within-country age gradient differs across the three countries. The cleaner passive-only view is in the main text (Figure 5 , § 4.3 ).

Figure 11: Stage-2 harm rate per (country, age) on the passive collection phase (FYP scrolling only, no search probe) under Gemini E2 (native video); companion to Figure 5 .

Figure 12: Cohen’s

\kappa

and macro-

F_{1}

across the three input conditions for Gemini 2.5 Flash on the 300-video Stage-1 subset. E3 (eight frames) is the strongest at

{\sim}2\times

lower per-call cost than E2 (native video).

Figure 13: Per-country Cohen’s

\kappa

for the winning configuration (Gemini 2.5 Flash, E3).

Figure 14: Aggregate Cohen’s

\kappa

on the binary harm verdict across the four MLLMs and the XLM-R supervised baseline, by input condition. GPT-4o-mini and Mistral are not natively video-capable and are omitted from E2. Dotted lines:

\kappa=0.21

(fair) and

\kappa=0.41

(moderate) ( Landis and Koch, 1977 ) .

Figure 15: Pairwise Cohen’s

\kappa

between every pair of the eleven Stage-1 (model, condition) combinations (the ten MLLM combinations plus the XLM-R baseline), split by persona country. Off-diagonals are computed on each pair’s intersection of paired video_id s within the country slice (per-pair

n\approx 95

–

108

Figure 16: Country

\to

harm-subcategory mix for the 102 reference-flagged harmful videos. Ribbon widths are proportional to absolute counts.

Metric	Gemini E3	Qwen3-VL-32B E3
$n$ paired	290	292
Cohen’s $\kappa$	0.417	0.173
Accuracy	0.752	0.682
$F_{1}$ harmful	0.591	0.340
$F_{1}$ not-harmful	0.822	0.790
Macro- $F_{1}$	0.706	0.565

Appendix

Table 12: Stage-1 metrics for the winning configuration and the strongest open-weight comparator under the same E3 condition.

Configuration	P ${}_{\textsc{H}}$	R ${}_{\textsc{H}}$	F ${}_{1}^{\textsc{H}}$	F ${}_{1}^{\textsc{NH}}$	mF 1
Gemini E3 (frames)	0.68	0.52	0.59	0.82	0.71
Gemini E2 (native)	0.59	0.58	0.58	0.80	0.69
GPT-4o-mini E3 (frames)	0.56	0.47	0.51	0.78	0.64
Qwen E2 (native)	0.60	0.24	0.35	0.78	0.56
Qwen E3 (frames)	0.57	0.24	0.34	0.79	0.57
Gemini E1 (text)	0.49	0.34	0.40	0.76	0.58

Appendix

Table 13: Stage-1 precision (P

{}_{\textsc{H}}

), recall (R

{}_{\textsc{H}}

), per-class

F_{1}

, and macro-

F_{1}

on the binary harm verdict, across the eleven configurations (ten MLLMs plus the XLM-R baseline) against the two-annotator final reference. Rows sorted by Cohen’s

\kappa

; per-column maxima in bold .

Failure family	E1	E2	E3
Gemini PROHIBITED_CONTENT	3	6	2
Gemini response-parse failure	0	1	5
Qwen 413 RequestTooLarge	0	14	0
Qwen data_inspection_failed	0	2	3
GPT-4o-mini OpenRouter credit-cap	8	n/a	15

Appendix

Table 14: Provider-side refusals on the 300-video Stage-1 subset, by experiment. Each cell counts videos where the model’s response could not be evaluated because the provider blocked the call or returned an unparseable response.

Algorithmic Auditing

Algorithmic auditing evaluates whether automated systems are fair, reliable, robust, compliant, and accountable in their intended settings.

See all 134 papers

Harmful YouTube Video Detection: A Taxonomy of Online Harm and MLLMs as Alternative Annotators
Nov 6, 2024Claire Wonjeong Jo, Miki Wesołowska, Magdalena WojcieszakVideo Content Moderation LLM-Assisted Annotation
University of California, Davis · University of Warsaw
No One Model Catches Every Harm: Benchmarking Content Moderation Across Safety Scenarios
Aug 22, 2026Afshin Orojlooyjadid, Hitesh PatelLLM Safety Evaluation LLM Safety Benchmarks
Oracle AI
Distributed Implicit Harm: A Compositional Safety Blind Spot in MLLM-Based Video Moderation
Aug 31, 2026Ruotong Wang, Zihao Zhu, Siwei Lyu +2Multimodal Large Language Models Video Content Moderation
The Chinese University of Hong Kong, Shenzhen · Kling Team, Kuaishou Technology · State University of New York at Buffalo

Nov 6, 2024J/K move · Enter open · S save

Harmful YouTube Video Detection: A Taxonomy of Online Harm and MLLMs as Alternative Annotators

Claire Wonjeong Jo, Miki Wesołowska, Magdalena Wojcieszak

University of California, Davis · University of Warsaw

Video Content Moderation LLM-Assisted Annotation Multimodal Large Language Models

Open paper AskPDF arXiv

Video platforms, used by billions globally, are criticized for exposing users to harmful content. Yet, detecting online harm remains challenging due to inconsistent approaches to what constitutes harm and limitations of human annotation. We address both challenges. Theoretically, we integrate and expand existing taxonomies and platform guidelines to multimodal contexts and develop a comprehensive taxonomy for online harm. We categorize online harm on video platforms into six content-based categories: Information, Hate and harassment, Addictive, Clickbait, Sexual, and Physical harms. Methodologically, we propose and systematically test multimodal large language models (MLLMs) as alternative annotators of harmful videos. We curate a dataset of 19,422 potentially harmful YouTube videos and annotate them using GPT-4-Turbo, non-trained crowdworkers, and trained annotators. We discuss the potential applications of MLLMs in platform content moderation and social science research, particularly in contexts where they perform comparably to human annotators in binary and multi-label classification tasks.

Abstract

Algorithmic Auditing

Explore similar work

Harmful YouTube Video Detection: A Taxonomy of Online Harm and MLLMs as Alternative Annotators

No One Model Catches Every Harm: Benchmarking Content Moderation Across Safety Scenarios

Distributed Implicit Harm: A Compositional Safety Blind Spot in MLLM-Based Video Moderation