The Punjabi Geet Toxicity Dataset (PGT-FG): Fine-Grained Multimodal Annotations for Toxic Content in Punjabi Songs

Lakhjeet Kaur; Jaspreet Kaur; Komal Rani; Yougal Attri; Ramakant KumarJaydeep Kishore; Pravin Kumar

cs.CLOct 23, 2026

The Punjabi Geet Toxicity Dataset (PGT-FG): Fine-Grained Multimodal Annotations for Toxic Content in Punjabi Songs

Authors: Lakhjeet Kaur, Jaspreet Kaur, Komal Rani, Yougal Attri, Ramakant KumarJaydeep Kishore, Pravin Kumar

Organizations: Department of Computer Science, Guru Gobind Singh Indraprastha University, Delhi, India. · Department of Computer Science, Malwa College Bathinda, Punjab, India. · Department of Management, Akal University Talwandi Sabo, Punjab, India. · Independent Researcher, New Delhi, India. ·

Abstract

Resources for toxicity detection in Punjabi remain limited, particularly for musical content that combines linguistic and multimodal information. We present the Punjabi Geet Toxicity Dataset (PGT-FG), a segment-level resource developed to support research on toxicity detection in Punjabi musical content. The dataset was constructed from publicly available Punjabi songs collected from YouTube. An initial collection of 620 songs resulted in 465 successfully downloaded songs and 3,402 preliminary segments from 450 songs. After quality control and annotation, the final dataset contains 2,396 unique segments from 392 songs. Four native Punjabi annotators produced 2,870 annotation records, including a shared subset of 158 segments annotated by all four. Each annotation includes binary toxicity, eight fine-grained toxicity categories, severity, sentiment, mode, and an uncertainty flag. Inter-annotator agreement was measured with Fleiss' kappa on the shared subset, and majority-based resolution produced the final segment-level labels. Of the 2,396 segments, 705 (29.42%) were labelled toxic and 1,691 (70.58%) non-toxic. Violence was the most frequent category, occurring in 539 toxic segments (76.45%), followed by gang references (227, 32.20%) and drug references (133, 18.87%). The overall toxicity label achieved substantial agreement (kappa = 0.6521). PGT-FG provides a fine-grained resource for Punjabi toxicity detection and multimodal content analysis. The resolved dataset, with one final record per segment, is publicly available on Zenodo (https://doi.org/10.5281/zenodo.23040156). Original lyrics, lyric excerpts, and copyrighted audio or video content are not redistributed.

Figures & tables

Dataset	Language	Modalities	Samples (Segments)	Label Types
ToxCMM/ToxVidLM	Hindi–English	Text, Audio, Video	4,021	Toxicity, Sentiment, Severity
HateMM	English	Text, Audio, Video	43 hrs	Hate/Non-hate
MultiHateClip	English, Chinese	Video	2,000	Hate/Offensive/Normal
HateXplain	English	Text	20,148	Hate/Offensive/Normal, Target, Rationales
OLID	English	Text	14,000+	Offensive, Target, Target Type
HASOC	English, Hindi, German	Text	17,657	Hate/Offensive/Non-offensive

Table 1: Comparison of PGT-FG with existing toxicity-related datasets.

Figure 1: Dataset construction and quality-control pipeline. The initial inventory contained 620 songs, of which 465 were successfully downloaded and 450 were retained after initial filtering. These songs produced 3,402 candidate segments. A lyric-availability check retained all 450 songs and 3,402 segments. Subsequent data-cleaning and multimodal quality-control procedures removed 53 songs and 1,006 segments, resulting in a final dataset of 397 unique songs and 2,396 unique segments. The final annotation collection contains 2,870 original annotation records because 158 shared segments were independently annotated by four annotators.

Label	Type	Allowed Values and Meaning
Toxicity	Binary	0 = Non-toxic; 1 = Toxic
Categories	Multi-label	Violence, Misogyny, Casteism, Religious, Drug, Gang References, Profanity, Other
Severity	Ordinal	$-1$ = N/A; 0 = Mild; 1 = Moderate; 2 = Severe
Sentiment	Categorical	0 = Negative; 1 = Neutral; 2 = Positive
Mode	Categorical	$-1$ = N/A; 0 = Explicit; 1 = Implicit; 2 = Mixed
Uncertainty Flag	Binary	0 = No uncertainty flagged; 1 = Uncertainty flagged

Table 2: Label schema used for annotation of PGT-FG.

Label	Decision Criterion
Severity = $-1$	Non-toxic or not applicable
Severity = 0	Mild content, such as a single profanity or a brief mention of harmful content
Severity = 1	Moderate content, including sustained glorification, objectification, or general threats
Severity = 2	Severe content, including explicit violence, named targets, graphic content, or incitement
Mode = $-1$	Not applicable
Mode = 0	Explicit harmful content stated directly

Table 3: Decision criteria for severity and mode annotation.

Label	$\kappa$	Interpretation
Toxicity	0.6521	Substantial
Violence	0.6354	Substantial
Misogyny	0.6651	Substantial
Casteism	-0.0016	Poor
Religious	N/A	Not estimable – no variation
Drug	0.7488	Substantial

Table 4: Inter-annotator agreement measured using Fleiss’ Kappa on the shared subset of 158 segments independently annotated by four annotators.

Statistic	Value
Total Songs Collected	620
Successfully Downloaded Songs	465
Initial Segments	3,402
Final Unique Songs	397
Final Unique Segments	2,396
Annotation Records	2,870

Table 5: Overall statistics of PGT-FG.

Toxicity	Count	Percentage
Non-toxic	1,691	70.58%
Toxic	705	29.42%
Total	2,396	100.00%

Table 6: Distribution of toxic and non-toxic segments.

Figure 2: Distribution of toxic and non-toxic segments in the final dataset.

Figure 3: Sentiment distribution within the toxic and non-toxic segments. Positive sentiment was the dominant category in both groups, accounting for 63.87% of non-toxic segments and 70.35% of toxic segments. Negative sentiment accounted for 19.46% of non-toxic and 13.62% of toxic segments, while neutral sentiment accounted for 16.68% and 16.03%, respectively. The distribution illustrates that sentiment and toxicity are distinct annotation dimensions and that toxic content can occur within segments expressing an overall positive sentiment.

Severity	Count	Percentage
Mild	112	15.89%
Moderate	357	50.64%
Severe	236	33.48%
Total	705	100.00%

Table 7: Severity distribution of toxic segments.

Sentiment	Count	Percentage
Negative	425	17.74%
Neutral	395	16.49%
Positive	1,576	65.78%
Total	2,396	100.00%

Table 8: Sentiment distribution of unique song segments.

Mode	Count	Percentage
Explicit	567	80.43%
Implicit	106	15.04%
Mixed	32	4.54%
Total	705	100.00%

Table 9: Mode distribution of toxic segments.

Toxicity Category	Count	Percentage
Violence	539	76.45%
Misogyny	20	2.84%
Casteism	2	0.28%
Religious	0	0.00%
Drug	133	18.87%
Gang References	227	32.20%

Table 10: Distribution of fine-grained toxicity categories among toxic segments.

Figure 4: Distribution of fine-grained toxicity categories among the 705 toxic segments in the final dataset. Since the categories are multi-label, a segment may belong to more than one category; therefore, the percentages do not sum to 100%.

Field	Data Type	Allowed Values	Description
verse_id	String	Unique identifier	Unique identifier assigned to each song segment.
song_id	String	Song identifier	Identifier linking the segment to its source song.
start_sec	Float	$\geq 0$	Start time of the segment in seconds.
end_sec	Float	$\geq 0$	End time of the segment in seconds.
duration_sec	Float	$>0$	Duration of the segment in seconds.
artist_name	String	Artist name	Name of the artist associated with the source song.

Table 11: Schema of the released PGT-FG annotation CSV file.

Multimodal Dataset

See all 48 papers

ToxSyn-PT: A Synthetic Fine-Grained Dataset of Minority-Targeted Toxic Language in Portuguese
Jun 11, 2025Iago Alves Brito, Julia Soares Dollis, Fernanda Bufon Farber +2Synthetic Data Generation Hate Speech Detection
Advanced Knowledge Center for Immersive Technologies (AKCIT) · Federal University of Goiás
Beyond Content: A Comprehensive Speech Toxicity Dataset and Detection Framework Incorporating Paralinguistic Cues
May 15, 2026Zhongjie Ba, Liang Yi, Peng Cheng +3Speech Processing Hate Speech Detection
The State Key Laboratory of Blockchain and Data Security, Zhejiang University · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security, Hangzhou, Zhejiang, China · School of Cyber Science and Engineering, Nanjing University of Science and Technology
ToxiREX: A Dataset on Toxic REasoning in ConteXt
Jun 26, 2026Stefan F. Schouten, Ilia Markov, Piek VossenLanguage Model Safety Evaluation Toxicity Detection
Vrije Universiteit Amsterdam, the Netherlands

Jun 11, 2025J/K move · Enter open · S save

ToxSyn-PT: A Synthetic Fine-Grained Dataset of Minority-Targeted Toxic Language in Portuguese

Iago Alves Brito, Julia Soares Dollis, Fernanda Bufon Farber, Diogo Fernandes+1

Advanced Knowledge Center for Immersive Technologies (AKCIT) · Federal University of Goiás

Synthetic Data Generation Hate Speech Detection Synthetic Benchmark Generation Low-Resource Language Processing

Open paper AskPDF arXiv

The development of robust hate speech detection systems remains limited by the lack of large-scale, fine-grained training data, especially for languages beyond English. Existing corpora typically rely on simplistic toxic and non-toxic labels, and the few that capture hate directed at specific minority groups lack the positive counterexamples required to distinguish genuine hate from mere discussion. In this work, we introduce ToxSyn-PT, the first Portuguese large-scale corpus explicitly designed for multi-label hate speech detection across nine protected minority groups, including the non-toxic counterexamples absent in all other public datasets. Generated via a controllable four-stage pipeline, ToxSyn contains discourse-type annotations to capture rhetorical strategies of toxic/non-toxic language, such as sarcasm, dehumanization, and cultural appreciation. Our experiments reveal a catastrophic, mutual generalization failure compared to existing datasets from social-media domains: models trained on social media struggle to generalize to minority-specific contexts, and vice-versa. This finding indicates they are distinct tasks and exposes summary metrics like Macro F1 can be unreliable indicators of true model behavior, as they completely mask model failure. We publicly release ToxSyn on HuggingFace to support reproducible research on synthetic data generation and benchmark progress in hate-speech detection for low- and mid-resource languages.

Abstract

Multimodal Dataset

Explore similar work

ToxSyn-PT: A Synthetic Fine-Grained Dataset of Minority-Targeted Toxic Language in Portuguese

Beyond Content: A Comprehensive Speech Toxicity Dataset and Detection Framework Incorporating Paralinguistic Cues

ToxiREX: A Dataset on Toxic REasoning in ConteXt