Organizations: School of Computer Science, Wuhan University, Wuhan, Hubei, China · Zhongguancun Academy, Beijing, China · Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Shandong Computer Science Center (National Supercomputer Center in Jinan), Qilu University of Technology (Shandong Academy of Sciences), Jinan, Shandong, China; Shandong Provincial Key Laboratory of Computing Power Internet and Service Computing, Shandong Fundamental Research Center for Computer Science, Jinan, Shandong, China
Toxicity detection mitigates the dissemination of toxic content (e.g., hateful comments, posts, and messages within online social actions) to safeguard a healthy online social environment. However, malicious users persistently develop evasive perturbations to disguise toxic content and evade detectors. Traditional detectors or methods are static over time and are inadequate in addressing these evolving evasion tactics. Thus, continual learning emerges as a logical approach to dynamically update detection ability against evolving perturbations. Nevertheless, disparities across perturbations hinder the detector's continual learning on perturbed text. More importantly, perturbation-induced noises distort semantics to degrade comprehension and also impair critical feature learning to render detection sensitive to perturbations. These amplify the challenge of continual learning against evolving perturbations. In this work, we present ContiGuard, the first framework tailored for continual learning of the detector on time-evolving perturbed text (termed continual toxicity detection) to enable the detector to continually update capability and maintain sustained resilience against evolving perturbations. Specifically, to boost the comprehension, we present an LLM-powered semantic enriching strategy, where we dynamically incorporate possible meaning and toxicity-related clues excavated by LLM into the perturbed text to improve the comprehension. To mitigate non-critical features and amplify critical ones, we propose a discriminability-driven feature learning strategy, where we strengthen discriminative features while suppressing the less-discriminative ones to shape a robust classification boundary for detection...
Figures & tables
Figure 1. Illustration of our ContiGuard framework.
Existing Detectors
Insert
Repeat
Swap
Remove
Homo
Mask
Abbr
Auth
Distract
Avg
PespectiveAPI
77.20
64.61
70.48
60.51
64.84
63.29
50.93
66.38
64.37
64.7325.9%↓
OpenAI Moderation
66.15
52.24
61.13
56.49
50.78
59.35
51.00
84.47
81.14
62.5325.2%↓
ToxBERT
60.90
60.97
58.81
58.34
53.55
55.87
58.35
73.26
75.12
61.6914.4%↓
ToxRoBERTa
50.93
50.39
51.24
52.32
50.15
50.31
51.39
55.26
79.83
54.6534.0%↓
DetoxifyOriginal
61.59
63.76
61.90
57.73
56.96
59.27
56.03
71.87
75.66
62.7529.8%↓
DetoxifyMultilingual
60.20
58.50
56.41
56.96
82.53
59.20
52.63
66.62
81.92
63.8826.6%↓
Table 1. The results of ContiGuard and existing detectors on perturbed text. The subscripts indicate the relative performance degradation compared with ordinary text. The best scores are highlighted in bold .
Continual
T1
T2
T3
T4
T5
T6
T7
T8
T9
OT91
OT92
OT93
OT94
Joint
84.14
86.63
85.43
85.07
84.56
85.05
85.03
85.31
85.88
85.63
85.51
86.31
86.06
Stream
84.23
84.64
76.52
77.36
78.17
79.55
79.73
74.28
78.26
79.49
74.08
78.95
80.49
LFL
83.46
82.88
77.15
76.59
75.30
78.25
78.44
75.01
78.15
79.48
74.19
79.07
79.85
EWC
84.66
84.88
76.41
78.48
77.89
80.89
78.64
76.01
79.32
79.81
75.43
80.25
81.80
LWF
83.46
85.08
79.39
79.86
76.31
76.92
75.56
72.58
72.30
72.29
74.94
65.40
76.56
EPI
84.12
81.46
81.60
79.13
79.90
78.23
76.98
78.26
79.20
79.03
79.50
79.19
79.07
Table 2. The results of ContiGuard and adapted CL methods. Ti denote averaged performance of all orders at i-th moment. OT9j denote performance of j-th order at T9 moment. * indicates significant difference at p<0.001.
Figure 2. Results of statically fine-tuned detectors. Rows: types used for fine-tuning. Columns: types used for testing.
Figure 3. Results on different types of perturbed text at T9 . The numbers denote the best accuracy for each type.
T1
T2
T3
T4
T5
T6
T7
T8
T9
ContiGuard
87.17
89.27
86.20
86.42
85.41
86.29
85.41
84.92
86.47
w/o aux
83.11
84.63
80.52
81.03
80.03
81.85
82.12
80.06
81.97
w/o coop
87.42
88.88
85.62
85.71
85.90
87.16
85.63
86.22
85.96
w/o disc
86.20
87.71
84.08
85.63
86.18
85.30
84.49
84.74
85.71
w/o more
86.36
88.84
85.61
86.07
86.22
85.78
84.75
84.59
85.89
w/o less
86.78
88.86
86.49
86.11
86.17
85.59
84.44
84.06
86.25
Table 3. Ablation results at different moments.
Figure 4. Results of different memory sample numbers.
Figure 5. Retention rate of critical features, which denotes the proportion of historical critical features in current critical features. Each column shows the changes over time.
Methods
T1
T2
T3
T4
T5
T6
T7
T8
T9
IG
87.17
89.27
86.20
86.42
85.41
86.29
85.41
84.92
86.47
SHAP
86.73
89.33
84.86
85.04
86.30
85.96
84.15
85.44
85.94
Table 4. Results of different contribution measuring methods.
Figure 6. Results on DynEscape and NoisyHate.
Figure 7. Results of different training data sampling rates.
Figure 8. Case study for examining the impact of the LLM.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Perturbations
Examples
Insert
moron → mo*ro#n; idiot → idi+ot
Remove (Mv)
moron → moro; idiot → idot
Repeat
moron → mooron; idiot → iddiot
Swap
fool → folo; idiot → idoit
Homoglyph (Homo)
idiot → id10t ; fool → f001
Maskword (Mask)
fool → f**l; idiot → i*iot
Appendix
Table 5. Examples of different perturbations.
DynEscape (ord.)
DynEscape
NoisyHate
4o-mini
85.81
69.5019%↓
80.06
Appendix
Table 6. Result of 4o-mini. DynEscape (ord.) represents the original ordinary sample corresponding to DynEscape.
Department of Information and Decision Sciences University of Minnesota Minneapolis, USA · Department of Industrial Engineering and Management Sciences Northwestern University Evanston, USA · Fuqua School of Business Duke University Durham, USA +1