Baszta: Data-Centric Fine-Tuning of a Polish Multi-Label Safety Classifier

Adam Górski; Mateusz Jąkalak; Rafał Jakubowski

cs.AISep 24, 2026

Baszta: Data-Centric Fine-Tuning of a Polish Multi-Label Safety Classifier

Authors: Adam Górski, Mateusz Jąkalak, Rafał Jakubowski

Organizations: Billennium S.A.

Abstract

We develop a multi-label Polish content-safety classifier by fine-tuning allegro/herbert-base-cased (124M) across five categories (hate, vulgarity, sexual content, crime, self-harm) using a Focal + R-Drop objective, and evaluate the resulting model against Bielik Guard (Sójka) on the shared out-of-distribution Gadzi Język benchmark. Both systems are given per-category threshold tuning on the same calibration split. Under that matched protocol our model holds a small but statistically significant lead in micro F1, while an apparent macro-F1 lead does not survive: it was an artifact of comparing a tuned model against an untuned one. We also report what that micro figure is worth. Because Gadzi Język is 97% crime-positive, a classifier that flags crime on every input and nothing else already scores 0.910 micro F1 on the same test split, so micro separates neither system from a degenerate strategy and macro is the column that does. Per-category and per-protocol figures are reported in Section 4. The residual out-of-distribution gap is one of calibration rather than discrimination. Ranking quality stays high while positive probabilities collapse, and per-category temperature scaling recovers the loss where Platt scaling and isotonic regression do not. That recovery turns out to be conditional on the calibration set containing safe text. Gadzi Język contains almost none, so thresholds fitted on it flag crime on every safe input, and a balanced refit buys a deployable operating point at the cost of adversarial recall. We report both operating points rather than only the flattering one. Two changes that are standard practice, per-class cost-sensitive weighting and mean pooling, each raise in-distribution macro F1 while lowering the out-of-distribution figure, which indicates that robustness has to be selected for directly rather than inherited from in-distribution accuracy.

Figures & tables

Component	Configuration
Encoder	allegro/herbert-base-cased (124M)
Head	Linear(768 → 5), dropout 0.22
Loss	Focal (γ = 1.5) + R-Drop (α = 0.20)
Optimizer	AdamW (lr = 9.4e-5, wd = 5e-4)
Schedule	Cosine, 8% warmup
Frozen layers	4 / 12 encoder layers

Table 1: Architecture and training configuration of Baszta 1.0 , the model carried through §4. Values are those of the sweep-selected configuration of §3.4. Focal

\gamma

controls how strongly easy examples are down-weighted, and R-Drop

\alpha

weights the KL consistency term between two dropout passes over the same input.

Encoder	OOD macro	OOD macro (augmented)
HerBERT ( allegro/herbert-base-cased )	0.601	0.611
MMLW-RoBERTa (Sójka’s backbone, sweep-tuned)	0.275	-
Polish RoBERTa	0.282	0.269
XLM-RoBERTa	0.158	0.211

Table 2: Backbone selection. OOD macro F1 on the full 520-sample Gadzi Język set at per-category oracle thresholds, for four Polish-capable encoders trained on the v1 pool. HerBERT, Polish RoBERTa and XLM-RoBERTa share settings exactly (lr 2e-5, γ 2.0, no frozen layers, 512 tokens, 10 epochs). The MMLW row is its best sweep-tuned variant (lr 5.9e-5, γ 3.0), so that encoder received tuning the others did not and the comparison is conservative in its favour. The augmented column applies the online augmentation of §3.1. These are early runs whose absolute values sit below the tuned figures of §4, but the ordering is what selected the backbone.

Name	Configuration
Baszta 0.1	Clean v1 pool, no augmentation
Baszta 0.2	Balanced pool, online augmentation
Baszta 0.3	Balanced pool, aggressive augmentation
Baszta 0.4	Adds style-matched synthetic crime data
Baszta 0.5	Adds synthetic crime , sex and self-harm data
Baszta 1.0	Sweep recipe (§3.4), low-γ focal, v2 synthetic data, per-category temperature scaling

Table 3: Configuration names used in this report. The 0.x series are data and augmentation variants sharing one recipe. The 1.0 series is the sweep-selected recipe of §3.4, with the suffixed entries being controlled single-factor variants of it.

Category	Train support
hate	~3,146
vulgar	~4,037
sex	~4,500 (incl. 501 synthetic)
crime	~4,250 (incl. 286 synthetic)
self-harm	~1,246 (incl. 507 synthetic)

Table 4: Positive-label support per category in the 26,248-sample training pool after v2 augmentation, with the style-matched synthetic contribution shown in parentheses. Multi-label overlap means the column does not sum to the pool size.

Category	Definition used
hate	Hostility, dehumanisation or incitement directed at a person or group, typically on a protected characteristic
vulgar	Profanity, obscenity or crude language, with no requirement that it be aimed at anyone
sex	Sexual content, including solicitation and sexualisation of minors, which also carries crime
crime	Requests for, or instruction in, illegal activity, including violence, weapons, drugs and fraud
self-harm	Content describing, encouraging or seeking means of self-injury or suicide

Table 5: Working definitions of the five categories, adopted from Sójka. The distinction that matters most in practice is hate against vulgar : profanity with no target belongs to vulgar alone, and §4.8 tests exactly this separation on HateCheck-PL.

Model	Hate	Vulgar	Sex	Crime	Self-harm	Macro F1
Baszta 0.2	0.743	0.969	0.985	0.965	0.835	0.900
Baszta 0.1	0.733	0.967	0.978	0.960	0.852	0.898
Baszta 0.3	0.745	0.960	0.981	0.957	0.838	0.896
Sójka 0.1B†	0.628	0.742	0.889	0.707	0.886	0.770

Table 6: Per-category and macro F1 on the 4,632-sample validation split at a fixed threshold

t=0.5

. Baszta 0.1 is the clean-data baseline, Baszta 0.2 adds online augmentation, and Baszta 0.3 uses the aggressive-augmentation recipe. Macro is the unweighted mean over the five categories. Sójka’s row is measured on its own test split and is not a head-to-head comparison.

Model	Hate	Vulgar	Sex	Crime	Self-harm	Macro	Micro
Baszta 1.0	0.575	0.571	0.606	0.965	0.820	0.707	0.927
Baszta 0.4	0.514	0.333	0.621	0.985	0.800	0.651	-
Baszta 0.5	0.435	0.333	0.703	0.985	0.750	0.641	-
Baszta 0.2	0.419	0.333	0.500	0.985	0.778	0.603	0.918
Sweep top-1 (ablation baseline, §4.6)	0.489	0.400	0.686	0.898	0.693	0.633	0.828
Baszta 0.1 (clean, no synthetic)	0.410	0.400	0.650	0.675	0.708	0.569	0.651

Table 7: OOD F1 on Gadzi Język with per-category thresholds fitted on the full 520-sample set. Positives per category are crime 505, hate 43, self-harm 31, sex 18, vulgar 4, so macro (unweighted over categories) is dominated by crime and unstable on vulgar , micro pools all label decisions. The last two rows are reference points used later: the sweep’s top-1 configuration is the baseline every ablation in §4.6 is measured against, and Baszta 0.1 is the clean-data baseline for the synthetic-data gain in §4.7.

Category	OOD AUC	OOD pos-prob mean
hate	0.86	0.55
vulgar	0.49	0.13
sex	0.93	0.21
crime	0.72	0.19
self-harm	0.94	0.27

Table 8: Ranking quality against probability scale on OOD for Baszta 0.2 , over all five categories. AUC is threshold-free, where 1.0 is perfect ranking and 0.5 is chance. Pos-prob mean is the average predicted probability on positive examples. High AUC alongside collapsed probabilities is the signature of a calibration shift rather than a discrimination failure. Vulgar sits at chance because it has only four OOD positives, and crime is the one category whose ranking is also genuinely weak.

Calibration	Gadzi Macro F1
Raw ( $t=0.5$ )	0.178
Platt scaling	0.218
Isotonic regression	0.216
Oracle per-category thresholds	0.603

Table 9: Gadzi Język macro F1 for Baszta 0.2 under post-hoc calibrators fitted on in-distribution validation data. The raw row is the model at a fixed

t=0.5

, while Platt and isotonic recover only

\sim

0.22 against 0.603 for per-category thresholds fitted on the full test set, indicating the domain shift is too large for output recalibration alone. The 0.603 here is this model’s oracle figure and is not the 0.603 reached by Baszta 1.0 under the non-oracle protocol of §4.5.

Protocol (172/348 split)	Test Macro F1
Raw ( $t=0.5$ )	0.493
+ per-category threshold	0.603
+ temperature + threshold	0.699
Sójka reference	0.619

Table 10: Cumulative effect of the protocol on test macro F1: per-category temperatures and thresholds fitted on the disjoint 172-sample calibration split and evaluated on the held-out 348 samples. Sójka’s 0.619 is its published figure at its default 0.5 threshold, not a matched-threshold comparison.

Metric (Test split, N = 348)	Estimate	95% CI
Macro F1 (5 categories)	0.699	[0.493, 0.797]
Macro F1 (4 categories, excl. vulgar)	0.674	[0.569, 0.763]
Micro F1	0.927	[0.903, 0.949]

Table 11: Baszta 1.0 test-split F1 on the 172/348 split, with 95% bootstrap confidence intervals (2,000 resamples of the 348-sample test split). Excluding vulgar (

n=4

positives) narrows the macro interval substantially, micro F1 pools per-label decisions and is therefore far tighter than macro.

Operating point	Balanced macro	Balanced micro	Safe-text FPR	Gadzi macro	Gadzi micro
Gadzi-fit (T_crime=4.3, t_crime=0.025)	0.742	0.541	1.000	0.700	0.932
Balanced re-fit	0.952	0.959	0.055	0.496	0.778

Table 12: The same checkpoint at two operating points. Balanced macro and micro are measured on the held-out half of sojka_val , which carries roughly 870 of its 1,741 safe rows. Safe-text FPR is the fraction of all-negative inputs receiving at least one flag. The Gadzi-fit point maximises adversarial F1 but flags crime on every safe input and is not deployable. The two Gadzi Język columns are measured on the full 520 samples, which is why the Gadzi-fit row reads 0.700 / 0.932 where §4.5 reports 0.699 / 0.927 for the same operating point on its 348-sample test split.

Metric (both systems tuned on same cal split)	Ours	Sójka	Diff (ours−Sójka)	95% CI	P(ours not better)
Micro F1	0.929	0.903	+0.026	[+0.004, +0.049]	0.011
Macro F1 (5-cat)	0.712	0.782	−0.070	[−0.263, +0.046]	0.875
Always-crime baseline (micro / macro)	0.910 / 0.197	-	-	-	-

Table 13: Paired bootstrap comparison against Sójka with both systems tuned on the same 187-sample calibration split (temperature + threshold for ours, threshold for Sójka), evaluated on the disjoint 333-sample test split over 2,000 resamples. “P(ours not better)” is the fraction of resamples in which the difference is

\leq 0

, which is the one-sided p-value, and a 95% CI excluding zero indicates significance. The final row is the degenerate reference described below, scored on the same test split.

Benchmark	Rows	In training	Status
PL-Guard (NASK) [ 12 ]	900	899	excluded - memorised
PL-Guard-adv (NASK)	900	765	excluded - memorised
KLEJ CBD (test) [ 25 ]	1,000	149	149 dropped → 851 held-out
BAN-PL_1 [ 9 ]	24,000	461	461 dropped → 23,539 held-out
HateCheck-PL [ 24 ]	3,815	0	fully held-out
PolyGuardPrompts-PL [ 13 ]	1,725	0	fully held-out

Table 14: Exact-text contamination audit against the v2 training pool (whitespace-normalised, case-folded matching). “In training” counts matched rows. Near-total overlap disqualifies a benchmark outright, partial overlap is handled by dropping the matched rows before evaluation.

Benchmark (held-out)	N	Task / head	F1 (95% CI)	Safe/neg FPR
KLEJ CBD (clean)	851	cyberbully / hate-head	0.671 [0.606, 0.735]	0.102
BAN-PL_1 (clean)	23,539	harmful / any-head	0.667 [0.661, 0.673]	0.855
PolyGuardPrompts-PL	1,725	harmful / any-head	0.562 [0.531, 0.591]	0.357
HateCheck-PL	3,815	hate / hate-head	0.644 (diagnostic)	0.334

Table 15: Held-out benchmark F1 for Baszta 1.0 at the deployable balanced operating point with no re-tuning (95% bootstrap CIs, 2,000 resamples). Each benchmark’s binary label is mapped to the named head, and safe/neg FPR is the false-positive rate on that benchmark’s own negative class, whose definition differs per set. BAN-PL’s “neutral” rows are unmoderated but profane rather than safe, which drives its outlier rate for a detector trained on presence of harmful language. Sójka reaches 0.230 on those same rows, so the rate reflects our training objective rather than a property of the benchmark. Reading (3) below takes this up.

Failure mode (hate head)	Worst functionalities (accuracy)
False positives (fires on non-hateful)	counter-speech quoting/referencing hate (0.48–0.61), positive/neutral identity mentions ( ident_pos_nh 0.57, target_group_nh 0.51)
False negatives (misses hateful)	slur-only hate ( slur_h 0.30), implicit/emotive derogation (0.37), space-obfuscated hate ( spell_space_add_h 0.41)

Table 16: Per-functionality diagnostic of the hate head on HateCheck-PL. Values are accuracy within each functional test template (not F1), the listed functionalities are the worst-scoring ones in each failure direction, isolating which phenomena misfire rather than summarising overall performance.

Figure 1: Held-out performance on the four contamination-audited public benchmarks. Every system is scored on the same rows after the same exact-text filtering, with the same task-to-head mapping, and at per-category thresholds fitted on the same balanced sojka_val source, so no system is compared tuned against untuned. Left: F1, higher is better. Right: false-positive rate on each benchmark’s own negative class, lower is better, with the dashed line at the 5.5% rate the same operating point achieves on genuinely safe text. Reading the two panels together matters: a tall left bar beside a tall right bar is recall bought with false positives.

Benchmark (N)	Head	Baszta 1.0	Sójka (0.5)	Sójka (matched)	FPR ours / theirs
KLEJ CBD (851)	hate	0.671	0.088	0.438	0.102 / 0.087
BAN-PL (23,539)	any	0.667	0.694	0.754	0.855 / 0.230
PolyGuard-PL (1,725)	any	0.562	0.290	0.475	0.357 / 0.197
HateCheck-PL (3,815)	hate	0.644	0.527	0.767	0.334 / 0.661

Table 17: The same comparison numerically. Head is the output used for each benchmark’s binary task. Sójka is shown at both its default 0.5 threshold and the matched sojka_val fit, because the two differ enough to change the conclusion. The matched fit is the column plotted in Figure 1 and is generous to Sójka, since sojka_val contains rows from the corpus it was trained on.

Figure 2: Every system reduced to one binary unsafe/safe decision, on identical contamination-masked rows. Ours and Sójka’s are at their sojka_val -fitted operating points, the third-party guards at their own default decision rule with no threshold fitting. Read the panels together: a tall left bar beside a tall right bar is recall bought with false positives. Gadzi Język is absent because all 520 of its rows are positive, so it has no false-positive axis, and its detection rates are in the table below. Two confounds run against us and are stated in the text, the broader taxonomies flag categories these benchmarks label unharmful, and only our own training pool could be contamination-audited.

System	KLEJ CBD	BAN-PL	PolyGuard-PL	HateCheck-PL	Gadzi detect.
Baszta 1.0 (124M)	0.669 / 0.109	0.667 / 0.855	0.562 / 0.357	0.710 / 0.578	0.906
Sójka 0.1B v1.1 (124M)	0.433 / 0.131	0.754 / 0.230	0.475 / 0.197	0.784 / 0.745	0.675
HerBERT-PL-Guard (124M)	0.459 / 0.266	0.762 / 0.354	0.816 / 0.070	0.786 / 0.593	0.992
Qwen3Guard-Gen (0.6B)	0.262 / 0.028	0.263 / 0.078	0.766 / 0.085	0.636 / 0.384	0.985

Table 18: The same comparison numerically, as F1 / false-positive rate on negatives per benchmark. The last column is the detection rate on all 520 Gadzi Język prompts, which are all positive, so it is recall and not F1 and an all-positive predictor scores 1.000 on it. Generated from reports/external_guards/*.json by scripts/plot_cross_taxonomy_guards.py , so no figure here is transcribed by hand. Qwen3Guard-Gen is scored strictly, with its “Controversial” tier counted as safe.

Model (Gadzi, macro over 5 cats)	ECE	Adaptive-ECE	MCE	Brier
Baszta 1.0 raw	0.092	0.096	0.691	0.075
Baszta 1.0 + temperature	0.107	0.111	0.753	0.068
Sójka 0.1B	0.136	0.139	0.619	0.114

Table 19: Calibration on the full 520-sample Gadzi Język set, macro-averaged over the five categories. ECE and Adaptive-ECE (equal-mass bins) measure the mean confidence - accuracy gap, MCE the worst single bin, Brier the mean squared probability error, lower is better throughout. Temperature scaling improves Brier while slightly worsening ECE.

Figure 3: Per-category detection reliability for Baszta 1.0 on the Gadzi Język OOD set: mean predicted probability (x-axis) against empirical positive frequency (y-axis) per confidence bin. A point above the diagonal is one where the observed positive rate exceeds the predicted probability, which is the signature of under-confidence, the calibration scale shift that per-category temperature scaling corrects. Points below the diagonal indicate the opposite, over-confidence.

Attack (Gadzi, balanced op-point)	No defence: macro / flip	Deobfuscated: macro / flip
homoglyph	0.477 / 0.055	0.496 / 0.000
leetspeak	0.415 / 0.071	0.471 / 0.024
space-insertion	0.409 / 0.100	0.485 / 0.024
char-repetition	0.296 / 0.146	0.476 / 0.030
diacritics-strip	0.486 / 0.026	0.486 / 0.026
combined-heavy	0.343 / 0.124	0.426 / 0.078

Table 20: Adversarial robustness at the deployable balanced operating point (clean-input reference macro 0.496). Macro is Gadzi macro F1 under each attack. Flip is the fraction of per-label predictions that change relative to the clean input, so lower is better. Diacritics-stripping is deliberately not reversed by the canonicalizer, hence the identical columns.

Baseline	Val macro / micro	Gadzi macro / micro
TF-IDF + LogReg	0.868 / 0.883	0.462 / 0.642
TF-IDF + Compl-NB	0.717 / 0.763	0.393 / 0.690
HerBERT (Baszta 1.0)	0.9111 / 0.9585 (val)	0.496 / 0.778 (balanced op-point)

Table 21: Classical lexical floor: TF-IDF (word 1–2 gram + char 3–5 gram) with one-vs-rest logistic regression and Complement Naive Bayes, thresholds tuned on validation only. The HerBERT row quotes its deployable balanced operating point, whose thresholds come from sojka_val rather than from the full validation split, so the three rows share a non-oracle protocol but are not fitted on identical data.

Model	Val macro	Gadzi macro	Gadzi micro	OOD ECE	OOD Brier
Baszta 1.0 (no alpha)	0.9111	0.699	0.927	0.092	0.075
Baszta 1.0-α (inverse-freq alpha)	0.9136	0.656	0.927	0.133	0.093

Table 22: Controlled effect of per-class focal alpha set to inverse training frequency, with data, schedule, and all other hyperparameters held fixed. OOD figures use the 172/348 protocol, lower ECE and Brier are better. Cost-sensitive weighting raises in-distribution macro while lowering OOD macro and degrading calibration.

Recipe	Val macro	OOD macro	OOD micro vs. Sójka	OOD ECE / Brier
Baszta 1.0 ([CLS], no alpha)	0.9111	0.699	0.929 (p = 0.011 ✓)	0.092 / 0.075
Baszta 1.0-α (per-class alpha)	0.9136	0.656	0.927 (p = 0.017 ✓)	0.133 / 0.093
Baszta 1.0-mp (mean pooling)	0.9138	0.606	0.913 (p = 0.218 ✗)	0.109 / 0.093

Table 23: The three training recipes ranked in- versus out-of-distribution. OOD figures use the 172/348 protocol. The OOD macro column uses the 172/348 split, while the OOD micro column and its p-value come from the paired bootstrap of §4.5.2, which uses that script’s own 187/333 split. This is why micro reads 0.929 here and 0.927 in the table above for the same checkpoint. Validation macro ranks the recipes in exactly the reverse order of OOD macro.

Figure 4: Detection reliability for Baszta 1.0-α (per-class cost-sensitive alpha). Compared with Baszta 1.0 the curves sit further from the diagonal, which is the visual form of the ECE rise from 0.092 to 0.133 reported above.

Figure 5: Detection reliability for Baszta 1.0-mp (mean pooling), the variant with the best validation macro and the worst OOD behaviour of the three.

Model / ensemble	OOD macro	OOD micro (vs. Sójka)	OOD Brier
Baszta 1.0 (single, best macro)	0.712	0.929 (p = 0.011)	0.075
Baszta 1.0 + Baszta 1.0-α + Baszta 1.0-mp (mean-prob)	0.632	0.932 (p = 0.005)	0.071
Baszta 1.0 + Baszta 1.0-α (mean-prob)	0.660	0.933 (p = 0.004)	0.070

Table 24: Deep ensembles formed by averaging per-category temperature-scaled probabilities, on the same split as §4.5.2. p-values are paired-bootstrap micro-F1 comparisons against Sójka. Lower Brier is better. The ensemble improves micro-F1 significance and probability quality but not macro, which the weaker members drag down.

Clean held-out benchmark	Baszta 1.0	Baszta 1.0-α	Baszta 1.0+Baszta 1.0-α ensemble
HateCheck-PL hate F1	0.644	0.613	0.645
KLEJ CBD hate F1	0.671	0.687	0.671
BAN-PL any-head F1	0.667	0.663	0.667
PolyGuard-PL binary F1	0.562	0.511	0.517

Table 25: The same three configurations on the contamination-audited external benchmarks, each at its own sojka_val -fit deployable operating point with no re-tuning. The ensemble’s Gadzi Język gain does not transfer: it matches single-model Baszta 1.0 on three of four sets and is worse on PolyGuard-PL.

Dimension	Ours (Baszta 1.0+temp)	Sójka 0.1B	Δ / note
Encoder	HerBERT (124M)	MMLW-RoBERTa (100M)	similar
Loss	Focal (γ=1.5) + R-Drop	BCE	-
Training data	26,248 (+synthetic)	~6.9K	larger
Compute	1× T4 and 2× GH200, FP16	A100 cluster	constrained
Gadzi micro F1	0.927 [0.903, 0.949]	0.582	Sójka untuned, see below
Gadzi macro (cal split)	0.699 [0.493, 0.797]	0.619	directional (81% boot.)

Table 26: Head-to-head summary against Sójka 0.1B. Every both-tuned row gives both systems per-category tuning on the same 187-sample calibration split and scores them on the disjoint 333 samples, and those are the only rows from which a head-to-head margin should be read. The rows above them use the 172/348 split of §4.5 and leave Sójka at its default 0.5 threshold, so their large apparent margins are operating-point effects rather than model differences. Under matched tuning the per-category picture changes substantially: Sójka’s crime F1 rises from 0.587 to 0.946 and its hate F1 from 0.045 to 0.478, which turns the previously reported +37.8 pp and +53.0 pp per-category gaps into +3.7 pp and +1.2 pp. The oracle row fits thresholds on the full test set and is an optimistic upper bound. The deployable row is in-distribution and has no Sójka counterpart, so it is not a comparison, and neither is the validation row.

Explore similar work

May 28, 2026cs.LG

Opir: Efficient Multi-Task Safety Classification for Toxicity, Jailbreaks, Hate Speech, and Harmful Content

Real-time safety filtering for large language model (LLM) applications requires classifiers that can detect unsafe prompts, toxic language, jailbreak attempts, and unsafe responses without the cost profile of large guardrail models, and that can distinguish benign sensitive text from genuinely covert harmful content. In this paper, we introduce Opir, a family of encoder-based guardrail models built on the GLiClass architecture. Opir includes multi-task models for binary safe/unsafe classification, multi-label toxicity classification, jailbreak classification, and zero-shot unsafe prompt and response categorization. We also release edge variants with fewer than 100M parameters dedicated to binary safe/unsafe categorization. The models are trained on a three-level taxonomy containing 996 categories across 16 top-level labels, 126 mid-level labels, and 854 leaf labels. Opir's training data combines taxonomy-grounded unsafe prompts, adversarially mined hard negatives, benign safety-preserving examples, generated response examples, multilingual translations, and portions of the Aegis2 and WildGuard training subsets. We also open-sourced an evaluation harness that supports GLiClass and GLiNER2 backends as well as decoder-based models, and covers binary safety classification, multi-label categorization, toxicity, jailbreak detection, prompt safety, response safety, response refusal, and prompt subcategory views across public benchmark families. Across an expanded comparison spanning 12 safety-classification tasks and 17 category tasks against eight contemporary guardrail systems -- including both GLiNER2-based and generative guardrail models -- Opir variants are competitive on or ahead of the strongest open-weight baselines on the majority of benchmark datasets while operating with a substantially smaller deployment footprint.

Ihor Stepanov, Aleksandr Smechov

1Knowledgator, Kyiv, Ukraine · 2Wordcab, New York, USA

Large Language Model Safety Guardrail Jailbreak Attacks Jailbreaks Harmful Content

Ask PDF

Jul 2, 2026cs.CL

HaloGuard 1.0: An Open Weights Constitutional Classifier for Multilingual AI Safety

We present HaloGuard 1.0, an open-weights implementation of the constitutional-classifier paradigm for input safety. It achieves state-of-the-art performance on English and multilingual prompt-safety benchmarks at roughly one-tenth the model size of current leading open guard models. The safety constitution is the organising structure of the corpus: a natural-language constitution of 46 policies and 2,940 subcategories drives synthetic data generation, with exhaustive one-to-one paired counterfactuals that hold topic and vocabulary fixed while flipping intent, a two-tier harmless design that separately targets boundary and baseline false positives (FPs), and balanced multilingual materialisation across 46 languages that treats language as a surface form appearing on both sides of the boundary rather than as an adversarial signal. Across seven prompt-safety benchmarks, HaloGuard 1.0-0.8B attains the best average F1 (90.9) of any open guard we evaluate, outperforming baselines up to 27B parameters (over 30 times larger) while holding false-positive rate (FPR) to 4.3 and false-negative rate (FNR) to 9.5. The HaloGuard 1.0-4B variant reaches average F1 of 92.1 and FPR of 3.5, spending its extra capacity on precision rather than recall. A structured adjudication of the remaining failures indicates that most apparent missed-harm cases are benchmark mislabels rather than genuine model misses. An always-on adversarial red-teaming protocol continuously hardens the guard against both content-level and agentic attacks. We release the models as open weights.

Navaneeth Sangameswaran, Preetham S, Ashmiya Lenin

Astroware AI, Delaware, USA · Karunya Institute of Technology and Sciences, Coimbatore, India

Large Language Model Safety Adversarial Robustness

Ask PDF

Jul 28, 2026cs.CL

Shieldstral

We introduce Shieldstral, a 3B-parameter policy-adaptive multimodal safety classifier that matches or outperforms models nearly 7

\times

its size on text safety benchmarks and sets a new state of the art on multimodal safety classification. Shieldstral formulates content moderation as a binary question-answering task. This simple formulation unifies diverse moderation tasks into a single yes/no problem, enabling heterogeneous safety datasets with divergent taxonomies to be consolidated under one training framework. We present the data construction recipe, covering curation and generation of approximately 54.1M samples and a fine-grained evaluation set to evaluate policy adaptability. Together, these enable a small adaptive model to match or outperform much larger models.

Antonia Calvi, Avinash Sooriyarachchi, Giada Pistilli +273

Content Moderation

Ask PDF