VARM-Bench: Benchmarking Verifiable Structured Reasoning in Chinese Abusive Speech Moderation

Mingyu Yuan; Shengtao Wen; Lingbing Guo; Zhen Bi; Xiang Chen

cs.AIAug 16, 2026

VARM-Bench: Benchmarking Verifiable Structured Reasoning in Chinese Abusive Speech Moderation

Authors: Mingyu Yuan, Shengtao Wen, Lingbing Guo, Zhen Bi, Xiang Chen

Organizations: MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics · Nanjing University · College of Computer Science, Huzhou Normal University

Abstract

The widespread circulation of abusive online content has increased the need for reliable moderation of Chinese social-media text. Existing Chinese benchmarks support label classification, fine-grained toxicity categorization, and target-aware extraction, but do not provide a unified representation for deterministically verifying the stated basis of a moderation decision. We introduce VARM-Bench, a benchmark for field-anchored chain-of-thought rationales in Chinese abusive-speech moderation. Each instance contains a concise natural-language rationale with explicit anchors for six decisions: target, target type, target explicitness, author stance, harmfulness label, and fine-grained category. Our deterministic protocol evaluates field correctness, target alignment, output validity, complete-record agreement, and hidden record errors conditioned on correct final decisions, without relying on an LLM judge. Under a common structured-output protocol, we evaluate language models across multiple model families using zero-shot prompting, taxonomy guidance, and structured CoT supervision, and analyze lexical-cue sensitivity and field-level errors. Results show that strong label-level performance can conceal substantial errors in complete moderation records. VARM-Bench provides an auditable and reproducible benchmark for evaluating verifiable moderation rationales in Chinese abusive-speech moderation.

Figures & tables

Figure 1: Motivation for VARM-Bench. Similar lexical cues can correspond to attacks, quotations, opposition, or neutral mentions. A final label alone cannot reveal whether the target, stance, and harm category are jointly correct.

Figure 2: Overview of the data construction, annotation, and validation pipeline. Multi-source Chinese comments are normalized, anonymized, and filtered, annotated through LLM-assisted pre-annotation and human review, and finally evaluated at the target, scope, stance, harm, and record levels.

Figure 3: Seven-way category distribution of VARM-Bench across the train, development, and test splits.

Model	Setting	Decision		Structured Fields				Validity		Joint	Hidden Error
		Label	Cat.	T-F1	Type	Exp.	Stance	Parse	Format	JREM	HER-C	HER-L
Qwen2.5-7B	Zero-shot	88.4	63.7	57.5	50.1	34.9	54.4	94.0	91.6	31.9	54.9	62.8
Qwen2.5-7B	+Taxonomy	90.6	76.7	58.0	48.9	36.8	53.2	97.1	94.0	36.4	54.8	59.3
Qwen2.5-7B	CoT-SFT	96.7	88.2	73.1	59.2	46.9	78.4	99.6	99.4	59.5	34.4	38.3
Llama-3.1-8B	Zero-shot	61.4	54.8	50.0	51.0	34.1	30.6	86.6	82.1	24.6	48.0	61.6
Llama-3.1-8B	+Taxonomy	80.1	63.6	52.2	53.9	34.6	44.3	94.2	91.3	30.4	51.7	61.7

Table 1: Performance comparison of different models on the deduplicated 1,600-item test set (%) across classification, target extraction, structural validity, and complete-record evaluation. Classification metrics use Macro-F1, while T-F1 uses mean character-F1. HER-C/L are lower-is-better; all other metrics are higher-is-better. Bold indicates the best result in each column.

Figure 4: JREM across four context-dependent lexical-cue families. Solid and hatched bars denote harmful (H) and non-harmful (NH) cases; overlapping families are evaluated independently.

Figure 5: Complete-record error analysis on the 1,600-item test set. (a) Field mismatch rates, with target F1 below 0.5 counted as a mismatch; (b) Shapley decomposition of the Parse–JREM gap across referent, stance, and decision fields.

Figure 6: Qualitative comparison of moderation representations. Case A shows that label/category and target-span outputs omit target properties and the linked decision basis. Case B shows that identifying quoted abuse without author stance can invert harmfulness. VARM-Bench makes all six decisions jointly inspectable.

Figure 7: Generated-CoT audit profiles for Qwen2.5-7B CoT-SFT and DeepSeek-V4-Pro Zero-Shot. The three axes show Groundedness, Adequacy, and Coherence across 600 CoTs from 300 shared inputs after three blinded GPT-5.5 passes and human review.

Content Moderation

Content moderation studies how to detect, interpret, and respond to harmful or policy-violating material across digital platforms.

See all 87 papers

VCB Bench: An Evaluation Benchmark for Audio-Grounded Large Language Model Conversational Agents
Oct 13, 2025Jiliang Hu, Wenfu Wang, Zuchao Li +6Audio-Language Model Evaluation Voice Agent Evaluation
School of Artificial Intelligence, Wuhan University, Wuhan, China · Tencent AI Lab, Beijing, China
ReasoningShield: Safety Detection over Reasoning Traces of Large Reasoning Models
May 22, 2025Changyi Li, Jiayi Wang, Xudong Pan +2Language Model Safety Evaluation LLM Safety Benchmarks
Fudan University · Shanghai Innovation Institute
CARO: Chain-of-Analogy Reasoning Optimization for Robust Content Moderation
Apr 12, 2026Bingzhe Wu, Haotian Lu, Yuchen MouAnalogical Reasoning for Language Models Content Moderation
Shenzhen University · Tsinghua University · National University of Singapore

Oct 13, 2025J/K move · Enter open · S save

VCB Bench: An Evaluation Benchmark for Audio-Grounded Large Language Model Conversational Agents

Jiliang Hu, Wenfu Wang, Zuchao Li, Chenxing Li+5

School of Artificial Intelligence, Wuhan University, Wuhan, China · Tencent AI Lab, Beijing, China

Audio-Language Model Evaluation Voice Agent Evaluation Audio-Language Models Speech Language Models Language Model Robustness

Open paper AskPDF arXiv

Recent advances in large audio language models (LALMs) have greatly enhanced multimodal conversational systems. However, existing benchmarks remain limited -- they are mainly English-centric, rely on synthetic speech, and lack comprehensive, discriminative evaluation across multiple dimensions. To address these gaps, we present Voice Chat Bot Bench (VCB Bench) -- a high-quality Chinese benchmark built entirely on real human speech. VCB Bench evaluates LALMs from three complementary perspectives: instruction following (including speech-level control beyond text commands), knowledge understanding (general knowledge, reasoning, and daily dialogue), and robustness (stability under perturbations in content, environment, and speaker traits). Experiments on representative LALMs reveal notable performance gaps and highlight future directions for improvement. VCB Bench provides a reproducible and fine-grained evaluation framework, offering standardized methodology and practical insights for advancing Chinese voice conversational models.

Abstract

Content Moderation

Explore similar work

VCB Bench: An Evaluation Benchmark for Audio-Grounded Large Language Model Conversational Agents

ReasoningShield: Safety Detection over Reasoning Traces of Large Reasoning Models

CARO: Chain-of-Analogy Reasoning Optimization for Robust Content Moderation