BrandEval: A Two-Track Multi-Agent Benchmark for Risk-Sensitive LLM Crisis Communication

Haotian Gan; Weidong Tang; Hanbin Sun; Yi Luo; Xintian Li; Jiayong Wang; WanYue Li; Yudong Li

cs.AIOct 18, 2026

BrandEval: A Two-Track Multi-Agent Benchmark for Risk-Sensitive LLM Crisis Communication

Authors: Haotian Gan, Weidong Tang, Hanbin Sun, Yi Luo, Xintian Li, Jiayong Wang, WanYue Li, Yudong Li

Organizations: Shaanxi University of Science and Technology · China Agricultural University · Tianjin University · Tsinghua University ·

Abstract

Evaluating LLMs for crisis communication requires more than judging isolated responses, because public feedback and competitor actions can reshape trust over time. Existing benchmarks remain largely static and single-turn, offering limited evidence about long-horizon risk-sensitive communication. We introduce BrandEval, a two-track benchmark for product-crisis communication, focusing on public brand responses to product-related safety, quality, service, and trust incidents. BrandEval-Static evaluates stakeholder-aware strategic analysis, while BrandEval-Dynamic stress-tests response policies in BrandPolis, a controlled multi-agent sandbox with consumer feedback, competitor responses, and trust-risk trajectories. We also construct BrandSRD, a Chinese crisis-response dataset, and instantiate a reference Strategic Agent using Strategic Rationale (SR). Across proprietary and open-source models, we find that static strategic quality is informative but incomplete: it aligns with some end-state outcomes but does not reliably predict volatility and delayed recovery; dynamic stress tests further expose tail-risk failures under manipulative or weakly grounded policies. The code and dataset will be available at https://github.com/htgan-ai/BrandEval.

Figures & tables

Figure 1: Divergent Futures. Under the same crisis context, different response formulations can lead to divergent long-horizon trust trajectories in simulation.

Figure 2: BrandSRD construction pipeline. Source collection is followed by atomic extraction and de-identification, counterfactual generation, and human validation and audits, yielding 41,058 filtered and audited records and a held-out evaluation pool for BrandEval.

Product	Weibo	Douyin	XHS	Total
Smartwatch	1078	786	482	2346
Refrigerator	1024	1125	741	2890
Air conditioner	1096	553	781	2430
Rice cooker	858	2517	676	4051
Smart door lock	1360	2268	449	4077
Translation pen	1265	1051	614	2930

Table 1: BrandSRD source distribution. Counts of posts used to construct BrandSRD across 12 product domains and three platforms (XHS = Xiaohongshu).

Figure 3: Overview of BrandEval. BrandEval pairs Static diagnosis on held-out scenarios with Dynamic policy evaluation in BrandPolis. Strategic Rationale guides the Strategic Agent, which serves as a reference policy within the Dynamic track.

Figure 4: BrandEval prototype distribution. Topic and stakeholder composition of 1,000 scenarios.

Model	C/B	P/M	Comp.	Strat.
GPT - 5.1	7.86	7.81	7.42	7.60
Gemini - 2.5 - Pro	7.61	7.64	7.28	7.34
Claude - Sonnet - 4.5	7.52	7.71	7.36	7.40
GLM - 4.5	7.11	7.16	6.93	6.89
DeepSeek - R1	6.74	6.80	6.46	6.47
R1 - Distill - Qwen - 7B	4.88	4.31	4.44	4.04

Table 2: BrandEval-Static results. Multi-LLM panel scores (1–10) averaged over all scenarios. C/B = Consumer & Brand; P/M = Public & Media; Comp. = Competitor; Strat. = Strategic Synthesis.

Policy	Min BRI $\uparrow$	CVaR 10 BRI $\uparrow$	Min NPS $\uparrow$	Fail rate $\downarrow$
Manipulative	-18.52	-16.20	-35.00	12%
Specious	-10.24	-9.15	-22.00	4%
Strategic Agent	-5.21	-4.82	-12.00	0%

Table 4: Tail risk analysis across 50 scenarios. Min BRI and Min NPS are the worst-case end-of-horizon outcomes; CVaR 10 is the mean of the bottom 10%

BRI_{T}

outcomes; Fail rate is the proportion of scenarios with

BRI_{T}<-10

Model	BRI $\uparrow$		NPS $\uparrow$		ARV $\downarrow$		Pol. $\downarrow$		CRS $\downarrow$		SoV
Model	Single	Dual	Single	Dual	Single	Dual	Single	Dual	Single	Dual	Single	Dual
Backbone baseline
Qwen3-8B	-3.54	-4.30	-3.00	-9.00	30.06	29.74	1.98	1.98	16	15	63.46	68.90
Prompting baselines (same backbone, different prompting)
Qwen3-8B + CoT	-2.98	-3.88	-1.00	-4.00	29.50	30.60	3.92	5.82	16	15	68.93	69.58
Qwen3-8B + ReAct	-2.89	-3.74	-5.00	-5.00	35.16	31.95	0.00	3.92	13	12	70.30	69.62

Table 5: Comparisons and ablations on BrandEval-Dynamic. Results are grouped into a backbone baseline, prompting baselines with the same backbone, and Strategic Agent variants that ablate SR.

Metric	Highway accident	Product recall	Spontaneous combustion
Macro trend alignment
Lagged $r_{\max}$ ( $\pm 2$ ) $\uparrow$	0.60	0.80	0.75
Spearman $\rho$ $\uparrow$	0.62	0.56	0.59
Norm-RMSE $\downarrow$	0.35	0.32	0.29
DTW (Z-norm) $\downarrow$	0.21	0.27	0.23
90% band coverage $\uparrow$	0.60	0.65	0.65

Table 6: BrandPolis sanity checks. Macro-level trend alignment and micro-level discourse reproduction across three crisis types.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Dimension	Operational definition
Consumer & Brand	Accurate diagnosis of consumer concerns; concrete brand actions; credible evidence (data, timelines, accountability); explicit linkage to brand equity dimensions (competence/value/emotion) when appropriate.
Public & Media	Anticipation of public/media narratives; risk-aware messaging; clarity on responsibility/safety; preventive escalation control; alignment with transparency and verifiability.
Competitor	Recognition of competitive dynamics (e.g., opportunistic attacks, SoV shifts); defensive/offensive counter-moves; avoids self-inflicted openings.
Strategic Synthesis	Coherent trade-offs across stakeholders; prioritized goals; operational plan with milestones; avoids generic PR; actionability and consistency with constraints.

Appendix

Table 7: BrandEval-Static rubric. Four dimensions scored on a 1–10 scale.

Judge	Mean $\overline{\Delta}$	95% CI	Win(%)	Paired $d$
Standard	$+0.44$	$[-0.33,1.33]$	$44$	$0.35$
Evidence-sensitive	$\mathbf{+3.11}$	$\mathbf{[1.67,4.56]}$	$\mathbf{89}$	$\mathbf{1.34}$

Appendix

Table 8: Audit Judge paired discriminability (

N=9

). The evidence-sensitive judge better separates Verifiable and Specious strategies under matched stress scenarios.

Signal	Corr. ( $r$ ) $\uparrow$	$\boldsymbol{\|\Delta t_{\mathrm{peak}}\|\downarrow}$	$\boldsymbol{\|\Delta\kappa\|\downarrow}$
Volume	0.56	0.3	0.48
Heat	0.59	0.4	0.52
SoV	-0.07	9.7	0.72

Appendix

Table 9: Diffusion-signal alignment. Metrics are averaged over three real-world crisis events.

Model / Policy	BRI $\uparrow$	ARV $\downarrow$	CRS $\downarrow$	Fail rate $\downarrow$
Qwen3-8B + CoT	$-4.85$	$32.4$	$16$	$6.7\%$
Strategic Agent	$\mathbf{-2.60}$	$\mathbf{28.1}$	$\mathbf{13}$	$\mathbf{0.0\%}$

Appendix

Table 10: Supplementary cross-domain dynamic check. Results are averaged over 30 additional held-out scenarios from four product domains with

K=5

seeds per scenario.

Strategy	Backfire Rate $\downarrow$	Final BRI (Mean) 95% CI
Verifiable	$\mathbf{6.78\%}$	$-3.08$ $[-8.69,\,-0.18]$
Specious	$9.24\%$	$-2.71$ $[-7.73,\,-0.20]$
Manipulative	$11.39\%$	$-3.06$ $[-8.79,\,-0.20]$

Appendix

Table 12: Action-grounding diagnostic aggregated over 3 scenarios

\times

5 seeds.

Event	Metrics	Lag	Spearman $\rho$	Mean JSD	$\Delta t_{\text{peak}}$
Safety Accident	Negativity vs. Negativity proxy Stance dist. vs. Stance dist.	$0$	$0.8214$	$0.3842$	$-2$
Product Recall	Negativity vs. Negativity proxy Stance dist. vs. Stance dist.	$-3$	$0.9850$	$0.2156$	$0$
Design Issue	Net sentiment vs. BRI Stance dist. vs. Stance dist.	$-3$	$0.9643$	$0.3105$	$+1$

Appendix

Table 13: Stance distribution alignment between real-world data and simulation. We report Spearman

\rho

for trend alignment, mean JSD for stance-distribution alignment, and

\Delta t_{\text{peak}}

for peak-time deviation.

Metric	Simulator	Shuffle control	Significance
Kendall’s $\tau_{b}$ (rank)	$0.7642$	$0.04$	$p<0.001$
Top-1 match rate	$85.4\%$	$33.3\%$	–
Impact separation (mean $\Delta$ BRI across decision points)
$\bullet$ Verifiable action: $+42.97$ (Rank 1)
$\bullet$ Specious statement: $+2.15$ (Rank 2)
$\bullet$ Original / baseline: $0.00$ (Rank 3)

Appendix

Table 14: Action ranking consistency. We report Kendall’s

\tau_{b}

, Top-1 match rate, and a shuffle control on

N=1000

held-out decision points.

Figure 5: Case-study episode: dynamic metric trajectories. We plot BRI, NPS, SoV, and Pol. across the 20-round horizon.

Figure 6: Discrepancy analysis. Risk profile stratification and latent state recovery paths.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,203 papers

SentryBench: A Cost-Sensitive Benchmark for Adversarial Robustness of Tool-Using LLM Agents in Public-Safety Deployments
Oct 25, 2026Md Abu Yusuf, Oyshi Tabassum Aditi, Md Faysal Hosain MazedLLM Agent Safety AI Agent Safety Benchmarks
Department of Computer Science, Maharishi International University, IA, USA · Department of Computer Science and Engineering, Daffodil International University, Dhaka, Bangladesh · Department of Information Technology, University of the Cumberlands, Williamsburg, KY, USA
Guide Me Out: A Framework to Benchmark VLM Operators Communication in Crisis Scenarios
Jun 8, 2026Giacomo Gonella, Stefano Menini, Marco GueriniVLM Evaluation Disaster Response
Fondazione Bruno Kessler, Italy · University of Trento, Italy
TIER: Threat Implicitness Benchmark for Evaluating LLM Safety Behaviors
Sep 4, 2026Thu-Hien Trinh-Thi, Hai-Yen Vong, Thanh-Ha Ung-Dung +1LLM Safety Evaluation LLM Safety Benchmarks
Faculty of Information Technology, University of Science, Vietnam National University, Ho Chi Minh City, Vietnam

Oct 25, 2026J/K move · Enter open · S save

SentryBench: A Cost-Sensitive Benchmark for Adversarial Robustness of Tool-Using LLM Agents in Public-Safety Deployments

Md Abu Yusuf, Oyshi Tabassum Aditi, Md Faysal Hosain Mazed

Department of Computer Science, Maharishi International University, IA, USA · Department of Computer Science and Engineering, Daffodil International University, Dhaka, Bangladesh · Department of Information Technology, University of the Cumberlands, Williamsburg, KY, USA

LLM Agent Safety AI Agent Safety Benchmarks Adversarial Robustness of Language Models Safety-Utility Trade-Off in Language Models

Open paper AskPDF arXiv

Large language model (LLM) agents are increasingly wired to consequential tools that send public alerts, dispatch responders, and query citizen records, where one erroneous tool call carries real-world cost. Existing agent-safety benchmarks measure generic harm or prompt-injection success and implicitly treat refusing as the safe default. In public-safety deployments that assumption breaks: an agent too cautious to dispatch help fails as surely as one that triggers a false alert. We present SentryBench, which scores agents on a two-sided, cost-sensitive frontier jointly measuring attack success, over-refusal (missing a real emergency), and over-reaction (unwarranted action) under adversarial tool use, with scenarios grounded in real emergency-call transcripts, disaster messages, and real injection/jailbreak payloads. We formalize deployment harm as an asymmetric cost functional, prove naive caution is harm-dominated whenever it refuses more genuine emergencies than the attack harm it averts, and prove a targeted trust gate adds zero over-refusal on clean input. Across 840 scored episodes over three frontier models and three defenses, refuse-more caution drives over-refusal to 100 percent on two of three models for at most a 3.7-point change in attack success and yields the worst harm everywhere, whereas our trust-gating wrapper dominates the frontier (lowest harm on every model, over-refusal at or below baseline, higher containment). Provider content filters also silently perform part of the defending and must be reported separately.

Abstract

LLM Agent Evaluation

Explore similar work

SentryBench: A Cost-Sensitive Benchmark for Adversarial Robustness of Tool-Using LLM Agents in Public-Safety Deployments

Guide Me Out: A Framework to Benchmark VLM Operators Communication in Crisis Scenarios

TIER: Threat Implicitness Benchmark for Evaluating LLM Safety Behaviors