Adversarial Humanities Benchmark: Results on Stylistic Robustness in Frontier Model Safety

How can we evaluate whether frontier AI systems recognize child-safety risks before they escalate into explicit harm? Existing child safety evaluations focus on child sexual abuse material, yet many child-safety failures begin earlier: in model assistance that helps adults manipulate, impersonate, profile, or isolate minors, and in model responses that deepen children's emotional dependence on AI systems rather than redirecting them toward human support. We introduce CAREBench (Child AI Risk Evaluation), a benchmark to assess such upstream child-safety risks in language models. CAREBench contains 500 prompts spanning twelve risk categories, including grooming and relationship engineering, deception and impersonation, surveillance and privacy, sextortion and sexual abuse, AI anthropomorphization, emotional dependency, and mental illness sensitivity. Developed with response annotations from parents and clinicians, the benchmark excludes explicit abuse material and imagery; instead, it evaluates whether models recognize, refuse, de-escalate, or redirect risky interactions before harm becomes overt. Evaluating seven frontier models on our benchmark, we find failure rates ranging from 2% to 58%, with failure patterns that vary across risk categories. CAREBench provides a responsibly scoped evaluation for LLM developers to identify and close gaps in child safety policies.

We evaluate the adversarial robustness of three frontier large language models (LLMs) developed by Anthropic, Opus 4.8, Fable 5 and its successor Fable 5.1, against four families of automated jailbreak attack across 7,826 harmful intents spanning a ten-category harm taxonomy. Using the HackAgent red-teaming framework, hundreds of thousands of adversarial attempts were generated and every apparent success was independently re-adjudicated by the same panel of five frontier judge models ($\geq 4/5$ agreement) for all three targets. All models resist the majority of attacks, but the residual surface is larger than aggregate framing suggests: it is dominated by adaptive iterative attacks, while static obfuscation is near-fully neutralised. On the two attack families run against all three targets (TAP and PAP), with identical denominators, the ordering does not follow release date: Fable 5 is most robust ($2.72\%$), Opus 4.8 follows ($5.76\%$) and the newest model, Fable 5.1, is least robust ($8.19\%$). The three fail in largely disjoint places, and conditioning on prompts each model actually answered shows why: adaptive tree search breaks all three at broadly similar rates ($10.1\%$, $9.7\%$, $12.1\%$), whereas one-shot persuasion separates them by more than an order of magnitude ($3.1\%$, $0.7\%$, $9.5\%$). What distinguishes these models adversarially is their resistance to framing, not to search. Even so, they produced 1,315 (Opus 4.8), 620 (Fable 5) and 1,282 (Fable 5.1) panel-confirmed harmful completions spanning every harm category, found automatically and cheaply, usually within one or two refinement steps, by an attacker model with no human expert in the loop. Even the best, most-tested frontier models remain reliably breakable under sustained automated pressure, and robustness does not monotonically improve from one version to the next.

Safety benchmark scores provide incomplete evidence of deployment readiness: aligned language models often adhere to rigid rules even when a situational update flips which action is safe. We term this failure brittle safety. To diagnose it, we introduce context-flip evaluation, testing 12 models across a safety benchmark (PacifAIst) and two commonsense controls using paired variants where the nominally safe action produces harm. Three findings emerge. First, brittle safety is safety-specific: all 12 models exhibit a safety-commonsense gap (mean +17.4 pp). Baseline accuracy fails to predict brittleness: among models above 90% baseline accuracy, brittleness rates range from 13.7% to 90.0%. Second, failures stem from policy override rather than miscomprehension: despite acknowledging the context change in every case, models persist via three distinct mechanisms that vary by update type and model family. Third, on a hand-audited probe of catastrophic consequence-flip scenarios, standard action-level guardrails catch none, while a state-aware validator catches all without false alarms on correct interventions. This indicates that action-level content moderation is systematically blind to consequence-flips, motivating state-aware architectural alternatives. We release our protocol, perturbed benchmarks, and deployment probe.