The Italian Institute of Artificial Intelligence (AI4I) · Corso Castelfidardo 22, 10129 Turin, Italy
We evaluate the adversarial robustness of three frontier large language models (LLMs) developed by Anthropic, Opus 4.8, Fable 5 and its successor Fable 5.1, against four families of automated jailbreak attack across 7,826 harmful intents spanning a ten-category harm taxonomy. Using the HackAgent red-teaming framework, hundreds of thousands of adversarial attempts were generated and every apparent success was independently re-adjudicated by the same panel of five frontier judge models (
≥4/5 agreement) for all three targets. All models resist the majority of attacks, but the residual surface is larger than aggregate framing suggests: it is dominated by adaptive iterative attacks, while static obfuscation is near-fully neutralised. On the two attack families run against all three targets (TAP and PAP), with identical denominators, the ordering does not follow release date: Fable 5 is most robust (
2.72%), Opus 4.8 follows (
5.76%) and the newest model, Fable 5.1, is least robust (
8.19%). The three fail in largely disjoint places, and conditioning on prompts each model actually answered shows why: adaptive tree search breaks all three at broadly similar rates (
10.1%,
9.7%,
12.1%), whereas one-shot persuasion separates them by more than an order of magnitude (
3.1%,
0.7%,
9.5%). What distinguishes these models adversarially is their resistance to framing, not to search. Even so, they produced 1,315 (Opus 4.8), 620 (Fable 5) and 1,282 (Fable 5.1) panel-confirmed harmful completions spanning every harm category, found automatically and cheaply, usually within one or two refinement steps, by an attacker model with no human expert in the loop. Even the best, most-tested frontier models remain reliably breakable under sustained automated pressure, and robustness does not monotonically improve from one version to the next.