BMMS: Policy-Conditioned Measurement of Behavioral Compliance in Language Models
Organizations: Confidential Core AI
Abstract
Behavioral evaluation of language models fixes its dimensions in advance: hallucination, sycophancy, toxicity, over-refusal. An operator in a regulated industry needs a different question answered: does a system do what a particular rulebook requires? We introduce the Behavioral Misalignment Metric Suite (BMMS), which measures policy-conditioned misalignment by taking a policy corpus as an argument and returning a compliance profile over it, for any published policy and any system that answers in natural language. The Compliance Attainment Score (CAS) measures severity-weighted attainment over the obligations that bind the system, and the Compliance Robustness Margin (CRM) the share of that attainment surviving graded adversarial pressure; Behavioral Invariance (BIV) and the Behavioral Disposition Index (BDI) are specified and released with the suite. Dimensions enter only by passing a published admission test whose binding criterion is attribution: a quantity that moves when the judge changes, holding the target fixed, is measuring the instrument. Applying it excluded one of the quantities the suite itself reported. Across 1,003 obligations from eleven documents in five registers, the admission criterion holds 94.4% precision against blinded human judgment on vendor specifications and 50.0% on analytical guidance, extraction fidelity is 94.6%, and re-scoring under judges from two other providers leaves target ordering invariant while the absolute score of a single target spans up to 0.22. We apply it to the two most detailed published behavioral specifications. Own-specification advantage is not resolved from zero once document difficulty is subtracted, but absolute self-compliance is another matter: GPT-4o fails 70.7% of the hard constraints binding it in the Model Spec published by OpenAI, where the two Claude models fail 0.0% and 9.1% of theirs.
Figures & tables
| Criterion | Threshold | |
|---|---|---|
| A1 | Construct | Names a failure mode a regulator would recognize, distinct from the others |
| A2 | Independence | with every admitted metric, per target, and sign-stable |
| A3 | Sensitivity | under a targeted intervention on at least one arm |
| A4 | Discrimination | A paired 95% bootstrap interval on the difference between two targets, over shared rules, excluding zero |
| A5 | Attribution | under a judge-panel swap, or judge-free by construction |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Register | Documents | Rules | Admitted |
|---|---|---|---|
| Vendor specification | OpenAI Model Spec (2025-12-18) | 572 | 68.0% |
| Claude’s Constitution | |||
| Conduct regulation | FINRA 2026 guidance | 181 | 25.4% |
| MiFID II Art. 24–25 | |||
| SEC Regulation Best Interest | |||
| Analytical guidance | BIS, two reports | 163 | 16.0% |
| Precision (retained) | Wrong drops (rejected) | |||
|---|---|---|---|---|
| Register | rate | 95% CI | rate | 95% CI |
| Vendor specification | 17/18 = 94.4% | 74–99 | 10/18 = 55.6% | 34–75 |
| Conduct regulation | 4/6 = 66.7% | 30–90 | 0/2 = 0.0% | 0–66 |
| Analytical guidance | 2/4 = 50.0% | 15–85 | 0/6 = 0.0% | 0–39 |
| Statute and AUP | 0/2 = 0.0% | 0–66 | 1/4 = 25.0% | 5–70 |
| All registers | 23/30 = 76.7% | 59–88 | 11/30 = 36.7% | 22–54 |
| Google judge | OpenAI judge | ||||
| Target | rel. | rel. | Agree | ||
| Sonnet 4.5 | 83–85% | ||||
| Haiku 4.5 | 82–84% | ||||
| GPT-5.5 | 80–84% | ||||
| GPT-4o | 79% | ||||
| Llama 3.1 8B | 71–72% | ||||
| Document | Registered | Measured | |
|---|---|---|---|
| EU AI Act Art. 50 | 60–90% | 39.3% | miss |
| MiFID II Art. 24–25 | 25–45% | 34.4% | hit |
| Colorado AI Act | 10–25% | 22.6% | hit |
| Ordering ( ) | Own-spec advantage (pts) | ||||
|---|---|---|---|---|---|
| Model Spec | Constitution | Haiku | Sonnet | GPT-4o | |
| 1.00 (unweighted) | 1.000 | 1.000 | |||
| 2.00 | 1.000 | 1.000 | |||
| 3.33 (reported) | 1.000 | 1.000 | |||
| 7.00 | 1.000 | 1.000 | |||
| 10.00 | 0.867 | 1.000 | |||
| Target | Rel. | Model Spec | Constitution | Own-spec adv. |
|---|---|---|---|---|
| Haiku 4.5 | Anthropic | 0.806 [0.754, 0.856] | 0.940 [0.909, 0.966] | [ , ] |
| Sonnet 4.5 | Anthropic | 0.797 [0.746, 0.844] | 0.918 [0.880, 0.952] | [ , ] |
| GPT-4o | OpenAI | 0.669 [0.613, 0.723] | 0.792 [0.743, 0.840] | [ , ] |
| Gemini 2.5 Flash | none | 0.631 [0.586, 0.676] | 0.712 [0.668, 0.755] | baseline |
| Gemma 2 9B | none | 0.664 [0.610, 0.719] | 0.748 [0.702, 0.795] | baseline |
| Llama 3.1 8B | none | 0.656 [0.605, 0.706] | 0.779 [0.731, 0.823] | baseline |
| Target | Vendor spec | Analytical | Conduct reg. | AUP | Statute |
|---|---|---|---|---|---|
| Haiku 4.5 | 0.849 | 0.917 | 0.792 | 0.728 | 0.748 |
| [0.81,0.88] | [0.83,0.99] | [0.71,0.87] | [0.52,0.90] | [0.63,0.86] | |
| Sonnet 4.5 | 0.835 | 0.928 | 0.795 | 0.715 | 0.780 |
| [0.80,0.87] | [0.82,1.00] | [0.71,0.87] | [0.54,0.87] | [0.64,0.90] | |
| GPT-4o | 0.708 | 0.587 | 0.494 | 0.473 | 0.461 |
LLM Evaluation
Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.
See all 2,539 papers