Behavioral evaluation of language models fixes its dimensions in advance: hallucination, sycophancy, toxicity, over-refusal. An operator in a regulated industry needs a different question answered: does a system do what a particular rulebook requires? We introduce the Behavioral Misalignment Metric Suite (BMMS), which measures policy-conditioned misalignment by taking a policy corpus as an argument and returning a compliance profile over it, for any published policy and any system that answers in natural language. The Compliance Attainment Score (CAS) measures severity-weighted attainment over the obligations that bind the system, and the Compliance Robustness Margin (CRM) the share of that attainment surviving graded adversarial pressure; Behavioral Invariance (BIV) and the Behavioral Disposition Index (BDI) are specified and released with the suite. Dimensions enter only by passing a published admission test whose binding criterion is attribution: a quantity that moves when the judge changes, holding the target fixed, is measuring the instrument. Applying it excluded one of the quantities the suite itself reported. Across 1,003 obligations from eleven documents in five registers, the admission criterion holds 94.4% precision against blinded human judgment on vendor specifications and 50.0% on analytical guidance, extraction fidelity is 94.6%, and re-scoring under judges from two other providers leaves target ordering invariant while the absolute score of a single target spans up to 0.22. We apply it to the two most detailed published behavioral specifications. Own-specification advantage is not resolved from zero once document difficulty is subtracted, but absolute self-compliance is another matter: GPT-4o fails 70.7% of the hard constraints binding it in the Model Spec published by OpenAI, where the two Claude models fail 0.0% and 9.1% of theirs.
Figures & tables
Figure 1: (a) BMMS is a measurement layer conditioned on a policy supplied at run time. Obligations are extracted with their verbatim spans, admitted by a bindability panel drawn from three providers and disjoint from the verdict judge (funnel, left to right: 1,003 extracted, 489 majority-bindable, 309 unanimous, 260 high-confidence, 231 non-mixed), and converted once into a frozen hashed probe set that every target answers. Because each record retains the rendered judge prompt and the raw response, the attribution check re-scores stored responses under a second judge and issues no target queries at all. (b) The four candidate dimensions, each with the failure mode it names and the quantity this paper measures for it; CAS and CRM are admitted on this corpus, BIV and BDI carry partial evidence.
Criterion
Threshold
A1
Construct
Names a failure mode a regulator would recognize, distinct from the others
A2
Independence
∣r∣<0.60 with every admitted metric, per target, and sign-stable
A3
Sensitivity
∣Δ∣>0.02 under a targeted intervention on at least one arm
A4
Discrimination
A paired 95% bootstrap interval on the difference between two targets, over shared rules, excluding zero
A5
Attribution
∣Δ∣<0.05 under a judge-panel swap, or judge-free by construction
Table 1: The admission test. A candidate dimension enters only on passing all five. A5 is the criterion the suite is designed around: a quantity that moves when the judge changes, holding the agent fixed, is measuring the instrument.
Figure 2: (a) CAS by register. Pooling would let 295 vendor-specification rules dominate and hide a 24.7-point within-target spread. (b) CAS by specification. Every target scores higher on the Constitution, including targets unrelated to either document, which is the difficulty baseline the own-specification advantage is measured against.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Register
Documents
Rules
Admitted
Vendor specification
OpenAI Model Spec (2025-12-18)
572
68.0%
Claude’s Constitution
Conduct regulation
FINRA 2026 guidance
181
25.4%
MiFID II Art. 24–25
SEC Regulation Best Interest
Analytical guidance
BIS, two reports
163
16.0%
Appendix
Table 2: Corpus manifest by register. Extraction profile is assigned from register before any output is inspected.
Precision (retained)
Wrong drops (rejected)
Register
rate
95% CI
rate
95% CI
Vendor specification
17/18 = 94.4%
74–99
10/18 = 55.6%
34–75
Conduct regulation
4/6 = 66.7%
30–90
0/2 = 0.0%
0–66
Analytical guidance
2/4 = 50.0%
15–85
0/6 = 0.0%
0–39
Statute and AUP
0/2 = 0.0%
0–66
1/4 = 25.0%
5–70
All registers
23/30 = 76.7%
59–88
11/30 = 36.7%
22–54
Appendix
Table 3: The operating characteristic of the admission criterion, 60 blinded items over two rounds, 30 drawn from the retained region and 30 from the rejected region. Precision is the fraction of panel-retained rules a human annotator also judges agent-binding. Wrong drops is the fraction of panel-rejected rules the same annotator judges binding, which is the criterion’s cost. The two columns move in opposite directions across registers. Wilson intervals.
Figure 3: (a) Target ordering does not depend on the reporting profile. Sampling 10,000 weight vectors over CAS on each specification and the CRM break-point hold rate leaves the ordering unchanged in every draw, because the ranking is Pareto-dominant on all three axes. This is why the suite reports a vector rather than asserting a composite. (b) Movement under the judge swap against the A5 band at ±0.05 , bar-labeled with judge agreement. Movement tracks agreement, which tracks target quality, so the two strongest targets move least and the weakest move most.
Google judge
OpenAI judge
Target
Δ
rel.
Δ
rel.
Agree
Sonnet 4.5
+0.020
−0.017
−0.053
+0.045
83–85%
Haiku 4.5
+0.045
+0.008
−0.090
+0.008
82–84%
GPT-5.5
+0.024
−0.013
−0.084
+0.014
80–84%
GPT-4o
+0.090
+0.053
−0.093
+0.005
79%
Llama 3.1 8B
+0.058
+0.021
−0.104
−0.006
71–72%
Appendix
Table 4: Movement under each swap judge, with agreement rates and movement relative to the panel mean. The two swap judges bracket the primary: the Google judge is more lenient, the OpenAI judge stricter. Ordering is identical between the two swap judges and differs from the primary judge’s by one inversion.
Document
Registered
Measured
EU AI Act Art. 50
60–90%
39.3%
miss
MiFID II Art. 24–25
25–45%
34.4%
hit
Colorado AI Act
10–25%
22.6%
hit
Appendix
Table 5: Pre-registered admission rates against measurement.
Ordering ( τ )
Own-spec advantage (pts)
whard/wsoft
Model Spec
Constitution
Haiku
Sonnet
GPT-4o
1.00 (unweighted)
1.000
1.000
+4.1
+4.2
−1.8
2.00
1.000
1.000
+4.0
+3.3
−2.3
3.33 (reported)
1.000
1.000
+3.8
+2.5
−2.7
7.00
1.000
1.000
+3.5
+1.3
−3.3
10.00
0.867
1.000
+3.4
+0.8
−3.6
Appendix
Table 6: CAS under every severity weighting. τ is Kendall’s rank correlation of target ordering against the reported weighting, whard/wsoft=3.33 . The last row is the degenerate case in which soft preferences are dropped and CAS ceases to be the quantity defined in Appendix A .
Target
Rel.
Model Spec
Constitution
Own-spec adv.
Haiku 4.5
Anthropic
0.806 [0.754, 0.856]
0.940 [0.909, 0.966]
+0.038 [ −0.016 , +0.091 ]
Sonnet 4.5
Anthropic
0.797 [0.746, 0.844]
0.918 [0.880, 0.952]
+0.025 [ −0.028 , +0.078 ]
GPT-4o
OpenAI
0.669 [0.613, 0.723]
0.792 [0.743, 0.840]
−0.027 [ −0.072 , +0.018 ]
Gemini 2.5 Flash
none
0.631 [0.586, 0.676]
0.712 [0.668, 0.755]
baseline
Gemma 2 9B
none
0.664 [0.610, 0.719]
0.748 [0.702, 0.795]
baseline
Llama 3.1 8B
none
0.656 [0.605, 0.706]
0.779 [0.731, 0.823]
baseline
Appendix
Table 7: CAS by target and specification, with paired bootstrap intervals over rules. Own-spec advantage is the difference in differences against the mean of all three no-relationship arms with complete coverage of both documents. Every interval on that quantity contains zero. The two-arm baseline of Section 6 gives +0.051 , +0.038 and −0.040 , which also contain zero.
Figure 4: The suite emits a profile, not a score. (a) Six measured axes for the three targets with complete coverage: attainment (CAS), hard-constraint attainment on the Model Spec, robustness ( 1−CRMΔ ), depth (fraction of rules held compliant at every pressure level), register uniformity ( 1 minus the CAS spread across registers), and verdict determinacy (judge agreement under the panel swap). (b) The same values as a ratio to Haiku 4.5. GPT-4o’s deficit ranges from 0.47 on depth to 0.94 on register uniformity, so the axes are not one factor at different radii. A suite whose dimensions were redundant would produce flat lines here, which is the visual form of admission criterion A2.
Target
Vendor spec
Analytical
Conduct reg.
AUP
Statute
n=295
n=26
n=46
n=11
n=19
Haiku 4.5
0.849
0.917
0.792
0.728
0.748
[0.81,0.88]
[0.83,0.99]
[0.71,0.87]
[0.52,0.90]
[0.63,0.86]
Sonnet 4.5
0.835
0.928
0.795
0.715
0.780
[0.80,0.87]
[0.82,1.00]
[0.71,0.87]
[0.54,0.87]
[0.64,0.90]
GPT-4o
0.708
0.587
0.494
0.473
0.461
Appendix
Table 8: CAS by register with 95% bootstrap intervals over rules and the number of rules in each stratum. Rule counts are the frozen probe set, 295 + 26 + 46 + 11 + 19 = 397; a cell’s CAS denominator is at or below its column count, because rules whose probes all returned na are excluded per target. Registers are never pooled: 295 vendor-specification rules would otherwise dominate the aggregate. The intervals are what the strata support: GPT-4o and Gemini 2.5 Flash separate vendor specification from the regulatory registers with non-overlapping intervals, while the Claude models’ spread across registers does not clear its own uncertainty.
Independent Researcher · Institute for Automation and Applied Informatics, Karlsruhe Institute of Technology, Karlsruhe · Duale Hochschule Baden-Württemberg, Ravensburg