Labels Override Definitions in Jev-Style Typed Decision Models

Seyedarmin Azizi; Erfan Baghaei Potraghloo; Massoud Pedram

cs.AIOct 1, 2026

Labels Override Definitions in Jev-Style Typed Decision Models

Authors: Seyedarmin Azizi, Erfan Baghaei Potraghloo, Massoud Pedram

Organizations: University of Southern California (USC), USA

Abstract

A typed decision model answers a fixed question about an input by returning a probability for each of several caller-defined options. Each option carries a short label and a written definition, which is where a developer states the rule the model should apply. Jev introduced this interface for routing, moderation and triage, open implementations followed, and the same operation occurs whenever a language model is used as a classifier by scoring label strings. We study the open implementations, whose weights we can inspect and patch, and ask whether the probability follows the definitions or the labels. A preference for the label we call option-label bias. Across four open-weight typed decision models, three ways of reading an answer from a Qwen2.5 backbone, eleven classification tasks and PolicyBench, a synthetic routing suite we introduce in which the rule appears only in the definitions, the answer is mostly the labels. Deleting every definition leaves accuracy unchanged (laya-td: 0.8559 against 0.8487), although those definitions support 0.7971 on their own, and renaming the options to A and B raises accuracy by +0.1511 [+0.1377, +0.1646]. One system, von, is unaffected, and the two code bases differ in one expression: laya writes each option as "{label}: {definition}", while von writes only the definition. Changing that expression in both directions, with no weight changed, makes all three laya checkpoints exactly invariant (+0.0000 [+0.0000, +0.0000]) and creates the effect in von, whose accuracy falls from 0.8511 to 0.2281 when a label contradicts its definition. Earlier work attributed this failure to the constrained decision head these models use in place of a text decoder; our results locate it in the prompt rendering. We give a two-call test that tells a practitioner which case applies to their model, and measure what four mitigations are worth.

Figures & tables

Figure 1: PolicyBench, the synthetic routing suite of § 4.3 , in its controlled form ( XF , in which the two options carry definitions of comparable specificity). The decision rule is stated only in the option definition, so the label carries no information by construction and the two right-hand groups are controls. Series are three typed decision models ( laya-td , laya-en , von ) and one language-model readout (Qwen2.5-7B scoring the option strings); laya-ml is near chance on this suite throughout and is reported in Table 2 rather than plotted. Every series except von loses accuracy when the options are given meaningful names rather than A and B , and laya-en falls to 0.5297 under names that point to the wrong class, close to the chance level of

0.5

. von is flat across all three label conditions because it never places the label in the model’s input.

n=3500{}

; error bars are bootstrap 95% intervals over items.

variant	labels	definitions	what it isolates
canonical	the task’s own class names	correct	the deployed configuration
letters	A , B , …	correct	the definition channel alone
letters_perm	B , A , …	correct	definition following, no contradiction
unrelated	alpha , beta , …	correct	whether neutrality needs a letter
digits	0 , 1 , …	correct	as above
synonym	close synonyms	correct	a low dose of mismatch

Table 1: The variants used throughout. The first column gives the identifier used in the released code and in the raw result files; the plain names introduced below are used in all prose, tables and figures.

Figure 3: What each channel contributes, on the fixed-label choice tasks (SST-2, RTE, CB;

n=833{}

). For every model except von , supplying only the labels (second bar) is as accurate as supplying both channels (first bar), so the definitions add nothing when a meaningful label is present. von shows the opposite pattern: it matches its full accuracy from the definitions alone and falls to chance when they are removed. Exact values with intervals are in Table 9 of Appendix A . Error bars are bootstrap 95% intervals.

model	aligned	arbitrary alpha / beta	arbitrary 0 / 1	arbitrary A / B	misleading	label only	neither
laya-td	0.742 $[0.728,\,0.756]$	0.854 $[0.842,\,0.865]$	0.874 $[0.863,\,0.885]$	0.893 $[0.883,\,0.903]$	0.836 $[0.824,\,0.849]$	0.569 $[0.553,\,0.585]$	0.528 $[0.512,\,0.545]$
laya-en	0.791 $[0.778,\,0.804]$	0.836 $[0.823,\,0.848]$	0.846 $[0.834,\,0.858]$	0.869 $[0.857,\,0.880]$	0.530 $[0.513,\,0.546]$	0.529 $[0.513,\,0.545]$	0.502 $[0.486,\,0.519]$
laya-ml	0.635 $[0.620,\,0.652]$	0.604 $[0.588,\,0.621]$	0.644 $[0.628,\,0.660]$	0.631 $[0.615,\,0.647]$	0.673 $[0.657,\,0.688]$	0.521 $[0.505,\,0.538]$	0.477 $[0.459,\,0.493]$
von	0.852 $[0.840,\,0.864]$	0.852 $[0.840,\,0.864]$	–	0.852 $[0.840,\,0.864]$	0.852 $[0.840,\,0.864]$	0.500 $[0.483,\,0.517]$	0.500 $[0.483,\,0.517]$
Qwen2.5-1.5B	0.500 $[0.479,\,0.521]$	0.500 $[0.479,\,0.521]$	–	0.513 $[0.492,\,0.534]$	0.511 $[0.490,\,0.532]$	0.500 $[0.479,\,0.521]$	0.508 $[0.487,\,0.529]$
Qwen2.5-7B	0.789 $[0.770,\,0.806]$	0.915 $[0.902,\,0.927]$	–	0.909 $[0.897,\,0.921]$	0.835 $[0.820,\,0.851]$	0.500 $[0.479,\,0.522]$	0.516 $[0.495,\,0.539]$

Table 2: PolicyBench-XF accuracy by label condition. The decision rule appears only in the option definition. Chance is 0.500. label only and neither are controls establishing that the definition is the sole information source.

model	arbitrary $-$ aligned	misleading $-$ aligned	arbitrary exch. $-$ arbitrary
laya-td	$+0.1511$ $[+0.1377,\,+0.1646]$	$+0.0942$ $[+0.0754,\,+0.1129]$	$+0.0017$ $[-0.0014,\,+0.0051]$
laya-en	$+0.0776$ $[+0.0631,\,+0.0923]$	$-0.2618$ $[-0.2851,\,-0.2386]$	$-0.0063$ $[-0.0114,\,-0.0014]$
laya-ml	$-0.0042$ $[-0.0217,\,+0.0131]$	$+0.0376$ $[+0.0143,\,+0.0611]$	$-0.0003$ $[-0.0154,\,+0.0149]$
von	$+0.0000$ $[+0.0000,\,+0.0000]$	$+0.0000$ $[+0.0000,\,+0.0000]$	$+0.0000$ $[+0.0000,\,+0.0000]$
Qwen2.5-1.5B	$+0.0133$ $[+0.0076,\,+0.0195]$	$+0.0111$ $[-0.0305,\,+0.0529]$	$+0.0796$ $[+0.0667,\,+0.0933]$
Qwen2.5-7B	$+0.1203$ $[+0.1019,\,+0.1395]$	$+0.0464$ $[+0.0195,\,+0.0733]$	$+0.0305$ $[+0.0210,\,+0.0405]$

Table 3: Paired contrasts on PolicyBench-XF, bootstrap over items.

spread across renderings: $P(\text{true def.})$ = 0.0e+00
rendering	$P(\text{false def.})$	$P(\text{true def.})$	confidence
canonical, hold =F, escalate =T	0.085900	0.914100	0.8280
swapped, escalate =F, hold =T	0.085900	0.914100	0.8280
letters, A =F, B =T	0.085900	0.914100	0.8280
letters permuted, B =F, A =T	0.085900	0.914100	0.8280
unrelated, alpha =F, beta =T	0.085900	0.914100	0.8280

Table 4: von answering one routing request under five labelings of the same question. Probabilities are read back by which definition each label carries. The returned values are identical to six decimal places in every row.

Request options hold and escalate , with the routing policy written in the definition of escalate
laya	[state] [instructions]
	hold : none of the above applies
	escalate : the customer is on the Enterprise plan and asks for money back
von	[state] [instructions]
	none of the above applies
	the customer is on the Enterprise plan and asks for money back

Figure 4: What reaches the encoder for the same request. laya prefixes each definition with its option label (shaded), so the label competes with the definition the caller wrote. von passes only the definitions and uses the label solely as a key for returning the answer. § 5.5 removes this prefix from laya and shows that the behavior reported in § 5.1 and § 5.2 goes with it.

suite	model	aligned	arbitrary	misleading
PolicyBench-XF
textsclaya-td	0.742 $[0.728,\,0.756]$	0.893 $[0.883,\,0.903]$	0.836 $[0.824,\,0.849]$	$+0.1511$ $[+0.1377,\,+0.1646]$
textsclaya-td, label removed	0.891 $[0.881,\,0.902]$	0.891 $[0.881,\,0.902]$	0.891 $[0.881,\,0.902]$	$+0.0000$ $[+0.0000,\,+0.0000]$
textsclaya-en	0.791 $[0.778,\,0.804]$	0.869 $[0.857,\,0.880]$	0.530 $[0.513,\,0.546]$	$+0.0776$ $[+0.0631,\,+0.0923]$
textsclaya-en, label removed	0.853 $[0.841,\,0.864]$	0.853 $[0.841,\,0.864]$	0.853 $[0.841,\,0.864]$	$+0.0000$ $[+0.0000,\,+0.0000]$
textsclaya-ml	0.635 $[0.620,\,0.652]$	0.631 $[0.615,\,0.647]$	0.673 $[0.657,\,0.688]$	$-0.0042$ $[-0.0217,\,+0.0131]$

Table 5: The rendering changed in both directions, on both suites. For each laya checkpoint the patched row renders an option as its definition alone instead of "{label}: {definition}" . For von the patched row does the reverse and prepends the label. Weights, decision head and calibration are unchanged within each pair, and the items are the same.

Figure 5: Accuracy on yes/no questions before and after negating the question, pooled over seven tasks. The returned probability is re-oriented for the negated form, so a model that reads the negation would score the same in both conditions. Every model instead falls from above chance to below it by about the same margin, which is what happens when the returned probability does not change and only the correct answer has been inverted. Exact values and flip rates are in Table 10 of Appendix A .

model	condition	accuracy	mean confidence	AUROC of confidence
laya-td	aligned	0.8487	0.4022	0.8123
	misleading	0.1357	0.3397	0.2433
laya-en	aligned	0.8667	0.6139	0.8236
	misleading	0.1236	0.5852	0.1940
laya-ml	aligned	0.7923	0.6830	0.7292
	misleading	0.2185	0.5944	0.2871

Table 6: Whether a model’s own confidence separates its correct from its incorrect answers, by condition, on fixed-label choice tasks. An AUROC below

0.5

means the signal is actively misleading about its own errors.

model	condition	raw	prior-corrected	$\Delta$	state-blind
laya-td	aligned	0.7420	0.7811	$+0.0391$	0.6340
	arbitrary A / B	0.8931	0.8509	$-0.0423$	0.5240
	misleading	0.8363	0.7877	$-0.0486$	0.4546
laya-en	aligned	0.7911	0.7163	$-0.0749$	0.4900
	arbitrary A / B	0.8689	0.8286	$-0.0403$	0.5003
	misleading	0.5297	0.5934	$+0.0637$	0.9383

Table 7: Null-state prior correction on PolicyBench-XF. state-blind is the fraction of decisions equal to the decision the same rendering makes with no state at all.

model	second rendering	flag rate	precision	recall	acc. unflagged	coverage
laya-td	arbitrary A / B	0.1900	0.8977	0.6611	0.8921	0.8100
	arbitrary alpha / beta	0.1386	0.9031	0.4850	0.8458	0.8614
laya-en	arbitrary A / B	0.1954	0.6988	0.6539	0.9102	0.8046
	arbitrary alpha / beta	0.2214	0.6013	0.6375	0.9028	0.7786
von	arbitrary A / B	0.0000	–	0.0000	0.8520	1.0000
	arbitrary alpha / beta	0.0000	–	0.0000	0.8520	1.0000

Table 8: Running the decision under a second rendering and flagging disagreement. The detector needs two renderings that behave differently, a precondition the state-blindness statistic checks.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Accuracy on PolicyBench for three readouts over one fixed backbone, by label condition. Shown on the uncontrolled suite, where the gap between the scoring and generative readouts is largest; § 5.3 reports what happens under the control. Error bars are bootstrap 95% intervals over items.

model	aligned	label only	arbitrary A / B	neither	misleading
laya-td	0.849 $[0.825,\,0.873]$	0.856 $[0.832,\,0.879]$	0.797 $[0.770,\,0.824]$	0.569 $[0.537,\,0.604]$	0.136 $[0.113,\,0.158]$
laya-en	0.867 $[0.844,\,0.890]$	0.862 $[0.839,\,0.885]$	0.812 $[0.784,\,0.838]$	0.552 $[0.519,\,0.587]$	0.124 $[0.102,\,0.146]$
laya-ml	0.792 $[0.765,\,0.820]$	0.759 $[0.729,\,0.788]$	0.753 $[0.724,\,0.782]$	0.559 $[0.526,\,0.594]$	0.218 $[0.190,\,0.247]$
von	0.851 $[0.827,\,0.875]$	0.510 $[0.477,\,0.544]$	0.851 $[0.827,\,0.875]$	0.510 $[0.477,\,0.544]$	0.851 $[0.827,\,0.875]$
Qwen2.5-0.5B	0.701 $[0.664,\,0.738]$	0.664 $[0.624,\,0.701]$	0.679 $[0.643,\,0.714]$	0.482 $[0.441,\,0.521]$	0.476 $[0.434,\,0.513]$
Qwen2.5-1.5B	0.755 $[0.722,\,0.787]$	0.768 $[0.735,\,0.798]$	0.776 $[0.742,\,0.807]$	0.547 $[0.506,\,0.585]$	0.207 $[0.175,\,0.240]$

Appendix

Table 9: Exact values for Figure 3 : what each channel contributes, on the fixed-label choice tasks (SST-2, RTE, CB;

n=833{}

). The ordinal task is excluded because a score question has no label channel separate from its definition.

model	as written	negated	negated + warning	decision flip rate
laya-td	0.642 $[0.621,\,0.663]$	0.358 $[0.337,\,0.378]$	0.355 $[0.334,\,0.376]$	0.983 $[0.977,\,0.988]$
laya-en	0.658 $[0.637,\,0.679]$	0.345 $[0.324,\,0.366]$	0.342 $[0.321,\,0.362]$	0.982 $[0.975,\,0.987]$
laya-ml	0.644 $[0.623,\,0.665]$	0.372 $[0.351,\,0.394]$	0.369 $[0.348,\,0.390]$	0.966 $[0.958,\,0.974]$
von	0.738 $[0.719,\,0.756]$	0.263 $[0.244,\,0.281]$	0.269 $[0.250,\,0.287]$	0.971 $[0.964,\,0.979]$
Qwen2.5-0.5B	0.559 $[0.534,\,0.585]$	0.411 $[0.385,\,0.436]$	0.425 $[0.400,\,0.451]$	0.933 $[0.920,\,0.946]$
Qwen2.5-1.5B	0.692 $[0.668,\,0.716]$	0.278 $[0.253,\,0.302]$	0.354 $[0.330,\,0.380]$	0.797 $[0.777,\,0.818]$

Appendix

Table 10: Exact values for Figure 5 , with the decision flip rate of Equation ( 4 ) in the final column.

		uncontrolled suite			controlled suite (XF)
backbone	readout	aligned	arbitrary A / B	misleading	aligned	arbitrary A / B	misleading
Qwen2.5-1.5B	label log-prob.	0.507 $[0.486,\,0.529]$	0.700 $[0.680,\,0.720]$	0.500 $[0.479,\,0.521]$	0.500 $[0.479,\,0.521]$	0.513 $[0.492,\,0.534]$	0.511 $[0.490,\,0.532]$
	first token	0.511 $[0.490,\,0.533]$	0.693 $[0.673,\,0.713]$	0.500 $[0.478,\,0.521]$	0.500 $[0.479,\,0.520]$	0.514 $[0.493,\,0.535]$	0.511 $[0.490,\,0.533]$
	generate and parse	0.530 $[0.509,\,0.552]$	0.704 $[0.685,\,0.724]$	0.740 $[0.720,\,0.759]$	0.591 $[0.570,\,0.612]$	0.536 $[0.514,\,0.557]$	0.523 $[0.501,\,0.544]$
Qwen2.5-7B	label log-prob.	0.751 $[0.733,\,0.770]$	0.760 $[0.742,\,0.778]$	0.501 $[0.480,\,0.522]$	0.789 $[0.770,\,0.806]$	0.909 $[0.897,\,0.921]$	0.835 $[0.820,\,0.851]$
	first token	0.760 $[0.742,\,0.779]$	0.752 $[0.734,\,0.771]$	0.501 $[0.480,\,0.522]$	0.800 $[0.783,\,0.817]$	0.907 $[0.894,\,0.919]$	0.832 $[0.816,\,0.848]$

Appendix

Table 11: One backbone, three readouts, on both PolicyBench suites. The two scoring readouts agree throughout; the comparison with the generative readout does not hold across suites (§ 5.3 ).

model	arbitrary $-$ aligned	misleading $-$ aligned	arbitrary exch. $-$ arbitrary
laya-td	$+0.1201$ $[+0.1080,\,+0.1317]$	$+0.0912$ $[+0.0706,\,+0.1120]$	$+0.0660$ $[+0.0577,\,+0.0749]$
laya-en	$+0.2852$ $[+0.2669,\,+0.3037]$	$-0.0305$ $[-0.0603,\,-0.0009]$	$+0.0072$ $[-0.0017,\,+0.0163]$
laya-ml	$+0.0353$ $[+0.0171,\,+0.0534]$	$+0.0839$ $[+0.0580,\,+0.1100]$	$+0.0359$ $[+0.0169,\,+0.0540]$
von	$+0.0000$ $[+0.0000,\,+0.0000]$	$+0.0000$ $[+0.0000,\,+0.0000]$	$+0.0000$ $[+0.0000,\,+0.0000]$
Qwen2.5-1.5B	$+0.1928$ $[+0.1681,\,+0.2171]$	$-0.0073$ $[-0.0500,\,+0.0352]$	$-0.1766$ $[-0.2005,\,-0.1529]$
Qwen2.5-7B	$+0.0080$ $[-0.0090,\,+0.0252]$	$-0.2511$ $[-0.2867,\,-0.2152]$	$+0.1526$ $[+0.1314,\,+0.1733]$

Appendix

Table 12: Contrasts on the uncontrolled PolicyBench suite, in which the false branch carries the catch-all definition “none of the above applies”. The third column is the artifact discussed in § 5.2.1 ; compare it with the same column of Table 3 .

group	task	type	items	classes	class prevalence
fixed-label choice	sst2	choice	500	2	0.51, 0.49
	rte	choice	277	2	0.53, 0.47
	cb	choice	56	3	0.41, 0.50, 0.09
ordinal	sst5	score	500	5	0.14, 0.31, 0.16, 0.23, 0.17
yes/no	boolq	boolean	500	2	0.40, 0.60
	sst2_noul	boolean	500	2	0.51, 0.49

Appendix

Table 13: Tasks used in the paper, with the number of items evaluated and the class prevalence after any rebalancing.

Language Model Bias Evaluation

Language model bias evaluation tests how models’ outputs, decisions, and representations vary across social groups, cultures, languages, and contexts.

See all 491 papers

Sep 22, 2026

Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to It

Typed decision models return structured results, but output-type correctness alone does not ensure that decisions follow explicit option definitions. Each option pairs a name with a definition that defines its intended meaning; the name, however, can provide a competing semantic cue. We study this conflict in Jev and two open-weight models by changing only the name-definition mapping, leaving the question, state, and the names and definition texts themselves unchanged. We measure decision flips at the level of the selected definition, rather than the returned name. On 1200 decision tasks with task-specific definitions, decision-flip rates are up to 70.4 pp higher with yes/no names than with the 0/1 control. This gap holds across all 4 binary decision rules. With yes/no names, reassignment also lowers their mean AUC from 93.8% to a below-chance 23.2%. In the binary evaluations, random strings used as option names yield mean flip rates close to those of neutral controls across all three models, with comparable balanced accuracy before reassignment. Together, these results support option-name polarity as a contributor to decision instability beyond reassignment alone. The type-error rate remains 0% throughout, showing that type-correct outputs can still fail to follow explicit option definitions.

Yu Sun, Junhao Xu, Jiajia Shi +1

Fudan University · University of Science and Technology of China

LLM Decision-Making Structured Output Generation

Ask PDF

Sep 27, 2026

Beyond Calibration: Do a Typed-Decision Model's Probabilities Obey the Probability Axioms?

Typed-decision models such as TypeSafe's Jev answer a declared yes/no or multiple-choice question about a state with a probability instead of text, and their evaluations report accuracy and calibration. Neither requires that the probabilities a model gives to logically related questions fit together item by item, which is what a system that acts on those probabilities needs. We test this property, coherence, with a battery of logically linked questions that needs no labels. For 160 items from ChaosNLI and PubMedQA, each with three mutually exclusive labels, we ask whether the label is X, whether it is not X, whether it is one of the other two labels, and which label applies. On 480 negation pairs, Jev's probabilities for "the label is X" and "the label is not X" miss summing to one by 0.064 on average (95% CI 0.055 to 0.072). Qwen3.8-27B, run from its official BF16 weights, misses by 0.293 with first-token probabilities and by 0.122 with verbalized probabilities. The gap to the first-token readout persists on pairs where both systems give similar probabilities, without double-negation labels, and after averaging Jev's repeated calls. Jev is not coherent either: its violations are about five times its repeat noise, and it over-endorses statements about single labels, so that its three single-label probabilities sum to 1.14 on average. The two systems also fail differently. Qwen3.8-27B's first-token readout under-endorses the complement of a label whether or not the question contains "not", rejecting both a statement and its negation in 196 of 480 pairs, and it does not become more coherent where it is more confident, whereas Jev's violations concentrate where its answer is uncertain. Because the checks need no labels, they expose biases that appear only when question forms are compared, and inconsistencies within items.

Keyi Li, Yihao He, Quanyi Li

Software College, Northeastern University Shenyang, China

Reasoning Consistency in Language Models

Ask PDF

Sep 30, 2026

AnyJev Technical Report

A typed decision is a choice among a fixed set of options, returned as a probability rather than as text. Systems that need typed decisions today use models trained for that purpose. This report describes AnyJev, which reads a typed decision from one prefill of a pretrained instruction-tuned language model. The readout restricts the next-token distribution at the answer position to the option tokens. It has two defects: the model assigns higher probability to some labels whatever the input, and to some positions in the option list. AnyJev corrects both with no gradient steps and no parameter changes: it divides out a label prior estimated from unlabelled inputs, and it averages log-probabilities over the K cyclic rotations of the option list. On two 20-option tasks the rotations lower the order-flip rate from 0.33 to 0.14 and from 0.33 to 0.18, and raise accuracy on 11 of 11 models on both. Reading every rotation requires K prefills. A stopping rule selected against the full-rotation decision on unlabelled states cuts that. Selecting the threshold on one unlabelled split and bounding its disagreement on a second, it reads 10.6 rotations of 18 at a verified 0.008 bound on two of four cells; selected and bounded on one split, as our serving run did, it reads 7.3 and serves 2.2 times as many decisions per second on vLLM. The code is open source.

Jiamu Zhang, Tianze Yang, Yucheng Shi +6

Nokia, Sunnyvale, CA, USA · Tencent Hunyuan · The Hong Kong Polytechnic University

Multiple-Choice Question Answering Language Model Calibration LLM Decision-Making

Ask PDF

Abstract

Language Model Bias Evaluation

Explore similar work

Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to It

Beyond Calibration: Do a Typed-Decision Model's Probabilities Obey the Probability Axioms?

AnyJev Technical Report