Which Differences Matter? Controlled Tests of Rule Transfer, Joint Reliability, and Information Use in Decision Models

Tianyou Lai; Wentao Yue; Zhiwei Zuo; Qilei Li

cs.AIOct 21, 2026

Which Differences Matter? Controlled Tests of Rule Transfer, Joint Reliability, and Information Use in Decision Models

Authors: Tianyou Lai, Wentao Yue, Zhiwei Zuo, Qilei Li

Organizations: Carnegie Mellon University · Shenzhen University · Central China Normal University

Abstract

Decision models map stated rules and observed states to discrete answers. Aggregate accuracy leaves unresolved which rule and information changes guide those answers. We connect controlled input changes to transfer, joint reliability, and directed information use through executable arithmetic rules. Semantic rule separation and crossed rule and state tests make transfer and joint success separately measurable. We compare equivalent wordings, calibrate joint correctness through marginal-preserving reassociation, and alter tables to define alternative answer targets. Adaptation gains increase with nearest-source answer agreement; fully conflicting quartets yield small pooled gains and different gain signs across source splits. Equivalent rewrites have opposing task-level effects, linking transfer gains to rule presentation. Joint-score departures depend on which rule and state associations the reference preserves. A decision service changes answers more across equivalent wordings than across identical-text repeats and usually follows the answer targets implied by altered table values or mappings.

Figures & tables

Figure 1: Overview of the three behavioral diagnostics: rule transfer and wording, joint reliability, and external information use. Scores and interventions illustrate the design.

Development rules
Method	Natural	Diag. item	QAcc
Controlled Base	38.67	33.81	0.52
Rank-4 readout	39.11	35.11	0.17
LoRA	58.03	53.13	13.19
Jev	60.44	55.34	22.05

Table 1: LoRA retains higher natural accuracy on development and reserved rules. Scores are percentages. Development averages two rule splits; reserved evaluation uses batch 16, equal weights for reserved blocks, and MLP readouts selected by source-validation NLL without reserved-rule fitting or any checkpoint selection.

Nearest-source agreement groups
Group	Base	LoRA	Gain	QAcc
Natural accuracy
Low	37.06	48.53	+11.48	—
Middle	38.31	57.71	+19.40	—
High	39.06	67.13	+28.07	—
Diagnostic item accuracy and QAcc

Table 2: Transfer depends on source-answer agreement and conflict. Accuracy and LoRA QAcc are percentages; gains are pp. Base QAcc is zero throughout. Agreement groups rank pairs within task/

K

; load counts conflicting answers per intact quartet. Pooled scores weight source splits equally. Fully conflicting cohorts contain 39 and 32 quartet memberships, covering eight and seven task/

K

strata. Source splits reuse target questions with potentially different cohort memberships; these finite-cohort scores describe adaptation under source–target conflict.

Figure 2: Equivalent wording changes task-level adaptation gains in opposite directions. Paired changes in the LoRA–Base advantage use batch 8 and 95% shared-state intervals. Zero denotes an unchanged advantage on the same semantically equivalent decision questions.

Figure 3: Jev changes answers more across equivalent wordings than across identical-text repeats. (a) Disagreement rates average items within splits. (b) The paired excess remains positive after interrupted quartets are excluded. Each wording repeats twice; brackets and whiskers give the 95% quartet-resampling intervals.

Maximum selection only
Method	QAcc	Ref. mean	95% reference range
Base	0.52	1.09	[0.00, 2.60]
Rank-4 readout	0.17	1.05	[0.17, 2.09]
LoRA	13.19	12.21	[10.07, 14.41]
Jev	22.05	21.29	[19.79, 22.92]
LoRA	13.19	7.60	[4.17, 10.42]

Table 3: Pooled LoRA and Jev QAcc lie within the marginal-preserving reference ranges, while maximum-selection scores exceed their ranges. Scores are percentages. Central 95% ranges describe 1,000 stratified reassociations that preserve prediction marginals; they summarize the reference distribution rather than confidence intervals for performance on newly sampled rules.

Figure 4: Complete value tables benefit Jev more than the local predictors on these paired inputs. Lines connect missing-value (D3) and complete-value (D2) scores. Endpoint labels use a shared 0–100% accuracy scale.

Jev intervention	Overall $T/N$ (%)	Initially correct $T/B$ (%)
Formula rows	169/220 (76.82)	169/191 (88.48)
Record columns	173/190 (91.05)	173/177 (97.74)
Target values	243/320 (75.94)	243/282 (86.17)

Table 4: Jev usually switches to the altered-table target. Each cell gives transitions over its denominator, followed by the percentage. Overall rates use all conflicts; conditional rates use the initially correct conflicts.

Appendix figures & tables36 assets

Supplementary material from the paper’s appendix.

Appendix

State $(x,y,z)$	$r_{0}$	Answer	$r_{1}$	Answer
$(0,-8,0)$	$0$	no / A	$8$	yes / B
$(0,6,-10)$	$6$	yes / B	$-16$	no / A

Appendix

Table 5: Changing either the rule or the state reverses the binary answer in this evaluated quartet. Exact values determine the gold answers; zero has the answer no.

Role per replication cell	Questions	States	Permitted use
Source natural training	2,560	320	Gradient fitting. Source-training capability report.
Source diagnostic training	320	160	Gradient fitting. Source-training quartet report.
Reversed source renderings	2,880	0 new	Same labels after semantic option remapping. Fitting augmentation.
Source natural validation	640	80	Checkpoint, architecture, feature, and regularization selection where those choices are in the protocol.
Source diagnostic validation	320	160	Joint-response reporting and earlier operational qualification. Not the checkpoint-selection objective.
Development natural	640	80	New-rule natural evaluation.

Appendix

Table 6: Training, selection, and evaluation use distinct data roles in each replication cell. Reversed renderings augment training without adding complete states; audit data remain excluded from all model-selection decisions.

Model repository	Frozen revision / response version
Qwen/Qwen3.5-2B-Base	b1485b2fa6dfa1287294f269f5fb618e03d52d7c
Qwen/Qwen3.5-2B	15852e8c16360a2fea060d615a32b45270f8a8fc
Mapika/decider-2b	533964dae8be954c5b5e19fa4948e48408094c1e
startlux-models/StartLux-Decision-2B	3e2e456409a7fe44be69eee231566f7830872ea2
TypeSafe Jev (request alias: jev-latest )	jev-1.13.0 (successful responses)

Appendix

Table 7: Fixed checkpoint revisions and the recorded service version identify the evaluated systems. Jev’s recorded version identifies the service response, while its internal checkpoint and parameter count remain undisclosed.

System	Source rules	Development rules
	Natural	Natural	Diagnostic item	QAcc
Qwen Base, native BF16	39.58	34.90	33.07	0.00
Qwen Instruct	37.63	35.03	31.51	3.13
Decider v11	43.10	39.32	38.80	3.13
StartLux 2B, FP32 research path	38.28	38.54	41.67	6.25
Controlled Base	38.15	35.68	33.33	0.00

Appendix

Table 8: Higher item accuracy coexists with lower all-four correctness across the tested systems. Scores are macro-averaged percentages across the tested tasks. Each system retains its interface and training history.

Figure 5: LoRA gains recur across rule splits and state seeds, while joint correctness varies across cells. Left panels retain all development systems; right panels compare source-selected MLPs with LoRA on reserved rules. Numbers give accuracy (%). Natural accuracy uses a shared 0–100 color scale, while QAcc uses a shared 0–30 scale across all development and reserved panels for the corresponding metric, keeping cell comparisons on the same scale.

		Base		Rank-4 readout		LoRA		Jev
Split	State seed	Natural	QAcc	Natural	QAcc	Natural	QAcc	Natural	QAcc
1	101	41.15	0.00	38.54	1.04	60.16	16.67	57.42	21.88
1	202	39.58	0.00	37.37	0.00	61.46	13.54	57.55	19.79
1	303	36.72	0.00	40.23	0.00	65.36	13.54	60.03	17.71
2	101	36.46	0.00	37.76	0.00	52.08	7.29	62.11	23.96
2	202	40.10	0.00	39.19	0.00	53.91	18.75	63.02	28.13

Appendix

Table 9: LoRA gains recur across development cells, while the relative ranking of LoRA and Jev changes across rule splits. Scores are percentages, with equal weight assigned to each task family within every evaluated cell.

Split	State seed	Readout natural	Readout QAcc	LoRA natural	LoRA QAcc	Gap (pp)
1	101	40.22	0.00	52.48	7.48	+12.26
1	202	42.69	1.92	53.83	7.37	+11.14
1	303	42.29	0.00	55.49	4.27	+13.20
2	101	42.29	2.03	61.98	5.77	+19.69
2	202	42.53	0.96	57.75	13.57	+15.22
2	303	44.95	3.31	60.96	18.91	+16.01

Appendix

Table 10: Every fixed LoRA model exceeds its selected readout on reserved-rule natural accuracy. Scores are percentages, and gaps are percentage points with equal weights across all of the reserved evaluation rule blocks.

Figure 6: Individual correctness and all-four correctness reveal different task profiles. Jev leads in binary QAcc; LoRA leads in rating and routing and ties Jev in selection. Both panels use the same development questions, task/split weighting, and 0–100% scale; labels give every individual and joint score to one decimal place.

Model	Task	Item accuracy	QAcc
Base	Binary	48.96	2.08
Base	Rating	26.39	0.00
Base	Routing	27.43	0.00
Base	Selection	32.47	0.00
Rank-4 readout	Binary	50.00	0.00
Rank-4 readout	Rating	29.69	0.00

Appendix

Table 11: Task-level item and joint scores distinguish binary strength from performance on ratings and record selection. Percentages give equal weight to option counts, state seeds, and rule splits within each evaluated task.

Figure 7: Partial correctness occupies much more probability mass than all-four success for most systems. Segment widths retain the macro-weighted percentage of quartets with zero to four correct answers; dark blue marks all-four success. Every model and seed is retained, and labels annotate segments with macro weights of at least 10%.

	Item accuracy		QAcc
Model	Missing values	Complete values	Missing values	Complete values
Base	35.72	40.19	0.00	2.08
Rank-4 readout	35.81	39.28	0.00	3.13
LoRA	49.48	52.95	7.47	11.98
Jev	54.43	90.63	20.31	75.69

Appendix

Table 12: Complete values improve joint correctness most for Jev on the paired development inputs. Scores are percentages under the same task, option-count, state-seed, and rule-split macro weighting used in Figure 4 .

Figure 8: Answer binding depends on context and instructions even when the correct semantic answer is supplied. Direct instructions raise local-model accuracy, while Jev performs best with minimal context and legacy instructions. Numbers give accuracy (%) using a single 0–100 sequential color scale shared across all systems, context lengths, and instruction conditions.

Figure 9: LoRA exceeds the prospectively fixed 5 pp margin over the selected readout in every reserved block. Bars begin at zero and labels give the natural-accuracy difference (pp); the dashed line marks the decision margin.

Figure 10: Probe gains over shuffled targets are stronger on the source audit than on development inputs at many LoRA locations. Numbers give real minus shuffled-target accuracy (pp) for value and answer targets at every tested location. The diverging scale is centered at zero and shared across every evaluated layer and model panel.

Figure 11: Source-selected patches produce little change in counterfactual target success relative to the native baseline. Numbers retain absolute success (%); shading shows the within-column change from the native row (pp) on a shared, zero-centered scale. Audit and development retain all of the positive and negative controls.

Task	Model	QAcc	Ref. mean	Ref. 95% range	Obs. minus ref.
binary decision	Base	2.08	4.29	[0.00, 10.42]	-2.20
binary decision	Jev	66.67	66.55	[62.50, 70.83]	0.11
binary decision	LoRA	22.92	27.90	[20.83, 35.42]	-4.98
minimum routing	Base	0.00	0.07	[0.00, 0.69]	-0.07
minimum routing	Jev	3.47	3.00	[0.69, 5.56]	0.47
minimum routing	LoRA	9.03	5.90	[3.45, 9.03]	3.13

Appendix

Table 13: Joint-score departures depend on task and on the associations retained by reassociation. Task-specific QAcc is compared with 1,000-draw reference ranges; pooled rule-pair and state-vector controls use 2,000 draws. Scores are percentages. The reported ranges describe the reference distribution for each set of preserved associations.

Figure 12: Validation trajectories approach the frozen readout at selected early steps, but independent audit gaps remain. Curves show saved LoRA–readout gaps (pp); dotted vertical lines mark selected steps and the horizontal zero line denotes equality. Below-panel labels give audit natural-accuracy / QAcc gaps at those selected steps.

Model	Intervention	$N$	$B$	$T$	$T/N$ (%)	$T/B$ (%)
Base	Formula-row mismatch	220	69	8	3.64	11.59
Base	Record-column mismatch	190	66	5	2.63	7.58
Base	Target-value counterfactual	320	116	12	3.75	10.34
LoRA	Formula-row mismatch	220	90	17	7.73	18.89
LoRA	Record-column mismatch	190	84	21	11.05	25.00
LoRA	Target-value counterfactual	320	147	19	5.94	12.93

Appendix

Table 14: Directed answer transitions distinguish following a supplied target from an undirected answer change. Counts preserve the applicable conflict set:

T/N

uses all conflicts and

T/B

conditions on original correctness.

Metric	Models	Original	Rewrite 1	Rewrite 2	Change 1 [95% CI]	Change 2 [95% CI]
Natural	All six	19.30	16.66	19.21	-2.64 [-4.80, -0.62]	-0.09 [-1.76, 1.62]
Natural	Split 1	16.10	15.15	16.87	-0.95 [-3.10, 1.10]	0.77 [-0.87, 2.51]
Natural	Split 2	22.50	18.18	21.55	-4.33 [-6.64, -2.20]	-0.95 [-2.64, 0.80]
QAcc	All six	9.99	5.84	9.44	-4.15 [-5.65, -2.80]	-0.55 [-1.44, 0.30]
QAcc	Split 1	6.37	4.88	6.16	-1.50 [-3.13, 0.11]	-0.21 [-1.28, 0.89]
QAcc	Split 2	13.60	6.80	12.71	-6.80 [-9.44, -4.31]	-0.89 [-2.17, 0.39]

Appendix

Table 15: Functional rewrites reduce the pooled adaptation advantage; verbal rewrites produce little pooled natural-accuracy change. All entries compare LoRA with Base at batch 8. Changes are paired differences (pp); 95% intervals condition on the fixed rule allocations and model checkpoints under shared, paired input resampling.

Figure 13: LoRA remains above Base across equivalent wordings, with a larger reduction under functional notation. Points connect evaluations of fixed systems at batch 8; labels give accuracy (%). Natural accuracy and QAcc use separately labeled zero-based scales. LoRA averages six instances, and 9B is a distinct instruction-tuned system.

Task	Model	Original	Rewrite 1	Rewrite 2	Change 1 [95% CI]	Change 2 [95% CI]
Binary	Base	69.95	50.96	58.17	-18.99 [-24.77, -12.74]	-11.78 [-17.07, -6.25]
Binary	LoRA	77.72	73.60	78.04	-4.13 [-5.77, -2.48]	0.32 [-0.48, 1.16]
Routing	Base	27.64	27.48	29.25	-0.16 [-3.77, 3.37]	1.60 [-0.88, 4.09]
Routing	LoRA	44.87	38.29	41.57	-6.58 [-7.71, -5.46]	-3.30 [-4.22, -2.34]
Rating	Base	22.84	23.48	24.20	0.64 [-0.88, 2.09]	1.36 [-0.24, 2.96]
Rating	LoRA	54.91	48.33	53.15	-6.58 [-7.76, -5.46]	-1.76 [-2.60, -0.95]

Appendix

Table 16: Binary-task increases offset adaptation-gain reductions in rating, routing, and selection. Natural accuracy is reported in percentages and paired changes in pp, at batch 8 with complete-state resampling intervals.

	Same-text padding changes		Wording disagreement at fixed padding, FP32
Model	BF16	FP32	Functional (%, 95% CI)	Verbal (%, 95% CI)
Base	26/960	0/960	44.27 [34.38, 54.69]	31.25 [23.44, 39.06]
LoRA, split 1	11/960	0/960	23.44 [16.67, 30.21]	12.50 [8.33, 16.67]
LoRA, split 2	8/960	1/960	25.00 [19.27, 31.25]	17.71 [12.50, 22.92]

Appendix

Table 17: Wording sensitivity persists with FP32 inference and fixed padding on the controlled subset. Wording-flip rates compare each rewrite with the original on natural questions; same-text padding comparisons retain the wording and measure changes in the selected answer under a different padding condition on identical question inputs.

Mapping	Split	Conflict $n$	Mass (%)	Conditional weighted	Conditional subset	Contribution conflict	Contribution other
native	1	1907	43.73	-2.43	-3.11	-1.06	16.93
native	2	1722	38.90	12.62	11.37	4.91	17.26
permuted	1	1595	35.96	1.91	1.70	0.69	15.18
permuted	2	1454	32.97	14.74	13.70	4.86	17.31

Appendix

Table 18: Nearest-source nonconflicts account for most of the full-set adaptation gain. Conditional gains and weighted contributions are in pp. Weighted estimates retain full-set conflict mass; subset estimates recompute the task/

K

means on retained questions before averaging scores equally across strata within each task family.

Mapping	Split	Shared conflict $n$	LoRA–Base (pp)
native	1	918	-0.77
native	2	918	10.08
permuted	1	727	7.11
permuted	2	727	10.12

Appendix

Table 19: The common conflict intersection permits both source splits to be compared on identical questions. All ten task/

K

strata remain represented; percentages use the subset-normalized task and option-count macro.

Natural questions
Split	Agreement	Questions	Base	LoRA	Gain	95% gain CI	Conflict (%)
1	Low	1600	37.03	44.29	+7.26	[4.29, 10.31]	52.50
1	Middle	1280	39.26	54.82	+15.56	[12.48, 18.55]	41.67
1	High	1280	38.15	65.10	+26.95	[23.96, 29.86]	34.83
2	Low	1600	37.08	52.78	+15.69	[12.93, 18.33]	46.82
2	Middle	1280	37.37	60.61	+23.24	[20.42, 26.24]	39.84

Appendix

Table 20: Low-agreement pairs retain positive pooled transfer gains but lower joint correctness than high-agreement pairs. Accuracy and QAcc are percentages; gains and their intervals are percentage points. Diagnostic Base and LoRA columns give item accuracy, and QAcc gives LoRA all-four correctness. Base QAcc is zero in every group. Pooled counts retain both source-reference memberships for questions shared across the two source pools.

Split	Source–target answers	Questions	Weight (%)	Base	LoRA	Readout	Gain (pp)
1	Conflict	1907	43.73	34.49	32.07	34.02	-2.43
1	Agree	2253	56.27	40.83	70.93	47.73	+30.09
2	Conflict	1722	38.90	31.10	43.72	34.50	+12.62
2	Agree	2438	61.10	42.49	70.74	48.83	+28.25

Appendix

Table 21: Conflict-conditioned absolute accuracy exposes different transfer outcomes across the two source splits. Scores are percentages; gains compare LoRA with Base under the original weights conditioned on each subset. The readout column reports the MLP selected separately using the validation inputs of each source-rule split.

Split	Load	Quartets	Task/ $K$	Weight (%)	Base	LoRA	Gain (pp)	QAcc
1	0	37	7	13.14	32.93	47.29	+14.36	4.88
1	1	19	4	11.22	46.43	58.33	+11.90	14.29
1	2	129	10	50.32	39.17	48.89	+9.71	7.15
1	3	30	4	11.75	24.09	41.74	+17.65	4.55
1	4	39	8	13.57	20.47	14.37	-6.10	0.00
2	0	46	7	17.74	36.90	59.69	+22.79	23.09

Appendix

Table 22: All-four conflict yields little pooled accuracy gain and low joint correctness. Load counts conflicting members per complete quartet. Base and LoRA columns give diagnostic item accuracy; QAcc gives LoRA all-four correctness. Base QAcc is zero throughout. Weight is original diagnostic mass, and Task/

K

counts represented strata, listed by source split in pooled rows. Pooled counts reuse the same quartets under two source references.

Actual start gap	Within pairs	Disagree (%)	Across pairs	Disagree (%)
$<2$ s	523	9.37	1991	35.26
2–4 s	324	9.88	1363	35.95
4–8 s	105	6.67	452	34.51
8–16 s	0	N/A	2	100.00
$\geq 16$ s	8	12.50	32	37.50

Appendix

Table 23: Cross-wording disagreement exceeds same-text disagreement in every elapsed-time stratum with both comparisons. The 8–16 s stratum has no same-text pairs; the counts identify the available paired coverage.

	Priority policy		Exception policy		Finite-state transitions
System	Natural	QAcc	Natural	QAcc	Natural	QAcc
Base	60.00	0.00	31.87	0.00	25.31	0.00
Arithmetic LoRA	51.88	0.00	43.12	2.50	21.88	0.00
Qwen3.5-9B	78.12	22.50	76.25	2.50	32.81	0.00
Always deny	58.13	0.00	78.13	0.00	Not applicable

Appendix

Table 24: The tested systems show different competence across Boolean policies and finite-state transitions. All systems have zero transition-task QAcc. Scores are descriptive finite-set percentages; the arithmetic LoRA is reused without new-task training and the always-deny baseline applies to both evaluated Boolean policy rule families.

Task	Model	Strata	Size range	Singletons	Weight (%)	Constant	Weight (%)
Binary	Base	24	8–8	0	0.00	0	0.00
Binary	Jev	24	8–8	0	0.00	12	50.00
Binary	LoRA	24	8–8	0	0.00	5	20.83
Routing	Base	262	1–6	102	17.71	157	47.05
Routing	Jev	262	1–6	102	17.71	130	30.73
Routing	LoRA	262	1–6	102	17.71	129	29.34

Appendix

Table 25: Small and constant-prediction strata constrain the variation available to reassociation. Counts pool disjoint strata across six cells; weights average cell-level task macro weights. Constant counts include singletons.

Split/state	$K$	LoRA QAcc	LoRA $\bar{d}$ [ $q_{.025},q_{.975}$ ]	Jev QAcc	Jev $\bar{d}$ [ $q_{.025},q_{.975}$ ]
1/101	2	50.00	+21.48 [+0.00, +37.50]	25.00	+2.75 [-12.50, +25.00]
1/101	4	0.00	+0.00 [+0.00, +0.00]	0.00	+0.00 [+0.00, +0.00]
1/101	8	0.00	+0.00 [+0.00, +0.00]	12.50	+9.09 [+0.00, +12.50]
1/202	2	25.00	+11.99 [-12.50, +25.00]	12.50	+8.77 [+0.00, +12.50]
1/202	4	0.00	-2.69 [-12.50, +0.00]	12.50	+10.86 [+0.00, +12.50]
1/202	8	0.00	+0.00 [+0.00, +0.00]	12.50	+10.90 [+0.00, +12.50]

Appendix

Table 26: Maximum-selection departures vary across cells and option counts. QAcc is in percentages and mean differences

\bar{d}

in pp; brackets give quantiles of

d_{b}

, with eight complete diagnostic quartets in every cell/

K

stratum.

Intervention	Micro		Task/ $K$ macro		Equal cell
	$T/N$	$T/B$	$T/N$	$T/B$	$T/N$	$T/B$
Formula rows	76.82	88.48	78.81	86.47	79.18	86.58
Record columns	91.05	97.74	91.10	97.46	91.08	97.41
Target values	75.94	86.17	77.34	84.62	77.34	84.35

Appendix

Table 27: Jev's directed transitions persist under micro, task-balanced, and equal-cell weighting. All weighting schemes reuse the same paired requests and archived integer transition counts, changing their aggregation rather than the observed answers or the conflict questions entering the archived counts of paired answer transitions.

Figure 14: The fixed native chat template lowers 9B natural accuracy under all three wordings. (a) Lines connect raw and native scores on paired inputs. (b) Paired native–raw differences are negative, with 95% intervals from resampling the 13 reserved blocks. The score axis in (a) is restricted to 29–49% for resolving paired changes.

Panel of Figure 15 — Figure 15: The original native instruction asks for the minimum-scoring record and specifies the tie-break. The evaluated native request string is retained exactly.

Panel of Figure 16 — Figure 16: The complete-value request supplies scores for every rule while retaining the decision task. The displayed strings preserve the evaluated request, including its instruction, records, formula labels, and numeric values.

Evaluation	Controlled factor	Evaluation setting
CheckList / contrast sets	Local input changes	Behavioral and contrastive tests
LogicBench	Logical inference patterns	Three kinds of logic
RuleArena	Rules and tool assistance	Real-world regulations
CFQ / COGS / Ordered CommonGen	Combinations and concept order	Compositional generalization
Mind Your Format / symbol binding	Templates and option symbols	Answer-interface variation
Cacheable decision models	Joint versus independent encoding	Rule-following decision scores

Appendix

Table 28: Controlled input changes provide different kinds of evidence across evaluation designs. Our oracle-checked setting crosses rules and states and intervenes on equivalent wording and supplied-value mappings.

Rule-Based Classification

See all 22 papers

Can a Cacheable Decision Model Follow Rules?
Sep 29, 2026Dushyant Rajput, Nirdesh Chauhan, Siddharth KosarajuLLM Inference Acceleration LLM Decision-Making
AltSlate Labs LLP
Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to It
Sep 22, 2026Yu Sun, Junhao Xu, Jiajia Shi +1LLM Decision-Making Structured Output Generation
Fudan University · University of Science and Technology of China
JevOut: Natural Context Can Flip Decision Models
Sep 24, 2026Zixiang Xu, Zirui Song, Chiyu Zhang +4Adversarial Attacks on LLMs LLM Decision-Making
University of Southern California

Sep 29, 2026J/K move · Enter open · S save

Can a Cacheable Decision Model Follow Rules?

Dushyant Rajput, Nirdesh Chauhan, Siddharth Kosaraju

AltSlate Labs LLP

LLM Inference Acceleration LLM Decision-Making Efficient Language Model Inference

Open paper AskPDF arXiv

Certo is a small non-generative decision model (Qwen3-4B): it scores candidate actions from their text and returns a probability, instead of generating an answer. The accurate design reads the state, the rules, and each candidate together (a joint scorer), so cost grows with the menu. Independent encoding lets each candidate be encoded once and reused across states (about 5x cheaper at 77 candidates), but separates state from candidate. We ask how much rule-sensitivity survives that move, and whether it can be trained back. Four experiments on Certo: (1) the tested conversion to cacheable scoring loses rule-sensitivity (recall@1 1.00 -> 0.24) while the joint scorer holds 1.00, and a shortlist+rerank rescue fails; (2) targeted counterfactual supervision restores strong performance on held-out synthetic rule tasks (paraphrase, counterfactual, composition; reproducible across seeds), though we do not isolate whether predictions depend on the supplied rule; (3) on real rules the added benefit is not established -- after fixing a truncation confound, the joint scorer wins significantly on the short tier (0.861 vs 0.500) and directionally on the hard tier (0.655 vs 0.483, n=29); (4) a matched cross-domain real-prose mixture did not help and reduced contract accuracy (-9.3, -16.2 points). A cacheable encoder can be made rule-sensitive on its training distribution, but transfer to unseen-source real rules is not established; the joint scorer keeps an edge at the cost of caching.

Abstract

Rule-Based Classification

Explore similar work

Can a Cacheable Decision Model Follow Rules?

Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to It

JevOut: Natural Context Can Flip Decision Models