Differentiable Fuzzy Inference Layer: A Monotone, Compositional Ordinal Reasoning Head for Large Language Models

Zhen Zhang; Amr Alanwar

cs.CLAug 17, 2026

Differentiable Fuzzy Inference Layer: A Monotone, Compositional Ordinal Reasoning Head for Large Language Models

Authors: Zhen Zhang, Amr Alanwar

Organizations: School of Computation, Information and Technology Technical University of Munich, Germany

Abstract

A state-of-the-art language model asked to interpret "most of most students passed" typically answers "most," though composing two instances of "most" yields a proportion closer to "some." We trace this failure to an architectural choice rather than a data deficit: standard classifier heads treat ordinal categories as independent labels, with no mechanism to respect their natural ordering or compose them algebraically. We introduce the Differentiable Fuzzy Inference Layer (DFIL), a dual-path prediction head pairing a standard classifier with a scalar-bottlenecked branch grounded in a bank of ordered membership functions. DFIL supplies two structural primitives that a label-only head cannot inherit: monotonicity in the underlying quantity, and compositional reasoning via t-norm operations without any compositional training data. The scalar branch additionally provides an interpretable interface for analyzing residual errors. We instantiate DFIL on ordinal natural-language tasks across diverse LLM families.

Figures & tables

Panel of Figure 1 — Figure 1: Composing two “most” should yield “some,” not “most.” A label-only fine-tuned classifier picks $\arg\max_{q}[W\mathbf{h}]_{q}\!=\!\texttt{most}$ (incorrect); DFIL composes via t-norm on learned ordered centers $c_{q}$ and decides by nearest-center rule (Thm. 2 , 1 ), yielding some . The label-only head has no analogous algebraic primitive. Shaded box marks our method.

Method	$p$ exposed	Ordered $c_{q}$	Mono. by const.	Composes	LLM head	Interpretable
LoRA fine-tune (label head)	✗	✗	✗	✗	✓	✗
CORAL [ 7 ]	✗	✗	✓ †	✗	partial ‡	✗
LaSQuE [ 10 ]	✗	✗	✗	✗	✓	partial
PRESQUE [ 19 ]	✓ §	fixed	✓ §	✗	no ¶	✓
ANFIS [ 15 ]	✓	✓	✓	✓ ∗	✗	✓
DFIL (ours)	✓	✓	✓	✓	✓	✓

Table 1: Closest baselines on head-level capabilities for ordinal and compositional reasoning. “Explicit

p

” = head exposes a scalar proportion; “Ordered

c_{q}

” = learnable, order-constrained class centers; “Mono. by const.” = predictions monotone in

p

by construction (deployed inference rule); “Composes” = supports algebraic composition without re-training; “LLM head” = used as a Transformer LLM head in published work; “Interpretable” = per-class quantities mapping to human-readable boundaries.

Panel of Figure 2 — Figure 2: DFIL dual-path architecture. LoRA-adapted backbone (hatched) produces hidden state $\mathbf{h}$ . Main path: classifier $\mathbf{h}\to\hat{y}$ . DFIL path (shaded): NumericalHead $\mathbf{h}\to\hat{p}\!\in\![0,1]$ , projected through $Q$ ordered Gaussian MFs to $\boldsymbol{\mu}$ , with t-norm (dashed) for compositional and entailment queries. Both paths trained jointly; inference path selected by downstream needs (Thm. 1 , 2 ).

Claim	Effect	Statistic / setting
Compositional from-text (Tab. 3 )	$\mathbf{+5.9}$ pp	paired $t$ $p{=}0.011$ , $d{=}0.63$ , $n{=}20$ seeds
Uniform-init compositional (§ 5.3 )	$\mathbf{+23.2}$ pp	alignment-controlled, $n{=}3$ seeds
Single-step parity, 6 backbones (App. D.7.1 )	$-0.1$ to $+1.9$ pp	Welch $p\!\in\![0.43,0.97]$ , FRoG-Hard
Sample efficiency, $N\!\leq\!64$ (App. D.1 )	$+11.7$ to $+16.5$ pp	MF branch dominates at low budget
CSMC consistency, $n{=}21$ (App. D.8 )	$\mathbf{+5.50}$ pp	paired $t$ $p{=}0.0001$ , Wilcoxon $p{=}0.0003$

Table 2: Headline empirical results. Six findings anchoring the paper’s contributions; full per-cell numbers in the indicated table or appendix. Multi-seed cells report sample std (ddof=1).

Self-consistent GT (leakage-free, $n{=}20$ seeds for FT+Compose / DFIL)
Method	From-text	Oracle †
Fine-tune (text-only)	$16.4\pm 7.3^{\natural}$	—
FT+Compose	$25.4\pm 3.5$	$100.0\pm 0.0^{\flat}$
DFIL (ours)	$\mathbf{31.2\pm 9.2}$	$100.0\pm 0.0^{\flat}$
$\Delta$ vs. Fine-tune	$+14.8$	—
$\Delta$ vs. FT+Compose	$\mathbf{+5.9}^{\ddagger}$	$0.0$

Table 3: Compositional quantifier reasoning , Qwen2.5-7B,

900

unseen compositions. We report two evaluation protocols. QuRe-based GT (literature default) labels from frozen QuRe centers and favours methods whose centers sit at QuRe. Self-consistent GT labels from each model’s own learned centers and favours methods with adaptive centers; it is leakage-free w.r.t. external generators but not w.r.t. the predictor itself. Oracle † : composition applied to ground-truth

p

. Under QuRe-based GT, FT+Compose’s oracle (

94.3

) reflects its frozen QuRe-aligned centers mechanically matching the GT generator (not a capability advantage); DFIL’s lower oracle there (

50.5

) reflects its learned-center drift from QuRe (not compositional weakness). The self-consistent oracle row above confirms both methods reach

100\%

against their own MF (Theorem 2 ).

Init	Method	From-text	Text-only
Uniform	DFIL	$\mathbf{41.6\pm 7.3}$	$13.4\pm 4.5$
Uniform	FT+Compose	$18.4\pm 6.2$	$19.5\pm 4.1$
QuRe	DFIL	$31.2\pm 9.2$	$18.5\pm 9.2$
QuRe	FT+Compose	$25.4\pm 3.5$	$17.6\pm 3.6$

Table 4: Self-consistent compositional accuracy (%); GT from each model’s own MF (oracle

=100\%

by construction). Mean

\pm

std, Qwen2.5-7B: uniform

n{=}3

, QuRe

n{=}20

. QuRe from-text paired

t

p{=}0.011

d{=}0.63

Backbone	DFIL (%)	Fine-tune (%)	$\Delta$	MF mono. ‡
Qwen2.5-7B	$50.4\pm 1.8$	$50.5\pm 1.8$	$-0.1$	$0.0\%$
Qwen2.5-14B	$57.3\pm 1.7$	$56.9\pm 2.8$	+0.4	$0.1\%$
Qwen2.5-32B ¶	$55.4\pm 5.6$	$53.7\pm 6.1$	+1.7	$0.1\%$
Qwen2.5-72B †	$\mathbf{77.2\pm 1.5}$	$76.5\pm 1.6$	$+0.7$	$0.1\%$
Cross-model generalization:
Mistral-7B-v0.3	$64.1\pm 0.9$	$62.2\pm 4.5$	$+1.9$	—

Table 5: Scaling on FRoG-Hard, mean

\pm

std (%), 3 seeds (6 at 32B/72B).

\Delta

= DFIL

-

FT. GPT-4 zero-shot: 33–48% [ 20 ] .

Method	1.5B (%)	7B (%)	Mono.	Extra Params
Fine-tune only (LoRA + Linear)	$48.0\pm 0.1$	$51.3\pm 2.1$	No	2.6M
DFIL-only (MF path only)	$41.6\pm 1.0$	$46.2\pm 2.6$	1 viol.	2.6M
Unordered MF (no ordering)	$39.7\pm 2.0$	$38.8\pm 7.6$	Unstable	2.6M
No MF (1D linear classifier)	$17.8\pm 8.6$	$39.3\pm 9.7$	No	2.6M
DFIL (dual-path, ours)	$\mathbf{48.2\pm 0.4}$	$\mathbf{49.9\pm 2.6}$	1 viol.	3.0M

Table 6: Ablation on FRoG-Hard, mean

\pm

sample std over 3 seeds. This table uses

\lambda_{\text{DFIL}}{=}0.5

, the high-regularization stress regime : it isolates the contribution of each component (ordering, dual-path) under strong MF-loss pressure. The deployed default for parity claims (Sec. 5.5 , App. D.7.1 ) is

\lambda_{\text{DFIL}}{=}0.1

, validation-tuned, where the 7B DFIL row becomes

50.4{\pm}1.8

vs. fine-tune

50.5{\pm}1.8

(

-0.1

pp). The two regimes are separately reported; we do not blend them.

Overall ( $n\!=\!21$ ): DFIL MF $\mathbf{96.40}$ vs FT main $90.90$ , $\Delta\!=\!\mathbf{+5.50}$ pp
Domain	DFIL	FT	$\Delta$	Domain	DFIL	FT	$\Delta$
Public health	$94.5$	$79.0$	$\mathbf{+15.5}$	Clinical trials	$99.5$	$97.4$	$+2.1$
Education	$94.8$	$80.5$	$\mathbf{+14.3}$	Manufacturing	$93.1$	$91.2$	$+1.9$
Environment	$98.6$	$88.6$	$\mathbf{+10.0}$	Research	$97.9$	$96.2$	$+1.7$
Governance	$91.2$	$86.2$	$+5.0$	Finance	$95.7$	$94.8$	$+1.0$
Election	$99.8$	$96.4$	$+3.3$	Sports	$99.0$	$98.8$	$+0.2$

Table 7: CSMC consistency per domain (%) via DFIL’s MF branch (Thm. 1 ) vs. FT main classifier.

20

pairs/domain, mean over

21

seeds; methodology in App. D.8 .

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Uniform Init from-text	QuRe Init from-text
DFIL	$\mathbf{41.6\pm 7.3}$	$31.2\pm 9.2$
FT+Compose	$18.4\pm 6.2$	$25.4\pm 3.5$
$\Delta$ (DFIL $-$ FT+Compose)	$\mathbf{+23.2}$	$+5.9$

Appendix

Table 8: Self-consistent compositional accuracy (%) under uniform vs QuRe MF initialization (Qwen2.5-7B,

n{=}3

uniform /

n{=}20

QuRe; mean

\pm

sample std). Each model’s own learned MF defines the ground-truth labels (oracle is

100\%

by construction; we report from-text instead). DFIL learns centers from data; FT+Compose freezes its centers at the initialization. Both initialization regimes use

20

epochs of training; per-seed results are in the supplementary archive.

Method	MF-based	Classifier-based
DFIL	$\mathbf{100.0\pm 0.0}$	$56.0\pm 2.8$
Fine-tune	—	$52.6\pm 2.7$

Appendix

Table 9: Quantifier entailment accuracy (%). MF-based uses learned center ordering (

c_{Q_{i}}{\leq}c_{Q_{j}}

); classifier-based uses argmax labels on sentence pairs. Mean

\pm

sample std over 3 seeds, Qwen2.5-7B.

Method	DFIL-from-text	Text-only
DFIL	$\mathbf{39.5\pm 3.3}$	$12.8\pm 2.4$
Fine-tune	—	$11.1\pm 0.9$
$\Delta$ (DFIL-from-text $-$ Fine-tune text-only)	+28.4

Appendix

Table 10: Naturalistic compositional reasoning accuracy (%).

320

examples from

8

domain templates (clinical trials, education, manufacturing, elections, environmental science, corporate governance, sports, urban planning), with QuRe-based ground-truth labels (composed proportion bucketed via fixed QuRe centers, independent of either model’s learned MF). Mean

\pm

sample std over 3 seeds (Qwen2.5-7B).

Backbone	DFIL (main cls)	DFIL (MF branch)	Fine-tune
Mistral-7B-v0.3	$\mathbf{19.0\pm 2.3}$	$14.2\pm 5.4$	$16.5\pm 0.9$
Qwen2.5-7B	$14.9\pm 2.5$	$\mathbf{31.3\pm 2.7}$	$16.6\pm 4.4$

Appendix

Table 11: Cross-benchmark transfer to QuRe (Sec. 5.4 ). Mean

\pm

sample std over

3

seeds (42, 123, 456), no retraining.

Budget $N$	$16$	$32$	$64$
Fine-tune	$4.6{\pm}1.3$	$9.6{\pm}3.6$	$19.4{\pm}5.7$
DFIL (best path)	$\mathbf{21.1{\pm}1.3}$	$\mathbf{21.3{\pm}6.0}$	$\mathbf{23.1{\pm}2.7}$
$\Delta$ (DFIL $-$ FT)	$\mathbf{+16.5}$	$\mathbf{+11.7}$	$\mathbf{+3.7}$

Appendix

Table 12: Few-shot on FRoG-Hard, Qwen2.5-7B, 3 seeds. “DFIL (best path)” is the best of {main, MF, val-

\alpha

ensemble} where the path and

\alpha

are selected on the held-out validation set (no test-set selection).

Headline (mean over $Q_{1}$ /sources, 3 seeds)
Stratum	$n$	DFIL from-text	Text-only	$\Delta$
DFIL ( dfil_reg )	249	$32.8\pm 3.7$	$12.6\pm 0.6$	$+20.2$
FT+Compose	249	$\mathbf{36.4\pm 1.9}$	$12.9\pm 0.4$	$\mathbf{+23.6}$
Per outer quantifier $Q_{1}$ (FT+Compose, mean over seeds)
most	80	$28.7$	$9.6$	$+19.2$
some	80	$31.2$	$14.2$	$+17.1$

Appendix

Table 13: External NL Compositional Benchmark.

n{=}249

naturally-occurring sentences mined from GenericsKB-Best, PubMedQA, and CNN/DailyMail; LLM-judge filtered for true compositional structure via a 4-judge ensemble spanning families and scales (Qwen2.5-7B, Mistral-7B-Instruct-v0.3, Llama-3.1-8B-Instruct, Qwen2.5-72B-Instruct), each judge with triple-vote at

T{=}0.4

, retaining items with

\geq 3

4

judges agreeing. Inter-judge pairwise agreement on

595

commonly judged candidates: Qwen-7B

\leftrightarrow

Mistral

81.2\%

, Qwen-7B

\leftrightarrow

Qwen-72B

80.4\%

, Qwen-7B

\leftrightarrow

Llama

77.3\%

, Mistral

\leftrightarrow

Llama

77.0\%

, Mistral

\leftrightarrow

Qwen-72B

74.6\%

, Llama

\leftrightarrow

Qwen-72B

70.0\%

(range

70

–

81\%

across

6

pairs spanning

10\!\times

scale);

69.3\%

majority TRUE on items where all 4 judges gave a definite vote. Per-judge TRUE rates span both strict and lenient regimes (Qwen-72B

58.3\%

, Qwen-7B

64.5\%

, Mistral

66.7\%

, Llama

89.0\%

);

\geq 3

-of-

4

majority voting calibrates the ensemble between the strict (Qwen-72B) and lenient (Llama) ends. Pipeline: regex match “

Q_{1}

(of) (the) class (who

|

that

|

which

|

|

from

|

with) clause” on

\sim

3{,}200

raw hits; balance to

\leq 80

per canonical

Q_{1}

. Ground-truth label:

c_{Q_{1}}\!\cdot\!p_{\mathrm{base}}

via fixed QuRe centers, with

p_{\mathrm{base}}

from judge annotation. Qwen2.5-7B FRoG-trained checkpoints, 3 seeds, sample std. Random baseline

12.5\%

Property	Product	Minimum	Łukasiewicz
Smoothness	$C^{\infty}$	$C^{0}$	$C^{0}$
Idempotent	No	Yes	No
Nilpotent	No	No	Yes
“most of most”	$<$ “most”	$=$ “most”	$<$ “most”
Error contraction	Multiplicative	No contraction	Additive
Parameters	0	0	0

Appendix

Table 14: Comparison of t-norms for compositional quantifier reasoning.

Quantifier	Train	Val	Test (Hard)	Proportion range
none	25	6	31	$[0.00,0.00]$
tiny amount	85	21	106	$[0.01,0.10]$
few	414	104	518	$[0.05,0.30]$
small amount	322	80	402	$[0.10,0.35]$
some	162	40	202	$[0.25,0.50]$
moderate amount	389	97	486	$[0.35,0.65]$

Appendix

Table 15: FRoG dataset statistics. The “easy” and “hard” splits differ in surface-level cues: the hard split removes explicit proportion mentions from the text.

$\lambda_{\text{DFIL}}$	0.1	0.5 (default)	1.0	2.0	5.0
Accuracy (%)	50.4	51.2	49.9	51.2	50.4

Appendix

Table 16:

\lambda_{\text{DFIL}}

single-seed sensitivity probe on FRoG-Hard (Qwen2.5-7B, seed 42); reports the magnitude of variation across

\lambda

values, not a multi-seed sensitivity analysis.

Budget	16	32	64	128	256
Fine-tune	$4.6\pm 1.3$	$9.6\pm 3.6$	$19.4\pm 5.7$	$27.5\pm 0.4$	$33.0\pm 1.7$
DFIL (main)	$5.1\pm 1.2$	$8.6\pm 0.4$	$23.2\pm 2.6$	$26.4\pm 1.8$	$34.0\pm 1.4$
DFIL (MF)	$\mathbf{21.1\pm 1.3}$	$\mathbf{21.3\pm 6.0}$	$\;\;8.5\pm 5.7$	$22.7\pm 4.2$	$25.7\pm 1.6$
DFIL (ens., val- $\alpha$ )	$\mathbf{21.1\pm 1.3}$	$\mathbf{21.0\pm 6.3}$	$\mathbf{23.1\pm 2.7}$	$26.7\pm 1.7$	$33.8\pm 1.5$
$\Delta$ (best DFIL $-$ FT)	$\mathbf{+16.5}$	$\mathbf{+11.7}$	$\mathbf{+3.7}$	$-0.8$	$+0.8$

Appendix

Table 17: Few-shot learning on FRoG-Hard (Qwen2.5-7B). Accuracy (%), mean

\pm

std over 3 seeds. Ensemble coefficient

\alpha

is selected on validation. Bold marks the best DFIL path per budget when it exceeds fine-tune.

Budget	32	64	128	256	512
Fine-tune	$34.9\pm 0.5$	$36.6\pm 1.4$	$40.2\pm 3.1$	$45.5\pm 2.3$	$47.6\pm 2.0$
DFIL (main)	$34.9\pm 0.1$	$35.1\pm 1.9$	$39.8\pm 2.5$	$46.0\pm 2.4$	$48.8\pm 0.9$
DFIL (MF)	$38.1\pm 1.6$	$39.0\pm 1.7$	$40.3\pm 7.5$	$46.8\pm 3.9$	$50.9\pm 1.2$
DFIL (ens., val- $\alpha$ )	$\mathbf{37.9\pm 1.4}$	$\mathbf{39.1\pm 1.6}$	$\mathbf{43.4\pm 2.3}$	$\mathbf{48.1\pm 2.9}$	$\mathbf{50.8\pm 1.1}$
$\Delta$ (ensemble $-$ FT)	$\mathbf{+3.0}$	$\mathbf{+2.5}$	$\mathbf{+3.2}$	$\mathbf{+2.6}$	$\mathbf{+3.2}$

Appendix

Table 18: Few-shot learning on SST-5 (Qwen2.5-7B). Accuracy (%), mean

\pm

std over 3 seeds,

\alpha

selected on validation.

Backbone	Task	DFIL main	DFIL MF	DFIL ens α=0.5	Fine-tune	Mono
Qwen2.5-7B-Instruct	SST-5	$57.78\pm 1.33$	$\mathbf{57.98\pm 0.82}$	$57.86\pm 1.41$	$57.47\pm 0.64$	$0.0\%$
Mistral-7B-Instruct-v0.3	SST-5	$60.47\pm 0.98$	$\mathbf{60.71\pm 0.35}$	$60.39\pm 0.96$	$59.22\pm 0.11$	$0.0\%$
Phi-3.5-mini-3.8B	SST-5	$53.80\pm 0.30$	$\mathbf{54.99\pm 0.35}$	$53.97\pm 0.15$	$54.31\pm 1.15$	$0.0\%$
Llama-3.1-8B-Instruct	SST-5	$58.51\pm 0.68$	$\mathbf{59.17\pm 0.69}$	$58.55\pm 0.71$	$58.60\pm 0.21$	$0.0\%$
Qwen2.5-7B-Instruct	STS22	$\mathbf{34.18\pm 0.59}$	$28.93\pm 1.52$	$34.86\pm 1.06$	$32.99\pm 0.88$	$0.1\%$
Mistral-7B-Instruct-v0.3	STS22	$\mathbf{36.21\pm 1.47}$	$29.78\pm 1.17$	$35.87\pm 1.06$	$35.03\pm 3.05$	$0.1\%$

Appendix

Table 19: Cross-task accuracy (%) on SST-5 and STS22 across four backbone families. Mean

\pm

std over

3

seeds. Bold marks the best of (DFIL main, DFIL MF, DFIL ens) per cell when it beats Fine-tune. “Mono” is the synthetic-grid violation rate of the MF branch.

Scale	Method	none	tiny	few	small	some	mod.	most	all
		(31)	(106)	(518)	(402)	(202)	(486)	(199)	(100)
7B	DFIL	43.0	30.5	66.2	45.7	30.9	60.4	24.0	44.7
7B	FT	36.6	31.1	72.8	45.1	36.1	60.2	25.6	28.3
14B	DFIL	48.4	58.5	72.7	56.9	33.8	62.9	38.0	39.7
14B	FT	43.0	50.3	78.9	52.1	32.2	65.3	28.6	39.3
32B	DFIL	47.3	56.6	80.1	56.7	42.9	65.2	41.5	56.3

Appendix

Table 20: Per-class accuracy (%) on FRoG-Hard across scales. Mean over 3 seeds. Sample count per class in parentheses in header row.

	Scale	none	tiny	few	small	some	mod.	most	all
CORAL	7B	$83.9$	$31.4$	$57.6$	$30.6$	$7.3$	$48.8$	$34.0$	$83.3$
		$\pm 8.5$	$\pm 1.4$	$\pm 9.8$	$\pm 7.3$	$\pm 7.7$	$\pm 4.0$	$\pm 1.0$	$\pm 1.5$
CORAL	14B	$92.5$	$28.0$	$68.9$	$38.8$	$10.9$	$59.9$	$44.9$	$83.7$
		$\pm 4.9$	$\pm 10.6$	$\pm 8.8$	$\pm 5.9$	$\pm 10.7$	$\pm 4.3$	$\pm 0.3$	$\pm 4.5$

Appendix

Table 21: CORAL ordinal regression per-class accuracy (%) on FRoG-Hard. Mean

\pm

std over 3 seeds. DFIL and FT 7B per-class numbers, for comparison, are reported in Table 20 .

	none	tiny	few	small	some	mod.	most	all
QuRe initialization:
$c_{\text{init}}$	.020	.080	.180	.280	.400	.580	.780	.970
Learned centers (7B):
seed 42	.020	.024	.089	.243	.353	.478	.832	.980
seed 123	.020	.024	.104	.239	.352	.490	.774	.980
seed 456	.020	.024	.082	.232	.347	.470	.736	.980

Appendix

Table 22: Learned MF centers and widths (Qwen2.5-7B, each seed). QuRe initialization shown for reference.

Method / Path	Scale	Pairwise viol. (%)	Mean magnitude	Total pairs
DFIL (MF branch)	7B	$\mathbf{0.10}$	—	1,000
DFIL (main classifier)	7B	10.8	1.69	1,959,395
Fine-tune (classifier)	7B	10.6	1.69	1,959,395
CORAL (classifier)	7B	8.2	2.03	1,959,395
CORAL (classifier)	14B	6.6	2.13	1,959,395
CORN [ 28 ] (classifier)	7B	9.0	1.62	1,959,395

Appendix

Table 23: Monotonicity violation analysis on FRoG-Hard. Mean over 3 seeds. “MF branch” refers to DFIL’s membership function predictions; all other rows report main classifier predictions.

	Overall				Level 1 / Level 2
Seed	FT-text	Compose-text	DFIL-text	DFIL-oracle	FT-text	Compose-text	DFIL-text	DFIL-oracle
42	9.2	32.0	31.9	54.3	7.0/13.7	25.0/46.0	30.8/34.0	65.7/31.7
123	23.9	25.9	31.6	36.0	34.0/3.7	19.0/39.7	30.5/33.7	37.0/34.0
456	16.1	25.6	31.1	61.2	9.5/29.3	22.0/32.7	28.7/36.0	69.3/45.0
Mean	16.4	27.8	31.5	50.5	16.8/15.6	22.0/39.5	30.0/34.6	57.3/36.9
Std	7.3	3.6	0.4	13.0	14.9/12.9	3.0/6.7	1.1/1.2	17.7/7.1

Appendix

Table 24: Compositional accuracy (%) per seed (Qwen2.5-7B). Three evaluation methods. Compose-text values are from the May 2026 re-run (batch 64) since earlier eval-pipeline results were not preserved; per-seed results are in the supplementary archive.

Variant	Accuracy (%)	Trainable params	Description
DFIL (full)	$48.2\pm 0.4$	2,974,489	Full dual-path model
Fine-tune only	$48.0\pm 0.1$	2,577,672	Standard classifier, no DFIL
DFIL-only	$39.2\pm 1.3$	2,575,889	MF branch with $\lambda_{\text{DFIL}}{=}0$ on dfil variant
No MF branch	$17.8\pm 8.6$	2,575,889	MF branch removed entirely ( no_dfil )
No ordering	$39.7\pm 2.0$	2,575,889	MF without cumulative softmax

Appendix

Table 25: Extended ablation on FRoG-Hard (Qwen2.5-1.5B). Mean

\pm

sample std over 3 seeds. Note: this table reports a later retrain run at batch size

8

; the main-text Table 6 reports the same DFIL-only/no-DFIL/no-ordering ablations from an earlier batch-

64

run, where DFIL-only accuracy is

41.6\pm 1.0\%

. The two runs differ only in batch size and resulting LoRA effective learning rate; the qualitative ranking (DFIL full

\approx

fine-tune only

\gg

no-ordering

\approx

DFIL-only

\gg

no-MF) is identical across both runs.

Scale	$\Delta$ (%)	DFIL $\sigma$ (%)	FT $\sigma$ (%)	Variance ratio
7B	$-0.1$	1.8	1.8	1.03
14B	+0.4	1.7	2.8	0.61
32B †	+1.7	5.6	6.1	0.92
72B	$+0.7$	1.5	1.6	0.94

Appendix

Table 26: DFIL advantage decomposition across scales.

Category	DFIL (text)	DFIL (oracle)	Fine-tune
Linguistic ( $n{=}12$ )	$22.2\pm 12.7$	$\mathbf{91.7\pm 0.0}$	$30.6\pm 4.8$
Approximate ( $n{=}12$ )	$16.7\pm 14.4$	$\mathbf{77.8\pm 4.8}$	$13.9\pm 4.8$
Comparative ( $n{=}8$ )	$4.2\pm 7.2$	$\mathbf{50.0\pm 0.0}$	$12.5\pm 21.7$
Emphatic ( $n{=}8$ )	$16.7\pm 19.1$	$\mathbf{62.5\pm 0.0}$	$0.0\pm 0.0$
Overall (exact)	$15.8\pm 7.6$	$\mathbf{73.3\pm 1.4}$	$15.8\pm 5.8$
Overall (off-by-one)	$48.3\pm 6.3$	$\mathbf{100.0\pm 0.0}$	$45.0\pm 2.5$

Appendix

Table 27: Per-category IPR accuracy (%); mean

\pm

sample std over 3 seeds. “DFIL (text)” is the standard text

\to

backbone

\to

numerical-head pipeline. “DFIL (oracle)” bypasses the numerical head and feeds the ground-truth proportion (e.g.,

0.85

for “vast majority”) directly into the MF branch, applying the nearest-center rule on the model’s learned centers; this isolates head-side competence from backbone-side proportion extraction. Fine-tuning has no analogous oracle column because the label classifier exposes no scalar interface to inject proportions into. Off-by-one counts a prediction as correct if it differs from the gold label by at most one quantifier class.

Configuration	FRoG Acc.	CSMC Consistency	Pairwise Viol.
DFIL baseline	$51.0\pm 1.3$	$92.3\pm 2.4$	$10.8$
+ Distillation	$54.4\pm 1.3$	$91.8\pm 0.8$	$\;\;9.9\pm 0.1$
$\Delta$	$+3.4$	$-0.5$	$-0.9$

Appendix

Table 28: Monotonicity distillation results (Qwen2.5-7B, 3 seeds). Canonical evaluation (best.pt checkpoint).

† The very large $+30.2$ pp FRoG gain is a red flag, not a clean win:
Configuration	FRoG Acc.	CSMC Cons.
DFIL baseline	$51.0\pm 1.3$	$92.3\pm 2.4$
+ Attention pool †	$81.1\pm 0.6$	$95.8\pm 2.0$
$\Delta$	$+30.2$	$+3.5$
the attention head appears to memorize easy-split lexical patterns of the FRoG training distribution.

Appendix

Table 29: Attention-weighted numerical head results (Qwen2.5-7B, 3 seeds, best.pt checkpoint).

Compositional Reasoning

Compositional reasoning studies how models combine facts, concepts, rules, and skills to solve tasks that require multiple structured steps.

See all 39 papers

Aug 21, 2026

Latent Ordinal Evidence, Misaligned Outputs: Inference-Time Ordinal Lens Alignment for Multimodal LLMs

Multimodal LLMs apply the language model interface to visual inputs, where ordinal regression tasks such as age estimation, image quality assessment, and disease grading require autoregressive decisions over ordered class labels. We ask whether MLLMs reliably convert internal ordinal evidence into ordered digit-token outputs. Across four ordinal benchmarks and four MLLM backbones, ordinal labels are linearly recoverable from hidden states with Spearman correlation up to 0.938, and a task-designed prompt further sharpens this structure. Yet native digit-token outputs weakly expose it: the unembedding matrix filters the ordinal direction, and the digit-token row space retains below 1.15% across all 16 model-dataset combinations, with a 16 to 77 absolute-point accuracy gap between linear-probe and native outputs. We introduce Ordinal Lens Alignment (OLA), a frozen-backbone inference-time method that trains lightweight W_S-anchored lenses on mid-to-deep decoder layers, fuses them into an ordinal distribution, and corrects only digit-token logits at generation. OLA outperforms the SOTA LoRA-tuned OrderChain baseline in most settings while keeping the MLLM frozen, surpasses discriminative ordinal baselines in most cells, and improves over an offline lens in every setting.

Haiming Li, Yingsheng Liu, Jingmin Zhu +5

Faculty of Information Technology, Monash University, Melbourne, Australia · Monash University, Melbourne, Victoria, Australia

Multimodal Large Language Models Ordinal Regression Inference-Time Language Model Alignment Multimodal Model Interpretability

Ask PDF

Oct 12, 2026

FunDM: Cheap & Label-Free Decision Models from Frozen Language Models

Decision models such as Jev score candidate answers or actions. A small decision head trained label-free can recover about 99% of a frozen language model's multiple-choice accuracy while using only two-thirds of its transformer layers. It learns the full model's answer probabilities with KL divergence, without correct-answer labels for training. One backbone pass reads the question and all options together; the head returns probabilities for every option. We call this model FunDM (Fun Decision Model). Using 24 of Qwen3-8B's 36 layers and skipping the LM head, its +1T variant reaches 75.0% mean accuracy over eight benchmarks, versus 75.7% for the full model. On MMLU-Pro, both heads achieve 1.56 to 1.57 times batched vLLM throughput with about one-third lower peak GPU allocator memory. Both heads score higher than released Contrastive Language Models (CLM), fine-tuned CLM, and CLM-shaped heads trained from scratch on the eight multiple-choice benchmarks and coding-agent selection. Separately trained agent heads learn judge probabilities without pass/fail labels and select passing runs for 74.4 to 74.6 of 100 held-out tasks on average, versus 74 for the full-model judge and 52.0 for a CLM-shaped action-trained head. Code is available at: https://github.com/Ujjawal-K-Panchal/FunDM

Ujjawal K. Panchal

Fundamental AI

LLM Decision-Making Efficient Language Model Inference LLM Inference Acceleration Multiple-Choice Question Answering LLM Evaluation Language Model Distillation

Ask PDF

Aug 9, 2026

Are LLMs Positionally Consistent Ordinal Classifiers? A Systematic Evaluation

Large language models are increasingly used for ordinal classification, yet semantically equivalent changes to prompt organization can alter their predictions. We conduct systematic experiments to characterize positional bias from label order, demonstration order, and demonstration placement. First, we apply the three probes to ten frontier LLMs on a common ordinal-classification task; every model is sensitive to all three positional sources, showing that the problem is pervasive. Second, we vary eight prompt-, task-, and model-level factors across five datasets; accuracy and stability are often misaligned, and only lower scale cardinality consistently improves both. Third, we compare pointwise, pairwise, and listwise inference, alternative aggregation and debiasing methods, and joint configurations; the tested corrections do not provide a reliable remedy, while a comparison-based listwise formulation offers the best balance but transfers unevenly across models and bias sources. These findings show that positional robustness depends on the full system configuration rather than the model alone. Ordinal-classification systems should therefore be selected jointly for predictive performance and stability.

Yu Wang, Zhe Zhou, Menglin Liu +1

Cornell University · University of Washington · The Chinese University of Hong Kong, Shenzhen +1

Language Model Bias Evaluation Classification Language Model Robustness Input Order Sensitivity in Language Models

Ask PDF

Abstract

Compositional Reasoning

Explore similar work

Latent Ordinal Evidence, Misaligned Outputs: Inference-Time Ordinal Lens Alignment for Multimodal LLMs

FunDM: Cheap & Label-Free Decision Models from Frozen Language Models

Are LLMs Positionally Consistent Ordinal Classifiers? A Systematic Evaluation