Latent Ordinal Evidence, Misaligned Outputs: Inference-Time Ordinal Lens Alignment for Multimodal LLMs

Haiming Li; Yingsheng Liu; Jingmin Zhu; Siyuan Yan; Xieji Li; Jiajun Sun; Zhen Yu; Zongyuan Ge

cs.CVAug 21, 2026

Latent Ordinal Evidence, Misaligned Outputs: Inference-Time Ordinal Lens Alignment for Multimodal LLMs

Authors: Haiming Li, Yingsheng Liu, Jingmin Zhu, Siyuan Yan, Xieji Li, Jiajun Sun, Zhen Yu, Zongyuan Ge

Organizations: Faculty of Information Technology, Monash University, Melbourne, Australia · Monash University, Melbourne, Victoria, Australia

Abstract

Multimodal LLMs apply the language model interface to visual inputs, where ordinal regression tasks such as age estimation, image quality assessment, and disease grading require autoregressive decisions over ordered class labels. We ask whether MLLMs reliably convert internal ordinal evidence into ordered digit-token outputs. Across four ordinal benchmarks and four MLLM backbones, ordinal labels are linearly recoverable from hidden states with Spearman correlation up to 0.938, and a task-designed prompt further sharpens this structure. Yet native digit-token outputs weakly expose it: the unembedding matrix filters the ordinal direction, and the digit-token row space retains below 1.15% across all 16 model-dataset combinations, with a 16 to 77 absolute-point accuracy gap between linear-probe and native outputs. We introduce Ordinal Lens Alignment (OLA), a frozen-backbone inference-time method that trains lightweight W_S-anchored lenses on mid-to-deep decoder layers, fuses them into an ordinal distribution, and corrects only digit-token logits at generation. OLA outperforms the SOTA LoRA-tuned OrderChain baseline in most settings while keeping the MLLM frozen, surpasses discriminative ordinal baselines in most cells, and improves over an offline lens in every setting.

Figures & tables

	Adience ( $C=8$ )			DR ( $C=5$ )			HCI ( $C=5$ )			Aesthetic ( $C=5$ )
Backbone	IT/N	LT/N	LT/D	IT/N	LT/N	LT/D	IT/N	LT/N	LT/D	IT/N	LT/N	LT/D
Qwen3-VL	0.916	0.872	0.938	0.427	0.276	0.758	0.617	0.556	0.843	0.859	0.864	0.916
Gemma-4	0.887	0.893	0.929	0.362	0.255	0.757	0.430	0.551	0.773	0.794	0.838	0.919
LLaVA-NeXT	0.917	0.920	0.924	0.421	0.365	0.756	0.554	0.586	0.800	0.822	0.860	0.906

Table 1: SRCC Diagnostics for Probe-Readable Ordinal Evidence

Figure 1: Spectral Filtering of the Probed Ordinal Direction by the Unembedding Matrix

W_{U}

Backbone	Adience	DR	HCI	Aesthetic
Qwen2.5-VL	1.05%	0.92%	0.99%	1.01%
Qwen3-VL	1.12%	1.06%	1.08%	1.14%
Gemma-4	0.97%	0.88%	0.93%	0.96%
LLaVA-NeXT	1.03%	0.95%	0.97%	1.02%

Table 2: Digit-Token Row-Space Retention of Probe-Derived Ordinal Directions

Figure 2: Architecture of Ordinal Lens Alignment for Inference-Time

		Adience		DR		HCI		Aesthetic
Backbone	Method	ACC $\uparrow$	MAE $\downarrow$	ACC $\uparrow$	MAE $\downarrow$	ACC $\uparrow$	MAE $\downarrow$	ACC $\uparrow$	MAE $\downarrow$	Mean ACC $\uparrow$
Discriminative	Ord2Seq	0.639	0.430	0.842	0.250	0.609	0.520	0.644	0.304	0.684
Discriminative	NumCLIP	0.651	0.410	0.831	0.270	0.696	0.350	0.665	0.296	0.711
Qwen2.5-VL 7B	Naive Prompt	0.572	0.544	0.156	1.567	0.278	0.952	0.372	0.772	0.345
	CAA	0.628	0.396	0.702	0.357	0.338	0.817	0.245	0.928	0.478
	OrderChain	0.765	0.282	0.870	0.140	0.653	0.347	0.705	0.295	0.748

Table 3: Main benchmark results

Backbone	Method	ACC $\uparrow$	MAE $\downarrow$	ACC $\uparrow$	MAE $\downarrow$	ACC $\uparrow$	MAE $\downarrow$	ACC $\uparrow$	MAE $\downarrow$	Mean ACC $\uparrow$
		Adience		DR		HCI		Aesthetic
Qwen2.5-VL 7B	Designed Prompt	0.601	0.399	0.830	0.227	0.483	0.610	0.454	0.681	0.592
	Probe Steering	0.650	0.364	0.725	0.331	0.386	0.749	0.429	0.666	0.548
	Probe Only	0.780	0.267	0.910	0.095	0.623	0.390	0.670	0.350	0.746
	Capacity Probe	0.792	0.272	0.894	0.082	0.616	0.401	0.687	0.329	0.747
	Single-Layer Logit	0.715	0.305	0.895	0.110	0.620	0.410	0.530	0.470	0.690

Table 4: Component ablation across the design space identified by the diagnostics.

Figure 3: OLA accuracy under varying numbers of fused mid-to-deep decoder layers

Figure 4: Row-Normalized Confusion Matrices

Figure 5: Layer-Wise PCA of Last-Token Hidden States

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

$y$	0	1	2	3	4	5	6	7
Range (yrs)	0–2	4–6	8–13	15–20	25–32	38–43	48–53	60+
Count	2,488	2,140	2,124	1,642	5,081	2,339	830	872

Appendix

Table 5: Adience class distribution.

$y$	0	1	2	3	4
Grade	No DR	Mild	Mod.	Severe	Prolif.
Count	25,810	2,443	5,292	873	708

Appendix

Table 6: DR class distribution.

$y$	0	1	2	3	4
Quality	Unaccept.	Flawed	Ordinary	Profess.	Except.
Count	23	552	7,603	4,574	232

Appendix

Table 7: Aesthetic class distribution.

Dataset	Attribute	$C$	Train+val	Test/fold	Folds
Adience	age	8	$\sim 14{,}900$	$\sim 3{,}700$	5 (official)
DR	disease grade	5	28,101	7,025	5 (random)
HCI	decade	5	1,060	265	5 (random)
Aesthetic	aesthetic rating	5	12,777	207	Balanced test

Appendix

Table 8: Per-dataset split summary.

Layer	Adience	DR	HCI	Aesthetic
2	0.6845	0.7522	0.5771	0.7405
4	0.7283	0.7433	0.3927	0.7175
6	0.7154	0.7503	0.4278	0.7301
8	0.7423	0.7530	0.4769	0.7607
10	0.7259	0.7539	0.4980	0.7773
12	0.7632	0.7542	0.5234	0.8155

Appendix

Table 9: Per-layer SRCC on Qwen3-VL.

Layer	Adience	DR	HCI	Aesthetic
2	0.7987	0.7485	0.3744	0.4907
9	0.8023	0.7492	0.2497	0.5670
12	0.8095	0.7541	0.2413	0.7309
15	0.8481	0.7554	0.5146	0.7909
18	0.8866	0.7564	0.7188	0.7934
20	0.8885	0.7563	0.7251	0.8610

Appendix

Table 10: Per-layer SRCC on Gemma-4.

Layer	Adience	DR	HCI	Aesthetic
2	0.7751	0.7495	0.3611	0.6679
4	0.8164	0.7487	0.3857	0.7171
6	0.8304	0.7504	0.3736	0.6762
8	0.8504	0.7504	0.3742	0.6867
10	0.8608	0.7525	0.3568	0.6788
12	0.8666	0.7557	0.3250	0.6950

Appendix

Table 11: Per-layer SRCC on LLaVA-NeXT.

Backbone $\backslash$ Dataset	Adience	DR	HCI	Aesthetic
Qwen3-VL	L20 / 0.9376	L20 / 0.7578	L28 / 0.8432	L24 / 0.9163
Gemma-4	L40 / 0.9293	L21 / 0.7567	L26 / 0.7728	L40 / 0.9192
LLaVA-NeXT	L26 / 0.9236	L14 / 0.7559	L30 / 0.8003	L16 / 0.9056

Appendix

Table 12: Best layer and best SRCC per backbone–dataset cell.

Figure 6: Layer-wise 3D PCA of last-token hidden states on Adience under Qwen3-VL. Points are coloured by the eight age groups; the ordinal arc emerges in the middle-to-late decoder layers.

Figure 7: Layer-wise 3D PCA on Adience under Qwen2.5-VL.

Figure 8: Layer-wise 3D PCA on Adience under Gemma-4.

Figure 9: Layer-wise 3D PCA on Adience under LLaVA-NeXT.

Figure 10: Layer-wise 3D PCA on DR under Qwen3-VL. Points are coloured by the five ICDR severity grades.

Figure 11: Layer-wise 3D PCA on HCI under Qwen3-VL. Points are coloured by the five decade classes.

Figure 12: Layer-wise 3D PCA on Aesthetic under Qwen3-VL. Points are coloured by the five aesthetic grades.

		Adience		DR		HCI		Aesthetic
Backbone	Method	ACC $\uparrow$	MAE $\downarrow$	ACC $\uparrow$	MAE $\downarrow$	ACC $\uparrow$	MAE $\downarrow$	ACC $\uparrow$	MAE $\downarrow$
Qwen2.5-VL	Naive Prompt	0.5722 $\pm$ 0.0168	0.5442 $\pm$ 0.0644	0.1564 $\pm$ 0.0058	1.5670 $\pm$ 0.0118	0.2788 $\pm$ 0.0027	0.9520 $\pm$ 0.0044	0.3723	0.7721
	CAA	0.6284 $\pm$ 0.1641	0.3956 $\pm$ 0.3129	0.7022 $\pm$ 0.0142	0.3574 $\pm$ 0.0211	0.3381 $\pm$ 0.0396	0.8167 $\pm$ 0.0882	0.2453	0.9283
	OrderChain	0.7650 $\pm$ 0.0344	0.2823 $\pm$ 0.0127	0.8700 $\pm$ 0.0098	0.1400 $\pm$ 0.0016	0.6528 $\pm$ 0.0423	0.3473 $\pm$ 0.0223	0.7053	0.2947
	OLA (ours)	0.8088 $\pm$ 0.0156	0.2327 $\pm$ 0.0045	0.9201 $\pm$ 0.0040	0.0799 $\pm$ 0.0004	0.6848 $\pm$ 0.0281	0.3153 $\pm$ 0.0132	0.6962	0.3039
Qwen3-VL	Naive Prompt	0.5805 $\pm$ 0.0340	0.4011 $\pm$ 0.0299	0.6882 $\pm$ 0.0121	0.5704 $\pm$ 0.0229	0.3281 $\pm$ 0.0201	0.9526 $\pm$ 0.0202	0.3969	0.8588

Appendix

Table 13: Main results on four ordinal benchmarks under four MLLM backbones, with mean

\pm

std over five folds (Adience, DR, HCI); Aesthetic uses one class-balanced fold.

		Adience		DR		HCI		Aesthetic
Backbone	Method	ACC $\uparrow$	MAE $\downarrow$	ACC $\uparrow$	MAE $\downarrow$	ACC $\uparrow$	MAE $\downarrow$	ACC $\uparrow$	MAE $\downarrow$
Qwen2.5-VL	Capacity Probe	0.7923 $\pm$ 0.0250	0.2721 $\pm$ 0.0086	0.8941 $\pm$ 0.0178	0.0821 $\pm$ 0.0016	0.6161 $\pm$ 0.0356	0.4009 $\pm$ 0.0232	0.6871	0.3291
	Probe Only	0.7800 $\pm$ 0.0286	0.2667 $\pm$ 0.0098	0.9100 $\pm$ 0.0106	0.0950 $\pm$ 0.0011	0.6230 $\pm$ 0.0251	0.3904 $\pm$ 0.0157	0.6700	0.3500
	Probe Steering	0.6496 $\pm$ 0.1851	0.3639 $\pm$ 0.2881	0.7246 $\pm$ 0.0071	0.3306 $\pm$ 0.0196	0.3864 $\pm$ 0.0619	0.7488 $\pm$ 0.0806	0.4293	0.6658
	Single-Layer Logit	0.7150 $\pm$ 0.0215	0.3050 $\pm$ 0.0092	0.8950 $\pm$ 0.0086	0.1100 $\pm$ 0.0011	0.6200 $\pm$ 0.0258	0.4100 $\pm$ 0.0171	0.5300	0.4701
	Designed Prompt	0.6012 $\pm$ 0.0280	0.3989 $\pm$ 0.0180	0.8300 $\pm$ 0.0054	0.2270 $\pm$ 0.0015	0.4825 $\pm$ 0.0248	0.6104 $\pm$ 0.0314	0.4541	0.6812

Appendix

Table 14: Ablation studies on four ordinal benchmarks under four MLLM backbones (mean

\pm

std over five folds for Adience, DR, HCI; Aesthetic uses one class-balanced fold).

Explore similar work

CORDIAL: Calibrating Ordinal LLM Outputs from Few Labels
Sep 24, 2026Xiangwei Wang, Peng Wang, Saman HalgamugeLanguage Model Calibration
The University of Melbourne, Australia · Shanghai Jiao Tong University, China
OrdinalBench: A Benchmark Dataset for Diagnosing Generalization Limits in Ordinal Number Understanding of Vision-Language Models
Mar 8, 2026Yusuke Tozaki, Hisashi MiyamoriVLM Evaluation VLM Reasoning
Division of Frontier Informatics, Kyoto Sangyo University, Kyoto, Japan
Same Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language Models
Jun 24, 2026Akshay Paruchuri, Sanmi Koyejo, Ehsan AdeliMultimodal Robustness LLM Auditing
Stanford University

Sep 24, 2026J/K move · Enter open · S save

CORDIAL: Calibrating Ordinal LLM Outputs from Few Labels

Xiangwei Wang, Peng Wang, Saman Halgamuge

The University of Melbourne, Australia · Shanghai Jiao Tong University, China

Language Model Calibration

Open paper AskPDF arXiv

A large language model (LLM) can turn a text into a distribution over an ordered scale, but that distribution is a noisy measurement: saturated, compressed or exaggerated, and biased in a consistent direction. We propose CORDIAL, which treats the model's output as a noisy reading of the true label and corrects it with a channel of five interpretable parameters. The channel is small enough for its posterior to be averaged from a handful of labels, and we prove that the resulting calibration preserves first-order stochastic order. On Amazon reviews and CMU-MOSEI transcripts with four LLMs, CORDIAL has the lowest log loss among nine calibrators in 76 of 80 settings with 5 to 100 labels; with 20 labels and the main 7B reader, it matches the strongest baseline using 28-54 labels. The same posterior lets us learn priors from other tasks and fuse several LLMs. Unrestricted calibrators such as Dirichlet calibration overtake it only as the calibration set grows into the hundreds or thousands.

Abstract

Explore similar work

CORDIAL: Calibrating Ordinal LLM Outputs from Few Labels

OrdinalBench: A Benchmark Dataset for Diagnosing Generalization Limits in Ordinal Number Understanding of Vision-Language Models

Same Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language Models