Vision-language models (VLMs) are widely used as zero-shot image quality assessors, yet their good-vs-poor answers agree only moderately with human opinion. We ask whether these models fail to see quality or fail to say it. Layer-wise probes on five VLMs point to the second: quality is linearly decodable from the vision encoder and still from the last language layer, far better than the answer reveals, and about a hundred labels recover most of it. The loss lies mainly in the final read-out, which partly reads something else: relative to human opinion, it over-rewards colorfulness and under-rewards sharpness on all five models. We test this by intervention: if the read-out is at fault, a closed-form edit of a single last-layer neuron, fitted to many prompts at once, should close the gap and no more. Both hold. Under prompts never used for fitting, the edit recovers 86--100% of what a ridge head on the last-layer state, fitted under the same prompts, finds beyond the answer on KonIQ. Where that gap is small, it gains little, too little for our transfer criterion on the two largest models; where the head itself is below the answer, it lowers the score; and it works only when written from the middle of the language model on. Unlike unconstrained fine-tuning, the edit leaves perception question answering intact. It does not carry over to yes-or-no or rating questions, and it reverses the good--poor contrast after sentiment-laden sentences. Code will be made available.
Figures & tables
Figure 2: Quality decodable from the model but missing from its answer (Qwen3.5-0.8B, KonIQ test). Percentile ranks among the 2,015 test images by human opinion, by the model’s good − poor answer and after ME (means over 12 held-out prompts). Of the 182 images the model misplaces by at least 40 percentiles, three per direction are drawn at random; ME brings 61 of them within 15 percentiles.
Figure 3: Where quality is decodable. KonIQ test SRCC of ridge probes on each vision block and language layer, against the answer before and after ME (one held-out prompt).
Figure 4: How little the read-out needs. (a) KonIQ test SRCC against the number of pool labels (0.8B; mean and range over five subsets; ME: 12 held-out prompts, probes: one). (b) The read-out written into one language layer at a time (0.8B solid, 4B dashed, base horizontal; 500 KonIQ test images, real forward passes).
KonIQ [ 14 ]
KADID [ 24 ]
LIVE-C [ 11 ]
BID [ 5 ]
AGIQA-3K [ 19 ]
R4-CGQA [ 23 ]
Model
Method
SRCC
PLCC
SRCC
PLCC
SRCC
PLCC
SRCC
PLCC
SRCC
PLCC
SRCC
PLCC
CLIP-IQA [ 35 ] (zero-shot)
0.642
0.725
0.513
0.535
0.702
0.720
0.590
0.567
0.649
0.702
0.214
0.251
LIQE [ 46 ] (trained on KonIQ)
0.897
0.905
0.722
0.722
0.864
0.866
0.705
0.695
0.725
0.792
0.128
0.216
CLIP + ridge (2,000 labels)
0.794
0.825
0.640
0.648
0.753
0.790
0.779
0.810
0.853 †
0.906
0.469 ‡
0.539
Qwen3.5-0.8B
Base
0.642
0.727
0.546
0.591
0.694
0.761
0.662
0.715
0.687
0.772
0.109
0.172
+MTC
0.773
0.831
0.578
0.612
0.774
0.825
0.755
0.787
0.800 †
0.858
0.306 ‡
0.343
Table 1: The model’s own good − poor answer under 12 held-out prompts (SRCC; PLCC after a 4-parameter logistic mapping). Base is zero-shot; ME and MTC (gradient tuning of the same weights): 2,000 KonIQ pool labels, means over five versions; † / ‡ refit on the AGIQA-3K / R4-CGQA pool ( Sec. 5.5 ). KonIQ–BID were consulted in development, AGIQA-3K and R4-CGQA added after the recipe was fixed. LIQE was trained on KonIQ-10k; its KonIQ values (gray) may include its training images.
Figure 5: The edit reaches the last-layer head. (a) SRCC of a ridge head on the state entering the final norm, read outside the model, against ME (same pool and prompts); (b) the same as gains over the base answer. Dashed: y=x .
Figure 6: Two ways to improve quality answers. KonIQ test SRCC against the change in LLVisionQA accuracy. Base and ME: 12 held-out prompts; trained models: their own Q-Align score, one marker per seed; KL: LoRA with a KL penalty.
Variant
KonIQ
KADID
LIVE-C
BID
Base
0.642
0.546
0.694
0.662
ME (ours)
0.800
0.593
0.781
0.763
Unconstrained gradient tuning
0.792
0.562
0.777
0.759
Rescale 300 neurons, no fit
0.685
0.586
0.723
0.700
Ridge head read outside the model
on layer-16 state
0.772
0.563
0.765
0.742
Table 2: Ablations on Qwen3.5-0.8B (SRCC, 12 held-out prompts, mean of five versions). Bold: best per column, ∗ excluded; ∗ KADID column (Base 0.552, ME 0.602) on 40 held-out reference images.
KonIQ
LLVQA Δ
Valence
SST val.
SST-2
0.8B
base
0.642
–
6.5
1.75
86.7
ME
0.800
[ − 0.2, + 0.3]
[ − 0.1, 0.4]
[ − 0.77, − 0.56]
[83.0, 84.2]
NME
[0.800, 0.801]
[ − 0.5, − 0.3]
[6.2, 6.3]
[1.49, 1.55]
[80.2, 80.8]
4B
base
0.742
–
7.4
2.25
93.8
ME
0.842
[ − 0.3, 0.0]
[3.3, 3.6]
[ − 2.11, − 1.83]
[93.8, 93.9]
NME
[0.842, 0.844]
[ − 0.3, + 0.2]
[8.2, 8.3]
[0.44, 0.58]
[93.8, 94.0]
Table 3: Effects of ME and of its text-constrained version NME (brackets: range over versions). KonIQ: SRCC; LLVQA: LLVisionQA accuracy change (points); Valence and SST val.: “good” − “poor” log-probability after positive minus after negative sentences (ten reviews; 872 SST-2 sentences, measured in a later run, not on Gemma); SST-2: accuracy (%).
Question
Answer prefix
Fit prompts (the only prompts any edit sees)
F1 a
Rate the quality of the image.
The quality of the image is
F2
How would you judge the technical quality of this photo?
The technical quality of this photo is
F3
Assess the perceptual quality of this image, considering blur, noise, exposure and compression.
The perceptual quality of this image is
F4 b
Describe the visual quality of this image.
The visual quality of this image is
F5 b
What is the image quality like?
The image quality is
Table S1 : The 24 prompts (question; answer prefix). a The agents’ prompt; b the nine audit prompts; c the held-out prompt of the layer-wise probes.
KonIQ
KADID
LIVE Challenge
BID
Model
Pr.
Blk
Ans.
best
last
CI
Ans.
best
last
CI
Ans.
best
last
CI
Ans.
best
last
CI
0.8B
F1
5
0.653
0.903
0.840
[0.23, 0.28] ✓
0.551
0.563
0.608
[ − 0.03, 0.06]
0.697
0.796
0.794
[0.07, 0.13] ✓
0.693
0.843
0.779
[0.11, 0.19] ✓
0.8B
H1
5
0.617
0.903
0.849
[0.26, 0.31] ✓
0.588
0.563
0.570
[ − 0.07, 0.02]
0.702
0.796
0.802
[0.06, 0.13] ✓
0.662
0.843
0.783
[0.14, 0.23] ✓
4B
F1
10
0.795
0.910
0.888
[0.10, 0.13] ✓
0.713
0.554
0.667
[ − 0.19, − 0.13]
0.793
0.825
0.838
[0.01, 0.05] ✓
0.810
0.856
0.857
[0.02, 0.08] ✓
4B
H1
10
0.735
0.910
0.882
[0.15, 0.20] ✓
0.732
0.554
0.698
[ − 0.21, − 0.15]
0.795
0.825
0.837
[0.01, 0.05]
0.802
0.856
0.861
[0.02, 0.09] ✓
9B
F1
12
0.728
0.917
0.891
[0.17, 0.21] ✓
0.674
0.588
0.693
[ − 0.12, − 0.05]
0.806
0.840
0.851
[0.01, 0.05] ✓
0.796
0.867
0.858
[0.04, 0.10] ✓
Table S2 : Answer vs. probes (KonIQ-fitted ridge probes on the 2,000-image pool, one prompt). Ans.: the model’s good − poor answer; best: probe on the selected vision block (Blk); last: probe on the last token of the last language layer; CI: 95% CI of best − answer; ✓ : meets the 0.03 gap with CI above zero.
Figure S1 : (a) Labels needed on 4B (as Fig. 4 a; mean and range over five subsets per size). (b) Vision-block probes on Gemma and 27B under H1 (KonIQ test SRCC against relative block depth); L: last-layer probe; A: the answer.
Qwen3.5-0.8B
Qwen3.5-4B
n
vision
last
ME
rand.
vision
last
ME
10
0.335
0.115
0.409
0.277
0.347
0.628
0.677
20
0.596
0.638
0.678
0.181
0.562
0.750
0.743
50
0.743
0.734
0.712
0.356
0.749
0.786
0.794
100
0.797
0.752
0.739
0.367
0.801
0.819
0.814
200
0.845
0.792
0.761
0.438
0.850
0.839
0.819
Table S3 : KonIQ test SRCC against the number of labels. Vision: probe on the best vision block; last: probe on the last language layer (one held-out prompt); ME: 12 held-out prompts; rand.: probe on the randomly initialized encoder. Answers: 0.8B 0.642, 4B 0.742.
Qwen3.5-0.8B, layers 1–24
0.579
0.739
0.601
0.733
0.708
0.710
0.739
0.775
0.733
0.780
0.732
0.782
0.789
0.785
0.786
0.809 ∗
0.810 ∗
0.808 ∗
0.795
0.800 ∗
0.813 ∗
0.815 ∗
0.807 ∗
0.815
Qwen3.5-4B, layers 2, 4, …, 32
0.702
0.711
0.682
0.757
0.809
0.807
0.847 ∗
0.847 ∗
0.870 ∗
0.877 ∗
0.867 ∗
0.868 ∗
0.860 ∗
0.864 ∗
0.862 ∗
0.859
Table S4 : Read-out written into one language layer (500 KonIQ test images, 12 held-out prompts, real forward passes). Base: 0.651 (0.8B), 0.760 (4B). ∗ Paired CI against the last-layer edit includes zero or lies above it.
Model
d
1/d
cos
halves
KonIQ
LIVE-C
0.8B
1,024
0.031
0.046
0.77
0.75 / 0.51
0.84 / 0.67
4B
2,560
0.020
0.053
0.77
0.81 / 0.54
0.90 / 0.75
Gemma
1,536
0.026
0.026
0.79
0.67 / 0.44
0.75 / 0.59
9B
4,096
0.016
0.045
0.77
0.80 / 0.54
0.91 / 0.75
27B
5,120
0.014
0.030
0.59
0.86 / 0.61
0.91 / 0.74
Table S5 : Answer read-out vs. last-layer head. cos: cosine of deff and the head’s weights; halves: cosine of heads fitted on two pool halves; output: Spearman correlation of the two scores, raw / given MOS, mean over held-out prompts.
Brightness
Contrast
Colorful.
Sharpness
Aesthetic
KonIQ
0.8B
0.25 / 0.27 / 0.27
− 0.08 / 0.12 / 0.14
0.14 / 0.05 / 0.03
− 0.22 / 0.13 / 0.16
0.16 / 0.18 / 0.13
4B
0.05 / 0.21 / 0.19
0.09 / 0.18 / 0.18
0.12 / 0.07 / 0.08
0.00 / 0.20 / 0.21
0.36 / 0.11 / 0.12
Gemma
0.09 / 0.20 / 0.19
− 0.06 / 0.21 / 0.21
0.17 / 0.03 / 0.01
− 0.15 / 0.26 / 0.27
0.29 / 0.22 / 0.20
9B
0.06 / 0.19 / 0.18
− 0.03 / 0.10 / 0.12
0.14 / 0.05 / 0.03
− 0.12 / 0.13 / 0.20
0.34 / 0.18 / 0.13
27B
0.11 / 0.20 / 0.18
0.10 / 0.16 / 0.16
0.11 / 0.05 / 0.04
0.04 / 0.17 / 0.18
0.31 / 0.16 / 0.14
Table S6 : Partial Spearman correlation with image statistics given MOS (answer / last-layer head / ME). Correlation of each statistic with MOS: KonIQ 0.36, 0.24, − 0.02, 0.60, 0.40; LIVE Challenge 0.22, 0.11, 0.17, 0.50, 0.50.
Model
Dataset
Base
ME
ME − Base
CI
Prompts
Per-prompt gain
MTC
MTC − ME
Sig.
0.8B
KonIQ
0.642
0.800
+ 0.158
[ + 0.132, + 0.184]
12
+ 0.112 to + 0.204
0.773
− 0.035 to − 0.024
5 / 0
KADID
0.546
0.593
+ 0.047
[ + 0.004, + 0.087]
10
− 0.047 to + 0.115
0.578
− 0.029 to + 0.000
1 / 0
LIVE-C
0.694
0.781
+ 0.087
[ + 0.058, + 0.117]
12
+ 0.035 to + 0.169
0.774
− 0.011 to − 0.004
0 / 0
BID
0.662
0.763
+ 0.101
[ + 0.069, + 0.136]
12
+ 0.057 to + 0.143
0.755
− 0.011 to − 0.005
0 / 0
4B
KonIQ
0.742
0.842
+ 0.100
[ + 0.064, + 0.131]
11
− 0.018 to + 0.190
0.822
− 0.030 to − 0.011
5 / 0
KADID
0.698
0.640
− 0.058
[ − 0.107, − 0.015]
1
− 0.206 to + 0.064
0.663
+ 0.007 to + 0.047
0 / 3
Table S7 : KonIQ-fitted ME and MTC on the four test sets (SRCC, 12 held-out prompts, means over five versions). CI: smallest lower and largest upper bound of the two-way 95% bootstrap CI of ME − Base over the five versions. Prompts: held-out prompts (of 12) on which ME’s mean SRCC exceeds the base model’s, and the range of these per-prompt gains. MTC − ME: range over the five paired versions; Sig.: versions in which ME / MTC is significantly higher.
Model
KonIQ
KADID
LIVE-C
BID
0.8B
0.158 / 0.158
0.022 / 0.047
0.091 / 0.087
0.098 / 0.101
4B
0.104 / 0.100
− 0.067 / − 0.058
0.034 / 0.031
0.041 / 0.040
Gemma
0.226 / 0.222
0.073 / 0.076
0.147 / 0.152
0.139 / 0.143
9B
0.142 / 0.126
− 0.023 / − 0.054
0.038 / 0.022
0.055 / 0.042
27B
0.087 / 0.075
− 0.096 / − 0.109
0.027 / 0.011
0.038 / 0.027
Table S8 : Read-out gap / ME gain (SRCC; gap = last-layer head − answer, gain = ME − answer).
Model
Selected γ
CV gain
Test Δ SRCC
0.8B
16, 16, 16, 16, 16
0.0025–0.0038
− 0.0014 to + 0.0025
4B
4, 4, 4, 2, 4
0.0000–0.0005
− 0.0010 to + 0.0016
Gemma
8, 4, 4, 4, 4
0.0002–0.0013
+ 0.0001 to + 0.0021
9B
4, 4, 4, 4, 8
0.0005–0.0016
− 0.0004 to + 0.0043
Table S9 : Extended γ grid. Selected γ per version; gain in pool cross-validated SRCC over γ=2 ; change in test SRCC of the refitted edit over the original ME (4 datasets, 5 versions, 12 held-out prompts).
Gate variation
Neurons
ME − random (KonIQ)
≤0.10
5
− 0.008 to − 0.003
0.10–0.50
18
− 0.005 to + 0.052
>0.50
17
+ 0.030 to + 0.226
Table S10 : Random neurons (four models; 40 neurons). KonIQ SRCC of ME minus the random neuron with the same version number, by the variation of the neuron’s gate (standard deviation over mean). ME’s own neurons vary by 0.02–0.10.
Format
0.8B
4B
“high” − “low”
0.607 → 0.660–0.663 ∗
0.706 → 0.739–0.740 ∗
“excellent” − “bad”
0.535 → 0.579–0.582 ∗
0.627 → 0.659–0.661 ∗
“sharp” − “blurry”
0.674 → 0.726–0.731 ∗
0.700 → 0.708 ∗
Q-Align five levels
0.469 → 0.489–0.513
0.691 → 0.788–0.792 ∗
Yes/no question
0.679 → 0.676–0.679
0.664 → 0.664
1–5 rating
0.621 → 0.627–0.628 ∗
0.731 → 0.726–0.729 ∘
Table S11 : Other answer formats (KonIQ SRCC, base → range over ME versions; ∗ CIs above zero for all versions, ∘ CIs below zero, otherwise CIs include zero).
Two images, one prompt
Single image
Model
Base
ME
Base
ME
0.8B
71.5%
71.7–73.0%
84.8%
93.8–94.2%
4B
87.6%
87.7–88.0%
88.4%
95.4–95.6%
Table S12 : Two-image comparison (accuracy, mean of three prompts; ME: range over five versions). “Single image”: each image scored alone under the 12 held-out prompts, then compared.
Data
Model
Base
ME
NME
Crit.
SPAQ
0.8B
0.833
0.863–0.864
0.863–0.864
5/5
4B
0.882
0.881–0.882
0.884–0.885
0/5
Gemma
0.781
0.850–0.852
0.849–0.851
5/5
9B
0.866
0.865–0.867
0.864–0.866
0/5
FLIVE
0.8B
0.428
0.472–0.477
0.470–0.471
5/5
4B
0.394
0.480–0.483
0.480–0.483
5/5
Table S13 : Test sets never used for evaluation before (SRCC, 12 held-out prompts; ranges over five versions; the KonIQ-fitted edits without refitting). Crit.: versions meeting the transfer criterion.
Model
Method
KonIQ
KADID
LIVE-C
BID
0.8B
Base
0.589
0.466
0.617
0.642
Q-Debias
0.534
0.549
0.535
0.650
ME
0.756–0.763
0.554–0.562
0.765–0.768
0.746–0.753
4B
Base
0.792
0.698
0.788
0.803
Q-Debias
0.646
0.679
0.780
0.788
ME
0.818–0.821
0.498–0.523
0.789–0.792
0.837–0.840
Table S14 : Q-Debias (SRCC under the single-image question; ME: range over five versions).
Figure S2 : Twelve KonIQ test images drawn at random (Qwen3.5-0.8B). Bars: percentile among the 2,015 test images by MOS (human), by the base model’s mean good − poor score over the 12 held-out prompts (model) and by ME (mean of five versions).
Model
Base
ME
ME − Base
CI
Prompts
Per-prompt gain
MTC
MTC low
ME > MTC
0.8B
0.687
0.825
+ 0.137
[0.097, 0.178]
12
+ 0.069 to + 0.265
0.800
0.073
5/5
4B
0.750
0.848
+ 0.098
[0.057, 0.139]
11
− 0.011 to + 0.225
0.836
0.049
5/5
Gemma
0.623
0.818
+ 0.195
[0.123, 0.275]
12
+ 0.063 to + 0.472
0.804
0.106
4/5
9B
0.718
0.842
+ 0.124
[0.064, 0.181]
11
− 0.057 to + 0.276
0.837
0.072
0/5
Table S15 : AGIQA-3K test set, edits refit on the AGIQA-3K pool (994 images, 12 held-out prompts, five versions). CI: smallest lower and largest upper bound of the two-way 95% CI of ME − Base over versions; Prompts: held-out prompts on which ME is above the base; MTC low: smallest lower CI bound of MTC − Base; ME > MTC: versions in which ME is significantly higher (MTC is never significantly higher).
Base
KonIQ ME
Refit on graphics
Model
SRCC / PLCC
Δ SRCC
SRCC / PLCC
Δ (CI low)
Val
0.8B
0.109 / 0.172
− 0.007 to + 0.006
0.36 / 0.38–0.39
+ 0.25 (0.11–0.12)
0.06 → 0.41–0.42
4B
0.184 / 0.243
− 0.078 to − 0.070
0.41–0.42 / 0.47–0.48
+ 0.23 (0.09–0.10)
0.17 → 0.51
Gemma
0.153 / 0.180
+ 0.028 to + 0.044
0.35–0.39 / 0.40–0.43
+ 0.20–0.24 (0.08–0.11)
0.13 → 0.41–0.44
9B
0.148 / 0.278
+ 0.012 to + 0.020
0.38 / 0.46–0.47
+ 0.23–0.24 (0.08)
0.07 → 0.47–0.48
Table S16 : R4-CGQA test set (220 images, 12 held-out prompts; ranges over 5 versions). KonIQ ME: the edit fitted on KonIQ, unchanged; refit: the same recipe fitted on 2,000 graphics images. CI: two-way bootstrap over images and prompts. Val: 90 validation images.
Own @512
Own @1,024
Held-out @1,024
Refit
Mini (0.8B)
0.126 / 0.106
0.232 / 0.231
0.079
0.360
Lite (4B)
0.170 / 0.127
0.293 / 0.321
0.102
0.416
Pro (9B)
0.270 / 0.289
0.299 / 0.329
0.013
0.382
Table S17 : Q-ReAlign on the R4-CGQA test set. Own: Q-Align prompt and level-token score (SRCC / PLCC); held-out: mean SRCC under our 12 prompts; refit: our edit refit on graphics, same base size (12 held-out prompts, mean of 5 versions).
Top-1
MOS percentile
Model
Base
MTC
ME
Base
MTC
ME
0.8B
0.24
0.43
0.51
0.77
0.88
0.90
4B
0.41
0.54
0.60
0.83
0.90
0.91
Gemma
0.18
0.51
0.53
0.73
0.89
0.90
9B
0.31
0.51
0.57
0.78
0.90
0.91
Table S18 : Best-of- N selection on AGIQA-3K (100 test prompts; ME and MTC: mean of five versions).
Model
ME
Last-layer probe
Vision probe
0.8B
0.50–0.52 / 0.90
0.55 / 0.90
0.52 / 0.89
4B
0.59–0.60 / 0.91
0.60 / 0.91
0.57 / 0.92
Gemma
0.52–0.53 / 0.89–0.90
0.53 / 0.90
–
9B
0.57 / 0.91
0.56 / 0.89
0.54 / 0.91
Table S19 : Probes vs. ME as judges on AGIQA-3K (top-1 / MOS percentile; ME: range over five versions; Gemma: no vision probe).
KonIQ
LLVQA
Facts
Model
Method
own
held-out
LIVE-C
Δ
/40
0.8B
Base
0.506
0.642
0.694
–
38
+ME
–
0.800
0.781
+ 0.1
38
Q-Align, 2k
0.893
0.893
0.817
− 25.6
0
Q-Align, 7k
0.919
0.834
0.832
− 25.3
0
Q-ReAlign
0.769 / 0.910
0.530
0.793
− 22.5
0
Table S20 : ME vs. training the whole model. Own: each trained model’s Q-Align prompt and level-token score (Q-ReAlign at 512 / 1,024 resolution); held-out: good − poor under our 12 prompts; LIVE-C: own score for trained models. Q-Align (2k or 7k KonIQ labels; LoRA on 4B): mean of three seeds, held-out for seed 1 (0.8B, 2k: own 0.890, 0.895, 0.895 over seeds, so the equal own and held-out values are a coincidence of rounding). KL: Q-Align LoRA (2k) with a per-token KL penalty, weight βKL , to the frozen base model’s greedy answers to four generic questions on pool images; three seeds (held-out: mean of three), § one seed. LLVQA: change in LLVisionQA accuracy (points).
Figure S3 : What training gains and loses along the way. One Q-Align-style training run on the 2,000 pool labels (seed 1), evaluated at steps 5–250: KonIQ test SRCC under the model’s own scoring (left) and the change in LLVisionQA accuracy (right). Blue lines: ME on the same base model (solid 0.8B, dashed 4B). Full fine-tuning of 0.8B loses LLVisionQA accuracy within five steps, long before KonIQ SRCC reaches ME’s level; LoRA on 4B matches ME on both at steps 10 and 25 and loses accuracy from step 50.
Step
5
10
25
50
100
250
Validation
0.804
0.846
0.854
0.878
0.884
0.903
Test
0.795
0.843
0.856
0.877
0.882
0.906
LLVQA Δ
+ 2.5
+ 0.5
− 0.3
− 5.6
− 6.9
− 6.1
Table S21 : 4B LoRA along training (seed 1). Validation and test: KonIQ SRCC under the model’s own score; LLVQA: change in LLVisionQA accuracy (points). ME: 0.841 validation, 0.842 test.
Model
Facts /40
Neutral /10
Val. match /20
Degenerate
0.8B
38 → 38
9–10
16 → 10
0 → 0.8–1.3%
4B
40 → 40
10
18 → 11
0 → 0
Gemma
40 → 40
4–5
17 → 12–13
0 → 5.5–13%
9B
40 → 40
9–10
17 → 13–14
0 → 0.25%
Table S22 : Text probes and descriptions (base → three ME versions). Neutral: sentences whose top next token is unchanged. Val. match: sentences whose “good”/“poor” preference matches their valence (of 20; not the valence gap of Tab. 3 ). Degenerate: descriptions that repeat one word to the length limit.
Perplexity
SST-2 (%)
MMBench (%)
Qwen3.5-0.8B
Base
19.82
86.7
75.2
ME
20.57 / 20.23 / 20.11
83.3 / 83.0 / 84.2
75.6 / 75.5 / 75.3
MTC
20.99 / 20.56 / 20.15
84.3 / 85.7 / 86.2
75.3 / 75.1 / 75.3
Qwen3.5-4B
Base
10.80
93.8
86.2
Table S23 : General benchmarks (versions 1/2/3; Gemma: ranges). On 0.8B, all perplexity increases and SST-2 drops of ME have CIs excluding zero; MMBench CIs include or touch zero. On 4B and 9B, all perplexity increases have CIs excluding zero. For Gemma, plain-text perplexity was not stable to measure (base above 190 even with a BOS token) and is omitted.
KonIQ
SST-2 (%)
SST-2 gap
PPL
Val.
0.8B
base
0.642
86.4
1.75 [1.59, 1.93]
19.81
6.50
ME
0.800
83.2
− 0.62 [ − 0.98, − 0.26]
20.24
0.21
NME
–
80.4
1.52 [1.22, 1.83]
20.17
6.23
NME-G
0.803
81.0
0.29 [ − 0.02, 0.60]
20.08
1.44
4B
base
0.742
93.8
2.25 [2.09, 2.42]
10.79
7.39
ME
0.842
93.8
− 1.97 [ − 2.31, − 1.62]
10.87
3.47
Table S24 : Text-side constraints (means of five versions; base, ME and NME re-measured in the same run as NME-G; SST-2 gap with 95% CI over sentences; Val.: ten-review gap; NME’s KonIQ is in Tab. 3 ).
Figure S4 : Accurate on their own prompt, aware of its limit. Left: for each agent editing Qwen3.5-0.8B, its own held-out estimate, the locked KonIQ test under its prompt, and the mean over nine prompts no agent saw (fit prompts written after the agents finished, disjoint from the 12 held-out prompts); run3, whose structure ME keeps, is highlighted. Right: lines from the agents’ notes.
Figure S5 : One prompt or twelve. SRCC gain of Qwen3.5-0.8B over the base model on the KonIQ test set, under the agents’ prompt (shaded) and the 12 held-out prompts, sorted by ME’s gain; dashed lines: held-out means. SE and ME share neurons and closed form and differ only in the fitting objective.
Edit
KonIQ
KADID
LIVE-C
BID
Valence
QA
Base
–
0.643
0.548
0.693
0.662
6.5
–
ME 1
last layer, 1 neuron
0.801
0.593
0.781
0.763
0.2
− 0.2
mo1
layer 23, rank 1
0.747
0.590
0.758
0.730
7.0
− 0.5
mo2
vision block 9, 1 neuron
0.759
0.545
0.771
0.742
6.5
+ 0.7
mo3
layer 23, rank 1
0.728
0.590
0.744
0.721
6.5
− 0.4
Table S25 : Agents under a sentiment constraint (Qwen3.5-0.8B; SRCC, mean of 12 held-out prompts, real forward passes, so Base and ME 1 differ from the cached values of Tab. 1 by at most 0.002). Valence: “good” − “poor” log-probability gap between positive and negative sentences. QA: LLVisionQA accuracy change (points).
Run
Calls
Minutes
Output tok.
Input tok.
run1
21 / 50
30
53k
2.34M
run2
14 / 50
28
57k
2.38M
run3
16 / 50
28
53k
2.35M
run4
13 / 25
24
40k
1.83M
run5
10 / 10
26
38k
2.17M
lf1–lf3
45–50 / 50
96–102
57–68k
2.46–3.61M
Table S26: Agent runs. Calls: logged forward calls / budget. Tokens: output tokens and all input tokens processed (mostly cached context re-read on each step).
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.
Psychophysical experiments remain the most reliable approach for perceptual image quality assessment (IQA), yet their cost and limited scalability motivate automated alternatives. This paper investigates whether Vision-Language Models (VLMs) can assist in assessing perceived image appearance and quality. We introduce a psychophysics-inspired framework to probe VLM perceptual sensitivity through controlled pairwise image comparisons of contrast, colorfulness, and overall preference. Six VLMs (four proprietary and two open-weight models) are compared against psychophysical data. Results reveal strong attribute-dependent variability: Claude exhibits the highest internal consistency, whereas GPT achieves the strongest agreement for overall preference. Claude and Qwen show the strongest alignment for colorfulness, while Qwen performs best for contrast. However, no model consistently matches human perception across all attributes. High self-consistency does not necessarily imply perceptual validity, and VLM--human agreement generally improves when perceptual differences among renderings are more pronounced.
Imran Mehmood, Imad Ali Shah, Ming Ronnier Luo +1
School of Engineering, University of Galway, Ireland · Lero the Research Ireland Centre for Software, Ireland · State Key Laboratory of Extreme Photonics and Instrumentation, Zhejiang University, Hangzhou, China.
Vision-language models (VLMs) often fail to match a point or an object across two images, even when the answer can be decoded from their own hidden states. We trace this failure from the visual tokens to the answer letter, in Qwen3-VL (2B-32B), InternVL3.5-8B, and Gemma-3-12B, on synthetic shape matching, same-instance point correspondence (HPatches), and semantic correspondence (SPair-71k), and find two distinct failure points, a small set of attention heads that carries the comparison, and a repair that touches 0.02% of the parameters. Two failure points: exact copies are compared but not read out. An answer-position probe exceeds the calibrated output by 15 to 25 points in every family, for nameable and unnameable shapes alike, and steering along the probe's direction moves no more answers than random directions. Semantic correspondences are present in the visual tokens but never reach the answer. One pathway: redistributing the cross-image attention of 16-32 heads, with its mass preserved, removes 17-22 points of synthetic and 15-32 points of HPatches accuracy, against at most 7 for random or attention-matched heads. A lever that training finds on its own: training only these heads' query and key projections raises SPair-71k accuracy by 28.3 points (Qwen3-VL-8B) and 25.1 points (InternVL3.5-8B), beats both layer-matched and attention-matched heads, and transfers to HPatches and BLINK. An unconstrained all-layer adapter routes 21-51% of its gain through the attention of the same heads, against at most 7% for random heads, and does so by changing what they read (83-86% of the effect) rather than their weights. Every decision rule was frozen before its run, and every reported number is recomputed by independent verification scripts.
Vision-language models (VLMs) are often asked several questions about the same image in a single prompt (co-asking), yet existing evaluations report conflicting accuracy gains and hallucinations at the prompt level without tracking how individual answers change. By pairing each question's single-question and co-asked answers across 22 open-weight VLMs (1B-35B), we find that erratic net shifts reflect two opposing effects: on Densely Captioned Images (DCI), every model repairs 15.4%-29.8% of its single-question errors under co-asking, while the share of correct answers turned wrong falls from 30.8% on weaker models to 4.7% on stronger ones. Across all 14 models supporting custom attention masks, we show through attention readouts and layer-window interventions that earlier same-image questions read visual information into their question tokens in the early-to-middle layers during prefill, helping later questions concentrate image attention onto their queried targets. This visual help accumulates across the scene even across non-overlapping regions, whereas blocking earlier questions from attending to the image lowers both target localization and accuracy in all 14 models. While cross-question help is established during shared question reading, part of the co-asking damage enters later during answer decoding when the decoder attends to other question stems. We therefore introduce Read Together, Answer Apart, a training-free decoding mask that keeps shared prefill intact and isolates each answer from other stems, improving co-asking accuracy on 12 of 14 models on DCI (up to +4.71 percentage points, closing 70% of the pooled gap to asking alone) and on 10, 8, and 5 of the 14 models on WebUI, FinTabNet, and GQA (up to +11.4, +5.3, and +1.1 points).
Xinyi Guo, Kaile Huang, Xitong Ling +5
South China Normal University · Tsinghua University · Beijing University of Chemical Technology +1