Lost in Read-out: Locating the Image Quality Gap in Vision-Language Models with a Closed-Form Edit
Organizations: Nanyang Technological University, Singapore · Tianjin University, Tianjin, China · Hefei University of Technology, Hefei, China
Abstract
Vision-language models (VLMs) are widely used as zero-shot image quality assessors, yet their good-vs-poor answers agree only moderately with human opinion. We ask whether these models fail to see quality or fail to say it. Layer-wise probes on five VLMs point to the second: quality is linearly decodable from the vision encoder and still from the last language layer, far better than the answer reveals, and about a hundred labels recover most of it. The loss lies mainly in the final read-out, which partly reads something else: relative to human opinion, it over-rewards colorfulness and under-rewards sharpness on all five models. We test this by intervention: if the read-out is at fault, a closed-form edit of a single last-layer neuron, fitted to many prompts at once, should close the gap and no more. Both hold. Under prompts never used for fitting, the edit recovers 86--100% of what a ridge head on the last-layer state, fitted under the same prompts, finds beyond the answer on KonIQ. Where that gap is small, it gains little, too little for our transfer criterion on the two largest models; where the head itself is below the answer, it lowers the score; and it works only when written from the middle of the language model on. Unlike unconstrained fine-tuning, the edit leaves perception question answering intact. It does not carry over to yes-or-no or rating questions, and it reverses the good--poor contrast after sentiment-laden sentences. Code will be made available.
Figures & tables
| KonIQ [ 14 ] | KADID [ 24 ] | LIVE-C [ 11 ] | BID [ 5 ] | AGIQA-3K [ 19 ] | R4-CGQA [ 23 ] | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | Method | SRCC | PLCC | SRCC | PLCC | SRCC | PLCC | SRCC | PLCC | SRCC | PLCC | SRCC | PLCC |
| CLIP-IQA [ 35 ] (zero-shot) | 0.642 | 0.725 | 0.513 | 0.535 | 0.702 | 0.720 | 0.590 | 0.567 | 0.649 | 0.702 | 0.214 | 0.251 | |
| LIQE [ 46 ] (trained on KonIQ) | 0.897 | 0.905 | 0.722 | 0.722 | 0.864 | 0.866 | 0.705 | 0.695 | 0.725 | 0.792 | 0.128 | 0.216 | |
| CLIP + ridge (2,000 labels) | 0.794 | 0.825 | 0.640 | 0.648 | 0.753 | 0.790 | 0.779 | 0.810 | 0.853 † | 0.906 | 0.469 ‡ | 0.539 | |
| Qwen3.5-0.8B | Base | 0.642 | 0.727 | 0.546 | 0.591 | 0.694 | 0.761 | 0.662 | 0.715 | 0.687 | 0.772 | 0.109 | 0.172 |
| +MTC | 0.773 | 0.831 | 0.578 | 0.612 | 0.774 | 0.825 | 0.755 | 0.787 | 0.800 † | 0.858 | 0.306 ‡ | 0.343 | |
| Variant | KonIQ | KADID | LIVE-C | BID |
|---|---|---|---|---|
| Base | 0.642 | 0.546 | 0.694 | 0.662 |
| ME (ours) | 0.800 | 0.593 | 0.781 | 0.763 |
| Unconstrained gradient tuning | 0.792 | 0.562 | 0.777 | 0.759 |
| Rescale 300 neurons, no fit | 0.685 | 0.586 | 0.723 | 0.700 |
| Ridge head read outside the model | ||||
| on layer-16 state | 0.772 | 0.563 | 0.765 | 0.742 |
| KonIQ | LLVQA | Valence | SST val. | SST-2 | ||
|---|---|---|---|---|---|---|
| 0.8B | base | 0.642 | – | 6.5 | 1.75 | 86.7 |
| ME | 0.800 | [ 0.2, 0.3] | [ 0.1, 0.4] | [ 0.77, 0.56] | [83.0, 84.2] | |
| NME | [0.800, 0.801] | [ 0.5, 0.3] | [6.2, 6.3] | [1.49, 1.55] | [80.2, 80.8] | |
| 4B | base | 0.742 | – | 7.4 | 2.25 | 93.8 |
| ME | 0.842 | [ 0.3, 0.0] | [3.3, 3.6] | [ 2.11, 1.83] | [93.8, 93.9] | |
| NME | [0.842, 0.844] | [ 0.3, 0.2] | [8.2, 8.3] | [0.44, 0.58] | [93.8, 94.0] |
| Question | Answer prefix | |
| Fit prompts (the only prompts any edit sees) | ||
| F1 a | Rate the quality of the image. | The quality of the image is |
| F2 | How would you judge the technical quality of this photo? | The technical quality of this photo is |
| F3 | Assess the perceptual quality of this image, considering blur, noise, exposure and compression. | The perceptual quality of this image is |
| F4 b | Describe the visual quality of this image. | The visual quality of this image is |
| F5 b | What is the image quality like? | The image quality is |
| KonIQ | KADID | LIVE Challenge | BID | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | Pr. | Blk | Ans. | best | last | CI | Ans. | best | last | CI | Ans. | best | last | CI | Ans. | best | last | CI |
| 0.8B | F1 | 5 | 0.653 | 0.903 | 0.840 | [0.23, 0.28] ✓ | 0.551 | 0.563 | 0.608 | [ 0.03, 0.06] | 0.697 | 0.796 | 0.794 | [0.07, 0.13] ✓ | 0.693 | 0.843 | 0.779 | [0.11, 0.19] ✓ |
| 0.8B | H1 | 5 | 0.617 | 0.903 | 0.849 | [0.26, 0.31] ✓ | 0.588 | 0.563 | 0.570 | [ 0.07, 0.02] | 0.702 | 0.796 | 0.802 | [0.06, 0.13] ✓ | 0.662 | 0.843 | 0.783 | [0.14, 0.23] ✓ |
| 4B | F1 | 10 | 0.795 | 0.910 | 0.888 | [0.10, 0.13] ✓ | 0.713 | 0.554 | 0.667 | [ 0.19, 0.13] | 0.793 | 0.825 | 0.838 | [0.01, 0.05] ✓ | 0.810 | 0.856 | 0.857 | [0.02, 0.08] ✓ |
| 4B | H1 | 10 | 0.735 | 0.910 | 0.882 | [0.15, 0.20] ✓ | 0.732 | 0.554 | 0.698 | [ 0.21, 0.15] | 0.795 | 0.825 | 0.837 | [0.01, 0.05] | 0.802 | 0.856 | 0.861 | [0.02, 0.09] ✓ |
| 9B | F1 | 12 | 0.728 | 0.917 | 0.891 | [0.17, 0.21] ✓ | 0.674 | 0.588 | 0.693 | [ 0.12, 0.05] | 0.806 | 0.840 | 0.851 | [0.01, 0.05] ✓ | 0.796 | 0.867 | 0.858 | [0.04, 0.10] ✓ |
| Qwen3.5-0.8B | Qwen3.5-4B | ||||||
|---|---|---|---|---|---|---|---|
| vision | last | ME | rand. | vision | last | ME | |
| 10 | 0.335 | 0.115 | 0.409 | 0.277 | 0.347 | 0.628 | 0.677 |
| 20 | 0.596 | 0.638 | 0.678 | 0.181 | 0.562 | 0.750 | 0.743 |
| 50 | 0.743 | 0.734 | 0.712 | 0.356 | 0.749 | 0.786 | 0.794 |
| 100 | 0.797 | 0.752 | 0.739 | 0.367 | 0.801 | 0.819 | 0.814 |
| 200 | 0.845 | 0.792 | 0.761 | 0.438 | 0.850 | 0.839 | 0.819 |
| Qwen3.5-0.8B, layers 1–24 | |||||||
| 0.579 | 0.739 | 0.601 | 0.733 | 0.708 | 0.710 | 0.739 | 0.775 |
| 0.733 | 0.780 | 0.732 | 0.782 | 0.789 | 0.785 | 0.786 | 0.809 ∗ |
| 0.810 ∗ | 0.808 ∗ | 0.795 | 0.800 ∗ | 0.813 ∗ | 0.815 ∗ | 0.807 ∗ | 0.815 |
| Qwen3.5-4B, layers 2, 4, …, 32 | |||||||
| 0.702 | 0.711 | 0.682 | 0.757 | 0.809 | 0.807 | 0.847 ∗ | 0.847 ∗ |
| 0.870 ∗ | 0.877 ∗ | 0.867 ∗ | 0.868 ∗ | 0.860 ∗ | 0.864 ∗ | 0.862 ∗ | 0.859 |
| Model | cos | halves | KonIQ | LIVE-C | ||
|---|---|---|---|---|---|---|
| 0.8B | 1,024 | 0.031 | 0.046 | 0.77 | 0.75 / 0.51 | 0.84 / 0.67 |
| 4B | 2,560 | 0.020 | 0.053 | 0.77 | 0.81 / 0.54 | 0.90 / 0.75 |
| Gemma | 1,536 | 0.026 | 0.026 | 0.79 | 0.67 / 0.44 | 0.75 / 0.59 |
| 9B | 4,096 | 0.016 | 0.045 | 0.77 | 0.80 / 0.54 | 0.91 / 0.75 |
| 27B | 5,120 | 0.014 | 0.030 | 0.59 | 0.86 / 0.61 | 0.91 / 0.74 |
| Brightness | Contrast | Colorful. | Sharpness | Aesthetic | |
| KonIQ | |||||
| 0.8B | 0.25 / 0.27 / 0.27 | 0.08 / 0.12 / 0.14 | 0.14 / 0.05 / 0.03 | 0.22 / 0.13 / 0.16 | 0.16 / 0.18 / 0.13 |
| 4B | 0.05 / 0.21 / 0.19 | 0.09 / 0.18 / 0.18 | 0.12 / 0.07 / 0.08 | 0.00 / 0.20 / 0.21 | 0.36 / 0.11 / 0.12 |
| Gemma | 0.09 / 0.20 / 0.19 | 0.06 / 0.21 / 0.21 | 0.17 / 0.03 / 0.01 | 0.15 / 0.26 / 0.27 | 0.29 / 0.22 / 0.20 |
| 9B | 0.06 / 0.19 / 0.18 | 0.03 / 0.10 / 0.12 | 0.14 / 0.05 / 0.03 | 0.12 / 0.13 / 0.20 | 0.34 / 0.18 / 0.13 |
| 27B | 0.11 / 0.20 / 0.18 | 0.10 / 0.16 / 0.16 | 0.11 / 0.05 / 0.04 | 0.04 / 0.17 / 0.18 | 0.31 / 0.16 / 0.14 |
| Model | Dataset | Base | ME | ME Base | CI | Prompts | Per-prompt gain | MTC | MTC ME | Sig. |
|---|---|---|---|---|---|---|---|---|---|---|
| 0.8B | KonIQ | 0.642 | 0.800 | 0.158 | [ 0.132, 0.184] | 12 | 0.112 to 0.204 | 0.773 | 0.035 to 0.024 | 5 / 0 |
| KADID | 0.546 | 0.593 | 0.047 | [ 0.004, 0.087] | 10 | 0.047 to 0.115 | 0.578 | 0.029 to 0.000 | 1 / 0 | |
| LIVE-C | 0.694 | 0.781 | 0.087 | [ 0.058, 0.117] | 12 | 0.035 to 0.169 | 0.774 | 0.011 to 0.004 | 0 / 0 | |
| BID | 0.662 | 0.763 | 0.101 | [ 0.069, 0.136] | 12 | 0.057 to 0.143 | 0.755 | 0.011 to 0.005 | 0 / 0 | |
| 4B | KonIQ | 0.742 | 0.842 | 0.100 | [ 0.064, 0.131] | 11 | 0.018 to 0.190 | 0.822 | 0.030 to 0.011 | 5 / 0 |
| KADID | 0.698 | 0.640 | 0.058 | [ 0.107, 0.015] | 1 | 0.206 to 0.064 | 0.663 | 0.007 to 0.047 | 0 / 3 |
| Model | KonIQ | KADID | LIVE-C | BID |
|---|---|---|---|---|
| 0.8B | 0.158 / 0.158 | 0.022 / 0.047 | 0.091 / 0.087 | 0.098 / 0.101 |
| 4B | 0.104 / 0.100 | 0.067 / 0.058 | 0.034 / 0.031 | 0.041 / 0.040 |
| Gemma | 0.226 / 0.222 | 0.073 / 0.076 | 0.147 / 0.152 | 0.139 / 0.143 |
| 9B | 0.142 / 0.126 | 0.023 / 0.054 | 0.038 / 0.022 | 0.055 / 0.042 |
| 27B | 0.087 / 0.075 | 0.096 / 0.109 | 0.027 / 0.011 | 0.038 / 0.027 |
| Model | Selected | CV gain | Test SRCC |
|---|---|---|---|
| 0.8B | 16, 16, 16, 16, 16 | 0.0025–0.0038 | 0.0014 to 0.0025 |
| 4B | 4, 4, 4, 2, 4 | 0.0000–0.0005 | 0.0010 to 0.0016 |
| Gemma | 8, 4, 4, 4, 4 | 0.0002–0.0013 | 0.0001 to 0.0021 |
| 9B | 4, 4, 4, 4, 8 | 0.0005–0.0016 | 0.0004 to 0.0043 |
| Gate variation | Neurons | ME random (KonIQ) |
|---|---|---|
| 5 | 0.008 to 0.003 | |
| 0.10–0.50 | 18 | 0.005 to 0.052 |
| 17 | 0.030 to 0.226 |
| Format | 0.8B | 4B |
|---|---|---|
| “high” “low” | 0.607 0.660–0.663 ∗ | 0.706 0.739–0.740 ∗ |
| “excellent” “bad” | 0.535 0.579–0.582 ∗ | 0.627 0.659–0.661 ∗ |
| “sharp” “blurry” | 0.674 0.726–0.731 ∗ | 0.700 0.708 ∗ |
| Q-Align five levels | 0.469 0.489–0.513 | 0.691 0.788–0.792 ∗ |
| Yes/no question | 0.679 0.676–0.679 | 0.664 0.664 |
| 1–5 rating | 0.621 0.627–0.628 ∗ | 0.731 0.726–0.729 ∘ |
| Two images, one prompt | Single image | |||
|---|---|---|---|---|
| Model | Base | ME | Base | ME |
| 0.8B | 71.5% | 71.7–73.0% | 84.8% | 93.8–94.2% |
| 4B | 87.6% | 87.7–88.0% | 88.4% | 95.4–95.6% |
| Data | Model | Base | ME | NME | Crit. |
|---|---|---|---|---|---|
| SPAQ | 0.8B | 0.833 | 0.863–0.864 | 0.863–0.864 | 5/5 |
| 4B | 0.882 | 0.881–0.882 | 0.884–0.885 | 0/5 | |
| Gemma | 0.781 | 0.850–0.852 | 0.849–0.851 | 5/5 | |
| 9B | 0.866 | 0.865–0.867 | 0.864–0.866 | 0/5 | |
| FLIVE | 0.8B | 0.428 | 0.472–0.477 | 0.470–0.471 | 5/5 |
| 4B | 0.394 | 0.480–0.483 | 0.480–0.483 | 5/5 |
| Model | Method | KonIQ | KADID | LIVE-C | BID |
|---|---|---|---|---|---|
| 0.8B | Base | 0.589 | 0.466 | 0.617 | 0.642 |
| Q-Debias | 0.534 | 0.549 | 0.535 | 0.650 | |
| ME | 0.756–0.763 | 0.554–0.562 | 0.765–0.768 | 0.746–0.753 | |
| 4B | Base | 0.792 | 0.698 | 0.788 | 0.803 |
| Q-Debias | 0.646 | 0.679 | 0.780 | 0.788 | |
| ME | 0.818–0.821 | 0.498–0.523 | 0.789–0.792 | 0.837–0.840 |
| Model | Base | ME | ME Base | CI | Prompts | Per-prompt gain | MTC | MTC low | ME MTC |
|---|---|---|---|---|---|---|---|---|---|
| 0.8B | 0.687 | 0.825 | 0.137 | [0.097, 0.178] | 12 | 0.069 to 0.265 | 0.800 | 0.073 | 5/5 |
| 4B | 0.750 | 0.848 | 0.098 | [0.057, 0.139] | 11 | 0.011 to 0.225 | 0.836 | 0.049 | 5/5 |
| Gemma | 0.623 | 0.818 | 0.195 | [0.123, 0.275] | 12 | 0.063 to 0.472 | 0.804 | 0.106 | 4/5 |
| 9B | 0.718 | 0.842 | 0.124 | [0.064, 0.181] | 11 | 0.057 to 0.276 | 0.837 | 0.072 | 0/5 |
| Base | KonIQ ME | Refit on graphics | |||
|---|---|---|---|---|---|
| Model | SRCC / PLCC | SRCC | SRCC / PLCC | (CI low) | Val |
| 0.8B | 0.109 / 0.172 | 0.007 to 0.006 | 0.36 / 0.38–0.39 | 0.25 (0.11–0.12) | 0.06 0.41–0.42 |
| 4B | 0.184 / 0.243 | 0.078 to 0.070 | 0.41–0.42 / 0.47–0.48 | 0.23 (0.09–0.10) | 0.17 0.51 |
| Gemma | 0.153 / 0.180 | 0.028 to 0.044 | 0.35–0.39 / 0.40–0.43 | 0.20–0.24 (0.08–0.11) | 0.13 0.41–0.44 |
| 9B | 0.148 / 0.278 | 0.012 to 0.020 | 0.38 / 0.46–0.47 | 0.23–0.24 (0.08) | 0.07 0.47–0.48 |
| Own @512 | Own @1,024 | Held-out @1,024 | Refit | |
|---|---|---|---|---|
| Mini (0.8B) | 0.126 / 0.106 | 0.232 / 0.231 | 0.079 | 0.360 |
| Lite (4B) | 0.170 / 0.127 | 0.293 / 0.321 | 0.102 | 0.416 |
| Pro (9B) | 0.270 / 0.289 | 0.299 / 0.329 | 0.013 | 0.382 |
| Top-1 | MOS percentile | |||||
|---|---|---|---|---|---|---|
| Model | Base | MTC | ME | Base | MTC | ME |
| 0.8B | 0.24 | 0.43 | 0.51 | 0.77 | 0.88 | 0.90 |
| 4B | 0.41 | 0.54 | 0.60 | 0.83 | 0.90 | 0.91 |
| Gemma | 0.18 | 0.51 | 0.53 | 0.73 | 0.89 | 0.90 |
| 9B | 0.31 | 0.51 | 0.57 | 0.78 | 0.90 | 0.91 |
| Model | ME | Last-layer probe | Vision probe |
|---|---|---|---|
| 0.8B | 0.50–0.52 / 0.90 | 0.55 / 0.90 | 0.52 / 0.89 |
| 4B | 0.59–0.60 / 0.91 | 0.60 / 0.91 | 0.57 / 0.92 |
| Gemma | 0.52–0.53 / 0.89–0.90 | 0.53 / 0.90 | – |
| 9B | 0.57 / 0.91 | 0.56 / 0.89 | 0.54 / 0.91 |
| KonIQ | LLVQA | Facts | ||||
| Model | Method | own | held-out | LIVE-C | /40 | |
| 0.8B | Base | 0.506 | 0.642 | 0.694 | – | 38 |
| +ME | – | 0.800 | 0.781 | 0.1 | 38 | |
| Q-Align, 2k | 0.893 | 0.893 | 0.817 | 25.6 | 0 | |
| Q-Align, 7k | 0.919 | 0.834 | 0.832 | 25.3 | 0 | |
| Q-ReAlign | 0.769 / 0.910 | 0.530 | 0.793 | 22.5 | 0 | |
| Step | 5 | 10 | 25 | 50 | 100 | 250 |
|---|---|---|---|---|---|---|
| Validation | 0.804 | 0.846 | 0.854 | 0.878 | 0.884 | 0.903 |
| Test | 0.795 | 0.843 | 0.856 | 0.877 | 0.882 | 0.906 |
| LLVQA | 2.5 | 0.5 | 0.3 | 5.6 | 6.9 | 6.1 |
| Model | Facts /40 | Neutral /10 | Val. match /20 | Degenerate |
|---|---|---|---|---|
| 0.8B | 38 38 | 9–10 | 16 10 | 0 0.8–1.3% |
| 4B | 40 40 | 10 | 18 11 | 0 0 |
| Gemma | 40 40 | 4–5 | 17 12–13 | 0 5.5–13% |
| 9B | 40 40 | 9–10 | 17 13–14 | 0 0.25% |
| Perplexity | SST-2 (%) | MMBench (%) | |
| Qwen3.5-0.8B | |||
| Base | 19.82 | 86.7 | 75.2 |
| ME | 20.57 / 20.23 / 20.11 | 83.3 / 83.0 / 84.2 | 75.6 / 75.5 / 75.3 |
| MTC | 20.99 / 20.56 / 20.15 | 84.3 / 85.7 / 86.2 | 75.3 / 75.1 / 75.3 |
| Qwen3.5-4B | |||
| Base | 10.80 | 93.8 | 86.2 |
| KonIQ | SST-2 (%) | SST-2 gap | PPL | Val. | ||
|---|---|---|---|---|---|---|
| 0.8B | base | 0.642 | 86.4 | 1.75 [1.59, 1.93] | 19.81 | 6.50 |
| ME | 0.800 | 83.2 | 0.62 [ 0.98, 0.26] | 20.24 | 0.21 | |
| NME | – | 80.4 | 1.52 [1.22, 1.83] | 20.17 | 6.23 | |
| NME-G | 0.803 | 81.0 | 0.29 [ 0.02, 0.60] | 20.08 | 1.44 | |
| 4B | base | 0.742 | 93.8 | 2.25 [2.09, 2.42] | 10.79 | 7.39 |
| ME | 0.842 | 93.8 | 1.97 [ 2.31, 1.62] | 10.87 | 3.47 |
| Edit | KonIQ | KADID | LIVE-C | BID | Valence | QA | |
|---|---|---|---|---|---|---|---|
| Base | – | 0.643 | 0.548 | 0.693 | 0.662 | 6.5 | – |
| ME 1 | last layer, 1 neuron | 0.801 | 0.593 | 0.781 | 0.763 | 0.2 | 0.2 |
| mo1 | layer 23, rank 1 | 0.747 | 0.590 | 0.758 | 0.730 | 7.0 | 0.5 |
| mo2 | vision block 9, 1 neuron | 0.759 | 0.545 | 0.771 | 0.742 | 6.5 | 0.7 |
| mo3 | layer 23, rank 1 | 0.728 | 0.590 | 0.744 | 0.721 | 6.5 | 0.4 |
| Run | Calls | Minutes | Output tok. | Input tok. |
|---|---|---|---|---|
| run1 | 21 / 50 | 30 | 53k | 2.34M |
| run2 | 14 / 50 | 28 | 57k | 2.38M |
| run3 | 16 / 50 | 28 | 53k | 2.35M |
| run4 | 13 / 25 | 24 | 40k | 1.83M |
| run5 | 10 / 10 | 26 | 38k | 2.17M |
| lf1–lf3 | 45–50 / 50 | 96–102 | 57–68k | 2.46–3.61M |
VLM Evaluation
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.
See all 1,060 papers