OmniScore-Bench: Evaluating Vision-Language Models on Musical Score Reading
Organizations: Central Conservatory of Music · Tsinghua University
Abstract
Vision-language models (VLMs) can answer questions about musical scores, but their ability to recognise notation, recover its structure and interpret its musical meaning requires separate evaluation. Existing optical music recognition (OMR) resources establish detection and transcription measurements; a transcription score alone leaves the strengths and bottlenecks of a VLM score reader unresolved. We introduce OmniScore-Bench, an evaluation of OMR capabilities in VLMs through 3,000 questions spanning six aspects and 24 task subtypes. Each question specifies a visible scope, reference annotations and an answer schema, enabling automatic evaluation from symbol localisation to musical consistency and evidence-linked question answering. Our assessment of twelve VLMs reveals operation-specific capability profiles and different transcription leaders across excerpt, single-page and multipage tasks. Controlled analyses examine the roles of score images, auxiliary givens and additional page context. On the tested subset, image removal exposes visual dependence in pitch and repeat interpretation while measuring how much performance remains with public context alone. Separate format and content analyses distinguish answer-format violations from musical errors. OmniScore-Bench supports diagnosing which score-reading operations VLMs perform reliably and which require improvement.
Figures & tables
| Dataset / benchmark | T1 | T2 | T3 | T4 | T5 | T6 |
|---|---|---|---|---|---|---|
| Localisation | Structure | Relations | Transcription | Consistency | Score QA | |
| DeepScoresV2 [ tuggener2021deepscoresv2 ] | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ |
| MUSCIMA++ [ hajic2017muscima ] | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ |
| PrIMuS [ calvo2018primus ] | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ |
| GrandStaff [ rios2023pianoform ] | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ |
| SMB [ martinez2025smb ] | ✗ | ✓ | ✗ | ✓ | ✗ | ✗ |
| T1: Symbols | T2: Structure | T3: Relations | T5: Consistency | T6: Score QA | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | 1.1 | 1.2 | 1.3 | 1.4 | 2.1 | 2.2 | 2.3 | 2.4 | 3.1 | 3.2 | 3.3 | 3.4 | 5.1 | 5.2 | 5.3 | 5.4 | 6.1 | 6.2 | 6.3 | 6.4 |
| DeepSeek V4.1 Flash | 0.4 | 1.7 | 0.3 | 0.5 | 36.0 | 41.4 | 54.1 | 32.5 | 27.0 | 26.6 | 27.5 | 18.4 | 28.4 | 5.6 | 0.0 | 24.5 | 93.1 | 59.0 | 20.7 | 47.1 |
| Qwen 3.7 Plus | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.3 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 42.5 | 9.8 | 0.6 | 0.7 |
| Qwen 3.8 Max | 7.5 | 37.0 | 4.1 | 41.3 | 86.5 | 84.4 | 46.9 | 91.1 | 65.4 | 52.5 | 70.2 | 52.2 | 58.3 | 37.1 | 8.2 | 62.3 | 97.3 | 95.1 | 54.5 | 62.0 |
| Kimi K3 | 20.9 | 29.9 | 8.2 | 48.8 | 91.7 | 86.9 | 66.0 | 88.3 | 52.4 | 47.2 | 60.9 | 49.5 | 57.2 | 23.8 | 13.1 | 58.5 | 97.7 | 39.3 | 43.7 | 55.4 |
| Kimi K3.1 Pro | 36.4 | 55.0 | 21.5 | 60.1 | 94.0 | 91.5 | 70.3 | 89.5 | 57.3 | 66.7 | 70.6 | 61.4 | 64.8 | 53.2 | 21.3 | 64.0 | 98.3 | 57.4 | 58.1 | 66.5 |
| Subtype | Model | A | B | [95% CI] | ||
| Specific assistance: A = full; B = one field removed. Exact/joint success . | ||||||
| T5.2 | DeepSeek V4.1 Flash | 20 | 15.00 | 5.00 | 13/11 | |
| T5.2 | GPT-6.1 Sol | 20 | 95.00 | 95.00 | 20/20 | |
| T6.3 | DeepSeek V4.1 Flash | 13 | 15.38 | 15.38 | 11/6 | |
| T6.3 | GPT-6.1 Sol | 13 | 100.00 | 100.00 | 13/13 | |
| Fixed-target context: A = target only; B = target plus source pages. Corpus NED . | ||||||
| GPT-6.1 Sol | DeepSeek Flash | ||||
|---|---|---|---|---|---|
| Subtype | Image | Removed | Image | Removed | |
| T2.3 | 30 | 90.0 | 53.3 | 66.7 | 30.0 |
| T5.1 | 30 | 76.7 | 3.3 | 13.3 | 3.3 |
| T5.2 | 30 | 83.3 | 0.0 | 3.3 | 6.7 |
| T5.3 | 30 | 59.6 | 0.0 | 0.0 | 0.0 |
| T5.4 | 30 | 92.0 | 0.0 | 42.0 | 10.0 |
| Model | T4.1 | T4.2 | T4.3 | T4.4 |
|---|---|---|---|---|
| DeepSeek V4.1 Flash | 89.6 | 86.1 | 86.4 | 89.0 |
| Qwen 3.7 Plus | 90.8 | 89.9 | 91.8 | 96.1 |
| Qwen 3.8 Max | 81.1 | 81.1 | 76.1 | 76.2 |
| Kimi K3 | 71.7 | 69.3 | 70.9 | 74.5 |
| Kimi K3.1 Pro | 64.4 | 64.1 | 62.8 | 67.0 |
| GLM 5.3 Flash | 89.6 | 86.9 | 83.7 | 93.1 |
| Model | Strict | Outer fence only | ||
|---|---|---|---|---|
| Scored | Failure | Scored | Failure | |
| DeepSeek V4.1 Flash | 2947 | 53 | 2947 | 53 |
| Qwen 3.7 Plus | 341 | 2658 | 2824 | 173 |
| Qwen 3.8 Max | 2953 | 47 | 2953 | 47 |
| Kimi K3 | 2962 | 37 | 2962 | 37 |
| Kimi K3.1 Pro | 2966 | 32 | 2966 | 32 |
| Domain | Collection | Questions |
|---|---|---|
| Rendered | Muse native corpus | 952 |
| Rendered | Additional Muse pairs | 4 |
| Rendered | OpenScore Lieder | 44 |
| Handwritten | CVC-MUSCIMA source images | 680 |
| Handwritten | MUSCIMA++ annotations | 320 |
| Printed scan | Schubert Winterreise / Peters scans | 543 |
| Type | Target | Scope and public context | Main measurement |
|---|---|---|---|
| T1.1 | Noteheads and rests | Classes and boxes in the declared image region | Localisation F1 |
| T1.2 | Clefs and accidentals | Class-aware symbol boxes; applicable labels are fixed | Localisation F1 |
| T1.3 | Stems, flags, beams and dots | Thin rhythmic marks within the visible scope | Localisation F1 |
| T1.4 | Dynamics, articulations and ornaments | Supported expression labels and their regions | Localisation F1 |
| T2.1 | Systems and staves | Page structure and normalised regions | Region F1 |
| T2.2 | Measures | Physical measure regions in the input extent | Region F1 |
| Model | T1.1 | T1.3 | ||
|---|---|---|---|---|
| Strict | 1 px | Strict | 1 px | |
| DeepSeek V4.1 Flash | 0.4 | 0.6 | 0.3 | 0.4 |
| Qwen 3.7 Plus | 0.0 | 0.0 | 0.0 | 0.0 |
| Qwen 3.8 Max | 7.5 | 11.5 | 4.1 | 5.7 |
| Kimi K3 | 20.9 | 26.8 | 8.2 | 10.1 |
| Kimi K3.1 Pro | 36.4 | 45.2 | 21.5 | 27.1 |
| Opus / Astra NED | Difference | |||
|---|---|---|---|---|
| Type | Groups | Item mean | Group mean | 95% interval |
| T4.1 | 52 | 42.0 / 23.1 | 31.8 / 18.6 | [11.5, 28.4] |
| T4.2 | 49 | 49.4 / 25.5 | 40.0 / 18.1 | [18.6, 45.0] |
| T4.3 | 69 | 41.4 / 23.6 | 32.4 / 23.4 | [8.8, 23.6] |
| T4.4 | 122 | 33.1 / 50.1 | 33.4 / 49.7 | [-18.9, -2.4] |
| Length | Ref. symbols | Pages | Opus | Astra | |
|---|---|---|---|---|---|
| Shorter | 42 | 404–2686 | 2–7 | 16.9 | 25.0 |
| Middle | 42 | 2724–4716 | 2–12 | 27.9 | 45.7 |
| Longer | 41 | 4720–54443 | 2–84 | 53.6 | 58.8 |
| Model | Stop | Length | Unknown | Failures |
|---|---|---|---|---|
| Opus | 108 (0) | 16 (16) | 1 (1) | 17 |
| Astra | 118 (31) | 7 (7) | 0 (0) | 38 |
| Model | T4.1 | T4.2 | T4.3 | T4.4 | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| All | Valid | All | Valid | All | Valid | All | Valid | |||||
| Claude Fable 5.1 | 64.4 | 124 | 64.0 | 69.2 | 121 | 68.3 | 59.3 | 122 | 58.7 | 77.1 | 60 | 53.1 |
| Claude Opus 5.5 | 46.3 | 125 | 46.3 | 55.1 | 123 | 54.2 | 43.5 | 124 | 43.0 | 39.9 | 108 | 31.0 |
| DeepSeek V4.1 Flash | 89.6 | 121 | 89.4 | 86.1 | 122 | 85.8 | 86.4 | 116 | 86.2 | 89.0 | 117 | 88.6 |
| Gemini 3.8 Flash | 84.4 | 106 | 83.0 | 79.8 | 110 | 78.3 | 85.7 | 80 | 77.1 | 98.0 | 35 | 94.5 |
| Gemini 4 Argon | 46.2 | 121 | 44.9 | 29.0 | 124 | 28.1 | 52.0 | 110 | 46.2 | 88.3 | 36 | 52.7 |
| Type | Opus | Astra | 95% CI | |
|---|---|---|---|---|
| T4.1 | 125/52 | 46.3 | 23.5 | [-28.4, -12.1] |
| T4.2 | 123/49 | 54.2 | 22.0 | [-42.8, -18.6] |
| T4.3 | 118/66 | 42.0 | 22.0 | [-25.9, -13.3] |
| T4.4 | 80/79 | 33.3 | 34.6 | [-3.0, 5.7] |
| Task | Model | NED | CER | SER | LER | TEDn 6k |
|---|---|---|---|---|---|---|
| T4.1 | DeepSeek V4.1 Flash | 89.6 | 66.1 | 73.8 | 46.3 | — (121/125) |
| Qwen 3.7 Plus | 90.8 | 72.2 | 77.9 | 57.5 | — (91/125) | |
| Qwen 3.8 Max | 81.1 | 59.6 | 65.8 | 43.7 | 42.6 | |
| Kimi K3 | 71.7 | 57.8 | 61.9 | 45.3 | 43.0 | |
| Kimi K3.1 Pro | 64.4 | 53.2 | 55.7 | 43.9 | 38.7 | |
| GLM 5.3 Flash | 89.6 | 73.7 | 78.1 | 53.0 | — (107/125) |
| Task | Model | NED | CER | SER | LER | TEDn 6k |
|---|---|---|---|---|---|---|
| T4.1 | DeepSeek V4.1 Flash | 89.6 | 66.1 | 73.8 | 46.3 | — (121/125) |
| Qwen 3.7 Plus | 90.1 | — (124/125) | — (124/125) | — (124/125) | — (116/125) | |
| Qwen 3.8 Max | 81.1 | 59.6 | 65.8 | 43.7 | 42.6 | |
| Kimi K3 | 71.7 | 57.8 | 61.9 | 45.3 | 43.0 | |
| Kimi K3.1 Pro | 64.4 | 53.2 | 55.7 | 43.9 | 38.7 | |
| GLM 5.3 Flash | 89.5 | 73.2 | 77.6 | 52.6 | — (108/125) |
| Model | T1.1 | T1.2 | T1.3 | T1.4 | T2.1 | T2.2 | T2.3 | T2.4 | T3.1 | T3.2 | T3.3 | T3.4 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| DeepSeek V4.1 Flash | 0.4 | 1.7 | 0.3 | 0.5 | 36.0 | 41.4 | 54.1 | 32.5 | 27.0 | 26.6 | 27.5 | 18.4 |
| Qwen 3.7 Plus | 0.1 | 0.6 | 0.8 | 2.2 | 30.7 | 35.1 | 12.7 | 37.5 | 21.7 | 14.7 | 37.8 | 10.6 |
| Qwen 3.8 Max | 7.5 | 37.0 | 4.1 | 41.3 | 86.5 | 84.4 | 46.9 | 91.1 | 65.4 | 52.5 | 70.2 | 52.2 |
| Kimi K3 | 20.9 | 29.9 | 8.2 | 48.8 | 91.7 | 86.9 | 66.0 | 88.3 | 52.4 | 47.2 | 60.9 | 49.5 |
| Kimi K3.1 Pro | 36.4 | 55.0 | 21.5 | 60.1 | 94.0 | 91.5 | 70.3 | 89.5 | 57.3 | 66.7 | 70.6 | 61.4 |
| GLM 5.3 Flash | 0.4 | 11.7 | 0.4 | 2.5 | 65.9 | 60.9 | 19.1 | 64.2 | 39.6 | 21.2 | 52.5 | 24.1 |
| T1.4 ( ) | T3.1 ( ) | T3.4 ( ) | ||||
|---|---|---|---|---|---|---|
| Model | Strict | Outer fence only | Strict | Outer fence only | Strict | Outer fence only |
| DeepSeek V4.1 Flash | 11.44 (0) | 11.44 (0) | 1.12 (0) | 1.12 (0) | 0.64 (0) | 0.64 (0) |
| Qwen 3.7 Plus | 0.00 (12) | 5.88 (0) | 0.00 (29) | 1.14 (2) | 0.00 (22) | 0.45 (3) |
| Qwen 3.8 Max | 4.38 (0) | 4.38 (0) | 0.39 (0) | 0.39 (0) | 0.05 (0) | 0.05 (0) |
| Kimi K3 | 1.56 (0) | 1.56 (0) | 0.54 (0) | 0.54 (0) | 0.14 (0) | 0.14 (0) |
| Kimi K3.1 Pro | 0.88 (0) | 0.88 (0) | 0.29 (0) | 0.29 (0) | 0.05 (0) | 0.05 (0) |
| Model | T5.1 | T5.2 | T5.3 | T5.4 | T6.1 | T6.2 | T6.3 | T6.4 |
|---|---|---|---|---|---|---|---|---|
| DeepSeek V4.1 Flash | 28.4 | 5.6 | 0.0 | 24.5 | 93.1 | 59.0 | 20.7 | 47.1 |
| Qwen 3.7 Plus | 26.6 | 11.1 | 0.7 | 18.3 | 97.5 | 45.9 | 22.3 | 36.7 |
| Qwen 3.8 Max | 58.3 | 37.1 | 8.2 | 62.3 | 97.3 | 95.1 | 54.5 | 62.0 |
| Kimi K3 | 57.2 | 23.8 | 13.1 | 58.5 | 97.7 | 39.3 | 43.7 | 55.4 |
| Kimi K3.1 Pro | 64.8 | 53.2 | 21.3 | 64.0 | 98.3 | 57.4 | 58.1 | 66.5 |
| GLM 5.3 Flash | 32.2 | 6.7 | 5.7 | 34.6 | 97.2 | 62.3 | 23.8 | 49.5 |
| Strict | Outer fence only | |||
|---|---|---|---|---|
| Model | Counts | Evidence | Counts | Evidence |
| DeepSeek V4.1 Flash | 24.4 | 0.3 | 24.4 | 0.3 |
| Qwen 3.7 Plus | 0.0 | 0.0 | 25.0 | 0.0 |
| Qwen 3.8 Max | 35.2 | 3.3 | 35.2 | 3.3 |
| Kimi K3 | 39.8 | 21.4 | 39.8 | 21.4 |
| Kimi K3.1 Pro | 41.5 | 32.7 | 41.5 | 32.7 |
| GPT-6.1 Sol | DeepSeek V4.1 Flash | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Subtype / measurement | Image | Removed | 95% CI | Image | Removed | 95% CI | |||
| T2.3 | 30 | 90.0 | 53.3 | -36.7 | 66.7 | 30.0 | -36.7 | ||
| T5.1 | 30 | 76.7 | 3.3 | -73.3 | 13.3 | 3.3 | -10.0 | ||
| T5.2 | 30 | 83.3 | 0.0 | -83.3 | 3.3 | 6.7 | +3.3 | ||
| T5.3 | 30 | 59.6 | 0.0 | -59.6 | 0.0 | 0.0 | 0.0 | ||
| T5.4 | 30 | 92.0 | 0.0 | -92.0 | 42.0 | 10.0 | -32.0 | ||
| Model | Rendered | Printed scan | Handwritten |
|---|---|---|---|
| Questions / groups | 43 / 43 | 24 / 16 | 58 / 10 |
| DeepSeek V4.1 Flash | 80.5 [76.5, 84.0] (10) | 89.9 [87.6, 91.8] (10) | 89.0 [86.5, 91.0] (11) |
| Qwen 3.7 Plus | 88.0 [83.6, 92.4] (12) | 93.9 [89.9, 97.7] (12) | 93.6 [91.5, 95.1] (12) |
| Qwen 3.8 Max | 66.2 [59.9, 72.1] (8) | 77.9 [72.9, 83.7] (7) | 84.0 [81.3, 86.6] (8) |
| Kimi K3 | 57.4 [51.6, 62.8] (7) | 80.7 [75.7, 84.9] (8) | 76.3 [71.0, 82.2] (7) |
| Kimi K3.1 Pro | 52.7 [46.9, 58.1] (6) | 75.6 [72.4, 79.5] (6) | 61.5 [54.7, 68.4] (5) |
| Model | Rendered | Printed scan | Handwritten |
|---|---|---|---|
| Questions / groups | 24 / 24 | 61 / 22 | 40 / 14 |
| DeepSeek V4.1 Flash | 8.3 [0.0, 20.8] (9) | 0.0 [0.0, 0.0] (10) | 9.5 [0.0, 21.4] (9) |
| Qwen 3.7 Plus | 0.0 [0.0, 0.0] (12) | 0.0 [0.0, 0.0] (10) | 0.0 [0.0, 0.0] (12) |
| Qwen 3.8 Max | 54.2 [33.3, 75.0] (7) | 25.4 [9.1, 43.6] (5) | 26.2 [11.9, 40.5] (8) |
| Kimi K3 | 37.5 [16.7, 58.3] (8) | 5.3 [0.0, 11.4] (8) | 29.2 [17.9, 40.5] (7) |
| Kimi K3.1 Pro | 83.3 [66.7, 95.8] (3) | 21.2 [7.6, 37.1] (6) | 51.8 [34.5, 69.0] (5) |
| Subtype | Target | Context | [95% CI] | |
|---|---|---|---|---|
| T4.1 | 8 | 35.02 | 41.49 | |
| T4.2 | 5 | 48.43 | 45.87 |
VLM Evaluation
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.
See all 1,055 papers