Do Multimodal Models Decide Consistently on Equivalent Optical Spectra?
Organizations: Independent Researcher · Georgia Institute of Technology · University of Massachusetts Amherst
Abstract
A spectrum can be given to a model as a table in nanometers or as a plot in photon energy, and the decision should not depend on which one it sees. When the data needed for an answer are missing, the model should say so. We test representation invariance and abstention on measured optical spectra of silicon metasurfaces (MOCLIP) and gold nanoparticles, with labels computed from the data. Each spectrum is shown as four tables (nm, m, scientific notation, eV) and two plots (nm, eV), and partial spectra remove data inside or outside the queried range. We evaluate the decision models d3-nano and Jev 1.13, three GPT-6 Luna configurations and the Qwen3.5-2B base model. Changing only the representation moves accuracy by up to 41 percentage points within one model, mostly on photon-energy tables and plots. With the same evidence, asking in photon energy instead of wavelength helps some model-view pairs by 16-50 points and hurts others by up to 25. When the relevant extremum is removed, unadapted models rarely answer "undetermined". LoRA fine-tuning of d3-nano improves comparisons on the trained table formats, but much of its gain on unanswerable questions comes from a shortcut: it abstains whenever the input is truncated. Training with answerable truncations avoids the shortcut in all six seeds, with a 90-100 point gap between abstention after inside and outside cuts on held-out structures; with the same amount of extra original-task data, five of six seeds keep the shortcut. On gold spectra the two training sets perform similarly. Multimodal language models used for scientific decisions should be tested on equivalent representations and on missing evidence, not only on average accuracy. We release the tasks, per-response results and analysis code.
Figures & tables
| Set | Instances | Unique spectra | Variants | Core variants | Requests |
|---|---|---|---|---|---|
| MOCLIP main | 192 | 336 | 720 | 528 | 3,552 |
| Gold main | 54 | 70 | 244 | 140 | 1,048 |
| MOCLIP scope | 54 | 54 | 162 | 162 | 972 |
| Gold scope | 60 | 60 | 180 | 180 | 1,080 |
| MOCLIP | Gold | |||
|---|---|---|---|---|
| Model | Tables | Plots | Tables | Plots |
| d3-nano | 56 [54, 59] | 60 [57, 63] | 64 [59, 69] | 55 [48, 63] |
| Jev | 78 [75, 81] | – | 81 [77, 85] | – |
| Luna Decisions | 65 [62, 67] | 70 [67, 73] | 82 [77, 87] | 80 [75, 85] |
| Luna gen. none | 74 [70, 77] | 60 [56, 63] | 86 [82, 90] | 74 [68, 79] |
| Luna gen. low | 95 [94, 96] | 93 [91, 94] | 97 [96, 99] | 96 [93, 99] |
| Model | m table | sci. table | eV table | nm plot | eV plot | |
|---|---|---|---|---|---|---|
| MOCLIP | d3-nano | +4 [+2, +7] | +5 [+2, +9] | +5 [+1, +9] | +7 [+3, +11] | +7 [+3, +11] |
| Jev | +1 [-1, +3] | +2 [-0, +5] | -5 [-8, -1] | – | – | |
| Luna Decisions | +5 [+2, +9] | +2 [-2, +6] | +0 [-4, +5] | +15 [+11, +19] | -0 [-5, +4] | |
| Luna gen. none | +1 [-2, +4] | +1 [-2, +4] | -5 [-9, +0] | -10 [-15, -4] | -20 [-25, -13] | |
| Luna gen. low | -2 [-3, -0] | -2 [-3, -0] | -10 [-13, -7] | -5 [-7, -3] | -6 [-9, -4] | |
| Qwen3.5-2B base | +2 [-1, +5] | -1 [-4, +3] | -3 [-8, +1] | +6 [+1, +11] | +6 [+1, +11] |
| MOCLIP | Gold | |||||||
| Model | All-four | All-six | Wrong | Indep. | All-four | All-six | Wrong | Indep. |
| d3-nano | 53 [48, 58] | 37 [32, 42] | 12 | 8 | 62 [53, 71] | 46 [38, 54] | 7 | 6 |
| Jev | 75 [71, 79] | – | – | – | 49 [39, 61] | – | – | – |
| Luna Decisions | 59 [55, 63] | 45 [41, 48] | 9 | 15 | 66 [60, 73] | 47 [39, 55] | 1 | 29 |
| Luna gen. none | 57 [52, 62] | 31 [27, 36] | 4 | 17 | 65 [57, 73] | 39 [32, 47] | 1 | 28 |
| Luna gen. low | 84 [81, 88] | 78 [74, 82] | 0 | 73 | 89 [84, 94] | 87 [82, 92] | 0 | 84 |
| MOCLIP compare | Gold compare | Gold threshold | ||||
|---|---|---|---|---|---|---|
| Model | eV table | eV plot | eV table | eV plot | eV table | eV plot |
| d3-nano | 53 52 (-1 [-5, +4]) | 51 54 (+3 [-4, +10]) | 59 34 (-25 [-42, -8]) | 52 62 (+11 [-3, +25]) | 50 100 (+50 [+25, +75]) | 50 98 (+48 [+25, +70]) |
| Jev | 50 66 (+16 [+9, +23]) | – | 28 77 (+48 [+30, +64]) | – | 50 100 (+50 [+25, +75]) | – |
| Luna Decisions | 48 52 (+4 [+1, +7]) | 45 64 (+19 [+9, +28]) | 72 72 (+0 [-9, +8]) | 77 88 (+11 [+2, +20]) | 68 75 (+8 [-12, +28]) | 55 85 (+30 [+0, +60]) |
| Luna gen. none | 52 58 (+6 [-5, +16]) | 45 51 (+6 [-2, +15]) | 59 62 (+3 [-8, +12]) | 55 70 (+16 [+0, +30]) | 72 85 (+12 [-2, +28]) | 60 82 (+22 [+8, +40]) |
| Luna gen. low | 80 99 (+20 [+14, +26]) | 96 96 (+1 [-4, +5]) | 88 95 (+8 [+0, +14]) | 100 98 (-2 [-5, +0]) | 100 90 (-10 [-20, -2]) | 100 98 (-2 [-8, +0]) |
| MOCLIP | Gold | |||||
|---|---|---|---|---|---|---|
| Model | Compare, wavelength | Compare, energy | Middle-gap abstention | Scope | Compare | Scope |
| d3-nano | 52 54 (+2 [-3, +7]) | 52 49 (-3 [-7, +1]) | 0 0 (+0 [+0, +0]) | +0 +0 (+0 [+0, +0]) | 48 56 (+9 [+2, +16]) | +0 +0 (+0 [+0, +0]) |
| Jev | 51 54 (+4 [-3, +10]) | 66 54 (-12 [-20, -6]) | 0 0 (+0 [+0, +0]) | +7 +7 (+0 [-9, +9]) | 50 83 (+33 [+20, +44]) | +43 +5 (-38 [-53, -23]) |
| Luna Decisions | 48 51 (+3 [-2, +8]) | 52 50 (-2 [-7, +4]) | 0 0 (+0 [+0, +0]) | +26 +9 (-17 [-26, -7]) | 72 68 (-4 [-11, +2]) | +40 +27 (-13 [-30, +3]) |
| Luna gen. none | 49 51 (+2 [-7, +10]) | 56 55 (-1 [-10, +8]) | 0 0 (+0 [+0, +0]) | +19 +2 (-17 [-28, -6]) | 59 74 (+16 [+7, +24]) | +58 +67 (+8 [+2, +15]) |
| Luna gen. low | 80 89 (+9 [+3, +17]) | 100 100 (+0 [+0, +0]) | 0 6 (+6 [+0, +19]) | +37 +15 (-22 [-35, -9]) | 93 97 (+4 [-0, +8]) | +75 +72 (-3 [-10, +3]) |
| MOCLIP tables | MOCLIP plots | Gold tables | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Model | Full | Full | Full | ||||||
| d3-nano | 69 | +0 | -3 | 67 | +3 | +1 | 79 | +0 | +5 |
| Jev | 99 | +14 | +1 | – | – | – | 100 | +45 | +0 |
| Luna Decisions | 88 | +18 | -1 | 90 | +4 | +7 | 68 | +42 | +4 |
| Luna gen. none | 95 | +19 | -2 | 88 | +4 | -1 | 96 | +65 | +3 |
| Luna gen. low | 98 | +55 | -1 | 99 | +16 | +1 | 98 | +82 | +2 |
| Arm | Training data and representations | Train | Dev | Duration |
|---|---|---|---|---|
| ftA | Original tasks, nm tables | 632 | 80 | 2 epochs |
| ftAc | As ftA | 632 | 80 | Selected epoch |
| ftBm | Original tasks, four tables, sampled to ftA size | 632 | 372 | 2 epochs |
| ftBmc | As ftBm | 632 | 372 | Selected epoch |
| ftB | Original tasks, four tables | 2,956 | 372 | 2 epochs |
| ftC | ftB data + 72 answerable/unanswerable scope instances | 3,736 | 456 | 2 epochs |
| Difference from Decisions | High-confidence error (%) | ||||
|---|---|---|---|---|---|
| Dataset | Gen. none | Gen. low | Decisions | Gen. none | Gen. low |
| MOCLIP | +3 [+1, +4] | +28 [+26, +31] | 6 [5, 7] | 31 [28, 34] | 6 [4, 7] |
| Gold | +0 [-2, +3] | +15 [+12, +19] | 3 [2, 5] | 18 [15, 22] | 3 [2, 4] |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Dataset | Modality | Full (%) | ||
|---|---|---|---|---|---|
| d3-nano | MOCLIP | table | 69 [61, 76] | +0 [+0, +0] | -3 [-9, +2] |
| d3-nano | MOCLIP | plot | 67 [57, 76] | +3 [+0, +6] | +1 [-8, +10] |
| d3-nano | Gold | table | 79 [73, 85] | +0 [+0, +1] | +5 [+2, +8] |
| d3-nano | Gold | plot | 50 [38, 62] | +9 [+4, +14] | +3 [+0, +8] |
| Jev | MOCLIP | table | 99 [97, 100] | +14 [+8, +21] | +1 [+0, +2] |
| Jev | Gold | table | 100 [100, 100] | +45 [+35, +55] | +0 [+0, +0] |
| MOCLIP main | Gold | MOCLIP scope | Gold scope | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Arm | Seed | CT | CP | ST | GT | F | F | ||||
| ftA | 0 | 51 | 51 | 31 | 75 | 85 | +45 | -27 | 67 | +49 | +7 |
| ftA | 1 | 50 | 50 | 33 | 75 | 87 | +66 | -4 | 67 | +60 | +12 |
| ftA | 2 | 51 | 50 | 34 | 74 | 56 | -7 | -56 | 26 | +33 | +10 |
| ftAc | 0 | 80 | 49 | 32 | 63 | 78 | -4 | -70 | 44 | +45 | +3 |
| ftAc | 1 | 73 | 45 | 35 | 73 | 87 | +29 | -15 | 73 | +42 | +1 |
| Arm | Seed | Dataset | Modality | Full (%) | ||
|---|---|---|---|---|---|---|
| ftA | 0 | MOCLIP | table | 85 [78, 91] | +45 [+34, +56] | -27 [-34, -20] |
| ftA | 0 | MOCLIP | plot | 76 [69, 82] | +1 [-10, +12] | -52 [-60, -43] |
| ftA | 0 | Gold | table | 67 [62, 73] | +49 [+41, +57] | +7 [+3, +11] |
| ftA | 0 | Gold | plot | 52 [41, 65] | +53 [+42, +65] | +0 [-4, +4] |
| ftA | 1 | MOCLIP | table | 87 [78, 94] | +66 [+55, +76] | -4 [-8, -0] |
| ftA | 1 | MOCLIP | plot | 77 [68, 85] | +35 [+21, +48] | -19 [-28, -12] |
Multimodal Model Evaluation
Multimodal model evaluation measures how reliably AI systems understand and generate content across text, images, audio, video, and other data.
See all 793 papers