From Portraits to Pathology: Probing Diagnostic Overreach in Vision-Language Models
Organizations: Oracle AI
Abstract
Many medical conditions require clinical assessment of symptoms, history, and functioning rather than judgments from appearance. We study whether vision-language models (VLMs) abstain from facial attribution of autism spectrum disorder (ASD), attention-deficit/hyperactivity disorder (ADHD), major depressive disorder (MDD), and generalized anxiety disorder (GAD). To support this study, we construct PARITY (Paired Assessment with Reused Identity), a demographically balanced dataset for testing whether changing facial expressions alters diagnostic judgments, while holding identity and other visual factors as constant as possible. Our comprehensive evaluation of 27 open- and closed-weight models reveals inconsistent abstention across expressions, conditions, and task formats. Models turn facial expressions into unsupported medical judgments: this behavior is widespread among open-weight VLMs and extends to safety-trained frontier models that abstain on individual portraits yet make diagnostic judgments when comparing portraits of the same identity. Safeguard checks reveal further gaps: clinical instructions do not consistently prevent these judgments, and a guardrail model rarely flags paired requests as unsafe. Semantic analysis further shows that most baseline open-model refusals fail to explain both the insufficiency of facial evidence and the need for clinical assessment. Unsupported judgments and seemingly authoritative explanations can cause distress, reinforce stigma, and offer false reassurance, underscoring the need for stronger safety measures in model training and deployment.
Figures & tables
| Configuration | Repository / provider identifier | Revision |
| Open-weight configurations | ||
| Phi-3.5 Vision | microsoft/Phi-3.5-vision-instruct | 12b77fb40b |
| Phi-4 Multimodal | microsoft/Phi-4-multimodal-instruct | 93f923e1a7 |
| Gemma4-12B | google/gemma-4-12B-it | 707f0a3b8a |
| Gemma4-31B | google/gemma-4-31B-it | 842da3794e |
| North Micro Vision | CohereLabs/North-Micro-Vision-Instruct | fa548d9aa6 |
Appendix figures & tables37 assets
Supplementary material from the paper’s appendix.
Appendix
| Design quantity | Value |
| Identity specification ( ) | 1,000 |
| Race/ethnicity gender age | |
| Minimum allocation per cell | 0.9% |
| Maximum allocation per cell | 1.0% |
| Requested expression | Target action units | Facial movements |
| Happy | AU6, AU12 | Cheek raise; lip-corner pull |
| Sad | AU1, AU4, AU15 | Inner-brow raise; brow lower; lip-corner depress |
| Angry | AU4, AU5, AU7, AU23 | Brow lower; upper-lid raise; lid and lip tighten |
| Fearful | AU1, AU2, AU4, AU5, AU20 | Brow raise/lower; upper-lid raise; lip stretch |
| Surprised | AU1, AU2, AU5, AU26 | Brow raise; upper-lid raise; jaw drop |
| Disgusted | AU9, AU10, AU25 | Nose wrinkle; upper-lip raise; lips part |
| Single-image: Yes (%) | Paired: selection (%) | |||||||
| Model | ASD | ADHD | MDD | GAD | ASD | ADHD | MDD | GAD |
| Aya Vision-8B | 95.1 | 83.1 | 87.0 | 92.1 | 98.3 | 99.1 | 98.8 | 98.4 |
| Command A Vision-112B | 0.9 | 0.4 | 7.5 | 20.8 | 90.0 | 91.2 | 92.6 | 92.5 |
| Gemini 2.5 Flash † | 0.0 | 0.0 | 0.0 | 0.1 | 41.7 | 12.0 | 91.3 | 80.5 |
| Gemini 2.5 Flash-Lite † | 0.0 | 0.0 | 0.1 | 4.6 | 0.9 | 7.6 | 71.8 | 92.0 |
| Gemini 2.5 Pro † | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 10.5 | 10.6 |
| Model | ASD | ADHD | MDD | GAD |
| Aya Vision-8B | 95.1 / 3.2 / 1.6 / 0.1 | 83.1 / 15.2 / 1.2 / 0.5 | 87.0 / 12.2 / 0.5 / 0.3 | 92.1 / 7.5 / 0.0 / 0.4 |
| Command A Vision-112B | 0.9 / 48.9 / 50.1 / 0.0 | 0.4 / 46.6 / 53.0 / 0.0 | 7.5 / 72.0 / 20.5 / 0.0 | 20.8 / 42.9 / 36.4 / 0.0 |
| Gemini 2.5 Flash † | 0.0 / 1.5 / 98.0 / 0.5 | 0.0 / 0.0 / 100.0 / 0.0 | 0.0 / 17.6 / 81.9 / 0.5 | 0.1 / 6.7 / 93.1 / 0.0 |
| Gemini 2.5 Flash-Lite † | 0.0 / 0.1 / 99.7 / 0.1 | 0.0 / 0.0 / 100.0 / 0.0 | 0.1 / 0.7 / 99.2 / 0.0 | 4.6 / 0.4 / 95.0 / 0.0 |
| Gemini 2.5 Pro † | 0.0 / 0.0 / 100.0 / 0.0 | 0.0 / 0.0 / 100.0 / 0.0 | 0.0 / 0.0 / 100.0 / 0.0 | 0.0 / 0.0 / 100.0 / 0.0 |
| Gemma3-4B | 69.0 / 31.0 / 0.0 / 0.0 | 95.4 / 4.6 / 0.0 / 0.0 | 73.9 / 26.1 / 0.0 / 0.0 | 84.4 / 15.6 / 0.0 / 0.0 |
| Model | ASD | ADHD | MDD | GAD |
| Aya Vision-8B | 98.3 / 1.2 / 0.5 | 99.1 / 0.3 / 0.6 | 98.8 / 0.2 / 1.1 | 98.4 / 0.1 / 1.5 |
| Command A Vision-112B | 90.0 / 10.0 / 0.0 | 91.2 / 8.8 / 0.0 | 92.6 / 7.4 / 0.0 | 92.5 / 7.5 / 0.0 |
| Gemini 2.5 Flash † | 41.7 / 54.9 / 3.5 | 12.0 / 87.1 / 0.9 | 91.3 / 5.9 / 2.9 | 80.5 / 17.0 / 2.5 |
| Gemini 2.5 Flash-Lite † | 0.9 / 0.3 / 98.8 | 7.6 / 15.2 / 77.2 | 71.8 / 8.2 / 20.0 | 92.0 / 4.9 / 3.2 |
| Gemini 2.5 Pro † | 0.0 / 99.9 / 0.1 | 0.0 / 100.0 / 0.0 | 10.5 / 89.5 / 0.0 | 10.6 / 89.4 / 0.0 |
| Gemma3-4B | 100.0 / 0.0 / 0.0 | 100.0 / 0.0 / 0.0 | 100.0 / 0.0 / 0.0 | 100.0 / 0.0 / 0.0 |
| Model | ASD | ADHD | MDD | GAD |
| Aya Vision-8B | 86.5 / 7.1 / 6.3 / 0.0 | 74.9 / 21.7 / 2.6 / 0.8 | 75.3 / 20.9 / 3.3 / 0.5 | 87.0 / 12.7 / 0.1 / 0.1 |
| Command A Vision-112B | 0.0 / 14.2 / 85.8 / 0.0 | 0.0 / 5.4 / 94.6 / 0.0 | 0.0 / 63.0 / 37.0 / 0.0 | 0.3 / 36.2 / 63.5 / 0.0 |
| Gemini 2.5 Flash † | 0.0 / 0.0 / 99.5 / 0.5 | 0.0 / 0.0 / 99.9 / 0.1 | 0.0 / 0.0 / 98.7 / 1.3 | 0.0 / 0.0 / 99.6 / 0.4 |
| Gemini 2.5 Flash-Lite † | 0.0 / 0.0 / 78.4 / 21.6 | 0.0 / 0.0 / 98.3 / 1.7 | 0.0 / 0.0 / 97.4 / 2.6 | 0.0 / 0.0 / 90.2 / 9.8 |
| Gemini 2.5 Pro † | 0.0 / 0.0 / 100.0 / 0.0 | 0.0 / 0.0 / 100.0 / 0.0 | 0.0 / 0.0 / 100.0 / 0.0 | 0.0 / 0.0 / 100.0 / 0.0 |
| Gemma3-4B | 0.0 / 61.2 / 38.6 / 0.1 | 1.1 / 58.1 / 40.9 / 0.0 | 29.6 / 48.5 / 21.8 / 0.0 | 53.6 / 37.4 / 9.0 / 0.0 |
| Model | ASD | ADHD | MDD | GAD |
| Aya Vision-8B | 95.9 / 2.9 / 1.2 | 97.8 / 1.2 / 1.0 | 98.1 / 0.4 / 1.5 | 97.1 / 0.5 / 2.3 |
| Command A Vision-112B | 17.7 / 82.3 / 0.0 | 19.4 / 80.6 / 0.0 | 77.0 / 23.0 / 0.0 | 84.6 / 15.4 / 0.0 |
| Gemini 2.5 Flash † | 0.0 / 96.7 / 3.3 | 0.0 / 98.5 / 1.5 | 0.0 / 97.2 / 2.8 | 0.0 / 97.4 / 2.6 |
| Gemini 2.5 Flash-Lite † | 0.0 / 0.1 / 99.9 | 0.0 / 27.2 / 72.8 | 0.0 / 15.0 / 85.0 | 0.0 / 16.0 / 84.0 |
| Gemini 2.5 Pro † | 0.0 / 100.0 / 0.0 | 0.0 / 100.0 / 0.0 | 0.0 / 100.0 / 0.0 | 0.0 / 100.0 / 0.0 |
| Gemma3-4B | 97.2 / 2.8 / 0.0 | 99.5 / 0.5 / 0.0 | 99.6 / 0.4 / 0.0 | 100.0 / 0.0 / 0.0 |
| Model | ASD | ADHD | MDD | GAD |
| Aya Vision-8B | 97.2 / 1.9 / 0.9 | 100.0 / 0.0 / 0.0 | 99.1 / 0.0 / 0.9 | 100.0 / 0.0 / 0.0 |
| Command A Vision-112B | 54.6 / 45.4 / 0.0 | 4.6 / 95.4 / 0.0 | 24.1 / 75.9 / 0.0 | 14.8 / 85.2 / 0.0 |
| Gemini 2.5 Flash † | 1.9 / 98.1 / 0.0 | 0.0 / 100.0 / 0.0 | 16.7 / 83.3 / 0.0 | 9.3 / 90.7 / 0.0 |
| Gemini 2.5 Flash-Lite † | 0.0 / 0.0 / 100.0 | 0.0 / 9.3 / 90.7 | 0.0 / 3.7 / 96.3 | 0.0 / 19.4 / 80.6 |
| Gemini 2.5 Pro † | 0.0 / 100.0 / 0.0 | 0.0 / 100.0 / 0.0 | 0.0 / 100.0 / 0.0 | 0.0 / 100.0 / 0.0 |
| Gemma3-4B | 100.0 / 0.0 / 0.0 | 99.1 / 0.9 / 0.0 | 100.0 / 0.0 / 0.0 | 100.0 / 0.0 / 0.0 |
| Single image (%) | Paired images (%) | |||||||
| Cohort | Condition | Yes | No | Abstain | Invalid | Select | Abstain | Invalid |
| Open | ASD | 11.9 | 32.9 | 55.1 | 0.1 | 79.1 | 20.8 | 0.1 |
| Open | ADHD | 14.3 | 25.2 | 60.5 | 0.1 | 78.1 | 21.8 | 0.1 |
| Open | MDD | 18.6 | 54.2 | 27.1 | 0.1 | 88.7 | 11.2 | 0.1 |
| Open | GAD | 26.2 | 48.5 | 25.2 | 0.1 | 85.3 | 14.5 | 0.2 |
| Closed | ASD | 0.0 | 0.1 | 99.8 | 0.0 | 13.4 | 69.7 | 16.9 |
| Expression | ASD | ADHD | MDD | GAD |
| Neutral | 10.5 | 5.1 | 17.4 | 23.8 |
| Angry | 20.0 | 19.7 | 44.3 | 51.5 |
| Fearful | 19.0 | 26.1 | 23.1 | 41.7 |
| Happy (closed) | 5.0 | 7.5 | 4.3 | 7.0 |
| Happy (open) | 4.9 | 13.0 | 3.8 | 7.2 |
| Attribution (%) | Abstention (%) | Invalid (%) | |||||
| Task | Cohort | Base | Clinical | Base | Clinical | Base | Clinical |
| Single | Open | 17.7 | 5.4 | 42.0 | 67.5 | 0.1 | 0.2 |
| Single | Closed | 0.1 | 0.0 | 98.9 | 98.8 | 0.1 | 1.2 |
| Paired | Open | 82.8 | 57.2 | 17.1 | 37.7 | 0.1 | 5.1 |
| Paired | Closed | 18.5 | 0.0 | 70.2 | 84.3 | 11.3 | 15.7 |
| Condition | Single unsafe (%) | Paired unsafe (%) |
| ASD | 46.8 | 0.0 |
| ADHD | 11.3 | 0.0 |
| MDD | 74.5 | 0.0 |
| GAD | 71.7 | 0.0 |
| Spearman [95% interval] | Pearson | ||||
| Task | Condition | All models | Open models | All | Open |
| Single | ASD | 0.936 [0.806, 0.994] | 0.936 [0.806, 0.987] | 0.959 | 0.955 |
| Single | ADHD | 0.964 [0.862, 1.000] | 0.966 [0.869, 1.000] | 0.971 | 0.969 |
| Single | MDD | 0.972 [0.916, 0.993] | 0.983 [0.934, 0.996] | 0.951 | 0.941 |
| Single | GAD | 0.980 [0.930, 0.996] | 0.977 [0.894, 0.999] | 0.976 | 0.969 |
| Paired | ASD | 0.918 [0.777, 0.968] | 0.861 [0.591, 0.971] | 0.975 | 0.977 |
| Model | ASD | ADHD | MDD | GAD |
| Aya Vision-8B | 8.3 | 0.0 | 0.0 | – |
| Command A Vision-112B | 26.1 | 22.4 | 67.7 | 65.5 |
| Gemini 2.5 Flash † | 55.5 | 35.8 | 62.4 | 45.2 |
| Gemini 2.5 Flash-Lite † | 92.4 | 63.6 | 93.9 | 83.6 |
| Gemini 2.5 Pro † | 97.4 | 92.7 | 98.9 | 97.1 |
| Gemma3-4B | – | – | – | – |
| Model | ASD | ADHD | MDD | GAD |
| Aya Vision-8B | 0.0 | 0.0 | 0.0 | 0.0 |
| Command A Vision-112B | 14.0 | 14.9 | 4.2 | 2.1 |
| Gemini 2.5 Flash † | 51.5 | 35.8 | 55.3 | 11.4 |
| Gemini 2.5 Flash-Lite † | 75.0 | 35.5 | 18.9 | 17.5 |
| Gemini 2.5 Pro † | 52.4 | 54.2 | 82.7 | 50.7 |
| Gemma3-4B | – | – | – | – |
| Model | ASD | ADHD | MDD | GAD |
| Aya Vision-8B | 2.1 | 15.0 | 4.0 | 0.0 |
| Command A Vision-112B | 78.9 | 90.6 | 82.1 | 81.2 |
| Gemini 2.5 Flash † | 100.0 | 100.0 | 100.0 | 100.0 |
| Gemini 2.5 Flash-Lite † | 100.0 | 94.5 | 100.0 | 98.1 |
| Gemini 2.5 Pro † | 99.3 | 99.1 | 99.7 | 100.0 |
| Gemma3-4B | 2.4 | 0.0 | 3.0 | 4.4 |
| Model | ASD | ADHD | MDD | GAD |
| Aya Vision-8B | 13.2 | 0.0 | 0.0 | 14.3 |
| Command A Vision-112B | 96.2 | 98.1 | 63.1 | 59.8 |
| Gemini 2.5 Flash † | 100.0 | 100.0 | 100.0 | 100.0 |
| Gemini 2.5 Flash-Lite † | 100.0 | 87.8 | 100.0 | 100.0 |
| Gemini 2.5 Pro † | 100.0 | 99.0 | 99.9 | 99.8 |
| Gemma3-4B | 0.0 | 0.0 | 0.0 | – |
| Model | ASD | ADHD | MDD | GAD |
| Aya Vision-8B | 0.0 | – | – | – |
| Command A Vision-112B | 10.2 | 11.7 | 2.4 | 1.1 |
| Gemini 2.5 Flash † | 42.5 | 34.3 | 21.1 | 8.2 |
| Gemini 2.5 Flash-Lite † | – | 30.0 | 100.0 | 81.0 |
| Gemini 2.5 Pro † | 93.5 | 97.2 | 76.9 | 74.1 |
| Gemma3-4B | – | 0.0 | – | – |
| Theme | Example keywords/phrases |
| Refusals | Clinically grounded: cannot diagnose , cannot be inferred from a photograph , requires comprehensive clinical/behavioral evaluation , qualified professionals , not based on visual cues alone |
| Generic or deflective: not enough information , insufficient information , I am not a medical professional , cannot provide a diagnosis | |
| Speculative reasoning | Affect-based heuristics: exaggerated expression , forced expression , intense expression , unnatural expression , tense facial muscles , rigid expression , reduced expressiveness , atypical expression |
| Social–behavioral stereotypes: lack of eye contact , difficulty processing social cues , social communication , social interactions , emotional regulation , sensory overload , restricted/repetitive , neurotypical |
| Rationale excerpt |
| Clinically grounded refusals |
| Autism cannot be determined from photos or facial expressions; it requires a comprehensive clinical/behavioral assessment by qualified professionals. |
| It would be inappropriate and potentially harmful to guess ASD from a photograph; this requires professional evaluation beyond what images provide. |
| Image-evidence limitation without a clinical-assessment statement |
| A diagnosis like ASD cannot be made from an image; there isn’t enough reliable information here to choose between the two people. |
| Generic or incomplete refusals |
| Task | Condition | Unsafe (%) | Safe (%) | Invalid (%) |
| Single | ASD | 54.10 | 45.90 | 0.00 |
| ADHD | 24.21 | 75.79 | 0.00 | |
| MDD | 81.75 | 18.25 | 0.00 | |
| GAD | 93.52 | 6.48 | 0.00 | |
| Paired | ASD | 0.00 | 100.00 | 0.00 |
| ADHD | 0.08 | 99.92 | 0.00 |
| Family / group | Main-figure representatives | Additional configurations in appendix figures |
| Gemma 4 | Gemma4-31B | Gemma4-12B |
| Gemma 3 | Gemma3-4B; Gemma3-27B | None |
| InternVL 3.5 | InternVL3.5-8B Instruct; InternVL3.5-38B | InternVL3.5-8B; 8B Flash; 38B Flash; 38B Instruct |
| Mistral | Mistral3.1-24B; Mistral4-119B; Pixtral-12B | Mistral3.2-24B |
| Phi | Phi-3.5 Vision; Phi-4 Reasoning Vision | Phi-4 Multimodal |
| Cohere | Aya Vision-8B; Command A Vision-112B | North Micro Vision |