A Deterministic Evidence Layer for Vision-Language Autism Screening from Naturalistic Home Video

Wenqi Li; Mindi Ruan; Chuanbo Hu; Shuo Wang; Xin Li

cs.CVOct 6, 2026

A Deterministic Evidence Layer for Vision-Language Autism Screening from Naturalistic Home Video

Authors: Wenqi Li, Mindi Ruan, Chuanbo Hu, Shuo Wang, Xin Li

Organizations: Department of Computer Science, University at Albany, SUNY, Albany, NY, USA · Lane Department of Computer Science and Electrical Engineering, West Virginia University, Morgantown, WV, USA · Department of Radiology, Washington University in St. Louis, St. Louis, MO, USA

Abstract

Autism spectrum disorder (ASD) is diagnosed through specialist observation of a child's social behavior, and access to that expertise is the bottleneck for early identification. Vision-language models (VLMs) describe a child's behavior from video well; the verdict drawn from the description is unstable: at temperature~0, across eight pipeline configurations on one backbone, 16--37% of clips change their predicted label between repeated runs, and the cause lies in the serving stack. We keep the VLM frozen and move the decision out of the model. Under our grounded perception constraints the VLM writes an event table of timestamped, glossary-labeled events that names the eliciting press and logs counter-evidence; a text-only stage age-calibrates the confidence of each row; a deterministic weight-of-evidence scorer sums it into an evidence total and stratifies it into a risk category, so every decision decomposes into named per-feature contributions and can be re-scored from the saved table. On 43 caregiver-recorded, protocol-free home free-play clips of preschool children, the pipeline reaches AUC $0.851 \pm 0.012$ , 86.0% accuracy, and F $_1$ 71.8 over three runs. It labels 74.4% of clips correctly in every run (60.5% for the zero-shot baseline) and flags no typically developing clip in every run (9 of 31 at zero-shot). An ablation on the same backbone attributes the gain to the grounded perception constraints read through the deterministic scorer.

Figures & tables

Figure 1: The screening pipeline. Stage 1, a VLM that sees the video, writes a timestamped event table under grounded perception constraints; Stage 2, a text-only model given the table and the child’s age, calibrates each row under the never-erase rule; the deterministic scorer sums the calibrated table into an evidence total and stratifies it into a low-, intermediate-, or high-risk category.

Figure 2: Classification performance of the full pipeline of our method. A : ROC curve of the full pipeline, mean over the three runs (bold) and single runs (thin), with its operating point (large marker: mean; small markers: single runs). B : evidence total of every clip under the full pipeline, mean over the three runs (dot) and range (bar). Annotations mark the three ASD clips missed in every run and the two TD clips whose total stops exactly at 1.5 in one run.

Configuration	Verdict	Acc (%)	Recall (%)	Spec (%)	F 1 (%)
Zero-shot baseline	VLM	69.0 $\pm$ 5.4	86.1 $\pm$ 4.8	62.4 $\pm$ 6.7	60.9 $\pm$ 4.8
+ Few-shot exemplars	VLM	66.7 $\pm$ 1.3	55.6 $\pm$ 4.8	71.0 $\pm$ 0.0	48.1 $\pm$ 3.2
+ Age calibration	VLM	65.9 $\pm$ 2.7	91.7 $\pm$ 0.0	55.9 $\pm$ 3.7	60.0 $\pm$ 1.9
+ Two-stage split	LLM	77.5 $\pm$ 3.6	41.7 $\pm$ 8.3	91.4 $\pm$ 1.9	50.7 $\pm$ 9.0
+ Grounded perception constraints	LLM	81.4 $\pm$ 2.3	47.2 $\pm$ 4.8	94.6 $\pm$ 4.9	58.7 $\pm$ 1.4
+ Deterministic scorer (our full pipeline)	scorer	86.0 $\pm$ 0.0	63.9 $\pm$ 4.8	94.6 $\pm$ 1.9	71.8 $\pm$ 1.6

Table 1: Cumulative ablation of the full pipeline on Gemini 3.1 Pro (temperature

0

, three seeds each, mean

\pm

s.d.; positive class ASD). Each row adds one module to the row above. Verdict names the source of the label: VLM (single call), LLM (the text-only Stage 2), or scorer.

Figure 3: Stable correctness of the three configurations of Section 4.1 . For each configuration, pass@1 and pass 3 as defined in Section 4.4 ; the shaded segment is the share of clips correct in some runs only.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Group	N	Age, mean $\pm$ s.d. (months)	Age range (months)
ASD (evaluation)	12	40.9 $\pm$ 6.6	30–48
TD (evaluation)	31	36.6 $\pm$ 6.1	26–47
Demonstration clips (2 ASD, 2 TD)	4	35.8 $\pm$ 5.2	32–43

Appendix

Table A1: Composition of the home-video corpus. Ages are the children’s ages at recording. The four demonstration clips supply the in-context exemplars (Section 3.3 ) and are excluded from every reported metric.

#	Feature	Domain	sens / spec	$w_{f}$	Basis
1	Unusual / avoidant eye contact	SC	0.85 / 0.80	1.447	eye-tracking meta-analysis [ 58 ]
2	Limited facial expressions & social smiling	SC	0.78 / 0.70	0.956	high-risk sibling study; specificity limited [ 44 ]
3	Reduced gestures & joint attention	SC	0.87 / 0.82	1.576	joint-attention discriminability [ 8 ] ; early joint-attention deficits predict 18-month screen risk [ 42 ]
4	Preference for solitude / limited peer interest	SC	0.80 / 0.79	1.338	author estimate; multifactorial
5	Impaired social reciprocity	SC	0.88 / 0.84	1.705	response-to-name marker studies [ 66 , 41 ]
6	Lack of imaginative / pretend play	SC	0.72 / 0.62	0.639	published pair, symbolic play test, ASD vs. language disorder [ 7 ]

Appendix

Table A2: Per-feature weights of the deterministic scorer. Sensitivity and specificity are the assumed discriminative strength of the feature when present;

w_{f}=\ln(\mathrm{sens}_{f}/(1-\mathrm{spec}_{f}))

in nats; the basis column names the type of evidence consulted for each pair.

Rule	Selection	TP/FN/TN/FP	Recall	Spec.	Acc.
Deployed rule (1.5)	set on an earlier run	23/13/88/5	63.9	94.6	86.0
High risk only (1.8)	set on an earlier run	22/14/91/2	61.1	97.8	87.6
Re-selected by leave-one-out	Youden index	21/15/86/7	58.3	92.5	82.9
	maximum F 1	21/15/88/5	58.3	94.6	84.5
	recall first, spec. $\geq$ 90%	22/14/85/8	61.1	91.4	82.9

Appendix

Table A3: Optimism of the fixed decision rule. Confusion counts, recall, specificity, and accuracy (%) are pooled over the 43 clips of the three runs (129 clip-runs). The leave-one-out rows re-select the cap and the cut-off on 42 clips in every fold, under the named criterion, and apply them to the held-out clip.

Configuration	Verdict	Flip (%)
Zero-shot baseline	VLM	16.3
+ Few-shot exemplars	VLM	18.6
Single call, multi-turn exemplars, no age calibration	VLM	37.2
+ Age calibration	VLM	23.3
+ Two-stage split	LLM	20.9
Two-stage split, read by the scorer	scorer	27.9

Appendix

Table A4: Cross-seed flip rates on Gemini 3.1 Pro (three repeated runs at temperature

0

). Flip is the fraction of the 43 clips whose label differs across the runs; Verdict as in Table 1 .

Run	Accuracy (%)	Recall (%)	Specificity (%)	F 1 (%)
run 1	72.1	58.3	77.4	53.8
run 2	69.8	58.3	74.2	51.9
run 3	76.7	75.0	77.4	64.3
run 4	69.8	66.7	71.0	55.2
run 5	76.7	75.0	77.4	64.3
mean $\pm$ s.d.	73.0 $\pm$ 3.5	66.7 $\pm$ 8.3	75.5 $\pm$ 2.9	57.9 $\pm$ 6.0

Appendix

Table A5: Run-to-run stability of the single-call baseline (Gemini 2.5 Pro, full prompt, four in-context videos, VLM verdict) over five repeated runs of the identical request at temperature

0

(43 clips; an empty API response counts as an error).

Figure A1: Anatomy of the Stage 1 and Stage 2 prompts. Top: the system prompt of each stage (persona, knowledge, rules, output mandate); the knowledge blocks are the glossary of Table A6 and the norms of Fig. A2 . Bottom: the conversation the model sees, with demonstrations as prior user/assistant turn pairs (four corpus clips with their expert tables for Stage 1; the two hand-written exchanges of Fig. A3 for Stage 2) and the final user turn; the dashed bubble is the stage’s output, and Stage 1’s event table becomes the content of Stage 2’s final turn.

#	Feature (merged diagnostic items)	Expanded behavioral description	Typical situations / contexts	Representative examples
1	Unusual / Avoidant Eye Contact (ADOS-2 SA “Unusual Eye Contact”, DSM-5 A2 component)	Limited, fleeting or sidelong gaze; may avoid looking at faces even while interacting.	Feeding, play, greetings, clinical exams.	Infant does not lock eyes during feeding; toddler continues playing when a visitor enters and does not look up despite name being called.
2	Limited Facial Expressions & Social Smiling	Flat or restricted affect; smiles and emotive expressions are infrequent, delayed, or context-inappropriate.	Peek-a-boo, praise moments, gift giving, celebrations.	Child watches peek-a-boo with neutral face; preschooler receives a gift with minimal facial change.
3	Reduced Gestures & Joint Attention	Pointing, showing, waving, and nodding are rare; may not coordinate gaze with gesture or follow others’ pointing.	Requesting help, picture-book reading, playground pointing.	Child reaches toward a toy but does not point or look to a caregiver to share the request.
4	Preference for Solitude / Limited Peer Interest	Appears content playing alone; seldom approaches peers or adults to share enjoyment or comfort.	Day-care free-play, playdates, family gatherings.	Child ignores a peer offering a toy and continues solitary play.
5	Impaired Social Reciprocity	Reduced back-and-forth interaction; may not respond to name or may show limited verbal turn-taking.	Name-call tests, greetings, Q&A exchanges.	Child does not orient when name is called after multiple attempts.
6	Lack of Imaginative / Pretend Play	Symbolic play is absent or limited; play remains concrete, repetitive, or focused on object properties.	Doll/kitchen sets, toy-vehicle play, cooperative make-believe.	Child spins toy-car wheels and does not use the car in pretend scenarios.

Appendix

Table A6: The 11-item behavioral glossary: feature (with ADOS-2 / DSM-5 mapping where applicable), behavioral description, typical contexts, and an example. It constrains both expert annotation and model prompting.

Figure A2: The age-referenced norms of Stage 2, read from the norms text. For each glossary feature the bar marks the ages at which the named behavior is age-typical, so the row is downgraded to possible with an age note; the light segment is a transition range the norms name; the line under the bar is the condition under which the feature is kept. Dotted lines are the developmental anchors the norms list. Pervasiveness overrides the window at any age.

Figure A3: The two hand-written Stage 2 demonstrations, drawn as the output tables ( A : 30 months; B : 42 months). Yellow cells are the age notes Stage 2 wrote into the Note column; the green outline in A marks the social overture that blocks the pervasive feature #4; the green row in B is the clip-spanning #4 row Stage 2 added after the counter-evidence check found no overture. Feature numbers refer to Table A6 .

Explore similar work

Jun 25, 2026cs.CV

Fine-tuning a multimodal large language model for clinician-grade autism behavioral scoring from short home videos

Autism spectrum disorder (ASD) affects 1 in 31 US children, yet median age at diagnosis exceeds four years. Artificial intelligence pipelines that provide quantified diagnosis using easy to access observational data (e.g., home videos) could help with earlier diagnosis, and timely delivery of early treatments. We fine-tuned Gemini 2.5 Pro on 400 clinician-rated home videos with low-rank adaptation, training only on 30 behavioral features previously validated to produce reliable predictions when passed to various ML models. On 99 held-out children (49 ASD, 50 neurotypical), inter-rater reliability with clinicians (per-feature weighted Cohen's kappa) improved by 40% (p<0.001), with 27 of 28 evaluable features improving. As an emergent zero-shot capability, direct ASD diagnosis F1 improved by 53% (p<0.001), matching or exceeding clinician outcomes. Classifier-assisted pipelines using fine-tuned LLM-derived behavioral features matched clinician-scored inputs across all tested pathways and achieved 77% accuracy (95% CI: 68-85%) and an AUC of 86% (95% CI: 78-92%). Fine-tuned multimodal LLMs can serve as scalable behavioral feature extractors for use in autism assessment and diagnosis.

Mohammadmahdi Honarmand, Parnian Azizian, Aaron Kline +9

Department of Mechanical Engineering, Stanford University, Stanford, CA 94305, USA · Department of Pediatrics, Stanford University, Stanford, CA 94305, USA · Department of Biomedical Data Science, Stanford University, Stanford, CA 94305, USA +2

Autism Spectrum Disorder LLM Fine-Tuning Multimodal Large Language Models

Ask PDF

Sep 15, 2026cs.CV

A multimodal large language model for evidence-based autism spectrum disorder screening

The clinical management of autism spectrum disorder (ASD) faces a bottleneck in early screening, mainly because trained specialists are scarce and conventional assessment tools are subjective. Here, we introduce ASDchat, a multimodal large language model designed for evidence-based ASD screening, which takes video, audio, and dialogue as input. ASDchat adopts a dual-branch architecture, where the decision branch generates screening probabilities and the evidence branch generates traceable, timestamped behavioral evidence aligned with standardized clinical criteria (ADOS-2). The model was trained and evaluated on a dataset of 1,035 participants from 27 sites in China, which covered typically developing (TD) children, children with ASD, and children with other disorders. For ASD versus TD, ASDchat reached an area under the receiver operating characteristic curve (AUC) of 0.953

\pm

0.021. On 9 held-out sites that were not used for training, the mean AUC was 0.932. Furthermore, unsupervised clustering of the behavioral dimensions split the ASD cases into six subtypes with different phenotypic profiles, and ASDchat suggests an intervention for each subtype. ASDchat provides a feasible path for large-scale, evidence-based early ASD screening in clinical practice.

Jun Chen, Qi Zhao, Yunliang Jiang +8

National Special Education Resource Center for Children with Autism, Zhejiang Normal University, China · School of Computer Science and Technology, Zhejiang Normal University, China · School of Mathematics and Statistics, Xi’an Jiaotong University, China +5

Autism Spectrum Disorder Multimodal Large Language Models Multimodal Classification

Ask PDF

Sep 25, 2026cs.CV

UniAR: A Unified Framework for Autism Recognition Enhanced by Multi-View Prompt Learning

Autism Spectrum Disorder (ASD) is a complex neurodevelopmental disorder for which early and accurate diagnosis is critical to improving long-term developmental outcomes. However, existing ASD recognition methods are often constrained by the scarcity of diagnostic text data, forcing them to rely mainly on visual analysis and limiting their ability to model clinically meaningful semantic reasoning. To address this challenge, we propose UniAR, a unified framework enhanced by multi-granularity prompt learning for robust ASD recognition under heterogeneous data variations. Specifically, UniAR leverages a large multimodal model to generate hierarchical diagnostic descriptions at the word, phrase, and sentence levels, compensating for the lack of paired clinical reports. To align the generated semantics with visual evidence, we further design a Mixture-of-Experts-based Multi-Scale Alignment Module, which dynamically matches vector-quantized visual prototypes with semantic representations at corresponding granularities. Extensive experiments on four benchmarks covering brain MRI and facial expression scenarios show that UniAR consistently outperforms existing state-of-the-art methods, achieving average accuracies of 75.9% on MRI benchmarks and 91.6% on facial benchmarks, while improving average Accuracy on MRI benchmarks by 1.5 percentage points and average Accuracy on facial benchmarks by 1.2 percentage points over baselines. These results demonstrate that UniAR offers a robust and interpretable framework for ASD screening under semantic scarcity.

Lei Xin, Zeheng Wang, Jiayin Zhu +6

Wuhan University Wuhan, Hubei, China · Northeast Normal University Changchun, Jilin, China · The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, Guangdong, China +4

Autism Spectrum Disorder Multimodal Learning

Ask PDF