ASD-FEAT: A Multi-Modal Infant Video-Derived Dataset for Early ASD Risk Prediction

Sidrah Liaqat; Halil Helvaci; Sen-Ching Cheung; Chongruo Wu; Dongjie Chen; Chen Nee Chuah; Sally Ozonoff

cs.CVOct 2, 2026

ASD-FEAT: A Multi-Modal Infant Video-Derived Dataset for Early ASD Risk Prediction

Authors: Sidrah Liaqat, Halil Helvaci, Sen-Ching Cheung, Chongruo Wu, Dongjie Chen, Chen Nee Chuah, Sally Ozonoff

Organizations: University of Kentucky · Department of Electrical and Computer Engineering University of Kentucky Lexington, KY 40506, USA · University of California Davis · Department of Electrical and Computer Engineering University of California Davis, CA 95616, USA · MIND Institute, University of California Davis · Department of Psychiatry and Behavioral Sciences MIND Institute University of California Davis, CA 95616, USA

Abstract

Accurate early screening for Autism Spectrum Disorder (ASD) is a precursor to timely intervention, which is critical for improving cognitive and behavioral outcomes. We present ASD-FEAT (ASD - Feature Extraction And Tracking), a multimodal dataset derived from video recordings of infant-adult interaction sessions. The key contribution of ASD-FEAT is the combination of longitudinal coverage from infancy through 36 months, repeated interaction sessions, clinically validated developmental outcomes, expert frame-level behavioral annotations, and privacy-conscious multimodal feature representations. To the best of our knowledge, existing ASD behavioral datasets do not jointly provide these characteristics at comparable scale. To demonstrate the utility of ASD-FEAT, we use it to evaluate a computer-vision-based end-to-end pipeline relying on machine learning techniques to automatically identify ASD risk. ASD-FEAT integrates both expert-defined and deep-learned features, including face and eye landmarks, facial action units, gaze direction, head position, mel-spectrogram audio representations, and optical flow, to identify behavioral markers of social interaction. Our automated pipeline achieves an ASD classification accuracy of 76.2% and an Area Under the Receiver Operating Characteristic (AUROC) of 0.82, compared to classifiers trained on manually labeled behaviors, which yielded 81.3% accuracy and an AUROC of 0.88. We further introduce a within-visit partner contrast: a per-visit signal contrasting examiner-directed and parent-directed social behavior which, when added to the classifier, lifts the fully automated Look Face + Smile and Look Face + Vocal configurations to Matthews correlations of 0.49 and 0.45 respectively, exceeding the human-coded single-partner baseline of 0.42.

Figures & tables

Dataset	No. of Subjects	Video Source	Ages (m)	Video Duration	Annotated Behaviors	ASD Diagnostic Method
MMDB ( Rehg et al., 2013 )	121	Lab	15–30	160 videos (5 min each)	Gaze, facial exp, gestures, vocal.	Rapid ABC
DE-ENIGMA ( Riva and Riva, 2020 )	50 (25 ASD)	Lab	60–144	$\sim$ 250 videos (15–20 min)	Affect, body gestures, vocal.	ADOS-2
ChildPlay Gaze ( Tafasca et al., 2023 )	95 videos (401 clips)	YouTube	36–144	$\sim$ 67 min (total duration)	Gaze point/class	–
MMASD ( Li et al., 2023 )	32 ASD	Interv.	–	1315 videos (7 sec each)	11 Body actions	ADOS-2
ASD-FEAT (ours)	318 (54 ASD)	Lab	6–36	1886 videos (3 min each)	Gaze to face/obj, smile, vocal.	ADOS-2

Table 1: Comparative analysis of multimodal datasets for pediatric ASD research, detailing sample sizes, video durations, behavioral annotations, and diagnostic assessment methods.

Figure 1: Computer-vision-based ASD risk prediction pipeline. Following pre-processing, the machine learning stage detects fine-grained ASD-related behaviors using features derived from clinical interaction videos. The last stage predicts ASD risk based on manually annotated or machine-predicted behavioral markers. To protect subject privacy, the images in this figure are generated examples rather than frames from actual clinical videos.

Behavior	Operational definition applied during manual coding
Look Face	(a) The child’s face is on-screen and directly facing the partner’s face, or (b) a portion of the partner’s body is visible to contextualize facial orientation, or (c) a gaze shift coincides with an off-screen partner’s voice onset.
Look Object	The child’s gaze is directed towards objects that: (a) are part of the current experimental task, or (b) were previously utilized in the task, or (c) were presented by the examiner or parent during the session, or (d) serve as potential reinforcers (e.g., toy cars, comfort objects).
Smile	(a) Observable upturning of at least one mouth corner, or visible eye muscle engagement, or (b) audible expressions of joy such as chuckling or explosive vocal sounds. The manual coding distinguishes genuine positive affect from incidental facial movements.
Vocalization	(a) non-verbal sounds (open vowels, consonant-vowel combinations), or (b) single word or word approximation articulations, or (c) multi-word phrases, or (d) communicative air gasps signaling excitement. Non-communicative physiological sounds are excluded.
Social Smile	Any smile event co-occurring or occurring within a two seconds of a look face event
Social Vocalization	Any vocalization event co-occurring or occurring within two seconds of look face

Table 2: Behavioral Coding Definitions

Age (m)	Total N	Interactive play with		Male	Female	ASD	Non-ASD	% ASD
		Examiner	Parent
6	334	84	250	181	153	0	334	0%
9	140	0	140	82	58	0	140	0%
12	477	119	358	259	218	4	473	0.8%
18	312	112	200	148	164	38	274	12.2%
24	293	98	195	157	136	41	252	14.0%

Table 3: Sample sizes for each assessment timepoint (6, 9, 12, 18, 24, 36 months). The dataset, comprising of 318 unique subjects, is longitudinal, featuring visits at different ages with up to three interactive play sessions recorded per visit for most subjects. In the age rows, all counts (including ASD/Non-ASD) are numbers of recorded play interactions and sum to Total N.

Category	Sub-Type	Field Details & Description
Gaze & Layout	3D Gaze	gaze0-2 : 3D gaze direction vector from Gaze360.
	Adult Head	tl_x, tl_y, br_x, br_y : Bounding box coordinates for the adult partner’s head.
Facial	Landmarks	C_x/y_0...67 : 68-point 2D facial landmarks (jaw, nose, eyes, lips).
	Eye Lmk.	C_eye_lmk_x/y : 56-point 2D landmarks for the eye region.
	Action Units	C_AUxx_r/c : 17 AU intensities and 18 AU presence flags (presence set includes AU28_c , lip suck, with no corresponding intensity channel).
Scene	Objects	toytlx, toytly, toybrx, toybry : Bounding box coordinates for toys/objects.

Table 4: Comprehensive specification of features in the ASD-FEAT dataset.

Category	Sub-Type	Description (Field Names Included)
Ground Truth	Behavior Signals	look_face, look_object, smile, vocal : Frame-level binary labels for gaze target, social smiles, and vocalizations.
	Frame Metadata	frame, timestamp, valid, C_confidence : Frame indices, validity flags, and tracking confidence scores.
	Session Metadata	sub_id, visit, partner : De-identified study-local subject identifier; visit age in months (6, 9, 12, 18, 24, 36); interacting adult for the session (Examiner or Parent).
	Diagnostic Outcome	asd_longitudinal, asd_36 : Best-estimate clinical diagnosis (ASD or Not ASD) at each visit ( asd_longitudinal ) and the final 36-month outcome ( asd_36 ).

Table 5: Ground truth labels available in the ASD-FEAT dataset.

Figure 2: (Left) The standard 68-point facial landmark configuration used for tracking the jawline, eyebrows, nose, and mouth. (Right) A detailed 56-point eye landmark model, including eyelid contours and iris boundaries, utilized for robust gaze estimation.

Tool / Model	License	Scope
OpenFace 2.0 ( Baltrusaitis et al., 2018 )	Custom academic, non-commercial	Facial landmarks, eye landmarks, head pose, action units
Gaze360 ( Kellnhofer et al., 2019 )	CC BY-NC 4.0	3D gaze direction estimation
I3D pretrained on Kinetics ( Carreira and Zisserman, 2017 )	Apache 2.0 (code); Kinetics labels CC BY 4.0	RGB and optical-flow spatiotemporal embeddings
librosa ( McFee et al., 2015 )	ISC	Mel-spectrogram extraction from session audio
ResNet-50 (vocalization classifier backbone)	BSD 3-Clause (torchvision)	Mel-spectrogram–based vocalization detection
DBSCAN (scikit-learn) ( Ester et al., 1996 )	BSD 3-Clause	Event clustering on frame-level probability streams

Table 6: Upstream tools and pretrained models used to construct ASD-FEAT features, with licensing terms relevant to downstream redistribution and use.

Figure 4: Input features (face and eye landmarks, facial AUs, head and object position, eye gaze) are encoded through MLP and concatenated together to form input to the transformer model. The transformer model output passed through an MLP resulting in frame level target behavior prediction. To protect subject privacy, the images in this figure are generated examples rather than frames from actual clinical videos.

Parameter	Value	Description
eps ( $\epsilon$ )	35 frames ( $\approx$ 1.17 sec)	Maximum temporal distance between candidate frames to be considered neighbors; equivalently, the largest gap bridged when merging detections into a single event
min_samples	8 frames ( $\approx$ 0.27 sec)	Minimum number of candidate frames to form an event cluster; equivalently, the shortest retained event duration

Table 7: Parameters for DBSCAN clustering

Beha- vior	Features							Metrics
	Face & eye lmks	Facial Action Units	Head Position	Object Position	3D Gaze	I3D (RGB, Flow)	Mel Spec.	PPV (%)	Recall (%)	AUC ROC
	$\checkmark$	$\checkmark$	$\checkmark$					36.2	50.6	0.67
Look Face	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$				37.9	57.2	0.68
	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$			49.7	57.1	0.72
	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$		48.4	67.3	0.76
Look Object	$\checkmark$	$\checkmark$	$\checkmark$					84.8	59.2	0.64

Table 8: Performance of our transformer-based model on Look Face, Look Object, Smile and Vocalization classification tasks. Model performance is evaluated incrementally as the following feature modalities are fused: Face & Eye Landmarks, Facial Action Units, 3D Gaze, Head Position, Object Position, I3D (RGB and Flow), and Mel Spectrogram. For comparison, vocalization classification was performed using a ResNet50-based model.

	Sens (%)	Spec (%)	PPV (%)	AUROC	Acc (%)
LR	69.3	86.0	29.1	0.81	84.4
	(64.3–74.3)	(83.5–88.5)	(24.5–33.7)	(0.78–0.84)	(82.2–86.6)
RF	61.1	84.2	39.6	0.78	81.2
	(54.3–67.9)	(80.0–88.4)	(34.3–44.9)	(0.74-0.82)	(77.8–84.6)
MLP	76.0	85.9	31.2	0.81	85.1
	(72.5–79.5)	(83.5–88.3)	(26.8–35.6)	(0.79–0.83)	(82.9–87.3)

Table 9: Performance metrics (sensitivity, specificity, positive predictive value, AUROC, and accuracy) of Logistic Regression (LR), Random Forest (RF), and Multilayer Perceptron (MLP) models for ASD behavior classification on ground truth labels. Models were trained on a balanced dataset using SMOTE and Tomek Links to address the 93:7 Non-ASD to ASD class imbalance, and evaluated via stratified 10-fold cross-validation. Values in parentheses represent the 95% confidence interval. These results use an earlier 10-fold CV protocol for classifier selection; the ground-truth baseline used in the subsequent component-validation, calibration, and within-visit partner-contrast analyses is the ground-truth row of Table 10 , computed under a new protocol.

	Sens (%)	Spec (%)	PPV (%)	AUROC	Acc (%)
Look Face from ML	76.3 (67.7–84.9)	79.9 (77.4–82.4)	23.5 (19.8–27.2)	0.87 (0.83–0.90)	79.7 (77.2–82.1)
Look Object from ML	80.9 (75.0–86.9)	80.1 (77.7–82.5)	24.7 (21.8–27.5)	0.88 (0.85–0.91)	80.2 (78.1–82.3)
Smile from ML	80.7 (74.6–86.9)	79.7 (77.1–82.2)	24.4 (21.5–27.4)	0.88 (0.85–0.90)	79.8 (77.5–82.0)
Vocal from ML	81.5 (74.8–88.2)	79.8 (76.9–82.7)	24.9 (21.7–28.1)	0.87 (0.84–0.90)	79.9 (77.2–82.6)
Look Face + Look Object	77.0 (68.6–85.5)	79.6 (77.1–82.1)	23.6 (19.6–27.7)	0.85 (0.81–0.89)	79.5 (76.8–82.2)
Look Face + Smile	81.0 (71.5–90.4)	78.4 (76.0–80.9)	23.3 (19.3–27.3)	0.86 (0.83–0.90)	78.6 (76.0–81.2)

Table 10: Component-validation sweep: ASD risk prediction when one or more ground-truth behavior channels are replaced by their ML-estimated counterparts, benchmarked against the all-ground-truth model. Sensitivity, specificity, PPV, AUROC, and accuracy are reported as mean over subject-disjoint stratified cross-validation folds (9 evaluation folds, 10-run ensemble per fold). Values in parentheses are 95% confidence intervals. Single-channel substitutions are near-lossless; degradation is confined to the fully ML based configuration.

Behavior	18m AUROC ( $\Delta$ )	24m AUROC ( $\Delta$ )	36m AUROC ( $\Delta$ )
smile rate	0.33 ( $-$ 0.06) [0.16–0.50]	0.28 (+0.08) [0.15–0.41]	0.59 (+0.23) [0.46–0.72]
smile prop	0.47 (+0.04) [0.21–0.73]	0.65 (+0.10) [0.49–0.80]	0.66 (+0.11) [0.55–0.78]
lookface rate	0.57 (+0.93) [0.36–0.78]	0.67 (+1.86) [0.58–0.77]	0.65 (+1.52) [0.55–0.75]
lookface prop	0.25 (+0.02) [0.13–0.38]	0.60 (+0.05) [0.49–0.71]	0.60 (+0.02) [0.49–0.71]
lookobject rate	0.56 (+0.78) [0.32–0.80]	0.65 (+1.56) [0.53–0.76]	0.65 (+1.55) [0.54–0.75]
lookobject prop	0.29 ( $-$ 0.02) [0.15–0.43]	0.59 ( $-$ 0.03) [0.46–0.72]	0.49 (+0.03) [0.35–0.63]

Table 11: Per-age cross-validated AUROC and raw effect magnitude (

\Delta=\bar{c}_{\text{ASD}}-\bar{c}_{\text{Non}}

events/min, Eq. 3 , under asd_longitudinal ) of the within-visit partner contrast on the twelve-feature set. AUROC values are mean over stratified folds (10-fold at 24m and 36m); 95% CIs in square brackets are derived from the fold distribution. Bold indicates AUROC 95% CI lower bound

\geq 0.55

. The 18-month column uses 9-fold CV, as only

n_{\text{ASD}}=9

dual-session instances are present at 18m; its AUROC estimates are correspondingly noisier (wider CIs) and no 18m cell clears the bolding threshold, but the raw effect magnitudes

\Delta

are reported on the same footing as the later ages.

		ASD			Non-ASD
Behavior	Age	Parent	Examiner	P $-$ E	Parent	Examiner	P $-$ E	$\Delta$
smile rate	24m	0.94	0.83	$+0.11$	1.63	1.60	$+0.03$	$+0.08$
	36m	1.39	1.38	$+0.01$	1.98	2.21	$-0.23$	$+0.23$
smile prop	24m	0.09	0.05	$+0.04$	0.14	0.21	$-0.07$	$+0.10$
	36m	0.15	0.14	$+0.01$	0.23	0.33	$-0.10$	$+0.11$
lookface rate	24m	1.94	2.17	$-0.23$	3.85	5.93	$-2.08$	$+1.86$

Table 12: Partner-stratified mean behavior rates (events/min for rate features; proportions for prop features) from ground-truth, expert-coded behaviors, by ASD label and age, in the matched dual-partner cohort (

n=188

subject-visit instances at 24m/36m, 32 ASD). For each group, Parent and Examiner are the per-(subject, visit) session means and P

-

E their within-group difference. The final column,

\Delta=(\text{ASD }P{-}E)-(\text{Non-ASD }P{-}E)=\bar{c}_{\text{ASD}}-\bar{c}_{\text{Non-ASD}}

(Eq. 3 ), is the between-group difference, identical to the

\Delta

reported in Table 11 . All quantities are rounded independently to two decimals, so within-row differences may differ from the displayed

\Delta

by up to 0.01.

	Ground Truth (G)			All ML (FOSV)
Behavior	18m	24m	36m	18m	24m	36m
smile rate	0.33 ( $-0.06$ ) [0.16, 0.50]	0.28 ( $+0.08$ ) [0.15, 0.41]	0.59 ( $+0.23$ ) [0.46, 0.72]	0.62 ( $+0.38$ ) [0.42, 0.82]	0.63 ( $+0.58$ ) [0.52, 0.75]	0.57 ( $+0.39$ ) [0.50, 0.64]
smile prop	0.47 ( $+0.04$ ) [0.21, 0.73]	0.65 ( $+0.10$ ) [0.49, 0.80]	0.66 ( $+0.11$ ) [0.55, 0.78]	0.54 ( $+0.06$ ) [0.31, 0.76]	0.67 ( $+0.08$ ) [0.54, 0.79]	0.67 ( $+0.10$ ) [0.51, 0.83]
lookface rate	0.57 ( $+0.93$ ) [0.36, 0.78]	0.67 ( $+1.86$ ) [0.58, 0.77]	0.65 ( $+1.52$ ) [0.55, 0.75]	0.75 ( $-1.29$ ) [0.57, 0.92]	0.67 ( $+1.32$ ) [0.52, 0.81]	0.60 ( $+0.52$ ) [0.47, 0.72]
lookface prop	0.25 ( $+0.02$ ) [0.13, 0.38]	0.60 ( $+0.05$ ) [0.49, 0.71]	0.60 ( $+0.02$ ) [0.49, 0.71]	0.57 ( $+0.02$ ) [0.39, 0.75]	0.68 ( $+0.06$ ) [0.61, 0.75]	0.42 ( $-0.01$ ) [0.32, 0.52]
lookobject rate	0.56 ( $+0.78$ ) [0.32, 0.80]	0.65 ( $+1.56$ ) [0.53, 0.76]	0.65 ( $+1.55$ ) [0.54, 0.75]	0.58 ( $+0.60$ ) [0.42, 0.74]	0.55 ( $+0.54$ ) [0.39, 0.70]	0.46 ( $-0.26$ ) [0.29, 0.63]

Table 13: Per-behavior within-visit partner-contrast AUROC under ML substitution: ground truth (G, identical to Table 11 ) vs. fully ML-estimated (FOSV) across 18, 24, and 36 months. Each cell shows AUROC (raw effect

\Delta=\bar{c}_{\text{ASD}}-\bar{c}_{\text{NonASD}}

, Eq. 3 ) over 95% CI. Bold = AUROC 95% CI lower bound

\geq 0.55

. The 18-month columns use reduced-fold CV (9-fold for G, 8-fold for FOSV, reflecting

n_{\text{ASD}}=9

and

8

dual-session instances respectively) and are correspondingly noisier.

\Delta

for FOSV is measured in ML-detected behavior units, which may differ in scale from human-coded units (e.g., ML lookface events/min vs. GT lookface events/min); the comparison is qualitative across columns and quantitative within column.

	Dim	Sens (%)	Spec (%)	PPV (%)	AUROC	Acc (%)
Ground Truth	26	63.3 (47.8–78.9)	85.1 (81.5–88.6)	42.1 (33.1–51.1)	0.83 (0.76–0.89)	82.1 (78.3–85.9)
Ground Truth	38	70.4 (60.5–80.2)	86.9 (81.7–92.1)	49.4 (40.6–58.3)	0.85 (0.79–0.91)	84.5 (80.4–88.6)
Look Face	26	60.2 (45.7–74.6)	82.8 (77.5–88.0)	40.2 (31.7–48.8)	0.79 (0.74–0.84)	79.4 (75.7–83.1)
Look Face	38	75.6 (59.2–91.9)	80.0 (74.4–85.6)	40.4 (32.7–48.1)	0.85 (0.80–0.89)	79.2 (74.8–83.6)
Look Object	26	63.3 (47.8–78.9)	80.6 (75.0–86.2)	37.0 (28.7–45.2)	0.81 (0.75–0.87)	78.2 (73.8–82.5)
Look Object	38	73.3 (56.7–90.0)	84.6 (79.5–89.8)	45.0 (37.1–52.9)	0.84 (0.80–0.88)	82.8 (78.7–86.9)

Table 14: Stage 2 risk prediction on the matched dual-partner cohort (279 subject-visit instances from 118 subjects, 41 ASD): standard 26-dim feature vector vs. 38-dim model augmented with the twelve within-visit partner-contrast features. Rows use ground-truth behaviors or the indicated ML substitution. Mean over subject-disjoint folds; 95% CI (

\pm 1.96\,\mathrm{SE}

) in parentheses. Bold marks Look Face + Smile and Look Face + Vocal, the two-channel ML configurations highlighted in the discussion (extended metrics in Table 15 , Panel C).

	MCC	LR +	LR -	[email protected]% (%)
Panel A: full cohort (component-validation sweep)
Ground Truth	0.389	4.45	0.23	12.8 (11.3–14.2)
Look Face	0.347	3.94	0.30	11.4 (9.6–13.3)
Look Object	0.376	4.18	0.24	12.1 (10.7–13.4)
Smile	0.370	4.07	0.24	11.8 (10.6–13.0)
Vocal	0.377	4.21	0.23	12.2 (10.5–13.9)

Table 15: Prevalence-calibrated and imbalance-robust metrics. [email protected]% recalibrates positive predictive value to the U.S. general-population ASD prevalence via Bayes’ rule (95% CI in parentheses);

\mathrm{LR}^{+}

\mathrm{LR}^{-}

are prevalence-independent likelihood ratios; MCC is the Matthews correlation coefficient evaluated at the cohort prevalence. Panel A: full Stage 2 cohort (Table 10 protocol). Panels B–C: matched dual-partner cohort, 26-dim baseline vs. 38-dim partner-contrast-augmented (Table 14 protocol).

Explore similar work

Jun 25, 2026

Fine-tuning a multimodal large language model for clinician-grade autism behavioral scoring from short home videos

Autism spectrum disorder (ASD) affects 1 in 31 US children, yet median age at diagnosis exceeds four years. Artificial intelligence pipelines that provide quantified diagnosis using easy to access observational data (e.g., home videos) could help with earlier diagnosis, and timely delivery of early treatments. We fine-tuned Gemini 2.5 Pro on 400 clinician-rated home videos with low-rank adaptation, training only on 30 behavioral features previously validated to produce reliable predictions when passed to various ML models. On 99 held-out children (49 ASD, 50 neurotypical), inter-rater reliability with clinicians (per-feature weighted Cohen's kappa) improved by 40% (p<0.001), with 27 of 28 evaluable features improving. As an emergent zero-shot capability, direct ASD diagnosis F1 improved by 53% (p<0.001), matching or exceeding clinician outcomes. Classifier-assisted pipelines using fine-tuned LLM-derived behavioral features matched clinician-scored inputs across all tested pathways and achieved 77% accuracy (95% CI: 68-85%) and an AUC of 86% (95% CI: 78-92%). Fine-tuned multimodal LLMs can serve as scalable behavioral feature extractors for use in autism assessment and diagnosis.

Mohammadmahdi Honarmand, Parnian Azizian, Aaron Kline +9

Department of Mechanical Engineering, Stanford University, Stanford, CA 94305, USA · Department of Pediatrics, Stanford University, Stanford, CA 94305, USA · Department of Biomedical Data Science, Stanford University, Stanford, CA 94305, USA +2

Autism Spectrum Disorder LLM Fine-Tuning Multimodal Large Language Models

Ask PDF

Sep 15, 2026

A multimodal large language model for evidence-based autism spectrum disorder screening

The clinical management of autism spectrum disorder (ASD) faces a bottleneck in early screening, mainly because trained specialists are scarce and conventional assessment tools are subjective. Here, we introduce ASDchat, a multimodal large language model designed for evidence-based ASD screening, which takes video, audio, and dialogue as input. ASDchat adopts a dual-branch architecture, where the decision branch generates screening probabilities and the evidence branch generates traceable, timestamped behavioral evidence aligned with the Autism Diagnostic Observation Schedule (ADOS-2). The model was trained and evaluated on a dataset of 1,035 participants from 27 sites in China, which covered typically developing (TD) children, children with ASD, and children with other developmental disorders. For ASD versus TD, ASDchat reached an area under the receiver operating characteristic curve (AUC) of 0.953

\pm

0.021. On 6 held-out sites that were not used for training, the mean AUC was 0.961. Furthermore, unsupervised clustering of the behavioral dimensions split the ASD cases into six subtypes with different phenotypic profiles, and ASDchat suggests an intervention for each subtype. ASDchat provides a feasible path for large-scale, evidence-based ASD screening in clinical practice.

Jun Chen, Qi Zhao, Yunliang Jiang +10

National Special Education Resource Center for Children with Autism, Zhejiang Normal University, China · School of Computer Science and Technology, Zhejiang Normal University, China · School of Mathematics and Statistics, Xi’an Jiaotong University, China +5

Autism Spectrum Disorder Multimodal Large Language Models Multimodal Classification

Ask PDF

Oct 6, 2026

A Deterministic Evidence Layer for Vision-Language Autism Screening from Naturalistic Home Video

Autism spectrum disorder (ASD) is diagnosed through specialist observation of a child's social behavior, and access to that expertise is the bottleneck for early identification. Vision-language models (VLMs) describe a child's behavior from video well; the verdict drawn from the description is unstable: at temperature~0, across eight pipeline configurations on one backbone, 16--37% of clips change their predicted label between repeated runs, and the cause lies in the serving stack. We keep the VLM frozen and move the decision out of the model. Under our grounded perception constraints the VLM writes an event table of timestamped, glossary-labeled events that names the eliciting press and logs counter-evidence; a text-only stage age-calibrates the confidence of each row; a deterministic weight-of-evidence scorer sums it into an evidence total and stratifies it into a risk category, so every decision decomposes into named per-feature contributions and can be re-scored from the saved table. On 43 caregiver-recorded, protocol-free home free-play clips of preschool children, the pipeline reaches AUC

0.851 \pm 0.012

, 86.0% accuracy, and F

_1

71.8 over three runs. It labels 74.4% of clips correctly in every run (60.5% for the zero-shot baseline) and flags no typically developing clip in every run (9 of 31 at zero-shot). An ablation on the same backbone attributes the gain to the grounded perception constraints read through the deterministic scorer.

Wenqi Li, Mindi Ruan, Chuanbo Hu +2

Department of Computer Science, University at Albany, SUNY, Albany, NY, USA · Lane Department of Computer Science and Electrical Engineering, West Virginia University, Morgantown, WV, USA · Department of Radiology, Washington University in St. Louis, St. Louis, MO, USA

Autism Spectrum Disorder Medical VLMs

Ask PDF