EDA: Observe, Verify, and Assess for Evidence-Grounded Multimodal Depression Analysis
Organizations: Northeastern University · Northeastern University at Qinhuangdao · Technical University of Munich · Shenzhen MSU-BIT University
Abstract
Existing multimodal depression assessment methods often summarize long recordings into global behavioral descriptions. This leaves the source and decision relevance of individual observations difficult to inspect. We propose EDA, an evidence-grounded framework organized around Observe, Verify, and Assess. In Observe, literature-informed acoustic and visual cues are localized into source-linked Evidence Records with temporal intervals and behavioral measurements. In Verify, frozen multimodal agents examine whether the cited media support each observation. The simplified Support admission policy retains positively verified records and their review grounds. In Assess, statistical descriptors and temporal events support reference-based behavioral comparison, while PHQ-9-guided text analysis provides complementary symptom evidence. Cross-modal disagreement triggers review of the original text before final selection. Experiments on D-Vlog and LMVD evaluate human-referenced cue agreement and downstream assessment utility separately. AI cue annotations achieve 91.02% accuracy against human consensus labels on shared source segments. With a frozen 7B backbone, EDA Support improves F1 over direct prediction by 7.61 and 9.88 percentage points on D-Vlog Leave-Query-Out and LMVD, respectively. These findings support explicit, source-linked behavioral evidence for inspectable multimodal depression assessment.
Figures & tables
| D-Vlog test | LMVD | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Method | Year | Acc. | Prec. | Rec. | F1 | Acc. | Prec. | Rec. | F1 |
| Depression Detector [ 32 ] | 2022 | – | 65.40 | 65.57 | 63.50 | – | – | – | – |
| TAMFN [ 40 ] | 2023 | – | 66.02 | 66.50 | 65.82 | – | – | – | – |
| Attention-MIL [ 21 ] | 2024 | – | 67.27 | 67.77 | 66.64 | – | – | – | – |
| Depressformer [ 10 ] | 2024 | 65.00 | 64.00 | 54.00 | 59.00 | – | – | – | – |
| DepMamba [ 31 ] | 2025 | 68.87 | 68.19 | 86.99 | 76.44 | 72.13 | 70.18 | 76.56 | 73.20 |
| D-Vlog Leave-Query-Out | LMVD | ||||
|---|---|---|---|---|---|
| Backbone | Workflow | Acc. | F1 | Acc. | F1 |
| 7B | Direct | 87.56 | 86.89 | 76.41 | 72.58 |
| 7B | DepressionAgent | 86.79 | 88.36 | 76.74 | 77.18 |
| 7B | EDA Support | 94.30 | 94.50 | 83.10 | 82.46 |
| 7B | EDA Evidence | 94.17 | 94.38 | 82.89 | 82.21 |
| 30B-A3B | Direct | 81.35 | 84.87 | 74.60 | 79.06 |
| D-Vlog Leave-Query-Out | LMVD | |||
|---|---|---|---|---|
| Representation | Acc. | F1 | Acc. | F1 |
| Statistical Descriptors | 67.36 | 70.35 | 69.28 | 69.47 |
| Temporal Events | 59.97 | 63.86 | 62.26 | 66.01 |
| Joint Statistical–Event Representation | 67.10 | 70.26 | 69.56 | 70.27 |
| Admission | Cue Retention (%) | AV F1 | AVT F1 |
|---|---|---|---|
| D-Vlog Leave-Query-Out | |||
| Evidence review | 76.38 | 70.26 | 94.38 |
| EDA Support | 77.64 | 71.63 | 94.50 |
| LMVD | |||
| Evidence review | 45.94 | 70.27 | 82.21 |
| EDA Support | 53.34 | 70.35 | 82.46 |
| D-Vlog Leave-Query-Out | LMVD | |||
|---|---|---|---|---|
| Trigger | Acc. | F1 | Acc. | F1 |
| Completion | 93.52 | 93.69 | 82.72 | 81.91 |
| Random | 93.91 | 94.08 | 82.47 | 81.60 |
| Uncertainty | 93.52 | 93.69 | 82.72 | 81.78 |
| AV disagreement | 94.17 | 94.38 | 83.16 | 82.51 |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| ID | Observable cue | Annotation criterion | Literature basis |
|---|---|---|---|
| A01 | Low loudness variation | Limited fluctuation in vocal intensity within a localized speech interval | Vocal emphasis/prosody [ 7 ] |
| A02 | Slow pitch change | Gradual pitch changes across voiced speech intervals | Pitch dynamics [ 19 , 20 ] |
| A03 | Slow articulation | Reduced articulation pace in an observable speech segment | Speech timing [ 2 ] |
| A04 | Long pause | A sustained silence between speech segments, interpreted in speech context | Pause timing [ 30 , 18 ] |
| A05 | Repeated long pauses | Multiple prolonged pauses grouped within a local speech context | Pausing/fluency [ 1 ] |
| V01 | Infrequent lip-corner lowering | Low occurrence of visibly lowered lip corners across valid facial observations | Facial activity [ 5 ] |
| Cue | Behavioral annotation or measurement |
|---|---|
| A01: Low loudness variation | Compute RMS loudness in dBFS and summarize its P90–P10 range, standard deviation, interquartile range, and adjacent changes to localize intervals with limited loudness variation. |
| A02: Slow pitch change | Compute the absolute difference between adjacent valid F0 frames, normalized by their mean F0 and temporal separation, to obtain the relative F0 change rate (%/s) and localize intervals with slow changes. |
| A03: Slow articulation | Divide the number of linguistic units by VAD speech duration to obtain an articulation-rate proxy. English uses words and Chinese uses Han characters; low-rate intervals are localized within the same language and counting unit. |
| A04: Long pause | Use speech activity boundaries to identify silence between adjacent speech segments within the same response, retaining pauses lasting at least 0.5 s. |
| A05: Repeated long pauses | Reuse A04 pause events to localize at least two long pauses separated by valid speech within the same response. |
| V01: Infrequent lip-corner lowering | Compute the proportion of AU15-positive valid frames among all valid AU15 frames to localize intervals with a low occurrence proportion of lip-corner lowering. |
| Cue | Measurement and candidate threshold | Valid-observation criteria |
|---|---|---|
| A01: Low loudness variation | Loudness P90–P10 ; SD dB; IQR dB; maximum adjacent change dB | Speech proportion |
| A02: Slow pitch change | Mean relative F0 change rate (%/s) | Valid-F0 proportion ; at least 3 baseline windows |
| A03: Slow articulation | Linguistic units per VAD speech second ; candidate duration s | Voiced duration s; at least 3 units; alignment quality ; at least 3 baseline intervals with the same language and counting unit |
| A04: Long pause | Within-response pause s | RMS-confirmed silence s; overlap ratio with VAD silence |
| A05: Repeated long pauses | At least 2 A04 pauses within one response; intervening speech s; local candidate duration s | Response duration s |
| V01: Infrequent lip-corner lowering | AU15-positive valid-frame proportion | Valid-face proportion |
| Cue | Sample-level statistics | Event-level measurements |
|---|---|---|
| A01 | Median of within-window loudness P90–P10 | Loudness P90–P10 (dB) |
| A02 | Median of within-window mean relative F0 change rate | Mean relative F0 change rate (%/s) |
| A03 | Median articulation rate within the same language and counting unit | Articulation rate (words/s or Han characters/s) |
| A04 | Pause-union duration divided by response-proxy duration; median pause duration; events per minute | Duration only |
| A05 | Pause-union duration divided by response-proxy duration | Pause count; total pause duration divided by event duration |
| V01 | AU15-positive valid-frame proportion | AU15-positive valid-frame proportion |
| Method | Acc. | Prec. | Rec. | F1 |
|---|---|---|---|---|
| DepressionAgent (30B-A3B) | 92.43 | 92.73 | 93.67 | 93.20 |
| EDA Support (7B) | 94.30 | 95.94 | 93.10 | 94.50 |
| EDA Evidence (7B) | 94.17 | 95.70 | 93.10 | 94.38 |
| D-Vlog Leave-Query-Out | D-Vlog test | LMVD | |||||
|---|---|---|---|---|---|---|---|
| System | Modality | Acc. | F1 | Acc. | F1 | Acc. | F1 |
| DepressionAgent (30B-A3B) | A | 57.13 | 39.75 | – | – | – | – |
| V | 44.68 | 0.00 | – | – | – | – | |
| AV | 63.84 | 54.80 | – | – | – | – | |
| T | 89.60 | 90.38 | – | – | – | – | |
| AVT | 92.43 | 93.20 | – | – | – | – | |
| D-Vlog Leave-Query-Out | D-Vlog test | LMVD | |||||
|---|---|---|---|---|---|---|---|
| Backbone | Workflow | Acc. | F1 | Acc. | F1 | Acc. | F1 |
| 7B | Direct | 87.56 | 86.89 | 88.89 | 88.48 | 76.41 | 72.58 |
| 7B | DepressionAgent | 86.79 | 88.36 | 87.72 | 89.23 | 76.74 | 77.18 |
| 7B | EDA Support | 94.30 | 94.50 | 95.32 | 95.45 | 83.10 | 82.46 |
| 7B | EDA Evidence | 94.17 | 94.38 | 95.32 | 95.45 | 82.89 | 82.21 |
| 30B-A3B | Direct | 81.35 | 84.87 | 83.63 | 86.54 | 74.60 | 79.06 |
| D-Vlog Leave-Query-Out | D-Vlog test | LMVD | ||||
|---|---|---|---|---|---|---|
| Representation | Acc. | F1 | Acc. | F1 | Acc. | F1 |
| Statistical Descriptors | 67.36 | 70.35 | 70.76 | 74.23 | 69.28 | 69.47 |
| Temporal Events | 59.97 | 63.86 | 66.08 | 69.79 | 62.26 | 66.01 |
| Joint Statistical–Event Representation | 67.10 | 70.26 | 71.35 | 74.87 | 69.56 | 70.27 |
| Dataset / protocol | Initial T | Internal T review | Final AVT (Support) | |||
|---|---|---|---|---|---|---|
| Acc. | F1 | Acc. | F1 | Acc. | F1 | |
| D-Vlog Leave-Query-Out | 90.28 | 90.54 | 93.52 | 93.69 | 94.30 | 94.50 |
| D-Vlog test | 90.64 | 91.01 | 94.15 | 94.25 | 95.32 | 95.45 |
| LMVD | 78.55 | 78.28 | 81.51 | 81.87 | 83.10 | 82.46 |
| A | V | AV | AVT | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Admission | Intervals | Acc. | F1 | Acc. | F1 | Acc. | F1 | Acc. | F1 |
| Support | 99,166 | 67.10 | 68.64 | 61.79 | 64.76 | 68.91 | 71.63 | 94.30 | 94.50 |
| + Evidence | 97,553 | 67.10 | 68.56 | 61.53 | 64.77 | 67.10 | 70.26 | 94.17 | 94.38 |
| A | V | AV | AVT | |||||
|---|---|---|---|---|---|---|---|---|
| Admission | Acc. | F1 | Acc. | F1 | Acc. | F1 | Acc. | F1 |
| Support | 65.50 | 68.78 | 63.16 | 64.80 | 71.35 | 74.61 | 95.32 | 95.45 |
| Evidence | 66.08 | 69.47 | 64.33 | 67.03 | 71.35 | 74.87 | 95.32 | 95.45 |
| D-Vlog Leave-Query-Out | LMVD | |||
|---|---|---|---|---|
| Configuration | Acc. | F1 | Acc. | F1 |
| EDA Support | 94.30 | 94.50 | 83.10 | 82.46 |
| Without conflict review | 93.52 | 93.69 | 82.72 | 81.91 |
| Without confidence gate | 94.17 | 94.38 | 83.16 | 82.51 |
| Rule | D-Vlog Leave-Query-Out | D-Vlog test | LMVD | |||
|---|---|---|---|---|---|---|
| Acc. | F1 | Acc. | F1 | Acc. | F1 | |
| T (unclassified retained) | 93.52 | 93.69 | 94.15 | 94.25 | 81.51 | 81.87 |
| + completion | 93.52 | 93.69 | 94.15 | 94.25 | 82.72 | 81.91 |
| + source review | 94.17 | 94.38 | 95.32 | 95.45 | 83.16 | 82.51 |
| + final gate (AVT) | 94.30 | 94.50 | 95.32 | 95.45 | 83.10 | 82.46 |
| Protocol | Samples | Revised | Corrected | Harmed | Revised (%) | Beneficial (%, 95% CI) | Acc. (pp) |
|---|---|---|---|---|---|---|---|
| D-Vlog Leave-Query-Out | 772 | 8 | 7 | 1 | 1.04 | 87.50 [52.91, 97.76] | +0.78 |
| D-Vlog test | 171 | 2 | 2 | 0 | 1.17 | 100.00 [34.24, 100.00] | +1.17 |
| LMVD | 1,823 | 39 | 23 | 16 | 2.14 | 58.97 [43.42, 72.92] | +0.38 |
| Dataset / protocol | Audio hours | Video hours | Candidates | Support | Evidence |
|---|---|---|---|---|---|
| D-Vlog Leave-Query-Out | 123.85 | 123.85 | 127,719 | 99,166 | 97,553 |
| D-Vlog test | 25.96 | 25.96 | 27,079 | 20,992 | 20,699 |
| LMVD | 236.73 | 214.62 | 193,710 | 103,332 | 88,994 |
Depression Detection
Depression detection uses language, behavior, speech, and biological signals to identify depressive symptoms and estimate their severity.
See all 80 papers