InertiaMPDD: Multimodal Personalized Depression Detection via Emotional Inertia-Inspired Dynamics Modeling
Organizations: Northeastern University · The University of Osaka · Southeast University · Xi’an University of Posts and Telecommunications
Abstract
Multimodal depression screening estimates risk from speech, visual behavior and language. Current methods mainly predict labels, with limited evidence of departures from a non-depressed reference. Recording-level supervision also leaves fine-grained emotional dynamics unannotated. Motivated by emotional inertia, the persistence of affect, InertiaMPDD learns healthy next-state patterns using a world-model-inspired latent predictor. Using pretrained multimodal representations, it predicts from observed history and context. Reference deviation, residual decay rate and a Jacobian spectral descriptor form reference-derived emotional dynamics descriptors, evaluated as candidate proxy biomarkers for standalone screening and personalized decision correction. Across five cohorts from four datasets, fusion reaches 98.41% F1 on D-Vlog and 84.98% on LMVD. Standalone descriptors outperform fixed additive fusion on MPDD but discriminate weakly on AVEC2014. These findings distinguish reference informativeness from fusion benefit and reveal context-dependent utility.
Figures & tables
| Item | Value |
|---|---|
| Backbone | Qwen2.5-Omni-3B thinker (bf16, frozen) |
| LoRA | rank 8, trained from scratch, lr , batch 1 grad-accum 8, linear schedule |
| Bias head | Linear( ), fp32, AdamW lr (separate parameter group), init |
| IIP | online encoder , 3-layer FiLM causal Transformer, width 256, 4 heads, |
| feed-forward width 512, dropout 0.1, pre-normalization, fixed semi-orthogonal target | |
| Conditioning MLP | , SiLU, final-layer initialization |
| Cohort | Training | Test |
|---|---|---|
| MPDD-elder | ||
| MPDD-young | ||
| AVEC2014 |
| Cohort | Quantity | F1 / F1 | 95% CI |
|---|---|---|---|
| D-Vlog | NSP | 97.38 | [94.84, 99.47] |
| Full | 98.41 | [96.38, 100.00] | |
| Gain | +1.03 | [0.00, +2.76] | |
| AVEC2014 | NSP | 50.00 | [0.00, 85.71] |
| Full | 66.67 | [0.00, 100.00] | |
| Gain | +16.67 | [ , +80.00] |
| Configuration | D-Vlog | LMVD | AVEC 2014 | MPDD elder | MPDD young |
|---|---|---|---|---|---|
| InertiaMPDD (full) | 98.41 | 84.98 | 66.67 | 85.71 | 80.00 |
| NSP only | 97.38 | 74.57 | 50.00 | 76.92 | 66.67 |
| w/o conditioning | 97.87 | 75.07 | 50.00 | 72.73 | 66.67 |
| Global correction | 97.35 | 80.84 | 66.67 | 72.73 | 66.67 |
| Full IIP (audio + visual) | 69.65 | 75.88 | 50.00 | 92.31 | 90.91 |
| Shuffled history | 67.02 | 59.68 | 35.29 | 77.78 | 90.91 |
| Configuration | D-Vlog | LMVD | AVEC 2014 | MPDD elder | MPDD young |
|---|---|---|---|---|---|
| InertiaMPDD | 98.41 | 84.98 | 66.67 | 85.71 | 80.00 |
| w/o conditioning | 97.87 | 75.07 | 50.00 | 72.73 | 66.67 |
| Global context | 97.87 | 75.07 | 54.55 | 72.73 | 66.67 |
| Context swapping | 97.35 | 76.55 | 33.33 | 77.78 | 90.91 |
| Global correction | 97.35 | 80.84 | 66.67 | 72.73 | 66.67 |
| Configuration | D-Vlog | LMVD | AVEC2014 | MPDD-elder | MPDD-young | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| Accuracy | F1 score | Accuracy | F1 score | Accuracy | F1 score | Accuracy | F1 score | Accuracy | F1 score | |
| Full IIP (causal Transformer) | 64.00 | 69.65 | 73.70 | 75.88 | 62.50 | 50.00 | 95.45 | 92.31 | 90.91 | 90.91 |
| Static MLP | 65.68 | 69.15 | 67.60 | 70.36 | 56.25 | 0.00 | 68.18 | 66.67 | 100.00 | 100.00 |
| Persistence | 62.72 | 63.16 | 64.10 | 67.82 | 56.25 | 46.15 | 68.18 | 58.82 | 45.45 | 33.33 |
| Linear AR predictor | 63.31 | 67.37 | 61.50 | 65.50 | 75.00 | 33.33 | 68.18 | 53.33 | 81.82 | 81.82 |
| Shuffled history | 62.72 | 67.02 | 57.20 | 59.68 | 31.25 | 35.29 | 81.82 | 77.78 | 90.91 | 90.91 |
| Configuration | Accuracy | F1 score |
|---|---|---|
| Visual-only NSP | 73.14 | 75.39 |
| Audio-only NSP | 92.57 | 92.82 |
| Audio–visual NSP | 95.43 | 95.74 |
| Audio–visual–text NSP | 97.14 | 97.38 |
| Audio–visual NSP + IIP | 95.43 | 95.74 |
| InertiaMPDD | 98.29 | 98.41 |
| Correction input | D-Vlog | AVEC 2014 | MPDD elder | MPDD young |
|---|---|---|---|---|
| NSP only | 97.38 | 50.00 | 76.92 | 66.67 |
| NSP + | 97.89 | 66.67 | 77.78 | 85.71 |
| NSP + | 97.35 | 54.55 | 66.67 | 66.67 |
| NSP + | 97.87 | 50.00 | 72.73 | 70.00 |
| NSP + | 97.38 | 28.57 | 77.78 | 85.71 |
| NSP + | 97.89 | 28.57 | 77.78 | 85.71 |
| Decision rule | D-Vlog | LMVD | AVEC 2014 | MPDD elder | MPDD young |
|---|---|---|---|---|---|
| NSP only | 97.38 | 74.57 | 50.00 | 76.92 | 66.67 |
| IIP only | 69.65 | 75.88 | 50.00 | 92.31 | 90.91 |
| Global correction | 97.35 | 80.84 | 66.67 | 72.73 | 66.67 |
| InertiaMPDD | 98.41 | 84.98 | 66.67 | 85.71 | 80.00 |
| Scaling | 97.87 | 75.20 | 50.00 | 48.28 | 66.67 |
| Convex fusion | 97.38 | 77.06 | 60.00 | 93.33 | 100.00 |
| Control | Observed relation to the diagnostic reference |
|---|---|
| Zero or global context | Same test decisions as true context in one setting |
| Persistence predictor | Reproduces the conditional predictor’s test decisions |
| PCA / Mahalanobis scores | Match confusion counts, each differing on two recordings |
| Global / block time shuffling | Retain screening performance |
| Repeated mean-state input | Retains strong screening performance without temporal variation |
| Setting | LMVD | MPDD elder | MPDD young |
|---|---|---|---|
| IIP reference | 75.88 | 92.31 | 90.91 |
| Duration matched | 56.07 | 100.00 | 90.91 |
| AV interval masking | 80.65 | 87.50 | 85.71 |
| Healthy-pool subsampling | 70.54 | 73.68 | 95.24 |
| Cohort | Estimator | Mean relative error (%) | Spearman correlation | Runtime (s) | Peak memory (GiB) |
|---|---|---|---|---|---|
| D-Vlog | Exact eigendecomposition | 0 | 1 | 0.322 | 0.059 |
| D-Vlog | 30 JVP steps | 4.25 | 0.6086 | 0.204 | 0.058 |
| D-Vlog | 60 JVP steps | 5.32 | 0.5547 | 0.444 | 0.058 |
| D-Vlog | 120 JVP steps | 4.28 | 0.5542 | 0.761 | 0.058 |
| LMVD | Exact eigendecomposition | 0 | 1 | 0.370 | 0.059 |
| LMVD | 30 JVP steps | 4.53 | 0.9423 | 0.224 | 0.058 |
Depression Detection
Depression detection uses language, behavior, speech, and biological signals to identify depressive symptoms and estimate their severity.
See all 79 papers