VIS-DICT: A Visual Dictionary for Missing Modality Imputation in Social Network Depression Detection

Hamed Marvi; Mohammad Mehdi Keikha; Abolfazl Nadi

cs.CVSep 2, 2026

VIS-DICT: A Visual Dictionary for Missing Modality Imputation in Social Network Depression Detection

Authors: Hamed Marvi, Mohammad Mehdi Keikha, Abolfazl Nadi

Organizations: Department of Computer Science, School of Mathematics, Statistics and Computer Science, College of Science, University of Tehran, Tehran, Iran · Department of Computer Science, University of Sistan and Baluchestan, Zahedan, Iran

Abstract

Tracking social media posts can help spot early signs of depression. Recent studies show that combining text and images works better for detecting depression than using text alone. However, many social media posts do not have images, which makes it hard to use multimodal models. Most existing methods fill in missing images using retrieval or generative models that need extra training. In this paper, we introduce Vis-Dict, a dictionary-based method that builds missing visual features by linking words to average image vectors from complete training posts. These estimated visual features are then combined with text to track changes in user behavior over time. We tested Vis-Dict on a social media dataset using user timelines of up to 512 posts and compared it with other missing-data methods. The results show that Vis-Dict performs on par with generative networks, reaching an F1-score of 0.9454 and an ROC-AUC of 0.9890. Most importantly, Vis-Dict achieves this strong performance with zero trainable parameters for image generation. These findings show that directly connecting words to visual features is an effective and practical way to handle missing images in depression detection systems.

Figures & tables

Figure 1. Overall architecture of the tweet-level pipeline. The system extracts text and image representations ( $h_{\text{txt}},h_{\text{img}}\in\mathbb{R}^{768}$ ), estimates missing visual embeddings using five alternative imputation methods when an image is absent, projects both representations to $\mathbb{R}^{512}$ , and concatenates them into a unified vector ( $z_{\text{fused}}\in\mathbb{R}^{1024}$ ) for single-tweet classification and user-level sequence modeling.

Figure 2. Architecture of the textual feature extraction pipeline using XLM-RoBERTa [ 32 ] to generate a 768-dimensional dense vector.

Figure 3. Architecture of the visual feature extraction pipeline using CLIP ViT [ 42 ] , illustrating the custom resizing and average-color padding strategy.

Figure 4. Architecture of the Text-to-Visual (T2V) translation module.

Figure 5. Overview of the Vis-Dict framework: dictionary construction from complete multimodal posts (Step 1) and feature imputation for posts without images (Step 2).

Split	Class 0 (Normal)	Class 1 (Depressed)	Avg Tweets / User	Missing Images
Training	1,401	1,401	235	90.6%
Validation	175	175	202	88.7%
Testing	176	176	290	90.1%
Overall	1,752	1,752	237	90.4%

Table 1. User-level dataset distribution and longitudinal timeline statistics across the experimental splits.

Component / Model	Learning Rate	Batch Size	Weight Decay	Dropout	Epochs
Text Encoder (XLM-R)	$2\times 10^{-5}$	64	0.01	-	2
T2V Mapping Network	$1\times 10^{-4}$	64	0.01	-	3
Multimodal Fusion Head	$2\times 10^{-5}$	64	0.01	0.30	2
User Temporal Transformer *	$1\times 10^{-4}$	32	0.01	0.20	2
* Transformer details: $d_{model}=256$ , Layers=2, Attention Heads=8, Max Sequence Length=512.

Table 2. Hyperparameter and architectural specifications across model components.

Modality	Evaluated Samples	Accuracy	ROC-AUC	Depressed F1
Text-Only	All Samples	0.7633	0.8578	0.7598
Image-Only (Unjoint)	Full Modality (Real Images)	$0.8676\pm 0.0165$	$0.9361\pm 0.0048$	0.7911
Image-Only (Joint)	Full Modality (Real Images)	$0.7937\pm 0.0280$	$0.8707\pm 0.0041$	0.7206

Table 3. Performance of single-modality baselines at the tweet level.

Imputation Strategy	Training	All Samples	Full Modality	Missing Modality
Zero Placeholder	Unjoint	0.7624 (0.8562)	0.8556 (0.9370)	0.7477 (0.8474)
	Joint	0.7596 (0.8540)	0.8166 (0.9050)	0.7477 (0.8474)
Constant Learnt	Unjoint	0.7589 (0.8552)	0.8556 (0.9370)	0.7514 (0.8475)
	Joint	0.7576 (0.8518)	0.8166 (0.9050)	0.7514 (0.8475)
MissRAG	Unjoint	0.7593 (0.8587)	0.8556 (0.9370)	0.7470 (0.8472)
	Joint	0.7577 (0.8559)	0.8166 (0.9050)	0.7470 (0.8474)

Table 4. Tweet-level multimodal classification results reported as Depressed

F_{1}

and (ROC-AUC) across data subsets.

Imputation Strategy	Training Paradigm	Dimensional Variance
Real Image (Oracle Baseline)	—	0.6842
MissRAG	Joint	0.4835
T2V Network	Joint	0.4288
MissRAG	Unjoint	0.2252
Vis-Dict (Ours)	Joint	0.1387
Vis-Dict (Ours)	Unjoint	0.0374

Table 5. Average dimensional variance of visual embeddings on missing-image samples.

Prediction Pattern	Class 0	Class 1	Real Images	Missing Images	Total
Text Correct, Image Incorrect $\rightarrow$ Correct	16,770	5,267	1,030	21,007	22,037
Image Correct, Text Incorrect $\rightarrow$ Correct	350	1,117	608	859	1,467
Text Incorrect Misleads Fusion	2,309	8,314	214	10,409	10,623
Image Incorrect Misleads Fusion	843	235	487	591	1,078
Both Incorrect $\rightarrow$ Correct Fusion	123	21	22	122	144
Both Correct $\rightarrow$ Incorrect Fusion	12	91	10	93	103

Table 6. Tweet-level modality agreement and error breakdown for the Unjoint Vis-Dict model across data subsets.

Figure 6. Majority Voting: Depressed $F_{1}$ -Score across various classification thresholds. Performance peaks at $\theta=0.6$ across all imputation strategies.

Imputation Strategy	Training Paradigm	Depressed $F_{1}$	ROC-AUC
Zero Placeholder	Unjoint	0.9003	0.9062
	Joint	0.9048	0.9091
Constant Learnt	Unjoint	0.8916	0.8977
	Joint	0.9015	0.9063
MissRAG	Unjoint	0.8936	0.9006
	Joint	0.8889	0.8949

Table 7. User-level classification results using Majority Voting (Threshold

\theta=0.6

Figure 7. Softmax Averaging: Depressed $F_{1}$ -Score across various classification thresholds. Performance peaks at $\theta=0.5$ across all imputation strategies.

Imputation Strategy	Training Paradigm	Depressed $F_{1}$	ROC-AUC
Zero Placeholder	Unjoint	0.9111	0.9091
	Joint	0.9061	0.9034
Constant Learnt	Unjoint	0.9136	0.9119
	Joint	0.9036	0.9006
MissRAG	Unjoint	0.9183	0.9176
	Joint	0.9076	0.9062

Table 8. User-level classification results using Softmax Probability Averaging (Threshold

\theta=0.5

1. Unimodal Baselines
Model Configuration	Training Paradigm	Depressed $F_{1}$	ROC-AUC
Text-Only (All Samples)	Frozen	$0.9359\pm 0.0127$	$0.9869\pm 0.0005$
Text-Only (Full Modality)	Frozen	$0.8915\pm 0.0023$	$0.9683\pm 0.0007$
Image-Only (Full Modality)	Unjoint	$0.8611\pm 0.0234$	$0.9361\pm 0.0048$
Image-Only (Full Modality	Joint	$0.7747\pm 0.0506$	$0.8707\pm 0.0041$
2. Multimodal (w/ Imputation)

Table 9. User-level classification performance using the Sequential Transformer across 5 runs.

Imputation Strategy	Positional Encoding	Depressed $F_{1}$	ROC-AUC
Vis-Dict (Unjoint)	Included	$0.9454\pm 0.0128$	$0.9890\pm 0.0009$
Vis-Dict (Unjoint)	Removed	$0.9416\pm 0.0102$	$0.9897\pm 0.0010$
T2V Network (Joint)	Included	$0.9468\pm 0.0048$	$0.9855\pm 0.0008$
T2V Network (Joint)	Removed	$0.9024\pm 0.0229$	$0.9762\pm 0.0057$
MissRAG (Joint)	Included	$0.9408\pm 0.0075$	$0.9855\pm 0.0008$
MissRAG (Joint)	Removed	$0.9249\pm 0.0080$	$0.9825\pm 0.0011$

Table 10. Sequential Transformer performance with and without Positional Encoding (5 runs).

1. Full Modality (Real Images)
Imputation Strategy	Training Paradigm	Depressed $F_{1}$	ROC-AUC
Real Image (Oracle)	Unjoint	$0.9062\pm 0.0014$	$0.9698\pm 0.0022$
Real Image (Oracle)	Joint	$0.8684\pm 0.0104$	$0.9475\pm 0.0050$
2. Missing Modality (Imputed)
MissRAG	Unjoint	$0.9423\pm 0.0062$	$0.9841\pm 0.0002$
MissRAG	Joint	$0.9425\pm 0.0057$	$0.9836\pm 0.0007$

Table 11. Sequential model performance separated by Full Modality and Missing Modality subsets (5 runs).

Explore similar work

PAT: A Personality-Driven Augmentation and Transfer Learning Framework for Depression Detection on Social Media
Oct 10, 2026Shengwu Zheng, Yuzhe Ding, Xiaohui Liang +3Depression Detection
School of Cyber Science and Engineering, Wuhan University · School of Public Health, Wuhan University
Context Matters: Vision-Based Depression Detection Comparing Classical and Deep Approaches
Apr 11, 2026Maneesh Bilalpur, Saurabh Hinduja, Sonish Sivarajkumar +4Depression Detection Domain Generalization
Intelligent Systems Program, University of Pittsburgh, Pittsburgh, USA · CGI Technologies and Solutions Inc, USA · University of Oregon, USA +3
Dynamic Summary Generation for Interpretable Multimodal Depression Detection
Apr 13, 2026Shiyu Teng, Jiaqing Liu, Hao Sun +6Depression Detection Depression Severity Estimation
College of Information Science and Engineering, Ritsumeikan University, Shiga, Japan · College of Computer Science and Technology, Zhejiang University, Hangzhou, China · Faculty of Engineering, University of the Ryukyus, Japan

Oct 10, 2026J/K move · Enter open · S save

PAT: A Personality-Driven Augmentation and Transfer Learning Framework for Depression Detection on Social Media

Shengwu Zheng, Yuzhe Ding, Xiaohui Liang, Fei Li+2

School of Cyber Science and Engineering, Wuhan University · School of Public Health, Wuhan University

Depression Detection

Open paper AskPDF arXiv

Depression is a prevalent mental disorder that affects millions worldwide, with early detection crucial for effective intervention. While exist?ing methods have achieved promising results in automated depression detection, they face two key limitations: (1) noise in post screening: symptom-based semantic screening is easily misled by superficial lexical patterns, thereby introducing false-positive posts; (2) weak op?timization of the post encoder: relying solely on coarse user-level supervision signals pre?vents models from capturing depressive cues within individual posts. To address these limita?tions, we propose PAT, a Personality-driven Augmentation and Transfer learning frame?work. In the first phase, PAT optimizes a post?level encoder via a fine-grained contrastive learning objective, utilizing LLM-based data augmentation driven by classic personality the?ories to alleviate data scarcity. In the second phase, the trained post-level scorer is deployed for high-quality post filtering, while its en?coder parameters are seamlessly transferred for user-level tasks. Extensive experiments demon?strate that PAT significantly outperforms exist?ing methods. Furthermore, PAT delivers multi?granular interpretability by tracing post-level mental-state trajectories and highlighting key symptoms, thereby offering reliable diagnostic evidence for clinical practice.

Abstract

Explore similar work

PAT: A Personality-Driven Augmentation and Transfer Learning Framework for Depression Detection on Social Media

Context Matters: Vision-Based Depression Detection Comparing Classical and Deep Approaches

Dynamic Summary Generation for Interpretable Multimodal Depression Detection