GeMi: A Graph-based, Multimodal Recommendation System for Narrative Scroll Paintings
Organizations: Department of Management Science and Systems, The State University of New York, Jacobs Management Center, Buffalo, 14260, NY, USA. · Institute for Artificial Intelligence and Data Science, The State University of New York, Lockwood Hall, Buffalo, 14260, NY, USA. · Department of Computer Science and Engineering, The State University of New York, Davis Hall, Buffalo, 14260, NY, USA. · Technology, Information, Internet, eBay∗, Embassy Tech Village, Bengaluru, 560103, Karnataka, India.
Abstract
Recommendation Systems are effective in managing the ever-increasing amount of multimodal data available today and help users discover interesting new items. These systems can handle various media types such as images, text, audio, and video data, and this has made it possible to handle content-based recommendation utilizing features extracted from items while also incorporating user preferences. Graph Neural Network (GNN)-based recommendation systems are a special class of recommendation systems that can handle relationships between items and users, making them particularly attractive for content-based recommendations. Their popularity also stems from the fact that they use advanced machine learning techniques, such as deep learning on graph-structured data, to exploit user-to-item interactions. The nodes in the graph can access higher-order neighbor information along with state-of-the-art vision-language models for processing multimodal content, and there are well-designed algorithms for embedding, message passing, and propagation. In this work, we present the design of a GNN-based recommendation system on a novel data set collected from field research. Designed for an endangered performing art form, the recommendation system uses multimodal content (text and image data) to suggest similar paintings for viewing and purchase. To the best of our knowledge, there is no recommendation system designed for narrative scroll paintings -- our work therefore serves several purposes, including art conservation, a data storage system for endangered art objects, and a state-of-the-art recommendation system that leverages both the novel characteristics of the data and preferences of the user population interested in narrative scroll paintings.
Figures & tables
| GeMi Variant | Parameters |
|---|---|
| GeMi-GCN | Graph encoder: 2-layer Graph Convolutional Network; GCN hidden dimension: 128; GCN dropout: 0.20; Training epochs: 450; Optimizer: Adam; Activation: ReLU; Learning rate: ; Weight decay: ; Gradient clipping: max norm 2.0; Loss (balanced): focal binary cross-entropy with logits; Focal parameters: , ; kNN neighbors ( ): 30; Tree-edge augmentation neighbors ( ): 25; Tree augmentation max nodes: 2500; Edge dropout: 0.10; interactions; Preference threshold: 0.2 |
| GeMi-GAE | Graph encoder: Graph Auto-Encoder (GAE) with a 2-layer GCN encoder; Hidden dimension: 128; Latent dimension ( ): 64; Dropout: 0.20; Training epochs: 400; Optimizer: Adam; Activation: ReLU; Learning rate: ; Weight decay: ; Gradient clipping: max norm 2.0; kNN neighbors ( ): 30; Edge dropout: 0.10; Self-supervised loss: GAE reconstruction loss ; Supervised loss: focal binary cross-entropy with logits; Focal parameters: , ; Joint objective: with ; Augmentation neighbors: , , ; Augmentation max nodes per label: 2500; interactions; Preference threshold: 0.2 |
| GeMi-VGAE | Graph encoder: Variational Graph Auto-Encoder (VGAE) with GCN-based parameterization; Hidden dimension: 256; Latent dimension ( ): 128; Dropout: 0.20; Log-standard-deviation clamp: ; Training epochs: 500; Optimizer: Adam; Activation: ReLU; Learning rate: ; Weight decay: ; Gradient clipping: max norm 2.0; kNN neighbors ( ): 25; Tree augmentation neighbors: ; Tree augmentation max nodes: 2500; Edge dropout: 0.10; Reconstruction loss: VGAE reconstruction loss ; KL regularization: VGAE KL loss ; KL annealing: linearly increased to ; Self-supervised VGAE objective: ; Supervised loss: focal binary cross-entropy with logits; Focal parameters: , ; Joint SSL objective: with ; interactions; Preference threshold: 0.2 |
| Model | Animal | Mythology | Tree |
|---|---|---|---|
| LATTICE | 0.57 0.21 | 0.47 0.25 | 0.20 0.18 |
| PMGT | 0.52 0.21 | 0.46 0.23 | 0.33 0.22 |
| HUIGN | 0.48 0.16 | 0.55 0.25 | 0.10 0.11 |
| DualGNN | 0.55 0.21 | 0.47 0.26 | 0.25 0.21 |
| FREEDOM | 0.58 0.22 | 0.41 0.24 | 0.22 0.19 |
| MICRO | 0.61 0.20 | 0.47 0.26 | 0.26 0.21 |
| Model | Animal | Mythology | Tree |
|---|---|---|---|
| LATTICE | 0.53 0.22 | 0.54 0.13 | 0.21 0.25 |
| PMGT | 0.47 0.19 | 0.52 0.17 | 0.25 0.20 |
| HUIGN | 0.49 0.20 | 0.51 0.12 | 0.23 0.22 |
| DualGNN | 0.41 0.18 | 0.56 0.32 | 0.19 0.27 |
| FREEDOM | 0.51 0.17 | 0.55 0.24 | 0.24 0.16 |
| MICRO | 0.45 0.25 | 0.58 0.23 | 0.26 0.19 |
| Model | Animal | Mythology | Tree |
|---|---|---|---|
| MambaRec | 0.50 0.24 | 0.60 0.17 | 0.12 0.13 |
| PGL | 0.43 0.26 | 0.60 0.20 | 0.25 0.18 |
| HPMRec | 0.47 0.26 | 0.63 0.21 | 0.34 0.21 |
| COHESION | 0.41 0.23 | 0.59 0.20 | 0.27 0.17 |
| SMORE | 0.40 0.20 | 0.66 0.24 | 0.24 0.16 |
| CMDL | 0.41 0.25 | 0.67 0.22 | 0.23 0.18 |
| Model | Animal | Mythology | Tree |
|---|---|---|---|
| MambaRec | 0.52 0.19 | 0.57 0.26 | 0.15 0.28 |
| PGL | 0.44 0.23 | 0.60 0.21 | 0.27 0.21 |
| HPMRec | 0.46 0.18 | 0.64 0.22 | 0.23 0.14 |
| COHESION | 0.50 0.24 | 0.55 0.11 | 0.25 0.26 |
| SMORE | 0.41 0.54 | 0.59 0.13 | 0.17 0.31 |
| CMDL | 0.57 0.28 | 0.61 0.15 | 0.16 0.25 |
| Model | Animal | Mythology | Tree |
|---|---|---|---|
| GraphSAGE | 0.48 ± 0.14 | 0.56 ± 0.23 | 0.34 ± 0.28 |
| PinSAGE | 0.52 ± 0.21 | 0.57 ± 0.22 | 0.49 ± 0.14 |
| GATNE-I | 0.52 ± 0.20 | 0.60 ± 0.21 | 0.37 ± 0.19 |
| GeMi GCN | 0.56 ± 0.24 | 0.66 ± 0.20 | 0.44 ± 0.20 |
| GeMi GAE | 0.50 ± 0.22 | 0.63 ± 0.15 | 0.34 ± 0.23 |
| GeMi VGAE | 0.52 ± 0.22 | 0.70 ± 0.21 | 0.44 ± 0.20 |
| Model | Animal | Mythology | Tree |
|---|---|---|---|
| GraphSAGE | 0.56 ± 0.25 | 0.58 ± 0.12 | 0.33 ± 0.16 |
| PinSAGE | 0.56 ± 0.25 | 0.57 ± 0.12 | 0.32 ± 0.17 |
| GATNE-I | 0.56 ± 0.25 | 0.60 ± 0.19 | 0.34 ± 0.22 |
| GeMi GCN | 0.46 ± 0.22 | 0.32 ± 0.19 | 0.56 ± 0.34 |
| GeMi GAE | 0.51 ± 0.29 | 0.61 ± 0.18 | 0.29 ± 0.23 |
| GeMi VGAE | 0.50 ± 0.27 | 0.61 ± 0.23 | 0.29 ± 0.23 |
| Model | Animal | Mythology | Tree |
|---|---|---|---|
| GraphSAGE | 0.63 ± 0.18 | 0.59 ± 0.19 | 0.26 ± 0.18 |
| PinSAGE | 0.50 ± 0.21 | 0.65 ± 0.17 | 0.21 ± 0.15 |
| GATNE-I | 0.51 ± 0.20 | 0.60 ± 0.21 | 0.36 ± 0.19 |
| GeMi GCN | 0.71 ± 0.25 | 0.39 ± 0.05 | 0.35 ± 0.13 |
| GeMi GAE | 0.47 ± 0.20 | 0.71 ± 0.20 | 0.20 ± 0.22 |
| GeMi VGAE | 0.52 ± 0.23 | 0.77 ± 0.11 | 0.53 ± 0.19 |
| Model | Animal | Mythology | Tree |
|---|---|---|---|
| GraphSAGE | 0.58 ± 0.24 | 0.52 ± 0.15 | 0.26 ± 0.12 |
| PinSAGE | 0.46 ± 0.20 | 0.61 ± 0.11 | 0.39 ± 0.22 |
| GATNE-I | 0.56 ± 0.24 | 0.61 ± 0.20 | 0.33 ± 0.22 |
| GeMi GCN | 0.36 ± 0.12 | 0.59 ± 0.07 | 0.35 ± 0.13 |
| GeMi GAE | 0.69 ± 0.25 | 0.77 ± 0.11 | 0.34 ± 0.15 |
| GeMi VGAE | 0.53 ± 0.19 | 0.78 ± 0.10 | 0.53 ± 0.19 |
| Model | Animal | Mythology | Tree |
|---|---|---|---|
| SigCLIP + GCN | 0.48 0.25 | 0.70 0.21 | 0.13 0.15 |
| SigCLIP + GAE | 0.54 0.28 | 0.52 0.17 | 0.20 0.17 |
| SigCLIP + VGAE | 0.51 0.25 | 0.62 0.19 | 0.18 0.18 |
| Model | Animal | Mythology | Tree |
|---|---|---|---|
| VAE + GCN | 0.49 0.23 | 0.58 0.20 | 0.38 0.24 |
| VAE + GAE | 0.33 0.20 | 0.47 0.21 | 0.23 0.20 |
| VAE + VGAE | 0.34 0.20 | 0.53 0.13 | 0.20 0.18 |
| Model | Animal | Mythology | Tree |
|---|---|---|---|
| SigCLIP + GCN | 0.52 ± 0.26 | 0.57 ± 0.14 | 0.39 ± 0.27 |
| SigCLIP + GAE | 0.50 ± 0.25 | 0.54 ± 0.23 | 0.37 ± 0.27 |
| SigCLIP + VGAE | 0.51 ± 0.35 | 0.50 ± 0.27 | 0.44 ± 0.21 |
| Model | Animal | Mythology | Tree |
|---|---|---|---|
| VAE + GCN | 0.47 ± 0.27 | 0.51 ± 0.28 | 0.45 ± 0.23 |
| VAE + GAE | 0.39 ± 0.23 | 0.61 ± 0.22 | 0.30 ± 0.25 |
| VAE + VGAE | 0.51 ± 0.25 | 0.33 ± 0.18 | 0.51 ± 0.30 |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Parameters |
|---|---|
| LATTICE | Graph convolution layers: 2; Graph mixing coefficient ( ): 0.6; Loss function: Bayesian Personalized Ranking (BPR); Optimizer: Adam; Learning rate: ; Weight decay: ; Activation: linear graph propagation; Embedding normalization: normalization; |
| PMGT | Neighbors per sequence: 32; Hidden size: 128; Transformer layers: 4; Attention heads: 4; Feed-forward dimension: 256; Activation: GELU; Dropout: 0.1; Loss functions: BCE-with-logits + MSE; Masking ratio: 0.16; Random replacement ratio: 0.02; Optimizer: AdamW; Learning rate: ; Weight decay: |
| HUIGN | Hierarchical intent layers ( ): [32, 8, 4]; Co-interaction threshold: 2; Loss terms: BPR + entropy regularization + independence regularization + weight decay; Entropy weight ( ): 0.1; Independence weight ( ): 0.1; Weight decay ( ): ; Optimizer: Adam; Learning rate: |
| DualGNN | Graph propagation depth: 2; Graph mixing coefficient ( ): 0.6; Contrastive temperature ( ): 0.2; Node feature dropout: 0.2; Loss terms: InfoNCE alignment + orthogonality regularization + graph smoothness regularization; Loss weights: ; Optimizer: Adam; Learning rate: ; Weight decay: |
| FREEDOM | Multimodal graph mixing: (image) and (text); Item–Item propagation layers ( ): 1; User–Item propagation layers ( ): 2; Degree-sensitive edge pruning ratio ( ): 0.8; Training loss: BPR + multimodal auxiliary BPR regularizers; Regularization weight ( ): ; Optimizer: Adam; Learning rate: ; Weight decay: |
| MICRO | Item-view layers: 1 (per modality); Graph mixing coefficient ( ): 0.7; Contrastive objective: InfoNCE (temperature ); Orthogonality regularizer: cross-view residual decorrelation; Loss weights: (contrastive), ; Optimizer: Adam; Learning rate: ; Weight decay: |
| Model | Parameters |
|---|---|
| MambaRec | UI graph layers ( ): 2; Item–Item layers ( ): 2; Dimensionality reduction factor: ; Normalization + dropout: LayerNorm( ) + Dropout(0.2); Loss: BPR pairwise ranking loss + regularization + contrastive InfoNCE + optional MMD alignment; Contrastive weight: 1.0; InfoNCE temperature: 0.2; MMD weight: ; MMD kernel: RBF with bandwidth ; Regularization weight: ; Optimizer: Adam; Learning rate: ; Weight decay: |
| PGL | UI graph layers ( ): 2; Item–Item layers ( ): 2; Dropout: 0.2; Loss: BPR pairwise ranking + contrastive InfoNCE loss; Contrastive temperature ( ): 0.2; Regularization weight ( ): on contrastive term; Optimizer: Adam; Learning rate: ; Weight decay: |
| HPMRec | Hypercomplex algebra dimension ( ): 4; UI graph layers ( ): 2; Item–Item multimodal layers ( ): 2; Hypercomplex interaction: Cayley–Dickson multiplication with scale ; Dropout: 0.2; Loss: BPR pairwise ranking + regularization + self-supervised hypercomplex loss; Reg weight: ; SSL weight: 0.05; Optimizer: Adam; Learning rate: ; Weight decay: |
| COHESION | UI graph layers ( ): 2; Multimodal item–item layers ( ): 2; UI edge dropout: 0.2; Post-fusion stabilizer: LayerNorm + Linear + LeakyReLU; User–User neighborhood aggregation: ; Loss: BPR pairwise ranking + regularization; Reg weight: ; Optimizer: Adam; Learning rate: ; Weight decay: |
| SMORE | UI graph layers ( ): 2; Item–Item layers ( ): 2; Spectrum (FFT) convolution: rFFT over feature dimension with learnable complex filters (3 branches: ); Nonlinear fusion branch: frequency-domain then iFFT; Dropout: 0.2; Auxiliary contrastive loss: InfoNCE; Contrastive weight: 0.15; Temperature ( ): 0.2; Regularization: light ; Reg weight: ; Optimizer: Adam; Learning rate: ; Weight decay: ; Gradient clipping: max norm 5.0 |
| CMDL | UI graph layers ( ): 2; Content-view augmentation: feature dropout + Gaussian noise ; Item content gating; Loss: BPR pairwise ranking + regularization + contrastive alignment; Contrastive weight: 0.1; Contrastive temperatures: (items), ; Regularization weight: ; Optimizer: AdamW; Learning rate: ; Weight decay: |
| Model | Parameters |
|---|---|
| GraphSAGE | Base percentile: , GNN layers: , SAGEConv aggregator: max, SAGEConv settings: normalize: False, Hidden channels: , Output channels: 2, Activation: ReLU (between layers), Loss: cross-entropy on item labels, Optimizer: Adam, Learning rate: . |
| PinSAGE | Percentile threshold: , Random-walk neighborhood: , walk length: , new neighbor per node(topm): , GNN encoder: same as GraphSAGE above (2 layers, hidden 12, out 2, Adam lr ; cross-entropy). |
| GATNE-I | kNN on training items (k = 30), Similarity floor: 0.0, Feature preprocessing: L2-normalize features before cosine, Embedding dim: 128, Negative samples: 10, Optimizer: Adam, Learning rate: , Weight decay: . |
| Model | Parameters |
|---|---|
| GraphSAGE | GNN layers: 2, SAGEConv aggregator: max, SAGEConv settings: project=True, normalize=False, Hidden channels: 12, Output channels: 2, Activation: ReLU, Loss: cross-entropy on item labels, Optimizer: Adam, Learning rate: 1e-4, Weight decay: default. |
| PinSAGE | Percentile threshold: 90, Random-walk neighborhood: 50, walk length: 5, new neighbor per node(topm): 20, GNN encoder same as GraphSAGE above (2 layers, hidden 12, out 2, Adam, cross-entropy) |
| GATNE-I | kNN on training items (k = 30), Similarity floor: 0.0, Feature preprocessing: L2-normalize features before cosine, Embedding dim: 128, Negative samples: 10, Optimizer: Adam, Learning rate: , Weight decay: . No learning on test interactions. |
| Model | Animal | Mythology | Tree |
|---|---|---|---|
| SIGCLIP + GCN | |||
| SIGCLIP + GAE | |||
| SIGCLIP + VGAE | |||
| VAE + GCN | |||
| VAE + GAE | |||
| VAE + VGAE |
| Model | Animal | Mythology | Tree |
|---|---|---|---|
| SIGCLIP + GCN | |||
| SIGCLIP + GAE | |||
| SIGCLIP + VGAE | |||
| VAE + GCN | |||
| VAE + GAE | |||
| VAE + VGAE |
Recommender Systems
Recommender systems predict and rank items for people using interaction histories, content, context, and feedback.
See all 514 papers