User-Aware Conditional Generative Total Correlation Learning for Multi-Modal Recommendation
Authors: Jing Du, Zesheng Ye, Congbo Ma, Feng Liu, Flora. D. Salim
Organizations: The University of New South Wales, Sydney, Australia · University of Melbourne, Melbourne, Australia · New York University Abu Dhabi, Abu Dhabi, UAE
Multi-modal recommendation (MMR) enriches item representations by introducing item content, e.g., visual and textual descriptions, to improve upon interaction-only recommenders. The success of MMR hinges on aligning these content modalities with user preferences derived from interaction data, yet dominant practices based on disentangling modality-invariant preference-driving signals from modality-specific preference-irrelevant noises are flawed. First, they assume a one-size-fits-all relevance of item content to user preferences for all users, which contradicts the user-conditional fact of preferences. Second, they optimize pairwise contrastive losses separately toward cross-modal alignment, systematically ignoring higher-order dependencies inherent when multiple content modalities jointly influence user choices. In this paper, we introduce GTC, a conditional Generative Total Correlation learning framework. We employ an interaction-guided diffusion model to perform user-aware content feature filtering, preserving only personalized features relevant to each individual user. Furthermore, to capture complete cross-modal dependencies, we optimize a tractable lower bound of the total correlation of item representations across all modalities. Experiments on standard MMR benchmarks show GTC consistently outperforms state-of-the-art, with gains of up to 28.30% in NDCG@5. Ablation studies validate both conditional preference-driven feature filtering and total correlation optimization, confirming the ability of GTC to model user-conditional relationships in MMR tasks. The code is available at: https://github.com/jingdu-cs/GTC.
Figures & tables
Figure 1. The representations from user-item interactions exhibit low cosine similarity (left) and high Euclidean distance (right) to representations from visual and textual content modalities, on Amazon Sports dataset. In contrast, visual and textual representations are more aligned, implying the gap between latent user preference and explicit item attributes.
Figure 2. The user-conditional nature of “appealing” feature relevance. The item on the right appeals to a user who likes “orange” and “cotton”, while a different user is more interested in the brand (“AW Apparel”) and style, for whom color and fabric are less important. Thus, what constitutes a preference-driving signal is not only determined by the item attributes themselves, but also by individual user preference.
Figure 3. Overall illustration of the proposed GTC framework.
Models
FREEDOM
SMORE
GRCN
LATTICE
BM3
PGL
SLMRec
LGMRec
MGCN
DRAGON
GTC
Improved(%)
NDCG
@5
0.0196
0.0181
0.0245
0.0262
0.0267
0.0294
0.0291
0.0291
0.0314
0 .0749
0.0961
28.30%
@10
0.0257
0.0226
0.0316
0.0335
0.0343
0.0376
0.0368
0.0375
0.0399
0 .0880
0.1106
25.68%
@20
0.0327
0.0274
0.0394
0.0420
0.0430
0.0472
0.0451
0.0468
0.0496
0 .1005
0.1242
23.58%
@50
0.0437
0.0336
0.0510
0.0544
0.0552
0.0611
0.0564
0.0601
0.0636
0 .1175
0.1411
20.09%
Recall
@5
0.0290
0.0268
0.0373
0.0392
0.0397
0.0441
0.0434
0.0442
0.0475
0 .1040
0.1307
25.67%
@10
0.0477
0.0407
0.0590
0.0613
0.0628
0.0694
0.0668
0.0698
0.0734
0 .1441
0.1749
21.37%
Table 1. Overall Performance in Sports (up), Baby (middle), and Cell (down) dataset. For each comparison, the best-performing method is bolded . Improved indicates the percentage improvement of the proposed GTC over the runner-up model . * denote statistically significant improvements, validated by a paired t-test at a significance level of p<0.05 against the runner-up model .
Datasets
# interaction
sparsity
user
item
num
avg interaction
num
avg interaction
Sports
296,337
99.95%
35,598
8.3245
18,357
16.1430
Baby
160,792
99.88%
19,445
8.2691
7,050
22.8074
Cell
194,439
99.93%
27,879
6.9744
10,429
18.6441
Table 2. Dataset statistics.
Module
Recall
NDCG
@5
@10
@20
@50
@5
@10
@20
@50
GTC Base
0.0995
0.1367
0.1841
0.2610
0.0709
0.0831
0.0953
0.1109
GTC Base w/ DN
0 .1229
0 .1663
0 .2180
0 .3020
0 .0894
0 .1037
0 .1170
0 .1340
GTC Base w/ TC
0.1206
0.1641
0.2164
0.2994
0.0884
0.1027
0.1162
0.1330
GTC w/o TC
0.1099
0.1506
0.1982
0.2794
0.0785
0.0918
0.1041
0.1206
GTC
0.1307
0.1749
0.2275
0.3108
0.0961
0.1106
0.1242
0.1411
Table 3. Ablation experiments on Sports (up), Baby dataset (middle), and Cell (bottom).
Figure 4. Impact of content features in Sports (up), Baby (middle), and Cell (bottom) Datasets.
Figure 5. Modality balance trend during training GTC.
Figure 6. User preference consistency in the Sports dataset (up) and Baby dataset (down).
Figure 7. Parameter evaluation in Sports (left), Baby (middle), and Cell (right) datasets.
Multimodal Graph Neural Networks have become standard for recommendation by augmenting sparse interaction data with content features. Yet current architectures face two bottlenecks: structural rigidity, from a reliance on static precomputed similarity graphs that cannot adapt to evolving preferences; and semantic fragility, where noisy modality signals are indiscriminately fused, distorting the collaborative signal. We propose MURAL (Multimodal Uncertainty-aware Recommendation via Adaptive edge Learning), a unified framework that shifts multimodal recommendation from fixed structural augmentation to dynamic topology discovery. To address structural rigidity, an Adaptive Edge Learner combines a differentiable retrieval-augmented strategy with an approximate nearest neighbor search to discover latent item-item correlations that are both semantically adaptive and computationally scalable (O(NlogN)). To address semantic fragility, an Uncertainty-Aware Fusion module models the aleatoric uncertainty of heterogeneous modalities, dynamically down-weighting unreliable features while prioritizing high-confidence signals as a defense against cross-modal noise. We further employ a contrastive teacher-student alignment that anchors modality-specific representations to stable behavioral signals, ensuring optimization stability without gradient leakage. Experiments on large-scale benchmarks including TikTok and Amazon show that MURAL significantly surpasses both structural and generative state-of-the-art baselines, achieving superior accuracy while offering interpretability through domain-specific modality dominance and robustness under extreme data corruption.
Ahmad Mousavi, Majid Alikhani, Yeon-Chang Lee +2
Department of Mathematics and Statistics American University · Independent Researcher Toronto, Ontario, Canada · Independent Researcher +4
Generative Recommendation (GR) has emerged as a promising paradigm by formulating item recommendation as a sequence-to-sequence generation task over item identifiers. Recent studies have incorporated multimodal signals to provide richer token-level evidence for generation. However, existing approaches largely rely on alignment-centric fusion and underexplore synergistic information across modalities. In practice, synergistic information plays a critical role in capturing emergent item properties that cannot be inferred from any single modality alone. Such properties encode intrinsic item semantics and guide user preferences, enabling models to move beyond surface-level feature matching. To address this limitation, we propose SynGR, a synergistic generative recommendation framework that explicitly encourages the exploitation of cross-modal dependencies during generation. By constraining overreliance on dominant modalities, SynGR enables the model to capture emergent item semantics beyond shared or modality-specific signals. Extensive experiments across three benchmark datasets demonstrate that SynGR achieves superior performance.
Wei Chen, Xingyu Guo, Shuang Li +6
School of Artificial Intelligence, Beihang University, Beijing, China · School of Computer Science and Engineering, Beihang University, Beijing, China
Personalized content systems depend on available UGC and struggle when suitable content is absent, delayed, or costly to create. Although multimodal generators can synthesize content on demand, how to translate behavioral traces into generation-ready preferences remains underexplored. We study personalized multimodal content generation: creating user-tailored multimodal content without existing item pools or waiting for matching UGC. We propose TailorMind, linking collaborative preference modeling with controllable multimodal generation. TailorMind enriches sparse user histories via hypergraph collaborative filtering and optimizes textual profiles with ranking-error feedback and textual gradient descent. Retrieval-augmented style control grounds outputs in authentic UGC patterns, while cross-modal cohesion reflection reduces semantic drift. We construct TailorBench, a benchmark from three mainstream platforms evaluated along five dimensions: coherence, novelty, aesthetic, hallucination, profiling. Experiments show that TailorMind achieves competitive or stronger coherence, improves novelty and aesthetic quality over representative generation baselines and ground-truth UGC, demonstrating advantages over retrieving available content or comparable UGC, while achieving up to 29% Recall gains in reranking. Our code is released at: https://github.com/iLearn-Lab/TailorMind.
Hengji Zhou, Ye Liu, Yufeng Liu +3
1South China University of Technology · 2Harbin Institute of Technology, Shenzhen