Multimodal Generative Model

Multimodal generative models aim to create coherent representations and generate data across multiple modalities (e.g., text, images, audio) by learning the relationships between them. Current research emphasizes improving the expressiveness of these models, often using energy-based priors or combining contrastive and reconstruction learning techniques within architectures like transformers and variational autoencoders. This field is significant for advancing artificial intelligence, enabling applications such as improved image captioning, radiology report generation, and more robust and efficient path planning in robotics, while also highlighting and mitigating biases present in training data.

Papers

October 24, 2023

GenKIE: Robust Generative Multimodal Document Key Information Extraction
Panfeng Cao, Ye Wang, Qiang Zhang, Zaiqiao Meng
End to End Token Level Optical Character Recognition Key Information Extraction Multimodal Generative Model

October 8, 2023

MULTISCRIPT: Multimodal Script Learning for Supporting Open Domain Everyday Tasks
Jingyuan Qi, Minqian Liu, Ying Shen, Zhiyang Xu, Lifu Huang
Generative Question Multimodal Dialogue Multimodal Generative Model Script Event Prediction Domain Task

June 15, 2023

Improving Path Planning Performance through Multimodal Generative Models with Local Critics
Jorge Ocampo Jimenez, Wael Suleiman
Motion Planning Variational Auto Wasserstein Generative Adversarial Network Configuration Space Multimodal Generative Model

June 2, 2023

Privacy Distillation: Reducing Re-identification Risk of Multimodal Diffusion Models
Virginia Fernandez, Pedro Sanchez, Walter Hugo Lopez Pinaya, Grzegorz Jacenków, Sotirios A. Tsaftaris, Jorge Cardoso
Knowledge Distillation Synthetic Data Multimodal Generative Model Identification Risk

April 26, 2023

Multimodal Composite Association Score: Measuring Gender Bias in Generative Multimodal Models
Abhishek Mandal, Susan Leavy, Suzanne Little
Cross Modal Multimodal Model Gender Bias Multimodal Generative Model Bias Measurement Generative Multimodal Model

December 29, 2022

Multimodal Sequential Generative Models for Semi-Supervised Language Instruction Following
Kei Akuzawa, Yusuke Iwasawa, Yutaka Matsuo
Semi Supervised Learning Semi Supervised Sequence to Sequence Task Multimodal Generative Model Speaker Modeling

October 26, 2022

Multimodal Generative Models for Bankruptcy Prediction Using Textual Data
Rogelio A. Mancisidor, Kjersti Aas
Prediction Model Multimodal Generative Model Bankruptcy Prediction

October 18, 2022

Aligning MAGMA by Few-Shot Learning and Finetuning
Jean-Charles Layoun, Alexis Roger, Irina Rish
LeArning Abstract Generative Model Vision Language Visual Question Answering Visual Language Model Multimodal Generative Model Multimodal Data Augmentation

October 13, 2022

Decoding Visual Neural Representations by Multimodal Learning of Brain-Visual-Linguistic Features
Changde Du, Kaicheng Fu, Jinpeng Li, Huiguang He
Multimodal Learning Neural Representation Multimodal Generative Model Multimodal Semantic

September 7, 2022

Benchmarking Multimodal Variational Autoencoders: CdSprites+ Dataset and Toolkit
Gabriela Sejnova, Michal Vavrecka, Karla Stepanova, Tadahiro Taniguchi
MAESTRO Dataset Easy to Use Toolkit Multimodal Generative Model Multimodal Variational AutoEncoders Multimodal VAE

August 3, 2022

Multimodal sensor fusion in the latent representation space
Robert J. Piechocki, Xiaoyang Wang, Mohammud J. Bocus
Sensor Fusion Multimodal Classification Multimodal Generative Model

July 5, 2022

A survey of multimodal deep generative models
Masahiro Suzuki, Yutaka Matsuo
Timely Survey Generative Model Multimodal Learning Multimodal Generative Model

June 29, 2022

Can Push-forward Generative Models Fit Multimodal Distributions?
Antoine Salmona, Valentin de Bortoli, Julie Delon, Agnès Desolneux
Generative Model Generative Adversarial Network Variational Autoencoders Generative Network Multimodal Generative Model Push Forward

May 25, 2022

Mutual Information Divergence: A Unified Metric for Multimodal Generative Models
Jin-Hwa Kim, Yunji Kim, Jiyoung Lee, Kang Min Yoo, Sang-Woo Lee
Text to Image Generation Mutual Information Multimodal Representation Learning Mutual Information Maximization Multimodal Generative Model

December 20, 2021

Multimodal Adversarially Learned Inference with Factorized Discriminators
Wenxue Chen, Jianke Zhu
Contrastive Learning Generative Adversarial Network Generative Modeling Multimodal Data Multimodal Phenomenon Multimodal Generative Model Dual Discriminator Driven Inference

Multimodal Generative Model

Papers

GenKIE: Robust Generative Multimodal Document Key Information Extraction

MULTISCRIPT: Multimodal Script Learning for Supporting Open Domain Everyday Tasks

Improving Path Planning Performance through Multimodal Generative Models with Local Critics

Privacy Distillation: Reducing Re-identification Risk of Multimodal Diffusion Models

Multimodal Composite Association Score: Measuring Gender Bias in Generative Multimodal Models

Multimodal Sequential Generative Models for Semi-Supervised Language Instruction Following

Multimodal Generative Models for Bankruptcy Prediction Using Textual Data

Aligning MAGMA by Few-Shot Learning and Finetuning

Decoding Visual Neural Representations by Multimodal Learning of Brain-Visual-Linguistic Features

Benchmarking Multimodal Variational Autoencoders: CdSprites+ Dataset and Toolkit

Multimodal sensor fusion in the latent representation space

A survey of multimodal deep generative models

Can Push-forward Generative Models Fit Multimodal Distributions?

Mutual Information Divergence: A Unified Metric for Multimodal Generative Models

Multimodal Adversarially Learned Inference with Factorized Discriminators