Paper ID: 2312.07478 • Published Dec 12, 2023
Double-Flow GAN model for the reconstruction of perceived faces from brain activities
Zihao Wang, Jing Zhao, Xuetong Ding, Hui Zhang
TL;DR
Get AI-generated summaries with premium
Get AI-generated summaries with premium
Face plays an important role in humans visual perception, and reconstructing
perceived faces from brain activities is challenging because of its difficulty
in extracting high-level features and maintaining consistency of multiple face
attributes, such as expression, identity, gender, etc. In this study, we
proposed a novel reconstruction framework, which we called Double-Flow GAN,
that can enhance the capability of discriminator and handle imbalances in
images from certain domains that are too easy for generators. We also designed
a pretraining process that uses features extracted from images as conditions
for making it possible to pretrain the conditional reconstruction model from
fMRI in a larger pure image dataset. Moreover, we developed a simple pretrained
model for fMRI alignment to alleviate the problem of cross-subject
reconstruction due to the variations of brain structure among different
subjects. We conducted experiments by using our proposed method and traditional
reconstruction models. Results showed that the proposed method is significant
at accurately reconstructing multiple face attributes, outperforms the previous
reconstruction models, and exhibited state-of-the-art reconstruction abilities.