Face completion with pyramid semantic attention and latent codes

Shilei Cao, Kouichi Sakurai

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

—Face completion, which is to reproduce the missing region of an incomplete face image, has achieved promising performance due to the adoption of generative adversarial network (GANs) and the development of GPU. However, current network frameworks always ignore that face images usually have strong semantic correlation and symmetry, failing to recover some semantically plausible details especially for filling in large contiguous holes. To produce visually realistic and semantically correct results, we propose a two-stage adversarial framework, the first stage is to produce coarse images and works as a prior for searching the most similar latent codes in reference sets, which are combined by composing their intermediate feature maps. Besides, the second stage captures texture information with our novel pyramid semantic attention block for fully using semantic information and embed the learned structure features into the inpainting process. Our attention layer considers not only the known contents but also our reconstructed parts, so that we can improve the realism of reconstructing parts, then apply this attention layer into a novel pyramid structure. In addition, we add weights in the loss function around the predicted boundary for encouraging our model to generate clearer contour lines and better interpolation properties. Empirically, the experiment on CelebA dataset shows our proposed method is effective to fill in large contiguous holes on the face images. Especially, the SSIM score of our model is nearly higher 0.1 than context encoder model.

Original languageEnglish
Title of host publicationProceedings - 2020 8th International Symposium on Computing and Networking, CANDAR 2020
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages1-8
Number of pages8
ISBN (Electronic)9781728182216
DOIs
Publication statusPublished - Nov 2020
Event8th International Symposium on Computing and Networking, CANDAR 2020 - Virtual, Naha, Japan
Duration: Nov 24 2020Nov 27 2020

Publication series

NameProceedings - 2020 8th International Symposium on Computing and Networking, CANDAR 2020

Conference

Conference8th International Symposium on Computing and Networking, CANDAR 2020
Country/TerritoryJapan
CityVirtual, Naha
Period11/24/2011/27/20

All Science Journal Classification (ASJC) codes

  • Artificial Intelligence
  • Computational Theory and Mathematics
  • Computer Networks and Communications
  • Computer Science Applications
  • Software

Fingerprint

Dive into the research topics of 'Face completion with pyramid semantic attention and latent codes'. Together they form a unique fingerprint.

Cite this