Real-time facial motion capture using RGB-D images under complex motion and occlusions

Joao Otavio De Lucena, Joao Paulo Lima, Diego Thomas, Veronica Teichrieb

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We present a technique for capturing facial performance in real time using an RGB-D camera. Such method can be applied to face augmentation by leveraging facial expression changes. The technique is able to perform both 3D facial modeling and facial motion tracking without the need of pre-scanning or training for a specific user. The proposed approach builds on an existing method that we refer as FaceCap, which uses a blendshape representation and a Bump image for tracking facial motion and capturing geometric details. The original FaceCap algorithm fails in some scenarios with complex motion and occlusions, mainly due to problems in the face detection and tracking steps. FaceCap also has problems with the Bump image filtering step that generates outliers, causing more distortion on the 3D augmented blendshape. In order to solve these problems, we propose two refinements: (a) a new framework for face detection and landmark localization based on the state-of-the-art methods MTCNN and CE-CLM, respectively; and (b) a simple but effective modification in the filtering step, removing reconstruction failures in the eye region. Experiments showed that the proposed approach can deal with unconstrained scenarios, such as large head pose variations and partial occlusions, while achieving real-time execution.

Original languageEnglish
Title of host publicationProceedings - 2019 21st Symposium on Virtual and Augmented Reality, SVR 2019
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages120-129
Number of pages10
ISBN (Electronic)9781728154343
DOIs
Publication statusPublished - Oct 2019
Event21st Symposium on Virtual and Augmented Reality, SVR 2019 - Rio de Janeiro, Brazil
Duration: Oct 28 2019Oct 31 2019

Publication series

NameProceedings - 2019 21st Symposium on Virtual and Augmented Reality, SVR 2019

Conference

Conference21st Symposium on Virtual and Augmented Reality, SVR 2019
CountryBrazil
CityRio de Janeiro
Period10/28/1910/31/19

All Science Journal Classification (ASJC) codes

  • Computer Science Applications
  • Media Technology
  • Modelling and Simulation

Fingerprint Dive into the research topics of 'Real-time facial motion capture using RGB-D images under complex motion and occlusions'. Together they form a unique fingerprint.

  • Cite this

    De Lucena, J. O., Lima, J. P., Thomas, D., & Teichrieb, V. (2019). Real-time facial motion capture using RGB-D images under complex motion and occlusions. In Proceedings - 2019 21st Symposium on Virtual and Augmented Reality, SVR 2019 (pp. 120-129). [8920911] (Proceedings - 2019 21st Symposium on Virtual and Augmented Reality, SVR 2019). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/SVR.2019.00034