Augmented blendshapes for real-time simultaneous 3D head modeling and facial motion capture

Research output: Chapter in Book/Report/Conference proceedingConference contribution

11 Citations (Scopus)

Abstract

We propose a method to build in real-time animated 3D head models using a consumer-grade RGB-D camera. Our framework is the first one to provide simultaneously comprehensive facial motion tracking and a detailed 3D model of the user's head. Anyone's head can be instantly reconstructed and his facial motion captured without requiring any training or pre-scanning. The user starts facing the camera with a neutral expression in the first frame, but is free to move, talk and change his face expression as he wills otherwise. The facial motion is tracked using a blendshape representation while the fine geometric details are captured using a Bump image mapped over the template mesh. We propose an efficient algorithm to grow and refine the 3D model of the head on-the-fly and in real-time. We demonstrate robust and high-fidelity simultaneous facial motion tracking and 3D head modeling results on a wide range of subjects with various head poses and facial expressions. Our proposed method offers interesting possibilities for animation production and 3D video telecommunications.

Original languageEnglish
Title of host publication2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016
PublisherIEEE Computer Society
Pages3299-3308
Number of pages10
ISBN (Electronic)9781467388511
Publication statusPublished - Jan 1 2016
Event2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016 - Las Vegas, United States
Duration: Jun 26 2016Jul 1 2016

Publication series

NameProceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
Volume2016-January
ISSN (Print)1063-6919

Other

Other2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016
CountryUnited States
CityLas Vegas
Period6/26/167/1/16

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Vision and Pattern Recognition

Fingerprint Dive into the research topics of 'Augmented blendshapes for real-time simultaneous 3D head modeling and facial motion capture'. Together they form a unique fingerprint.

  • Cite this

    Thomas, D. G. F., & Taniguchi, R-I. (2016). Augmented blendshapes for real-time simultaneous 3D head modeling and facial motion capture. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016 (pp. 3299-3308). (Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition; Vol. 2016-January). IEEE Computer Society.