Learning-based active 3D measurement technique using light field created by video projectors

Yuki Shiba, Satoshi Ono, Ryo Furukawa, Shinsaku Hiura, Hiroshi Kawasaki

Research output: Contribution to journalArticle

Abstract

The combination of a pattern projector and a camera is widely used for 3D measurement. To recover shape from a captured image, various kinds of depth cues are extracted from projected patterns in the image, such as disparities from active stereo or blurriness for depth from defocus. Recently, several techniques have been proposed to improve 3D quality using multiple depth cues by installing coded apertures in projectors or by increasing the number of projectors. However, superposition of projected patterns forms a complicated light field in 3D space, which makes the process of analyzing captured images challenging. In this paper, we propose a learning-based technique to extract depth information from such a light field, which includes multiple depth cues. In the learning phase, prior to the 3D measurement of unknown scenes, projected patterns as they appear at various depths are prepared from not only actual images but also ones generated virtually using computer graphics and geometric calibration results. Then, we use principal component analysis (PCA) to extract features of small patches. In the 3D measurement (reconstruction) phase, the same features of patches are extracted from a captured image of a target scene and compared with the learned data. By using the dimensional reduction by feature extraction, an efficient search algorithm, such as an approximated nearest neighbor (ANN), can be used for the matching process. Another important advantage of our learning-based approach is that we can use most known projection patterns without changing the algorithm.

Original languageEnglish
Article number6
JournalIPSJ Transactions on Computer Vision and Applications
Volume11
Issue number1
DOIs
Publication statusPublished - Dec 1 2019

Fingerprint

Computer graphics
Principal component analysis
Feature extraction
Cameras
Calibration

All Science Journal Classification (ASJC) codes

  • Computer Vision and Pattern Recognition

Cite this

Learning-based active 3D measurement technique using light field created by video projectors. / Shiba, Yuki; Ono, Satoshi; Furukawa, Ryo; Hiura, Shinsaku; Kawasaki, Hiroshi.

In: IPSJ Transactions on Computer Vision and Applications, Vol. 11, No. 1, 6, 01.12.2019.

Research output: Contribution to journalArticle

@article{d6e625bec15b43aeb2215cbd326dd231,
title = "Learning-based active 3D measurement technique using light field created by video projectors",
abstract = "The combination of a pattern projector and a camera is widely used for 3D measurement. To recover shape from a captured image, various kinds of depth cues are extracted from projected patterns in the image, such as disparities from active stereo or blurriness for depth from defocus. Recently, several techniques have been proposed to improve 3D quality using multiple depth cues by installing coded apertures in projectors or by increasing the number of projectors. However, superposition of projected patterns forms a complicated light field in 3D space, which makes the process of analyzing captured images challenging. In this paper, we propose a learning-based technique to extract depth information from such a light field, which includes multiple depth cues. In the learning phase, prior to the 3D measurement of unknown scenes, projected patterns as they appear at various depths are prepared from not only actual images but also ones generated virtually using computer graphics and geometric calibration results. Then, we use principal component analysis (PCA) to extract features of small patches. In the 3D measurement (reconstruction) phase, the same features of patches are extracted from a captured image of a target scene and compared with the learned data. By using the dimensional reduction by feature extraction, an efficient search algorithm, such as an approximated nearest neighbor (ANN), can be used for the matching process. Another important advantage of our learning-based approach is that we can use most known projection patterns without changing the algorithm.",
author = "Yuki Shiba and Satoshi Ono and Ryo Furukawa and Shinsaku Hiura and Hiroshi Kawasaki",
year = "2019",
month = "12",
day = "1",
doi = "10.1186/s41074-019-0058-y",
language = "English",
volume = "11",
journal = "IPSJ Transactions on Computer Vision and Applications",
issn = "1882-6695",
publisher = "Information Processing Society of Japan",
number = "1",

}

TY - JOUR

T1 - Learning-based active 3D measurement technique using light field created by video projectors

AU - Shiba, Yuki

AU - Ono, Satoshi

AU - Furukawa, Ryo

AU - Hiura, Shinsaku

AU - Kawasaki, Hiroshi

PY - 2019/12/1

Y1 - 2019/12/1

N2 - The combination of a pattern projector and a camera is widely used for 3D measurement. To recover shape from a captured image, various kinds of depth cues are extracted from projected patterns in the image, such as disparities from active stereo or blurriness for depth from defocus. Recently, several techniques have been proposed to improve 3D quality using multiple depth cues by installing coded apertures in projectors or by increasing the number of projectors. However, superposition of projected patterns forms a complicated light field in 3D space, which makes the process of analyzing captured images challenging. In this paper, we propose a learning-based technique to extract depth information from such a light field, which includes multiple depth cues. In the learning phase, prior to the 3D measurement of unknown scenes, projected patterns as they appear at various depths are prepared from not only actual images but also ones generated virtually using computer graphics and geometric calibration results. Then, we use principal component analysis (PCA) to extract features of small patches. In the 3D measurement (reconstruction) phase, the same features of patches are extracted from a captured image of a target scene and compared with the learned data. By using the dimensional reduction by feature extraction, an efficient search algorithm, such as an approximated nearest neighbor (ANN), can be used for the matching process. Another important advantage of our learning-based approach is that we can use most known projection patterns without changing the algorithm.

AB - The combination of a pattern projector and a camera is widely used for 3D measurement. To recover shape from a captured image, various kinds of depth cues are extracted from projected patterns in the image, such as disparities from active stereo or blurriness for depth from defocus. Recently, several techniques have been proposed to improve 3D quality using multiple depth cues by installing coded apertures in projectors or by increasing the number of projectors. However, superposition of projected patterns forms a complicated light field in 3D space, which makes the process of analyzing captured images challenging. In this paper, we propose a learning-based technique to extract depth information from such a light field, which includes multiple depth cues. In the learning phase, prior to the 3D measurement of unknown scenes, projected patterns as they appear at various depths are prepared from not only actual images but also ones generated virtually using computer graphics and geometric calibration results. Then, we use principal component analysis (PCA) to extract features of small patches. In the 3D measurement (reconstruction) phase, the same features of patches are extracted from a captured image of a target scene and compared with the learned data. By using the dimensional reduction by feature extraction, an efficient search algorithm, such as an approximated nearest neighbor (ANN), can be used for the matching process. Another important advantage of our learning-based approach is that we can use most known projection patterns without changing the algorithm.

UR - http://www.scopus.com/inward/record.url?scp=85069210607&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85069210607&partnerID=8YFLogxK

U2 - 10.1186/s41074-019-0058-y

DO - 10.1186/s41074-019-0058-y

M3 - Article

AN - SCOPUS:85069210607

VL - 11

JO - IPSJ Transactions on Computer Vision and Applications

JF - IPSJ Transactions on Computer Vision and Applications

SN - 1882-6695

IS - 1

M1 - 6

ER -