Wide-area shape reconstruction by 3D endoscopic system based on CNN decoding, shape registration and fusion

Ryo Furukawa, Masaki Mizomori, Shinsaku Hiura, Shiro Oka, Shinji Tanaka, Hiroshi Kawasaki

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

For effective in situ endoscopic diagnosis and treatment, dense and large areal shape reconstruction is important. For this purpose, we develop 3D endoscopic systems based on active stereo, which projects a grid pattern where grid points are coded by line gaps. One problem of the previous works was that success or failure of 3D reconstruction depends on the stability of feature extraction from the images captured by the endoscope camera. Subsurface scattering or specularities on bio-tissues make this problem difficult. Another problem was that shape reconstruction area was relatively small because of limited field of view of the pattern projector compared to that of the camera. In this paper, to solve the first problem, learning-based approach, i.e., U-Nets, for efficient detection of grid lines and codes at the detected grid points under severe conditions, is proposed. To solve the second problem, an online shape-registration and merging algorithm for sequential frames is proposed. In the experiments, we have shown that we can train U-Nets to extract those features effectively for three specimens of cancers, and also conducted 3D scanning of shapes of a stomach phantom model and a surface inside a human mouth, in which wide-area surfaces are successfully recovered by shape registration and merging.

Original languageEnglish
Title of host publicationOR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis - 1st International Workshop, OR 2.0 2018 5th International Workshop, CARE 2018, 7th International Workshop, CLIP 2018, 3rd International Workshop, ISIC 2018 Held in Conjunction with MICCAI 2018
EditorsAnand Malpani, Marco A. Zenati, Cristina Oyarzun Laura, M. Emre Celebi, Duygu Sarikaya, Noel C. Codella, Allan Halpern, Marius Erdt, Lena Maier-Hein, Luo Xiongbiao, Stefan Wesarg, Danail Stoyanov, Zeike Taylor, Klaus Drechsler, Kristin Dana, Anne Martel, Raj Shekhar, Sandrine De Ribaupierre, Tobias Reichl, Jonathan McLeod, Miguel Angel González Ballester, Toby Collins, Marius George Linguraru
PublisherSpringer Verlag
Pages139-150
Number of pages12
ISBN (Print)9783030012007
DOIs
Publication statusPublished - Jan 1 2018
Event1st International Workshop on OR 2.0 Context-Aware Operating Theaters, OR 2.0 2018, 5th International Workshop on Computer Assisted Robotic Endoscopy, CARE 2018, 7th International Workshop on Clinical Image-Based Procedures, CLIP 2018, and 1st International Workshop on Skin Image Analysis, ISIC 2018, held in conjunction with the 21st International Conference on Medical Imaging and Computer-Assisted Intervention, MICCAI 2018 - Granada, Spain
Duration: Sep 16 2018Sep 20 2018

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11041 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Other

Other1st International Workshop on OR 2.0 Context-Aware Operating Theaters, OR 2.0 2018, 5th International Workshop on Computer Assisted Robotic Endoscopy, CARE 2018, 7th International Workshop on Clinical Image-Based Procedures, CLIP 2018, and 1st International Workshop on Skin Image Analysis, ISIC 2018, held in conjunction with the 21st International Conference on Medical Imaging and Computer-Assisted Intervention, MICCAI 2018
CountrySpain
CityGranada
Period9/16/189/20/18

Fingerprint

Shape Reconstruction
Merging
Registration
Decoding
Fusion
Fusion reactions
Cameras
Grid
Endoscopy
Feature extraction
Camera
Problem-based Learning
Scattering
Tissue
Scanning
Endoscope
Line
3D Reconstruction
Projector
Phantom

All Science Journal Classification (ASJC) codes

  • Theoretical Computer Science
  • Computer Science(all)

Cite this

Furukawa, R., Mizomori, M., Hiura, S., Oka, S., Tanaka, S., & Kawasaki, H. (2018). Wide-area shape reconstruction by 3D endoscopic system based on CNN decoding, shape registration and fusion. In A. Malpani, M. A. Zenati, C. Oyarzun Laura, M. E. Celebi, D. Sarikaya, N. C. Codella, A. Halpern, M. Erdt, L. Maier-Hein, L. Xiongbiao, S. Wesarg, D. Stoyanov, Z. Taylor, K. Drechsler, K. Dana, A. Martel, R. Shekhar, S. De Ribaupierre, T. Reichl, J. McLeod, M. A. González Ballester, T. Collins, ... M. G. Linguraru (Eds.), OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis - 1st International Workshop, OR 2.0 2018 5th International Workshop, CARE 2018, 7th International Workshop, CLIP 2018, 3rd International Workshop, ISIC 2018 Held in Conjunction with MICCAI 2018 (pp. 139-150). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11041 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-030-01201-4_16

Wide-area shape reconstruction by 3D endoscopic system based on CNN decoding, shape registration and fusion. / Furukawa, Ryo; Mizomori, Masaki; Hiura, Shinsaku; Oka, Shiro; Tanaka, Shinji; Kawasaki, Hiroshi.

OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis - 1st International Workshop, OR 2.0 2018 5th International Workshop, CARE 2018, 7th International Workshop, CLIP 2018, 3rd International Workshop, ISIC 2018 Held in Conjunction with MICCAI 2018. ed. / Anand Malpani; Marco A. Zenati; Cristina Oyarzun Laura; M. Emre Celebi; Duygu Sarikaya; Noel C. Codella; Allan Halpern; Marius Erdt; Lena Maier-Hein; Luo Xiongbiao; Stefan Wesarg; Danail Stoyanov; Zeike Taylor; Klaus Drechsler; Kristin Dana; Anne Martel; Raj Shekhar; Sandrine De Ribaupierre; Tobias Reichl; Jonathan McLeod; Miguel Angel González Ballester; Toby Collins; Marius George Linguraru. Springer Verlag, 2018. p. 139-150 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11041 LNCS).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Furukawa, R, Mizomori, M, Hiura, S, Oka, S, Tanaka, S & Kawasaki, H 2018, Wide-area shape reconstruction by 3D endoscopic system based on CNN decoding, shape registration and fusion. in A Malpani, MA Zenati, C Oyarzun Laura, ME Celebi, D Sarikaya, NC Codella, A Halpern, M Erdt, L Maier-Hein, L Xiongbiao, S Wesarg, D Stoyanov, Z Taylor, K Drechsler, K Dana, A Martel, R Shekhar, S De Ribaupierre, T Reichl, J McLeod, MA González Ballester, T Collins & MG Linguraru (eds), OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis - 1st International Workshop, OR 2.0 2018 5th International Workshop, CARE 2018, 7th International Workshop, CLIP 2018, 3rd International Workshop, ISIC 2018 Held in Conjunction with MICCAI 2018. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11041 LNCS, Springer Verlag, pp. 139-150, 1st International Workshop on OR 2.0 Context-Aware Operating Theaters, OR 2.0 2018, 5th International Workshop on Computer Assisted Robotic Endoscopy, CARE 2018, 7th International Workshop on Clinical Image-Based Procedures, CLIP 2018, and 1st International Workshop on Skin Image Analysis, ISIC 2018, held in conjunction with the 21st International Conference on Medical Imaging and Computer-Assisted Intervention, MICCAI 2018, Granada, Spain, 9/16/18. https://doi.org/10.1007/978-3-030-01201-4_16
Furukawa R, Mizomori M, Hiura S, Oka S, Tanaka S, Kawasaki H. Wide-area shape reconstruction by 3D endoscopic system based on CNN decoding, shape registration and fusion. In Malpani A, Zenati MA, Oyarzun Laura C, Celebi ME, Sarikaya D, Codella NC, Halpern A, Erdt M, Maier-Hein L, Xiongbiao L, Wesarg S, Stoyanov D, Taylor Z, Drechsler K, Dana K, Martel A, Shekhar R, De Ribaupierre S, Reichl T, McLeod J, González Ballester MA, Collins T, Linguraru MG, editors, OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis - 1st International Workshop, OR 2.0 2018 5th International Workshop, CARE 2018, 7th International Workshop, CLIP 2018, 3rd International Workshop, ISIC 2018 Held in Conjunction with MICCAI 2018. Springer Verlag. 2018. p. 139-150. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-030-01201-4_16
Furukawa, Ryo ; Mizomori, Masaki ; Hiura, Shinsaku ; Oka, Shiro ; Tanaka, Shinji ; Kawasaki, Hiroshi. / Wide-area shape reconstruction by 3D endoscopic system based on CNN decoding, shape registration and fusion. OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis - 1st International Workshop, OR 2.0 2018 5th International Workshop, CARE 2018, 7th International Workshop, CLIP 2018, 3rd International Workshop, ISIC 2018 Held in Conjunction with MICCAI 2018. editor / Anand Malpani ; Marco A. Zenati ; Cristina Oyarzun Laura ; M. Emre Celebi ; Duygu Sarikaya ; Noel C. Codella ; Allan Halpern ; Marius Erdt ; Lena Maier-Hein ; Luo Xiongbiao ; Stefan Wesarg ; Danail Stoyanov ; Zeike Taylor ; Klaus Drechsler ; Kristin Dana ; Anne Martel ; Raj Shekhar ; Sandrine De Ribaupierre ; Tobias Reichl ; Jonathan McLeod ; Miguel Angel González Ballester ; Toby Collins ; Marius George Linguraru. Springer Verlag, 2018. pp. 139-150 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{0b90b50b24cd46818721e03886b8d983,
title = "Wide-area shape reconstruction by 3D endoscopic system based on CNN decoding, shape registration and fusion",
abstract = "For effective in situ endoscopic diagnosis and treatment, dense and large areal shape reconstruction is important. For this purpose, we develop 3D endoscopic systems based on active stereo, which projects a grid pattern where grid points are coded by line gaps. One problem of the previous works was that success or failure of 3D reconstruction depends on the stability of feature extraction from the images captured by the endoscope camera. Subsurface scattering or specularities on bio-tissues make this problem difficult. Another problem was that shape reconstruction area was relatively small because of limited field of view of the pattern projector compared to that of the camera. In this paper, to solve the first problem, learning-based approach, i.e., U-Nets, for efficient detection of grid lines and codes at the detected grid points under severe conditions, is proposed. To solve the second problem, an online shape-registration and merging algorithm for sequential frames is proposed. In the experiments, we have shown that we can train U-Nets to extract those features effectively for three specimens of cancers, and also conducted 3D scanning of shapes of a stomach phantom model and a surface inside a human mouth, in which wide-area surfaces are successfully recovered by shape registration and merging.",
author = "Ryo Furukawa and Masaki Mizomori and Shinsaku Hiura and Shiro Oka and Shinji Tanaka and Hiroshi Kawasaki",
year = "2018",
month = "1",
day = "1",
doi = "10.1007/978-3-030-01201-4_16",
language = "English",
isbn = "9783030012007",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Verlag",
pages = "139--150",
editor = "Anand Malpani and Zenati, {Marco A.} and {Oyarzun Laura}, Cristina and Celebi, {M. Emre} and Duygu Sarikaya and Codella, {Noel C.} and Allan Halpern and Marius Erdt and Lena Maier-Hein and Luo Xiongbiao and Stefan Wesarg and Danail Stoyanov and Zeike Taylor and Klaus Drechsler and Kristin Dana and Anne Martel and Raj Shekhar and {De Ribaupierre}, Sandrine and Tobias Reichl and Jonathan McLeod and {Gonz{\'a}lez Ballester}, {Miguel Angel} and Toby Collins and Linguraru, {Marius George}",
booktitle = "OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis - 1st International Workshop, OR 2.0 2018 5th International Workshop, CARE 2018, 7th International Workshop, CLIP 2018, 3rd International Workshop, ISIC 2018 Held in Conjunction with MICCAI 2018",
address = "Germany",

}

TY - GEN

T1 - Wide-area shape reconstruction by 3D endoscopic system based on CNN decoding, shape registration and fusion

AU - Furukawa, Ryo

AU - Mizomori, Masaki

AU - Hiura, Shinsaku

AU - Oka, Shiro

AU - Tanaka, Shinji

AU - Kawasaki, Hiroshi

PY - 2018/1/1

Y1 - 2018/1/1

N2 - For effective in situ endoscopic diagnosis and treatment, dense and large areal shape reconstruction is important. For this purpose, we develop 3D endoscopic systems based on active stereo, which projects a grid pattern where grid points are coded by line gaps. One problem of the previous works was that success or failure of 3D reconstruction depends on the stability of feature extraction from the images captured by the endoscope camera. Subsurface scattering or specularities on bio-tissues make this problem difficult. Another problem was that shape reconstruction area was relatively small because of limited field of view of the pattern projector compared to that of the camera. In this paper, to solve the first problem, learning-based approach, i.e., U-Nets, for efficient detection of grid lines and codes at the detected grid points under severe conditions, is proposed. To solve the second problem, an online shape-registration and merging algorithm for sequential frames is proposed. In the experiments, we have shown that we can train U-Nets to extract those features effectively for three specimens of cancers, and also conducted 3D scanning of shapes of a stomach phantom model and a surface inside a human mouth, in which wide-area surfaces are successfully recovered by shape registration and merging.

AB - For effective in situ endoscopic diagnosis and treatment, dense and large areal shape reconstruction is important. For this purpose, we develop 3D endoscopic systems based on active stereo, which projects a grid pattern where grid points are coded by line gaps. One problem of the previous works was that success or failure of 3D reconstruction depends on the stability of feature extraction from the images captured by the endoscope camera. Subsurface scattering or specularities on bio-tissues make this problem difficult. Another problem was that shape reconstruction area was relatively small because of limited field of view of the pattern projector compared to that of the camera. In this paper, to solve the first problem, learning-based approach, i.e., U-Nets, for efficient detection of grid lines and codes at the detected grid points under severe conditions, is proposed. To solve the second problem, an online shape-registration and merging algorithm for sequential frames is proposed. In the experiments, we have shown that we can train U-Nets to extract those features effectively for three specimens of cancers, and also conducted 3D scanning of shapes of a stomach phantom model and a surface inside a human mouth, in which wide-area surfaces are successfully recovered by shape registration and merging.

UR - http://www.scopus.com/inward/record.url?scp=85054882518&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85054882518&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-01201-4_16

DO - 10.1007/978-3-030-01201-4_16

M3 - Conference contribution

AN - SCOPUS:85054882518

SN - 9783030012007

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 139

EP - 150

BT - OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis - 1st International Workshop, OR 2.0 2018 5th International Workshop, CARE 2018, 7th International Workshop, CLIP 2018, 3rd International Workshop, ISIC 2018 Held in Conjunction with MICCAI 2018

A2 - Malpani, Anand

A2 - Zenati, Marco A.

A2 - Oyarzun Laura, Cristina

A2 - Celebi, M. Emre

A2 - Sarikaya, Duygu

A2 - Codella, Noel C.

A2 - Halpern, Allan

A2 - Erdt, Marius

A2 - Maier-Hein, Lena

A2 - Xiongbiao, Luo

A2 - Wesarg, Stefan

A2 - Stoyanov, Danail

A2 - Taylor, Zeike

A2 - Drechsler, Klaus

A2 - Dana, Kristin

A2 - Martel, Anne

A2 - Shekhar, Raj

A2 - De Ribaupierre, Sandrine

A2 - Reichl, Tobias

A2 - McLeod, Jonathan

A2 - González Ballester, Miguel Angel

A2 - Collins, Toby

A2 - Linguraru, Marius George

PB - Springer Verlag

ER -