Augmented reality navigation system for endoscopic surgery based on three-dimensional ultrasound and computed tomography: Application to 20 clinical cases

In the endoscopic surgical fields, more information which could be used intraoperatively is strongly demanded. Three-dimensional reconstructed images are useful imaging modalities to assist the surgeon in endoscopic surgery. We expected a new magneto-optic hybrid 3-D sensor configuration, and have developed an augmented reality navigation system using an accurate three-dimensional sensory system that can be utilized in endoscopic surgery. The system has accommodated oblique endoscope, real-time distortion correction of magnetic fields. We have preliminary established 'combined' navigation system. The complete system consists of a laparoscope, two (magnetic and optic) 3D digitizers, ultrasonographic device, a laparoscopic US probe, and a workstation with a volume rendering accelerator. Augmented reality (AR) visualization, which superimposed the visualized 3D-US images and segmented and rendered CT-based images on captured laparoscopic live images, was achieved. We applied our system to various clinical cases. The system provides us real-time anatomical information which cannot be visualized without navigation system. 3D-US as an imaging modality has the advantage that real-time imaging is acquirable, regardless of organ shift or distortion.