Enhanced Endoscopic Navigation

Overlay of a registered CT segmentation of the middle turbinate on a video sequence (left); and showing camera pose and motion (right).


Project Description:

The significance of the endoscopic visualization and navigation is the introduction of a paradigm shift in surgical navigation by using a device present in every endoscopic surgery, namely the endoscope, to improve registration and visualization of anatomy. This will have numerous positive impacts. Most importantly, it will provide an in-expensive, non-invasive, radiation-free method to enhance registration accuracy at any point of the procedure. Enhancements in registration will reduce ambiguity for the surgeon during surgery, enhancing confidence, and improve workflow by reducing the need to re-register or re-image the patient. The endoscope will also be used as a measurement device to update anatomic models during a procedure. This not only will improve the ability of the surgeon to visualize the progress of the surgery, but it will accrue additional benefits to the patient and hospital, as it may reduce the level of radiation exposure and cost by eliminating the need for intra-operative CT imaging.

Research Objectives:

  1. Develop video-CT registration algorithms that are accurate to CT resolution
  2. Develop methods for surface shape estimation from endoscopic images.
  3. Perform comparative evaluation of video-CT-based navigation on patient data.
  4. Assess the accuracy and reliability of intra-operative surface estimation on patient data.

People Involved:



Gregory D. Hager, Ph.D.
Mandell Bellmore Professor of Computer Science


Russell H. Taylor, Ph.D.
John C. Malone Professor of Computer Science


Masaru Ishii, M.D., Ph.D.
Associate Professor of Otolaryngology – Head and Neck Surgery


Austin Reiter, Ph.D.
Assistant Research Professor of Computer Science


Simon Leonard, Ph.D.
Assistant Research Professor of Computer Science


Balázs P. Vágvölgyi
Associate Research Engineer in LCSR


Jeff Siewerdsen, Ph.D.
Professor of Biomedical Engineering


Yoshi Otake, Ph.D.
Associate Professor at Nara Institute of Science and Technology

Ph.D. Students


Ayushi Sinha
Computer Science


Xiang Xiang
Computer Science


Daniel Mirota, Ph.D.
(Research Engineer at Intel)

Data Sets:

We developed a data collection system to simultaneously capture both the endoscopic video and external motion tracking data. Patient data was collected at Johns Hopkins Outpatient Center.

Endoscopic sensor and data collection devices. The top box is the processor produced by NDI. The bottom left is a high-precision optically tracked endoscope, The bottom middle and right form a EM tracked scope for use in airway data collection.

A sample of patient endoscopic video data. In this sample, we make use of frame 3 to frame 66 for testing. All frames in this continuous part represent endoscopic scenarios.

A snippet from a sinus endoscopy video.

We also collect CT data from these patients, which we automatically segment to extract structures of interest.

Reconstruction from Video:

Obtaining accurate and dense reconstructions from video frames is a crucial step in enhancing registration accuracy. This will improve the visualisation of the progress of the surgery, and also provide a radiation-free alternative to intra-operative CT imaging.

Sparse reconstruction using structure from motion.

Learning based photometric reconstruction produces dense reconstructions.

Segmentation and Modeling from CT:

The different Hounsfield Units (or image intensity values) corresponding to different tissue types in CT images allow us to automatically segment structures in these images. Segmenting structures in several images allows us to model these shapes and build statistical atlases. Statistical atlases of the sinus can help us understand the average sinus anatomy in a given population, as well as the variations observed in that population. This knowledge can help in automatic segmentation of sinuses, deformable registration, pathology detection, etc.


SinusInferior TurbinateMiddle Turbinate

We built statistical atlases of the sinus using 52 head CTs. The image on the left shows the direction and magnitude of the largest variation in our sinus dataset up to +/- 3 standard deviations. The middle and right images show the same information for the inferior and middle turbinates, respectively. The variations in the turbinates show the natural process of decongestion.


Finally, in order to allow the surgeon to location the endoscope in the pre-operative CT image, the endoscopy video reconstruction and the CT segmentation need to be registered. The video at the top of this page shows a registration using trimmed ICP, while the video below shows a registration using the V-IMLOP algorithm for a 1 second sequence.

A 1 second sequence showing the alignment produced by the V-IMLOP registration algorithm. The while contours are occluding contours extracted from the endoscopy video frames, and the green contours are occluding contours computed on the triangle mesh extracted from CT segmentation.


A Sinha, A Reiter, S Leonard, M Ishii, RH Taylor, GD Hager. “Simultaneous segmentation and correspondence improvement using statistical modes“, Accepted to SPIE, Medical Imaging 2017: Image Processing, To be presented in Orlando, FL (February 14, 2017)

SD Billings, A Sinha, A Reiter, S Leonard, M Ishii, GD Hager, RH Taylor. “Anatomically Constrained Video-CT Registration via the V-IMLOP Algorithm“, Lecture Notes in Computer Science, Vol. 9902, pp 133-141, Medical Image Computing and Computer-Assisted Intervention — MICCAI 2016, Athens, Greece (October 02, 2016)

A Reiter, S Leonard, A Sinha, M Ishii, RH Taylor, GD Hager. “Endoscopic-CT: learning-based photometric reconstruction for endoscopic sinus surgery“, Proc. SPIE 9784, Medical Imaging 2016: Image Processing, 978418, San Diego, CA (March 21, 2016).

S Leonard, A Reiter, A Sinha, M Ishii, RH Taylor, GD Hager. “Image-based navigation for functional endoscopic sinus surgery using structure from motion“, Proc. SPIE 9784, Medical Imaging 2016: Image Processing, 97840V, San Diego, CA (March 21, 2016)

A Sinha, S Leonard, A Reiter, M Ishii, RH Taylor, GD Hager. “Automatic segmentation and statistical shape modeling of the paranasal sinuses to estimate natural variations“, Proc. SPIE 9784, Medical Imaging 2016: Image Processing, 97840D, San Diego, CA (March 21, 2016)

Y Otake, S Leonard, A Reiter, P Rajan, JH Siewerdsen, GL Gallia, M Ishii, RH Taylor, GD Hager. “Rendering-based video-CT registration with physical constraints for image-guided endoscopic sinus surgery “, Proc. SPIE 9415, Medical Imaging 2015: Image-Guided Procedures, Robotic Interventions, and Modeling, 94150A, Orlando, FL (March 18, 2015)

Funding Sources:

National Institutes of Health Grant No. R01 EB015530: Enhanced Navigation for Endoscopic Sinus Surgery through Video Analysis, 7/1/2012-6/30/2017

Comments are closed