Enhanced Endoscopic Navigation

Overlay of a registered CT segmentation of the middle turbinate on a video sequence (left); and showing camera pose and motion (right).

 

Project Description:

The significance of the endoscopic visualization and navigation is the introduction of a paradigm shift in surgical navigation by using a device present in every endoscopic surgery, namely the endoscope, to improve registration and visualization of anatomy. This will have numerous positive impacts. Most importantly, it will provide an in-expensive, non-invasive, radiation-free method to enhance registration accuracy at any point of the procedure. Enhancements in registration will reduce ambiguity for the surgeon during surgery, enhancing confidence, and improve workflow by reducing the need to re-register or re-image the patient. The endoscope will also be used as a measurement device to update anatomic models during a procedure. This not only will improve the ability of the surgeon to visualize the progress of the surgery, but it will accrue additional benefits to the patient and hospital, as it may reduce the level of radiation exposure and cost by eliminating the need for intra-operative CT imaging.

Research Objectives:

1. Develop video-CT registration algorithms that are accurate to CT resolution
2. Develop methods for surface shape estimation from endoscopic images.
3. Perform comparative evaluation of video-CT-based navigation on patient data.
4. Assess the accuracy and reliability of intra-operative surface estimation on patient data.

People Involved:

Faculty/Postdocs

Gregory D. Hager, Ph.D. Mandell Bellmore Professor of Computer Science	Russell H. Taylor, Ph.D. John C. Malone Professor of Computer Science	Masaru Ishii, M.D., Ph.D. Associate Professor of Otolaryngology – Head and Neck Surgery	Austin Reiter, Ph.D. Assistant Research Professor of Computer Science
Ayushi Sinha, Ph.D. Provost’s Postdoctoral Fellow in LCSR	Mathias Unberath, Ph.D. Assistant Research Professor of Computer Science	Simon Leonard, Ph.D. Assistant Research Professor of Computer Science	Balázs P. Vágvölgyi Associate Research Engineer in LCSR
	Jeff Siewerdsen, Ph.D. Professor of Biomedical Engineering	Yoshi Otake, Ph.D. Associate Professor at Nara Institute of Science and Technology

Ph.D. Students

Xingtong Liu Computer Science

Former Ph.D. Students

Ayushi Sinha, Ph.D. (Provost’s Postdoctoral Fellow in LCSR)

Xiang Xiang, Ph.D. (Applied Scientist at Amazon AI)

Daniel Mirota, Ph.D. (Research Engineer at Intel)

Data Sets:

We developed a data collection system to simultaneously capture both endoscopic video and external endoscopic motion tracking data. Patient data was collected at Johns Hopkins Outpatient Center under an IRB approved protocol. We also collect high resolution CT image data from these patients, which we automatically segment to extract structures of interest.

A snippet from a sinus endoscopy video.

Reconstruction from Video:

Obtaining accurate and dense reconstructions from video frames is a crucial step in enhancing video-CT registration accuracy. This will improve the visualization of the progress of the surgery, and also provide a radiation-free alternative to intra-operative CT imaging.

Sparse reconstruction using structure from motion.

Learning based photometric reconstruction produces dense reconstructions.

Segmentation and Modeling from CT:

The different Hounsfield Units (or image intensity values) corresponding to different tissue types in CT images allow us to automatically segment structures in these images. Segmenting structures in several images allows us to model these shapes and build statistical atlases. Statistical atlases of the sinus can help us understand the average sinus anatomy in a given population, as well as the variations observed in that population. This knowledge can help in automatic segmentation of sinuses, deformable registration, pathology detection, etc.

 

We built statistical atlases of the sinus using 52 head CTs. The image on the left shows the direction and magnitude of the largest variation in our sinus dataset up to +/- 3 standard deviations. The middle and right images show the same information for the inferior and middle turbinates, respectively. The variations in the turbinates show the natural process of decongestion.

Registration:

Finally, in order to allow the surgeon to location the endoscope in the pre-operative CT image, the endoscopy video reconstruction and the CT segmentation need to be registered. The video at the top of this page shows a registration using trimmed ICP, while the video below shows a registration using the V-IMLOP algorithm for a 1 second sequence.

A 1 second sequence showing the alignment produced by the V-IMLOP registration algorithm. The while contours are occluding contours extracted from the endoscopy video frames, and the green contours are occluding contours computed on the triangle mesh extracted from CT segmentation.

We have also developed a family of deformable registration algorithms that register dense point clouds extracted from video to a statistical model. The registration begins with the statistically mean shape and optimises both transformation parameters (like rotation, translation, scale) and shape parameters that deform the mean shape according to the modes of variation learned by the statistical model.

Publications:

X Liu, A Sinha, M Unberath, M Ishii, GD Hager, RH Taylor, A Reiter. “Self-supervised Learning for Dense Depth Estimation in Monocular Endoscopy“, in Lecture Notes in Computer Science, Vol. 11041, pp 128-138, International Workshop on Computer-Assisted and Robotic Endoscopy — CARE 2018, Granada, Spain (October 02, 2018)

A Sinha, X Liu, A Reiter, M Ishii, GD Hager, RH Taylor. “Endoscopic navigation in the absence of CT imaging“, in Lecture Notes in Computer Science, Vol. 11073, pp 64-71, Medical Image Computing and Computer-Assisted Intervention — MICCAI 2018, Granada, Spain (September 13, 2018)

S Leonard, A Sinha, A Reiter, M Ishii, GL Galia, RH Taylor, GD Hager. “Evaluation and Stability Analysis of Video-Based Navigation System for Functional Endoscopic Sinus Surgery on In-Vivo Clinical Data“, in IEEE Transactions on Medical Imaging, Vol. 37, no. 10, pp. 2185-2195 (May 07, 2018)

A Sinha, A Reiter, S Leonard, M Ishii, RH Taylor, GD Hager. “Simultaneous segmentation and correspondence improvement using statistical modes“, in Proc. SPIE 10133, Medical Imaging 2017: Image Processing, 101331B, Orlando, FL (February 24, 2017)

SD Billings, A Sinha, A Reiter, S Leonard, M Ishii, GD Hager, RH Taylor. “Anatomically Constrained Video-CT Registration via the V-IMLOP Algorithm“, in Lecture Notes in Computer Science, Vol. 9902, pp 133-141, Medical Image Computing and Computer-Assisted Intervention — MICCAI 2016, Athens, Greece (October 02, 2016)

A Reiter, S Leonard, A Sinha, M Ishii, RH Taylor, GD Hager. “Endoscopic-CT: learning-based photometric reconstruction for endoscopic sinus surgery“, in Proc. SPIE 9784, Medical Imaging 2016: Image Processing, 978418, San Diego, CA (March 21, 2016)

S Leonard, A Reiter, A Sinha, M Ishii, RH Taylor, GD Hager. “Image-based navigation for functional endoscopic sinus surgery using structure from motion“, in Proc. SPIE 9784, Medical Imaging 2016: Image Processing, 97840V, San Diego, CA (March 21, 2016)

A Sinha, S Leonard, A Reiter, M Ishii, RH Taylor, GD Hager. “Automatic segmentation and statistical shape modeling of the paranasal sinuses to estimate natural variations“, in Proc. SPIE 9784, Medical Imaging 2016: Image Processing, 97840D, San Diego, CA (March 21, 2016)

Y Otake, S Leonard, A Reiter, P Rajan, JH Siewerdsen, GL Gallia, M Ishii, RH Taylor, GD Hager. “Rendering-based video-CT registration with physical constraints for image-guided endoscopic sinus surgery “, in Proc. SPIE 9415, Medical Imaging 2015: Image-Guided Procedures, Robotic Interventions, and Modeling, 94150A, Orlando, FL (March 18, 2015)

Funding Sources:

National Institutes of Health Grant No. R01 EB015530: Enhanced Navigation for Endoscopic Sinus Surgery through Video Analysis, 7/1/2012-6/30/2017 (PI: Hager)