Research

Research Objectives

Our research involves interdisciplinary collaboration between engineers, surgeons, and methodologists/biostatisticians. For details on the principal investigators and researchers involved in the project, please refer to the people page.

Surgical Activity Recognition

Psychomotor skills are essential for skillfully performing surgery. Surgical motion, i.e., hand and tool motion while performing surgery, like other human activity, is structured. Our research is based on the proposition that surgical motion is comprised of units of motion, which may be tasks, maneuvers, and gestures. Recognizing what unit of surgical motion is happening in the operating field is essential for developing tools to assess skill and provide automated feedback to individualize surgical skills training. Our goal is to develop time series models that capture the inherent structure in surgical motion. Our research to achieve this goal is focused on (1) developing new techniques to automatically represent continuous motion as a string of discrete symbols; (2) investigating properties of different representations of surgical tool motion in terms of their universality and generalization to higher level grammatical structures, and (3) develop time series methods such as hidden Markov models (HMMs) and hybrid dynamical systems, and string-motif based models of surgical tool motion.
We evaluate the effect of different representations of motion data and accuracy of models trained with and without supervision on the following tasks: (1) segmentation of the task into its components and (2) detection of the segment and its boundaries.

Objective Skill Assessment

Assessment of trainees’ surgical skill is typically based upon subjective evaluations by supervising surgeons, on checklists, or simple quantitative metrics such as total task time or frequency of errors. Our goals are to determine what constitutes surgical expertise and skill, to determine what constitutes surgical skill or expertise, to develop and evaluate objective measures of surgical skill, and to identify representations of the data and models that are invariant to operator style.

Automated Feedback and Individualized Learning

Our research aims to develop automated tools that provide objective, learning-directed feedback to surgeons in training. Evaluating surgical skill within a framework of decomposing surgical motion enables development of targeted feedback to facilitate surgical skill acquisition.
In an interdisciplinary collaboration study funded by the Science of Learning Institute at Johns Hopkins, we are working with investigators at the Johns Hopkins Minimally Invasive Surgery Training and Innovation Center (MISTIC) to integrate our technological developments into an ongoing robotic surgical skills training curriculum for automated objective assessment of surgical skills.

Human Machine Collaboration

Our research on human-machine collaboration aims to re-define robot-assisted surgery by developing systems that allow the da Vinci surgical robot to collaborate with the surgeon. Such collaborative systems can be applied in the laboratory to train surgeons’ technical skills and eventually in the operating room to provide safe, effective, efficient, and high quality patient care. Our work on human-machine collaboration has involved training the da Vinci surgical system (Intuitive Surgical, Inc., Sunnyvale, CA) to complete maneuvers initiated by a human operator (video).

Data Sets

To accomplish our research objectives, we rely on multiple data sets, comprising kinematic data describing surgical tool motion, video recordings of operators performing surgical tasks, and annotations for surgical tasks, maneuvers, gestures, and surgical skill. These data sets have been captured on various surgical platforms including robotic, endoscopic, open surgery, and simulation (da Vinci Skills Simulator).

Go here to see our publicly available datasets.

Robotic Surgery

For our research, we capture data from the da Vinci robotic surgery system via its application programming interface (API) using custom-made software. We record streaming kinematic data of each arm on the patient end and at the master console end in three-dimensions, system events, and stereo video from the endoscopic cameras. The data recording software we developed captures synchronized data from the video stream and the API stream at 30-50 Hz and assigns a timestamp to identify each frame. We have successfully deployed our software to collect data in research studies on operators performing surgical tasks using the da Vinci surgical system on bench-top models in the laboratory and on patients in the operating room. Detailed descriptions of the data sets collected for each of our research studies are available in the private pages.

Robotic surgery data set I (Surgical task performance on bench-top models): Data Set I includes data from eight operators performing multiple repetitions of three different tasks – suturing, knot tying, and needle passing.
Robotic surgery data set II (Surgical task performance on bench-top models): Data Set II includes data from 18 operators performing multiple repetitions of three tasks – throwing a surgeon’s knot, transection, and blunt dissection.
Robotic surgery data set III (Surgical procedures on patients): Data Set III includes data from an ongoing study evaluating the effect of warm-up on a da Vinci surgical skills simulator prior to a robot-assisted laparoscopic hysterectomy on trainees’ intra-operative skill on the surgical robot.

Open Surgery

We are collaborating with surgeons in the department of otolaryngology at Johns Hopkins on an ongoing study for developing objective measures of surgical skill among trainees performing septoplasty in the nose.

Septoplasty (Surgical procedure on patients): The nasal septum is the partition that divides the nasal cavity into right and left nostrils. Septoplasty refers to a surgical correction for a deviated nasal septum typically to alleviate symptoms such as difficulty in breathing or to correct a deformation in the nose. Septoplasty is an “index” case for residents in otolaryngology, i.e., the trainees are expected to achieve certain level of proficiency in performing the procedure by the end of their training. Current methods for evaluating trainees’ surgical skills in performing septoplasty include subjective assessments by the supervising surgical educators and the number of procedures that the trainee either assisted or performed under supervision by an attending surgeon.
To develop an objective measure of surgical skill among trainees performing septoplasty, we are capturing kinematic data and videos of trainee and attending surgeons performing the task. The data recording system involves generating an electromagnetic field to track in three dimensions, a sensor attached to the Cottle (an instrument used to elevate flaps of mucosa overlying the nasal septum). In addition, we capture stereo video using Microsoft Kinects (Microsoft, Inc., Redmond, CA).

Endoscopic Surgery

Functional Endoscopic Sinus Surgery (FESS) (Surgical task performance on cadavers): FESS is a surgical procedure typically performed to restore ventilation in patients with sinusitis or polyps obstructing the airway. FESS involves use of flexible nasal endoscopes to do the surgery. In a study to develop objective measures for surgical skill, including hand-eye coordination, we captured data from 20 surgeons while they performed nine different FESS tasks. We recorded tool and endoscope motion data through electromagnetic tracking of sensors attached to the part of the tools that remained outside the body. In addition, we also recorded the location of the surgeon’s eye gaze using an infrared eye tracking camera.

Simulation in Surgery

daVinci surgical skills simulator (Surgical task performance in simulation): The data we capture from the da Vinci surgical skills simulator include motion data from the console-side, the simulated tool-tip positions, and quantitative metrics computed and displayed by the in-built software.

Methods

Our research involves applying existing methods and developing new ones for representation of data, statistical models of surgical activity, and classifiers for surgical skill.

Data Representation

Statistical models that we use in our research studies rely upon both continuous and discrete representation of surgical motion data. For continuous representations of surgical motion data, we use dimensionality-reduction methods such as linear discriminant analysis or principal components analysis. We assume that the low-dimensional representation of our sample data originate from a population described by Gaussian mixture models.
For discrete representations of surgical motion data, we quantize the data using two different approaches. In the first approach, we use simple vector quantization methods such as identifying clusters (e.g., using k-means) and labeling each frame of the data based on its nearest cluster. This approach is sensitive to the coordinates of the reference frame from which the data were collected. In the second approach, which is coordinate invariant, we segment the motion trajectory into symbols using accumulated Frenet frames (AFF). In this method, called descriptive curve coding (DCC), the motion trajectory is encoded based on a local coordinate system defined at each point on the trajectory. DCC using AFF encodes changes in direction of the motion trajectory within short spatial or time windows based on an alphabet, which can be defined as a variable number of segments.

Models

We apply and develop various methods to model the data to recognize component units of surgical motion including hybrid dynamical systems, language models, and models based on string motifs and geometry of motion. Our first goal for each of the models is to recognize smaller units of surgical motion such as gestures and maneuvers.

Hybrid Dynamic Systems

Our research attempts to model the observed kinematic and video data describing surgical motion using dynamical systems to create an underlying dictionary of motion models. Surgical activities are expressed as a sequence of models taken from this dictionary. In particular, our work is focused on the use of sparse systems as a unifying theory for modeling and identification. A detailed description can be found under the Hybrid Dynamic Systems section.

Language Models

We presume that surgical motion can be modeled as a series of transitions from one component activity (e.g., task, maneuver, gesture) to another. This is similar to how speech is modeled as a transition from one phoneme to another. Our research in area of language modeling includes supervised and unsupervised training of various types of HMMs and linear dynamical systems with time varying parameters to model the structure of surgical motion.

String Motif-based Models

Surgical motion, when quantized into discrete symbols using either vector quantization methods or DCC, is represented as a string of such symbols. Information specific to operators, components of surgical motion, and surgical skill are encoded as motifs within the string. To recognize segments of surgical motion, our models extract information on the longest common substrings that comprise the segments and use a measure of distance from such models as a decision rule. We adopt a similar approach to identify class labels for surgical skill.

Models based on Fractal Geometry of Surgical Motion

Our work in this area is related to our ongoing study on surgical skill while performing septoplasty. The task of separating the mucosal flaps on the septum involves finding and breaking adhesions between the mucosa and the septum. We believe that expert surgeons are more efficient at finding and breaking the adhesions. We hypothesize that expert surgeons move more heuristically and the distribution of their movements imitate Levy flights, represented by a distribution with heavy tails. In contrast, trainee surgeons may exhibit a Brownian motion closer to a random Gaussian motion with shorter tails. We aim to develop models of the surgeons’ motion and use them to objectively assess surgical skill acquisition in trainees.