Augmented Reality for Brain Tumor Surgery

Wearable augmented reality (AR) is an emerging technology with enormous potential for use in the medical field, from training and procedure simulations to image-guided surgery. Medical AR seeks to enable surgeons to see different types of tissue in real time. With the objective of achieving real-time guidance, the emphasis on speed produces the need for a fast method for imaging and classification. Hyperspectral imaging (HSI) is a non-contact, optical imaging modality that rapidly acquires hundreds of images of tissue at different wavelengths, which can be used to generate spectral data of the tissue. Combining HSI information and machine-learning algorithms allows for effective tissue classification. The objective of the research is to minimize the error between the overlaid segmentations and the region of interest while greatly reducing the time between acquisition and visualization with a reasonable degree of correction for movement. We performed a tissue-based segmentation of an optical brain tumor phantom from its corresponding HSI hypercube using a neural network by examining the optical properties of each tissue type. Then, we accurately superimposed the segmentation labels over the original phantom using a Unity script uploaded to the HoloLens. The designed joint HSI and AR surgical guidance system has been tested for hologram registration accuracy through a series of positional and rotational tests. Our system virtually superimposes the HSI channels and segmentation labels of a brain tumor phantom onto the real scene using the HoloLens AR headset. The user can manipulate and interact with the segmentation labels and HSI channels by repositioning, rotating, changing visibility, and switching between them. All actions can be performed through either hand or voice controls. This creates a convenient and multifaceted visualization of brain tissue in real time with minimal user restrictions. We demonstrate the feasibility of this fast and practical HSI-AR technique for potential use of image-guided brain surgery.

Augmented reality brain tumor

Figure 1. The images above demonstrate hyperspectral imaging (HSI) directed, augmented reality (AR) guided surgery. The HoloLens application creates a convenient and untethered method to visualize tissue segmentations in real time. The top two images shows the segmentation labels overlaid on top of the brain tumor phantom. From the images, it can be seen that the boundaries of the blood vessel and tumor are clearly defined and easily distinguishable from the surrounding tissue. The image on the bottom left shows the scene setup and basic interaction with the application through a finger tap movement. On the bottom right is the application interface as seen through the HoloLens. There is a band selection menu which contains each of the spectral bands in the hypercube as well as a smaller segmentation label menu which allows the user to see the tissue segmentations overlaid on the phantom.