Wearable augmented reality (AR) is an emerging technology with enormous potential for use in the medical field, from training and procedure simulations to image-guided surgery. Medical AR seeks to enable surgeons to see different types of tissue in real time. With the objective of achieving real-time guidance, the emphasis on speed produces the need for a fast method for imaging and classification. Hyperspectral imaging (HSI) is a non-contact, optical imaging modality that rapidly acquires hundreds of images of tissue at different wavelengths, which can be used to generate spectral data of the tissue. Combining HSI information and machine-learning algorithms allows for effective tissue classification. The objective of the research is to minimize the error between the overlaid segmentations and the region of interest while greatly reducing the time between acquisition and visualization with a reasonable degree of correction for movement. We performed a tissue-based segmentation of an optical brain tumor phantom from its corresponding HSI hypercube using a neural network by examining the optical properties of each tissue type. Then, we accurately superimposed the segmentation labels over the original phantom using a Unity script uploaded to the HoloLens. The designed joint HSI and AR surgical guidance system has been tested for hologram registration accuracy through a series of positional and rotational tests. Our system virtually superimposes the HSI channels and segmentation labels of a brain tumor phantom onto the real scene using the HoloLens AR headset. The user can manipulate and interact with the segmentation labels and HSI channels by repositioning, rotating, changing visibility, and switching between them. All actions can be performed through either hand or voice controls. This creates a convenient and multifaceted visualization of brain tissue in real time with minimal user restrictions. We demonstrate the feasibility of this fast and practical HSI-AR technique for potential use of image-guided brain surgery.