Quantitative Imaging Analysis

In medical applications, multiple images are acquired from the same subject at different times or from different subjects. The critical stage for utilizing these images is to align them in order to visualize the combined information. Image registration includes the processing to resolve the mapping between images so that the features or structures in one image correspond to those in the other image. The transformation between two image scenes can be of either rigid or non-rigid. Rigid body transformation has six degrees of freedom in three dimensions, i.e. three translations and three rotations. For most body organs, the motion is non-rigid and requires more degrees of freedom to accurately describe tissue motion. Therefore, deformable registration becomes necessary for many image applications. One example is the registration of pre- and post-contrast enhanced breast MRI images. Deformable registration is required for this application as soft tissue, such as breast tissue, always undergoes non-rigid motion between images. Other similar applications can be found during medical imaging diagnosis where different modality images or the same modality at different times, are acquired and require deformable registration because of the non-rigid tissue deformation between images. Examples include heart imaging and chest PET-CT imaging where non-rigid motion can be a major tissue motion source. In image-guided radiation therapy, because of the type of treatment or patient respiration, a non-rigid shape or position change in an organ is unavoidable. Deformable image registration is critical for delivering an appropriate radiation dose and in order to avoid the damage to adjacent healthy tissue. An example is image-guided radiation therapy of prostate tumors or tumors in other organs.

We created and evaluated an almost fully automated, 3D non-rigid registration algorithm using mutual information and a thin-plate spline (TPS) transformation for MR images of the prostate and pelvis. In the first step, an automatic rigid-body registration with special features was used to capture the global transformation. In the second step, local feature points were registered. An operator entered only five feature points (FPs) located at the prostate center, in the left and right hip joints, and in the left and right distal femurs. The program automatically determined and optimized other FPs on the external pelvic skin surface and along the femurs. More than 600 control points were used to establish a TPS transformation for deformation of the pelvic region and the prostate. Ten volume pairs were acquired from three volunteers in the diagnostic (supine) and treatment positions (supine with legs raised). Various visualization techniques showed that warping rectified the significant pelvic misalignment caused by the rigid body method. Gray-value measures of registration quality, including mutual information, correlation coefficient, and intensity difference, all improved with warping. The distance between prostate 3D centroids was 0.7 ± 0.2 mm following warping compared to 4.9 ± 3.4 mm with rigid-body registration. The semiautomatic non-rigid registration works better than rigid body registration when the patient position is changes significantly between acquisitions; it could be a useful tool for many applications of prostate diagnosis and therapy.

Our semi-automatic segmentation method for prostate MR images. The blue curves are the ground truth labeled by a radiologist, while the red curves are the segmentations of the proposed method (Tian Z, Liu L, Zhang Z, Fei B. Superpixel-Based Segmentation for 3D Prostate MR Images. IEEE Trans Med Imaging. 2016 Mar;35(3):791-801).

We implemented mutual information B-spline deformation registration algorithms. Mutual information does not assume a linear intensity relationship between images and has been used for registration of images of either the same modality or different modalities. A motion constraint is optimized in order to achieve a smooth function instead of an unrealistic result. A gradient-based minimization method is used to find the B-splines control coefficients for optimal transformation. Multiresolution strategy is applied to register the image from the downsampled low-resolution image to the original high-resolution images. The number of control points also hierarchally increase along the multiresolution framework. The deformation computed at low resolution is the initial transformation for the optimization at the high resolution.

Figure 2. Deformable registration of 3D brain MR data. With deformable transformation, Image (a) is transformed into Image (b) with shape deformation in order to match the reference image (c).

For soft tissue, e.g. tumor, registration, we developed a finite element model (FEM)-based deformable registration method. We have applied this FEM registration method to tumor MRI and PET images. In the first step, we applied a rigid registration algorithm to align the cropped MRI and PET images using three translations and three rotations. After registration, we manually segmented the tumor slice-by-slice on both high-resolution MRI and PET image volumes. We then applied the deformable registration algorithm. The registration approach deforms the tumor surface from the MRI volume toward that from the PET image. The displacements at the surface vertices are the force that drives the elastic surface from MRI toward that from the PET image. The tumor was modeled as a linear isotopic elastic material. The FEM model was used to infer volumetric deformation of the tumor from the surface. The force is then integrated over each element and is distributed over the nodes belonging to the element using its shape functions. After obtaining the displacement field for all vertices, we used a linear interpolation to obtain the deformed image volume of the tumor.

Figure 3. Three-dimensional meshes of a tumor. (a) Tumor segmented from a high-resolution MR volume. (b) Same tumor from the corresponding microPET emission images. (c) Color overly of the tumor from MRI (yellow) and PET (red). The tumor deformed during the two imaging sessions.

The 2D registration program can align 2D images using rigid transformation. It integrates manual and automatic registration. It has the following features: (1) Register two 2D images automatically by intensity-based methods, (2) Register two 2D images manually. (3) Load and register floating multiple slices to the reference slice and display the registration parameters for each floating slice.

Figure 4. The 2D image registration software.

The 3D registration program can align image volumes from CT, PET, MRI, and/or other imaging modalities. It has the following features: (1) Resize the floating volume according to the reference volume size and resolution so as to keep the same volume resolution in every direction for the two volumes; (2) Automatic registration based on mutual information; (3) Manual registration using 3D translation and rotation; (4) It includes two deformable registration approaches; (5) Displays volume in all directions as well as the fusion results; and (6) Displays the location line in 3D space and easily locates each point in 3D space. The user interface is also very friendly and direct.

Figure 5. The 3D image registration software.

This project implements a three-dimensional (3D) to two-dimensional (2D) registration for computed tomography (CT) and dual-energy digital radiography (DR) for the detection of coronary artery calcification. In order to utilize CT as the “gold standard” to evaluate the ability of DR images to detect and localize calcium, we developed an automatic intensity-based 3D-to-2D registration method for 3D CT volumes and 2D DR images. To generate digital rendering radiographs (DRR) from the CT volumes, we developed three projection methods, including Gaussian-weighted projection, threshold-based projection, and average-based projection, were developed. Cross correlation (NCC) and normalized mutual information (NMI) are used as the similarity measurement.

The software program has the following capabilities: (1) Simulate DR images from the reference CT volume at any angle and generate the projection image using Gaussian-weighted projection, threshold-based projection, and average-based projection methods; and (2) Perform registration between the original DR images and the DRR image reconstructed from the CT volume.

Figure 6. Graphic user interface (GUI) for the 3D to 2D registration software.

Slice to volume registration is used to register a two-dimensional image slice to a three-dimensional image volume. In this study, we registered live-time interventional magnetic resonance imaging (iMRI) slices with a previously obtained, high resolution MRI volume which in turn can be registered with a variety of functional images, e.g. PET and SPECT, for tumor targeting. We created and evaluated a slice-to-volume registration algorithm with special features for its potential use in iMRI-guided, radiofrequency (RF) thermal ablation. The algorithm features included a multi-resolution approach, two similarity measures, and automatic restarting in order to avoid local minima. Imaging experiments were performed on volunteers using a conventional diagnostic MR scanner and an interventional MRI system under realistic conditions. Both high-resolution MR volumes and actual iMRI image slices were acquired from the same volunteers. Actual and simulated iMRI images were used to test the dependence of slice-to-volume registration on image noise, coil inhomogeneity, and RF needle artifacts. To quantitatively assess registration, we calculated the mean voxel displacement over a volume of interest between slice-to-volume registration and volume-to-volume registration, which was previously shown to be quite accurate. More than 800 registration experiments were performed. For transverse image slices covering the prostate, the slice-to-volume registration algorithm was 100% successful with an error of < 2 mm, and the average and standard deviation was only 0.4 mm ± 0.2 mm. Visualizations such as combined sector display and contour overlay showed excellent registration of the prostate and other organs throughout the pelvis. Error was greater when an image slice was obtained at other orientations and positions, mostly because of inconsistent image content such as that obtained from variable rectal and bladder filling. These preliminary experiments indicate that MR slice-to-volume registration is sufficiently accurate to be able to aid image-guided therapy. [caption id="attachment_91" align="aligncenter" width="566"] Figure 7. Slice to Volume Registration[/caption]

Selected Publications

Yogananda CG, Shah BR, Vejdani-Jahromi M, Nalawade SS, Murugesan GK, Yu FF, Pinho MC, Wagner BC, Emblem KE, Bjørnerud A, Fei BW (Corresponding author). A fully automated deep learning network for brain tumor segmentation. Tomography. 2020 Jun; 6(2):186.
[PDF]
Ortega S, Halicek M, Fabelo H, Camacho R, Plaza MD, Godtliebsen F, M Callicó G, Fei BW (Corresponding author). Hyperspectral imaging for the detection of glioblastoma tumor cells in H&E slides using convolutional neural networks. Sensors; 20(7):1911.
[PDF]
Nalawade S, Fang FY, Yogananda CG, Murugesan GK, Shah BR, Pinho MC, Wagner BC, Mickey B, Patel TR, Fei BW, Madhuranthakam AJ, Maldjian JA. Brain tumor IDH, 1p/19q, and MGMT molecular classification using MRI-based deep learning: effect of motion and motion correction. bioRxiv 2020.
[PDF]

Tran CT, Halicek M, Dormer JD, Tandon A, Hussain T, Fei BW (Corresponding author). Fully automated segmentation of the right ventricle in patients with repaired Tetralogy of Fallot using U-Net. Medical Imaging 2020: Biomedical Applications in Molecular, Structural, and Functional Imaging; 11317(113171M). International Society for Optics and Photonics,

[PDF]
Ma L, Halicek M, Fei BW. In vivo cancer detection in animal model using hyperspectral image classification with wavelet feature extraction. Medical Imaging 2020: Biomedical Applications in Molecular, Structural, and Functional Imaging; 11317(113171C). International Society for Optics and Photonics,
[PDF]
Edwards K, Chhabra A, Dormer J, Jones P, Boutin RD, Lenchik L, Fei BW (Corresponding author). Abdominal muscle segmentation from CT using a convolutional neural network. Medical Imaging 2020: Biomedical Applications in Molecular, Structural, and Functional Imaging; 11317(113170L). International Society for Optics and Photonics,
[PDF]
Halicek M, Dormer JD, Little JV, Chen AY, Fei BW (Corresponding author). Tumor detection of the thyroid and salivary glands using hyperspectral imaging and deep learning. Biomedical Optics Express; 11(3):1383-400.
[PDF]

Bangalore Yogananda CG, Shah BR, Vejdani-Jahromi M, Nalawade SS, Murugesan GK, Yu FF, Pinho MC, Wagner BC, Mickey B, Patel TR, Fei BW (Corresponding author). A novel fully automated MRI-based deep-learning method for classification of IDH mutation status in brain gliomas. Neuro-oncology; 22(3):402-11.

[PDF]

Shahedi M, Halicek M, Dormer JD, Fei BW (Corresponding author). Incorporating minimal user input into deep learning based image segmentation. Medical Imaging 2020: Image Processing; 11313(1131313). International Society for Optics and Photonics,

[PDF]

Mavuduru A, Halicek M, Shahedi M, Little JV, Chen AY, Myers LL, Fei BW (Corresponding author). Using a 22-layer U-Net to perform segmentation of squamous cell carcinoma on digitized head and neck histological images. Medical Imaging 2020: Digital Pathology; 11320(113200C). International Society for Optics and Photonics,

[PDF]

Shahedi M, Dormer JD, TT AD, Do QN, Xi Y, Lewis MA, Madhuranthakam AJ, Twickler DM, Fei BW (Corresponding author). Segmentation of uterus and placenta in MR images using a fully convolutional neural network. Medical Imaging 2020: Computer-Aided Diagnosis; 11314(113141R). International Society for Optics and Photonics,

[PDF]

Hao D, Ding S, Qiu L, Lv Y, Fei BW, Zhu Y, Qin B. Sequential vessel segmentation via deep channel attention network. Neural Networks.

[PDF]

Ortega S, Halicek M, Fabelo H, Callico GM, Fei BW (Corresponding author). Hyperspectral and multispectral imaging in digital and computational pathology: a systematic review. Biomedical Optics Express; 11(6): 3195-3233.

[PDF]

Fabelo H, Halicek M, Ortega S, Shahedi M, Szolna A, Pineiro JF, Sosa C, O’Shanahan AJ, Bisshopp S, Espino C, Marquez M, Hernandez M, Carrera D, Morera J, Callicó GM, Sarmiento R, Fei BW (Corresponding author). Deep learning-based framework for in-vivo identification of glioblastoma tumor using hyperspectral images of human brain. Sensors;19(4).

[PDF]

Halicek M, Shahedi M, Little JV, Chen AY, Myers LL, Sumer BD, Fei BW, Tomaszewski JE, Ward AD. Detection of squamous cell carcinoma in digitized histological images from the head and neck using convolution neural networks. Proceedings of SPIE Medical Imaging 2019: Digital Pathology.

[PDF]

Yogananda CGB, Nalawade SS, Murugesan GK, Wagner B, Pinho MC, Fei BW, Madhuranthakam AJ, Maldjian JA. Fully Automated Brain Tumor Segmentation and Survival Prediction of Gliomas using Deep Learning and MRI. bioRxiv (2019):760157.

[PDF]

Lu G, Wang D, Qin X, Muller S, Little JV, Wang X, Chen AY, Chen G, Fei BW (Corresponding author). Histopathology feature mining and association with hyperspectral imaging for the detection of squamous neoplasia. Scientific reports; 9(1): 1-13.

[PDF]

Halicek M, Dormer J, Little JV, Chen AY, Myers L, Sumer BD, Fei BW (Corresponding author). Hyperspectral Imaging of Head and Neck Squamous Cell Carcinoma for Cancer Margin Detection in Surgical Specimens from 102 Patients Using Deep Learning. Cancers (2019);11(9):1367.

[PDF]

Halicek M, Fabelo H, Ortega S, Callicó GM, Fei BW (Corresponding author). In-vivo and ex-vivo tissue analysis through hyperspectral imaging techniques: Revealing the invisible featuers of cancer. Cancers;11(6).

[PDF]

Shahedi M, Halicek M, Guo R, Zhang G, Schuster DM, Fei BW (Corresponding author). A semiautomatic prostate segmentation in CT images using a deep learning approach. The Annual Meeting of the Biomedical Engineering Society.

Tian Z, Liu L, Fei BW. Deep convolutional neural network for prostate MR segmentation. International Journal of Computer Assisted Radiology and Surgery;13(11):1687.

[PDF]

Lu G, Wang DS, Qin X, Muller S, Wang X, Chen AY, Chen ZG, Fei BW (Corresponding author). Detection and delineation of squamous neoplasia with hyperspectral imaging in a mouse model of tongue carcinogenesis. Journal of Biophotonics;11(3):e201700078.

[PDF]

Tian Z, Liu L, Zhang Z, Fei BW (Corresponding author). PSNet: prostate segmentation on MRI based on a convolutional neural network. Journal of Medical Imaging;5(2):021208.

[PDF]

Sarno, A., Dance, DR., van Engen, RE., Young, KC., Russo, P., Di Lillo, F., Mettivier, G., Bliznakova, K., Fei, BW., Sechopoulos, I.(2017). “A Monte Carlo model for mean glandular dose evaluation in spot compression mammography.”Medical Physics 44(7): 3848-3860.

[PDF] [DOI]

Ma L, Guo R, Zhang G, Tade F, Schuster DM, Fei BW, Automatic segmentation of the prostate on CT images using deep learning and multi-atlas fusion, Proc. SPIE 10133, Medical Imaging 2017: Image Processing, 101332O, February 24, 2017, Orlando, FL.

[PDF]

Fei BW, Lu G, Wang X, Zhang HZ, Little JV, Patel MR, Griffith CC, El-Diery MW, Chen AY. Label-free reflectance hyperspectral imaging for tumor margin assessment: a pilot study on surgical specimens of cancer patients. Journal of Biomedical Optics;22(8):086009.

[PDF]

Chung H, Lu G, Tian Z, Wang D, Chen ZG, Fei BW (Corresponding author). Superpixel-based spectral classification for the detection of head and neck cancer with hyperspectral imaging. Proceedings of SPIE – The International Society for Optical Engineering 2016; 9788:978813. PubMed PMID:27656035.

[PDF]

Pike R, Lu G, Wang D, Chen ZG, Fei BW (Corresponding author). A minimum spanning forest based classification method for dedicated breast CT images. Medical Physics 2015; 42:6190-202.

[PDF]

Pike R, Sechopoulos I, Fei BW (Corresponding author). A minimum spanning forest-based method for noninvasive cancer detection wtih hyperspectral imaging. IEEE Transactions on Biomedical Engineering 2015; 63(3):653-663.

[PDF]

Qin X, Wang S, Shen M, Zhang X, Lerakis S, Wagner MB, Fei BW (Corresponding author). Register cardiac fiber orientations from 3D DTI volume to 2D ultrasound image of rat hearts. Proceedings of SPIE – The International Society for Optical Engineering 2015; 9415:94152M. PubMed PMID:26855466.

[PDF]

Lu G, Halig L, Wang D, Chen ZG, Fei BW (Corresponding author). Spectral-spatial classification using tensor modeling for cancer detection with hyperspectral imaging. Proceedings of SPIE – The International Society for Optical Engineering 2014; 9034:903413. PubMed PMID:25328639.

[PDF]

Akbari H, Fei BW (Corresponding author). Automatic 3D segmentation of the kidney in MR images using wavelet feature extraction and probability shape model. Proceedings of SPIE – The International Society for Optical Engineering 2013; 8314:83143D. PubMed PMID:24027620.

[PDF]

Halig LV, Wang D, Wang AY, Chen ZG, Fei BW (Corresponding author).Biodistribution study of nanoparticle encapsulated photodynamic therapy drugs using multispectral imaging. Proceedings of SPIE – The International Society for Optical Engineering 2013; 8672. PubMed PMID:24236230.

[PDF]

Yang X, Fei BW (Corresponding author).3D prostate segmentation of ultrasound images combining longitudinal image registration and machine learning. Proceedings of SPIE – The International Society for Optical Engineering 2012; 8316:83162O. PubMed PMID:24027622.

[PDF]

Wang H, Fei BW (Corresponding author). An MR image-guided, voxel-based partial volume correction method for PET images. Medical Physics 2012; 39:179-195.

[PDF]

Akbari H, Halig LV, Schuster DM, Osunkoya A, Master VA, Nieh PT, Chen GZ, and Fei BW (Corresponding author). Hyperspectral imaging and quantitative analysis for prostate cancer detection, Journal of Biomedical Optics 2012; 17:076005.

[PDF]

Yang X, Akbari H, Halig L, Fei BW (Corresponding author). 3D non-rigid registration using surface and local salient features for transrectal ultrasound image-guided prostate biopsy. Proceedings of SPIE – The International Society for Optical Engineering 2011; 7964:79642V. PubMed PMID:24027609.

[PDF]

Yang X, Schuster D, Master V, Nieh P, Fenster A, Fei BW (Corresponding author). Automatic 3D segmentation of ultrasound images using atlas registration and statistical texture prior. Proceedings of SPIE – The International Society for Optical Engineering 2011; 7964:796432. PubMed PMID:22708024. (Cum Laude Poster Award – First Place).

[PDF]

Fei BW, Yang X, Nye JA, Aarsvold JN, Meltzer CC, Votaw JR. PET-MRI quantification tools – registration, segmentation, classification, and attenuation correction. IEEE Nuclear Science Symposium and Medical Imaging Conference Focused Workshop on PET/MRI, November 1, 2010, Knoxville, TN.

Wang H, Feyes D, Mulvihill J, Oleinick N, Maclennan G, Fei BW (Corresponding author). Multiscale fuzzy C-means image classification for multiple weighted MR images for the assessment of photodynamic therapy in mice. Proceedings of SPIE – The International Society for Optical Engineering 2007; 6512. PubMed PMID:24386526.

[PDF]

Haaga JR, Exner A, Fei BW, Seftel A. Semiquantitative imaging measurement of baseline and vasomodulated normal prostatic blood flow using sildenafil. International Journal of Impotence Research 2007; 19:110-3.

[PDF]

Fei BW (Corresponding author), Chen X, Wang H, Sabol JM, DuPont E, Gilkeson RC. Automatic registration of CT volumes and dual-energy digital radiography for detection of cardiac and lung diseases. Proceedings of IEEE Engineering in Medicine and Biology Society 2006; 1:1976-9. PubMed PMID:17945687.

[PDF]

Fei BW (Corresponding author), Flask C, Wang H, Pi A, Wilson D, Shillingford J, Murcia N, Weimbs T, Duerk J. Image segmentation, registration and visualization of serial MR images for therapeutic assessment of polycystic kidney disease in transgenic mice. Proceedings of IEEE Engineering in Medicine and Biology Society 2005; 1:467-9. PubMed PMID:17282217.

[PDF]

Fei BW, Zhuang T. Computer-assisted surgery and its localization methods. Foreign Medical Science 1997; 20:199-204.