PurposeEye morphology varies significantly across the population, especially for the orbit and optic nerve. These variations limit the feasibility and robustness of generalizing population-wise features of eye organs to an unbiased spatial reference.ApproachTo tackle these limitations, we propose a process for creating high-resolution unbiased eye atlases. First, to restore spatial details from scans with a low through-plane resolution compared with a high in-plane resolution, we apply a deep learning-based super-resolution algorithm. Then, we generate an initial unbiased reference with an iterative metric-based registration using a small portion of subject scans. We register the remaining scans to this template and refine the template using an unsupervised deep probabilistic approach that generates a more expansive deformation field to enhance the organ boundary alignment. We demonstrate this framework using magnetic resonance images across four different tissue contrasts, generating four atlases in separate spatial alignments.ResultsWhen refining the template with sufficient subjects, we find a significant improvement using the Wilcoxon signed-rank test in the average Dice score across four labeled regions compared with a standard registration framework consisting of rigid, affine, and deformable transformations. These results highlight the effective alignment of eye organs and boundaries using our proposed process.ConclusionsBy combining super-resolution preprocessing and deep probabilistic models, we address the challenge of generating an eye atlas to serve as a standardized reference across a largely variable population.
PurposeCells are building blocks for human physiology; consequently, understanding the way cells communicate, co-locate, and interrelate is essential to furthering our understanding of how the body functions in both health and disease. Hematoxylin and eosin (H&E) is the standard stain used in histological analysis of tissues in both clinical and research settings. Although H&E is ubiquitous and reveals tissue microanatomy, the classification and mapping of cell subtypes often require the use of specialized stains. The recent CoNIC Challenge focused on artificial intelligence classification of six types of cells on colon H&E but was unable to classify epithelial subtypes (progenitor, enteroendocrine, goblet), lymphocyte subtypes (B, helper T, cytotoxic T), and connective subtypes (fibroblasts). We propose to use inter-modality learning to label previously un-labelable cell types on H&E.ApproachWe took advantage of the cell classification information inherent in multiplexed immunofluorescence (MxIF) histology to create cell-level annotations for 14 subclasses. Then, we performed style transfer on the MxIF to synthesize realistic virtual H&E. We assessed the efficacy of a supervised learning scheme using the virtual H&E and 14 subclass labels. We evaluated our model on virtual H&E and real H&E.ResultsOn virtual H&E, we were able to classify helper T cells and epithelial progenitors with positive predictive values of 0.34±0.15 (prevalence 0.03±0.01) and 0.47±0.1 (prevalence 0.07±0.02), respectively, when using ground truth centroid information. On real H&E, we needed to compute bounded metrics instead of direct metrics because our fine-grained virtual H&E predicted classes had to be matched to the closest available parent classes in the coarser labels from the real H&E dataset. For the real H&E, we could classify bounded metrics for the helper T cells and epithelial progenitors with upper bound positive predictive values of 0.43±0.03 (parent class prevalence 0.21) and 0.94±0.02 (parent class prevalence 0.49) when using ground truth centroid information.ConclusionsThis is the first work to provide cell type classification for helper T and epithelial progenitor nuclei on H&E.
Understanding the way cells communicate, co-locate, and interrelate is essential to understanding human physiology. Hematoxylin and eosin (H&E) staining is ubiquitously available both for clinical studies and research. The Colon Nucleus Identification and Classification (CoNIC) Challenge has recently innovated on robust artificial intelligence labeling of six cell types on H&E stains of the colon. However, this is a very small fraction of the number of potential cell classification types. Specifically, the CoNIC Challenge is unable to classify epithelial subtypes (progenitor, endocrine, goblet), lymphocyte subtypes (B, helper T, cytotoxic T), or connective subtypes (fibroblasts, stromal). In this paper, we propose to use inter-modality learning to label previously un-labelable cell types on virtual H&E. We leveraged multiplexed immunofluorescence (MxIF) histology imaging to identify 14 subclasses of cell types. We performed style transfer to synthesize virtual H&E from MxIF and transferred the higher density labels from MxIF to these virtual H&E images. We then evaluated the efficacy of learning in this approach. We identified helper T and progenitor nuclei with positive predictive values of 0.34 ± 0.15 (prevalence 0.03 ± 0.01) and 0.47 ± 0.1 (prevalence 0.07 ± 0.02) respectively on virtual H&E. This approach represents a promising step towards automating annotation in digital pathology.
Magnetic resonance images are often acquired as several 2D slices and stacked into a 3D volume, yielding a lower through-plane resolution than in-plane resolution. Many super-resolution (SR) methods have been proposed to address this, including those that use the inherent high-resolution (HR) in-plane signal as HR data to train deep neural networks. Techniques with this approach are generally both self-supervised and internally trained, so no external training data is required. However, in such a training paradigm limited data are present for training machine learning models and the frequency content of the in-plane data may be insufficient to capture the true HR image. In particular, the recovery of high frequency information is usually lacking. In this work, we show this shortcoming with Fourier analysis; we subsequently propose and compare several approaches to address the recovery of high frequency information. We test a particular internally trained self-supervised method named SMORE on ten subjects at three common clinical resolutions with three types of modification: frequency-type losses (Fourier and wavelet), feature-type losses, and low-resolution re-gridding strategies for estimating the residual. We find a particular combination to balance between signal recovery in both spatial and frequency domains qualitatively and quantitatively, yet none of the modifications alone or in tandem yield a vastly superior result. We postulate that there may either be limits on internally trained techniques that such modifications cannot address, or limits on modeling SR as finding a map from low-resolution to HR, or both.
Managing patients with hydrocephalus and cerebrospinal fluid disorders requires repeated head imaging. In adults, this is typically done with computed tomography (CT) or less commonly magnetic resonance imaging (MRI). However, CT poses cumulative radiation risks and MRI is costly. Transcranial ultrasound is a radiation-free, relatively inexpensive, and optionally point-of-care alternative. The initial use of this modality has involved measuring gross brain ventricle size by manual annotation. In this work, we explore the use of deep learning to automate the segmentation of brain right ventricle from transcranial ultrasound images. We found that the vanilla U-Net architecture encountered difficulties in accurately identifying the right ventricle, which can be attributed to challenges such as limited resolution, artifacts, and noise inherent in ultrasound images. We further explore the use of coordinate convolution to augment the U-Net model, which allows us to take advantage of the established acquisition protocol. This enhancement yielded a statistically significant improvement in performance, as measured by the Dice similarity coefficient. This study presents, for the first time, the potential capabilities of deep learning in automating hydrocephalus assessment from ultrasound imaging.
KEYWORDS: Lawrencium, Magnetic resonance imaging, Super resolution, Image processing, Education and training, Diffusion, Image quality, Radon, Data modeling, Denoising
Anisotropic Low-Resolution (LR) Magnetic Resonance (MR) images are fast to obtain but hinder automated processing. We propose to use Denoising Diffusion Probabilistic Models (DDPMs) to super-resolve these 2D-acquired LR MR slices. This paper introduces AniRes2D, a novel approach combining DDPM with a residual prediction for 2D Super-Resolution (SR). Results demonstrate that AniRes2D outperforms several other DDPM-based models in quantitative metrics, visual quality, and out-of-domain evaluation. We use a trained AniRes2D to super-resolve 3D volumes slice by slice, where comparative quantitative results and reduced skull aliasing are achieved compared to a recent state-of-the-art self-supervised 3D super-resolution method. Furthermore, we explored the use of Noise Conditioning Augmentation (NCA) as an alternative augmentation technique for DDPM-based SR models, but it was found to reduce performance. Our findings contribute valuable insights to the application of DDPMs for SR of anisotropic MR images.
KEYWORDS: Education and training, Magnetic resonance imaging, Super resolution, Interpolation, Animals, Lawrencium, Animal model studies, 3D acquisition, Brain, Image quality
Animal models are pivotal in disease research and the advancement of therapeutic methods. The translation of results from these models to clinical applications is enhanced by employing technologies which are consistent for both humans and animals, like Magnetic Resonance Imaging (MRI), offering the advantage of longitudinal disease evaluation without compromising animal welfare. However, current animal MRI techniques predominantly employ 2D acquisitions due to constraints related to organ size, scan duration, image quality, and hardware limitations. While 3D acquisitions are feasible, they are constrained by longer scan times and ethical considerations related to extended sedation periods. This study evaluates the efficacy of SMORE, a self-supervised deep learning super-resolution approach, to enhance the through-plane resolution of anisotropic 2D MRI scans into isotropic resolutions. SMORE accomplishes this by self-training with high-resolution in-plane data, thereby eliminating domain discrepancies between the input data and external training sets. The approach is tested on mouse MRI scans acquired across a range of through-plane resolutions. Experimental results show SMORE substantially outperforms traditional interpolation methods. Additionally, we find that pre-training offers a promising approach to reduce processing time without compromising performance.
Deep learning algorithms using Magnetic Resonance (MR) images have demonstrated state-of-the-art performance in the automated segmentation of Multiple Sclerosis (MS) lesions. Despite their success, these algorithms may fail to generalize across sites or scanners, leading to domain generalization errors. Few-shot or one-shot domain adaptation is an option to reduce the generalization error using limited labeled data from the target domain. However, this approach may not yield satisfactory performance due to the limited data available for adaptation. In this paper, we aim to address this issue by integrating one-shot adaptation data with harmonized training data that includes labels. Our method synthesizes new training data with a contrast similar to that of the test domain, through a process referred to as “contrast harmonization” in MRI. Our experiments show that combining one-shot adaptation data with harmonized training data outperformed the use of either one of the data sources alone. Domain adaptation using only harmonized training data achieved comparable or even better performance compared to one-shot adaptation. In addition, all adaptations only required light fine-tuning of two to five epochs for convergence.
Linear registration to a standard space is a crucial early step in the processing of magnetic resonance images (MRIs) of the human brain. Thus an accurate registration is essential for subsequent image processing steps, as well as downstream analyses. Registration failures are not uncommon due to poor image quality, irregular head shapes, and bad initialization. Traditional quality assurance (QA) for registration requires a substantial manual assessment of the registration results. In this paper, we propose an automatic quality assurance method for the rigid registration of brain MRIs. Without using any manual annotations in the model training, our proposed QA method achieved 99.1% sensitivity and 86.7% specificity in a pilot study on 537 T1-weighted scans acquired from multiple imaging centers.
Generative priors for magnetic resonance (MR) images have been used in a number of medical image analysis applications. Due to the plethora of deep learning methods based on 2D medical images, it would be beneficial to have a generator trained on complete, high-resolution 2D head MR slices from multiple orientations and multiple contrasts. In this work, we trained a StyleGAN3-T model for head MR slices for T1 and T2-weighted contrasts on public data. We restricted the training corpus of this model to slices from 1mm isotropic volumes corresponding to three standard radiological views with set pre-processing steps. In order to retain full applicability to downstream tasks, we did not skull-strip the images. Several analyses of the trained network, including examination of qualitative samples, interpolation of latent codes, and style mixing, demonstrate the expressivity of the network. Images from this network can be used for a variety of downstream tasks. The weights are open-sourced and are available at https://gitlab.com/iacl/high-res-mri-head-slice-gan.
Fully automatic classification of magnetic resonance (MR) brain images into different contrasts is desirable for facilitating image processing pipelines, as well as for indexing and retrieving from medical image archives. In this paper, we present an approach based on a Siamese neural network to learn a discriminative feature representation for MR contrast classification. The proposed method is shown to outperform a traditional deep convolutional neural network method and a template matching method in identifying five different MR contrasts of input brain volumes with a variety of pathologies, achieving 98.59% accuracy. In addition, our approach permits one-shot learning, which allows generalization to new classes not seen in the training set with only one example of each new class. We demonstrate accurate one-shot learning performance on a sixth MR contrast that was not included in the original training.
Deep learning promises the extraction of valuable information from traumatic brain injury (TBI) datasets and depends on efficient navigation when using large-scale mixed computed tomography (CT) datasets from clinical systems. To ensure a cleaner signal while training deep learning models, removal of computed tomography angiography (CTA) and scans with streaking artifacts is sensible. On massive datasets of heterogeneously sized scans, time-consuming manual quality assurance (QA) by visual inspection is still often necessary, despite the expectation of CTA annotation (artifact annotation is not expected). We propose an automatic QA approach for retrieving CT scans without artifacts by representing 3D scans as 2D axial slice montages and using a multi-headed convolutional neural network to detect CT vs CTA and artifact vs no artifact. We sampled 848 scans from a mixed CT dataset of TBI patients and performed 4-fold stratified cross-validation on 698 montages followed by an ablation experiment—150 stratified montages were withheld for external validation evaluation. Aggregate AUC for our main model was 0.978 for CT detection, 0.675 for artifact detection during cross validation and 0.965 for CT detection, 0.698 for artifact detection on the external validation set, while the ablated model showed 0.946 for CT detection, 0.735 for artifact detection during cross-validation and 0.937 for CT detection, 0.708 for artifact detection on the external validation set. While our approach is successful for CT detection, artifact detection performance is potentially depressed due to the heterogeneity of present streaking artifacts and a suboptimal number of artifact scans in our training data.
White matter lesion (WML) segmentation applied to magnetic resonance imaging (MRI) scans of people with multiple sclerosis has been an area of extensive research in recent years. As with most tasks in medical imaging, deep learning (DL) methods have proven very effective and have quickly replaced existing methods. Despite the improvement offered by these networks, there are still shortcomings with these DL approaches. In this work, we compare several DL algorithms, as well as methods for ensembling the results of those algorithms, for performing MS lesion segmentation. An ensemble approach is shown to best estimate total WML and has the highest agreement with manual delineations.
KEYWORDS: Image segmentation, Magnetic resonance imaging, Data acquisition, Brain, Image processing algorithms and systems, Image processing, Data modeling, Tissues, Medical imaging, Neural networks
Purpose: Generalizability is an important problem in deep neural networks, especially with variability of data acquisition in clinical magnetic resonance imaging (MRI). Recently, the spatially localized atlas network tiles (SLANT) can effectively segment whole brain, non-contrast T1w MRI with 132 volumetric labels. Transfer learning (TL) is a commonly used domain adaptation tool to update the neural network weights for local factors, yet risks degradation of performance on the original validation/test cohorts.
Approach: We explore TL using unlabeled clinical data to address these concerns in the context of adapting SLANT to scanning protocol variations. We optimize whole-brain segmentation on heterogeneous clinical data by leveraging 480 unlabeled pairs of clinically acquired T1w MRI with and without intravenous contrast. We use labels generated on the pre-contrast image to train on the post-contrast image in a five-fold cross-validation framework. We further validated on a withheld test set of 29 paired scans over a different acquisition domain.
Results: Using TL, we improve reproducibility across imaging pairs measured by the reproducibility Dice coefficient (rDSC) between the pre- and post-contrast image. We showed an increase over the original SLANT algorithm (rDSC 0.82 versus 0.72) and the FreeSurfer v6.0.1 segmentation pipeline (rDSC = 0.53). We demonstrate the impact of this work decreasing the root-mean-squared error of volumetric estimates of the hippocampus between paired images of the same subject by 67%.
Conclusion: This work demonstrates a pipeline for unlabeled clinical data to translate algorithms optimized for research data to generalize toward heterogeneous clinical acquisitions.
Generalizability is an important problem in deep neural networks, especially in the context of the variability of data acquisition in clinical magnetic resonance imaging (MRI). Recently, the Spatially Localized Atlas Network Tiles (SLANT) approach has been shown to effectively segment whole brain non-contrast T1w MRI with 132 volumetric labels. Enhancing generalizability of SLANT would enable broader application of volumetric assessment in multi-site studies. Transfer learning (TL) is commonly to update neural network weights for local factors; yet, it is commonly recognized to risk degradation of performance on the original validation/test cohorts. Here, we explore TL by data augmentation to address these concerns in the context of adapting SLANT to anatomical variation (e.g., adults versus children) and scanning protocol (e.g., non-contrast research T1w MRI versus contrast-enhanced clinical T1w MRI). We consider two datasets: First, 30 T1w MRI of young children with manually corrected volumetric labels, and accuracy of automated segmentation defined relative to the manually provided truth. Second, 36 paired datasets of pre- and post-contrast clinically acquired T1w MRI, and accuracy of the post-contrast segmentations assessed relative to the pre-contrast automated assessment. For both studies, we augment the original TL step of SLANT with either only the new data or with both original and new data. Over baseline SLANT, both approaches yielded significantly improved performance (pediatric: 0.89 vs. 0.82 DSC, p<0.001; contrast: 0.80 vs 0.76, p<0.001 ). The performance on the original test set decreased with the new-data only transfer learning approach, so data augmentation was superior to strict transfer learning.
Multiple instance learning (MIL) is a supervised learning methodology that aims to allow models to learn instance class labels from bag class labels, where a bag is defined to contain multiple instances. MIL is gaining traction for learning from weak labels but has not been widely applied to 3D medical imaging. MIL is well-suited to clinical CT acquisitions since (1) the highly anisotropic voxels hinder application of traditional 3D networks and (2) patch-based networks have limited ability to learn whole volume labels. In this work, we apply MIL with a deep convolutional neural network to identify whether clinical CT head image volumes possess one or more large hemorrhages (>; 20cm3 ), resulting in a learned 2D model without the need for 2D slice annotations. Individual image volumes are considered separate bags, and the slices in each volume are instances. Such a framework sets the stage for incorporating information obtained in clinical reports to help train a 2D segmentation approach. Within this context, we evaluate the data requirements to enable generalization of MIL by varying the amount of training data. Our results show that a training size of at least 400 patient image volumes was needed to achieve accurate per-slice hemorrhage detection. Over a five-fold cross-validation, the leading model, which made use of the maximum number of training volumes, had an average true positive rate of 98.10%, an average true negative rate of 99.36%, and an average precision of 0.9698. The models have been made available along with source code1 to enabled continued exploration and adaption of MIL in CT neuroimaging.
Diffusion weighted magnetic resonance imaging (DW-MRI) is interpreted as a quantitative method that is sensitive to tissue microarchitecture at a millimeter scale. However, the sensitization is dependent on acquisition sequences (e.g., diffusion time, gradient strength, etc.) and susceptible to imaging artifacts. Hence, comparison of quantitative DW-MRI biomarkers across field strengths (including different scanners, hardware performance, and sequence design considerations) is a challenging area of research. We propose a novel method to estimate microstructure using DW-MRI that is robust to scanner difference between 1.5T and 3T imaging. We propose to use a null space deep network (NSDN) architecture to model DW-MRI signal as fiber orientation distributions (FOD) to represent tissue microstructure. The NSDN approach is consistent with histologically observed microstructure (on previously acquired ex vivo squirrel monkey dataset) and scan-rescan data. The contribution of this work is that we incorporate identical dual networks (IDN) to minimize the influence of scanner effects via scan-rescan data. Briefly, our estimator is trained on two datasets. First, a histology dataset was acquired on three squirrel monkeys with corresponding DW-MRI and confocal histology (512 independent voxels). Second, 37 control subjects from the Baltimore Longitudinal Study of Aging (67-95 y/o) were identified who had been scanned at 1.5T and 3T scanners (b-value of 700 s/mm2 , voxel resolution at 2.2mm, 30-32 gradient volumes) with an average interval of 4 years (standard deviation 1.3 years). After image registration, we used paired white matter (WM) voxels for 17 subjects and 440 histology voxels for training and 20 subjects and 72 histology voxels for testing. We compare the proposed estimator with super-resolved constrained spherical deconvolution (CSD) and a previously presented regression deep neural network (DNN). NSDN outperformed CSD and DNN in angular correlation coefficient (ACC) 0.81 versus 0.28 and 0.46, mean squared error (MSE) 0.001 versus 0.003 and 0.03, and general fractional anisotropy (GFA) 0.05 versus 0.05 and 0.09. Further validation and evaluation with contemporaneous imaging are necessary, but the NSDN is promising avenue for building understanding of microarchitecture in a consistent and deviceindependent manner.
Machine learning models are becoming commonplace in the domain of medical imaging, and with these methods comes an ever-increasing need for more data. However, to preserve patient anonymity it is frequently impractical or prohibited to transfer protected health information (PHI) between institutions. Additionally, due to the nature of some studies, there may not be a large public dataset available on which to train models. To address this conundrum, we analyze the efficacy of transferring the model itself in lieu of data between different sites. By doing so we accomplish two goals: 1) the model gains access to training on a larger dataset that it could not normally obtain and 2) the model better generalizes, having trained on data from separate locations. In this paper, we implement multi-site learning with disparate datasets from the National Institutes of Health (NIH) and Vanderbilt University Medical Center (VUMC) without compromising PHI. Three neural networks are trained to convergence on a computed tomography (CT) brain hematoma segmentation task: one only with NIH data, one only with VUMC data, and one multi-site model alternating between NIH and VUMC data. Resultant lesion masks with the multi-site model attain an average Dice similarity coefficient of 0.64 and the automatically segmented hematoma volumes correlate to those done manually with a Pearson correlation coefficient of 0.87, corresponding to an 8% and 5% improvement, respectively, over the single-site model counterparts.
Magnetic Resonance (MR) imaging allows the acquisition of images with different contrast properties depending on the acquisition protocol and the magnetic properties of tissues. Many MR brain image processing techniques, such as tissue segmentation, require multiple MR contrasts as inputs, and each contrast is treated differently. Thus it is advantageous to automate the identification of image contrasts for various purposes, such as facilitating image processing pipelines, and managing and maintaining large databases via content-based image retrieval (CBIR). Most automated CBIR techniques focus on a two-step process: extracting features from data and classifying the image based on these features. We present a novel 3D deep convolutional neural network (CNN)- based method for MR image contrast classification. The proposed CNN automatically identifies the MR contrast of an input brain image volume. Specifically, we explored three classification problems: (1) identify T1-weighted (T1-w), T2-weighted (T2-w), and fluid-attenuated inversion recovery (FLAIR) contrasts, (2) identify pre vs postcontrast T1, (3) identify pre vs post-contrast FLAIR. A total of 3418 image volumes acquired from multiple sites and multiple scanners were used. To evaluate each task, the proposed model was trained on 2137 images and tested on the remaining 1281 images. Results showed that image volumes were correctly classified with 97.57% accuracy.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.