Convolutional neural networks (CNNs) are known to fail if a difference exists in the data they are trained and tested on, known as domain shifts. This sensitivity is particularly problematic in computational pathology, where various factors, such as different staining protocols and stain providers, introduce domain shifts. Many solutions have been proposed in the literature to address this issue, with data augmentation being one of the most popular approaches. While data augmentation can significantly enhance the performance of a CNN in the presence of domain shifts, it does not guarantee robustness. Therefore, it would be advantageous to integrate generalization to specific sources of domain shift directly into the network’s capabilities when known to be present in the real world. In this study, we draw inspiration from roto-translation equivariant CNNs and propose a customized layer to enhance domain generalization and the CNN’s ability to handle variations in staining. To evaluate our approach, we conduct experiments on two publicly available, multi-institutional datasets: CAMELYON17 and MIDOG.
SignificanceAlthough the registration of restained sections allows nucleus-level alignment that enables a direct analysis of interacting biomarkers, consecutive sections only allow the transfer of region-level annotations. The latter can be achieved at low computational cost using coarser image resolutions.PurposeIn digital histopathology, virtual multistaining is important for diagnosis and biomarker research. Additionally, it provides accurate ground truth for various deep-learning tasks. Virtual multistaining can be obtained using different stains for consecutive sections or by restaining the same section. Both approaches require image registration to compensate for tissue deformations, but little attention has been devoted to comparing their accuracy.ApproachWe compared affine and deformable variational image registration of consecutive and restained sections and analyzed the effect of the image resolution that influences accuracy and required computational resources. The registration was applied to the automatic nonrigid histological image registration (ANHIR) challenge data (230 consecutive slide pairs) and the hyperparameters were determined. Then without changing the parameters, the registration was applied to a newly published hybrid dataset of restained and consecutive sections (HyReCo, 86 slide pairs, 5404 landmarks).ResultsWe obtain a median landmark error after registration of 6.5 μm (HyReCo) and 24.1 μm (ANHIR) between consecutive sections. Between restained sections, the median registration error is 2.2 and 0.9 μm in the two subsets of the HyReCo dataset. We observe that deformable registration leads to lower landmark errors than affine registration in both cases (p < 0.001), though the effect is smaller in restained sections.ConclusionDeformable registration of consecutive and restained sections is a valuable tool for the joint analysis of different stains.
Deep learning is a state-of-the-art pattern recognition technique that has been found extremely powerful for the analysis of digitized histopathological slides. The number of studies presenting highly promising results for solving diagnostic tasks in histopathology has grown exponentially over the last few years. Examples are subtyping of lung and skin tumors, breast and prostate cancer grading, and detection of metastases. Unfortunately, few studies so far include an external validation using large, independent cohorts, let alone study the true clinical usefulness in prospective studies. As a result, the balance between promise and hype in public opinion may be skewed.
In this talk, I will present some of the current possibilities of AI for histopathology, and discuss potential future developments. I will also address the challenges that have to be overcome before we can deliver true value to pathologists, patients, and the healthcare system. These are in many cases of a non-technical nature. Issues related to the availability of large heterogeneous data sets, possibilities to obviate expensive manual labeling of data, workflow integration, ethical, legal and regulatory issues, explainability, and reimbursement models all lie on the way forward for full adoption of computational pathology.
In this work, we propose a deep learning system for weakly supervised object detection in digital pathology whole slide images. We designed the system to be organ- and object-agnostic, and to be adapted on-the-fly to detect novel objects based on a few examples provided by the user. We tested our method on detection of healthy glands in colon biopsies and ductal carcinoma in situ (DCIS) of the breast, showing that (1) the same system is capable of adapting to detect requested objects with high accuracy, namely 87% accuracy assessed on 582 detections in colon tissue, and 93% accuracy assessed on 163 DCIS detections in breast tissue; (2) in some settings, the system is capable of retrieving similar cases with little to none false positives (i.e., precision equal to 1.00); (3) the performance of the system can benefit from previously detected objects with high confidence that can be reused in new searches in an iterative fashion.
Classification of non-small-cell lung cancer (NSCLC) into adenocarcinoma (LUAD) and squamous cell carcinoma (LUSC) via histopathology is a vital prerequisite to select the appropriate treatment for lung cancer patients. Most machine learning approaches rely on manually annotating large numbers of whole slide images (WSI) for training. However, manually delineating cancer areas or even single cancer cells on hundreds or thousands of slides is tedious, subjective and requires highly trained pathologists. We propose to use Neural Image Compression (NIC), which requires only slide-level labels, to classify NSCLC into LUSC and LUAD. NIC consists of two phases/networks. In the first phase the slides are compressed with a convolutional neural network (CNN) acting as an encoder. In the second phase the compressed slides are classified with a second CNN. We trained our classification model on >2,000 NIC-compressed slides from the TCGA and TCIA databases and evaluated the model performance additionally on several internal and external cohorts. We show that NIC approaches state of the art performance on lung cancer classification, with an average AUC of 0.94 on the TCGA and TCIA testdata, and AUCs between 0.84 and 0.98 on other independent datasets.
Prostate cancer is generally graded by pathologists based on hematoxylin and eosin (H and E) stained slides. Because of the large size of the tumor areas in radical prostatectomies (RP), this task can be tedious and error prone with known high interobserver variability. Recent advancements in deep learning have enabled development of automated systems that may assist pathologists in prostate diagnostics. As prostate cancer originates from glandular tissue, an important prerequisite for development of such algorithms is the possibility to automatically differentiate between glandular tissue and other tissues. In this paper, we propose a method for automatically segmenting epithelial tissue in digitally scanned prostatectomy slides based on deep learning. We collected 30 single-center whole mount tissue sections, with reported Gleason growth patterns ranging from 3 to 5, from 27 patients that underwent RP. Two different network architectures, U-Net and regular fully convolutional networks with varying depths, were trained using a set of sparsely annotated slides. We evaluated the trained networks on exhaustively annotated regions from a separate test set. The test set contained both healthy and cancerous epithelium with different Gleason growth patterns. The results show the effectiveness of our approach given a pixel-based AUC score of 0.97. Our method contains no prior assumptions on glandular morphology, does not directly rely on the presence of lumina, and all features are learned by the network itself. The generated segmentation can be used to highlight regions of interest for pathologists and to improve cancer annotations to further enhance an automatic cancer grading system.
The number of mitotic figures per tumor area observed in hematoxylin and eosin (H and E) histological tissue sections under light microscopy is an important biomarker for breast cancer prognosis. Whole-slide imaging and computational pathology have enabled the development of automatic mitosis detection algorithms based on convolutional neural networks (CNNs). These models can suffer from high generalization error, i.e. trained networks often underperform on datasets originating from pathology laboratories different than the one that provided the training data, mainly due to the presence of inter-laboratory stain variations. We propose a novel data augmentation strategy that exploits the properties of the H and E color space to simulate a broad range of realistic H and E stain variations. To our best knowledge, this is the first time that data augmentation is performed directly in the H and E color space, instead of RGB. The proposed technique uses color deconvolution to transform RGB images into the H and E color space, modifies the H and E color channels stochastically, and projects them back to RGB space. We trained a CNN-based mitosis detector on homogeneous data from a single institution, and tested its performance on an external, multicenter cohort that contained a wide range of unseen H and E stain variations. We compared CNNs trained with and without the proposed augmentation strategy and observed a significant improvement in performance and robustness to unseen stain variations when the new color augmentation technique was included. In essence, we have shown that CNNs can be made robust to inter-lab stain variation by incorporating extensive stain augmentation techniques.
Assessment of immunohistochemically stained slides is often a crucial diagnostic step in clinical practice. However, as this assessment is generally performed visually by pathologists it can suffer from significant inter-observer variability. The introduction of whole slide scanners facilitates automated analysis of immunohistochemical slides. Color deconvolution (CD) is one of the most popular first steps in quantifying stain density in histopathological images. However, color deconvolution requires stain color vectors for accurate unmixing. Often it is assumed that these stain vectors are static. In practice, however, they are influenced by many factors. This can cause inferior CD unmixing and thus typically results in poor quantification. Some automated methods exist for color stain vector estimation, but most depend on a significant amount of each stain to be present in the whole slide images. In this paper we propose a method for automatically finding stain color vectors and unmixing IHC stained whole slide images, even when some stains are sparsely expressed. We collected 16 tonsil slides and stained them for different periods of time with hematoxylin and a DAB-colored proliferation marker Ki67. RGB pixels of WSI images were converted to the hue saturation density (HSD) color domain and subsequently K-means clustering was used to separate stains and calculate the stain color vectors for each slide. Our results show that staining time affects the stain vectors and that calculating a unique stain vector for each slide results in better unmixing results than using a standard stain vector.
Diagnoses in kidney disease often depend on quantification and presence of specific structures in the tissue. The progress in the field of whole-slide imaging and deep learning has opened up new possibilities for automatic analysis of histopathological slides. An initial step for renal tissue assessment is the differentiation and segmentation of relevant tissue structures in kidney specimens. We propose a method for segmentation of renal tissue using convolutional neural networks. Nine structures found in (pathological) renal tissue are included in the segmentation task: glomeruli, proximal tubuli, distal tubuli, arterioles, capillaries, sclerotic glomeruli, atrophic tubuli, in ammatory infiltrate and fibrotic tissue. Fifteen whole slide images of normal cortex originating from tumor nephrectomies were collected at the Radboud University Medical Center, Nijmegen, The Netherlands. The nine classes were sparsely annotated by a PhD student, experienced in the field of renal histopathology (MH). Experiments were performed with three different network architectures: a fully convolutional network, a multi-scale fully convolutional network and a U-net. We assessed the added benefit of combining the networks into an ensemble. We performed four-fold cross validation and report the average pixel accuracy per annotation for each class. Results show that convolutional neural net- works are able to accurately perform segmentation tasks in renal tissue, with accuracies of 90% for most classes.
Currently, histopathological tissue examination by a pathologist represents the gold standard for breast lesion diagnostics. Automated classification of histopathological whole-slide images (WSIs) is challenging owing to the wide range of appearances of benign lesions and the visual similarity of ductal carcinoma in-situ (DCIS) to invasive lesions at the cellular level. Consequently, analysis of tissue at high resolutions with a large contextual area is necessary. We present context-aware stacked convolutional neural networks (CNN) for classification of breast WSIs into normal/benign, DCIS, and invasive ductal carcinoma (IDC). We first train a CNN using high pixel resolution to capture cellular level information. The feature responses generated by this model are then fed as input to a second CNN, stacked on top of the first. Training of this stacked architecture with large input patches enables learning of fine-grained (cellular) details and global tissue structures. Our system is trained and evaluated on a dataset containing 221 WSIs of hematoxylin and eosin stained breast tissue specimens. The system achieves an AUC of 0.962 for the binary classification of nonmalignant and malignant slides and obtains a three-class accuracy of 81.3% for classification of WSIs into normal/benign, DCIS, and IDC, demonstrating its potential for routine diagnostics.
The detection of cell nuclei plays a key role in various histopathological image analysis problems. Considering the high variability of its applications, we propose a novel generic and trainable detection approach. Adaption to specific nuclei detection tasks is done by providing training samples. A trainable deconvolution and classification algorithm is used to generate a probability map indicating the presence of a nucleus. The map is processed by an extended watershed segmentation step to identify the nuclei positions. We have tested our method on data sets with different stains and target nuclear types. We obtained F1-measures between 0.83 and 0.93.
Automated detection of prostate cancer in digitized H and E whole-slide images is an important first step for computer-driven grading. Most automated grading algorithms work on preselected image patches as they are too computationally expensive to calculate on the multi-gigapixel whole-slide images. An automated multi-resolution cancer detection system could reduce the computational workload for subsequent grading and quantification in two ways: by excluding areas of definitely normal tissue within a single specimen or by excluding entire specimens which do not contain any cancer. In this work we present a multi-resolution cancer detection algorithm geared towards the latter. The algorithm methodology is as follows: at a coarse resolution the system uses superpixels, color histograms and local binary patterns in combination with a random forest classifier to assess the likelihood of cancer. The five most suspicious superpixels are identified and at a higher resolution more computationally expensive graph and gland features are added to refine classification for these superpixels. Our methods were evaluated in a data set of 204 digitized whole-slide H and E stained images of MR-guided biopsy specimens from 163 patients. A pathologist exhaustively annotated the specimens for areas containing cancer. The performance of our system was evaluated using ten-fold cross-validation, stratified according to patient. Image-based receiver operating characteristic (ROC) analysis was subsequently performed where a specimen containing cancer was considered positive and specimens without cancer negative. We obtained an area under the ROC curve of 0.96 and a 0.4 specificity at a 1.0 sensitivity.
KEYWORDS: Tissues, 3D image reconstruction, 3D image processing, 3D modeling, Image segmentation, Breast, Brain, Binary data, Medical imaging, Tolerancing
There is currently an increasing interest in combining the information obtained from radiology and histology with the intent of gaining a better understanding of how different tumour morphologies can lead to distinctive radiological signs which might predict overall treatment outcome. Relating information at different resolution scales is challenging. Reconstructing 3D volumes from histology images could be the key to interpreting and relating the radiological image signal to tissue microstructure. The goal of this study is to determine the minimum sampling (maximum spacing between histological sections through a fixed surgical specimen) required to create a 3D reconstruction of the specimen to a specific tolerance. We present initial results for one lumpectomy specimen case where 33 consecutive histology slides were acquired.
This paper presents a new algorithm for automatic detection of regions of interest in whole slide histopathological images. The proposed algorithm generates and classifies superpixels at multiple resolutions to detect regions of interest. The algorithm emulates the way the pathologist examines the whole slide histopathology image by processing the image at low magnifications and performing more sophisticated analysis only on areas requiring more detailed information. However, instead of the traditional usage of fixed sized rectangular patches for the identification of relevant areas, we use superpixels as the visual primitives to detect regions of interest. Rectangular patches can span multiple distinct structures, thus degrade the classification performance. The proposed multi-scale superpixel classification approach yields superior performance for the identification of the regions of interest. For the evaluation, a set of 10 whole slide histopathology images of breast tissue were used. Empirical evaluation of the performance of our proposed algorithm relative to expert manual annotations shows that the algorithm achieves an area under the Receiver operating characteristic (ROC) curve of 0.958, demonstrating its efficacy for the detection of regions of interest.
This paper presents data on the sources of variation of the widely used hematoxylin and eosin (H&E) histological
staining, as well as a new algorithm to reduce these variations in digitally scanned tissue sections. Experimental
results demonstrate that staining protocols in different laboratories and staining on different days of the week are
the major factors causing color variations in histopathological images. The proposed algorithm for standardizing
histology slides is based on an initial clustering of the image into two tissue components having different absorption
characteristics for different dyes. The color distribution for each tissue component is standardized by aligning
the 2D histogram of color distribution in the hue-saturation-density (HSD) model. Qualitative evaluation of the
proposed standardization algorithm shows that color constancy of the standardized images is improved. Quantitative
evaluation demonstrates that the algorithm outperforms competing methods. In conclusion, the paper
demonstrates that staining variations, which may potentially hamper usefulness of computer assisted analysis of
histopathological images, can be reduced considerably by applying the proposed algorithm.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.