Invasive ductal carcinoma (IDC) represents the most prevalent form of breast cancer. Automatic and precise IDC grading is essential for the clinical assessment of tumor status. However, current methodologies struggle with accurate grading due to the intricate spatial structure within IDC tumor regions, coupled with significant similarities between different classes of images and high variability within the same class, posing a substantial challenge for IDC grading. To tackle these challenges, we propose a novel multi-scale attention model named MS-HiFuse. This model integrates the multi-scale convolution and the multi-head attention mechanism into the local feature block of the HiFuse network, creating an enhanced multi-scale local feature block that more effectively captures the nuanced features of the tumor area and facilitates the learning of fine-grained feature representations. Furthermore, to overcome the limited number of original data and to ensure data quality, we augment the dataset through operations such as maximum rectangle cropping, subdivision of plots, and image flipping for each sample. Comparison experiments demonstrate that our proposed MSHiFuse network attains an AUC of 80.84%, outperforming both the current competing networks and the original model. Hence, the application of the MS-HiFuse network proposed herein holds significant promise for the grading of IDC pathology images.
In computational pathology, training and inference of conventional deep convolutional neural networks (CNN) are usually limited to patches of small sizes (e.g., 256 × 256) sampled from whole slide images. In practice, however, diagnostic and prognostic information could lie within the context of tumor microenvironment across multiple regions, far beyond the scope of individual patches. For instance, the spatial relationship of tumor-infiltrating lymphocytes (TIL) across regions of interest might be prognostic for non-small cell lung cancer (NSCLC). This poses a multi-instance learning (MIL) problem, and a single-patch-driven CNN typically fails to learn spatial information and context between multiple patches, especially their spatial relationship. In this work, we present a cell graph-based MIL framework to predict the risk of death for early-stage NSCLC by aggregating feature representation of TIL-enclosing patches according to their spatial relationship. Inspired by PATCHY-SAN, a graph-embedding framework for CNNs, we use graph kernel-based approaches to embed a bag of patches into a sequence with their spatial information encoded into the sequence order. A transformer model was then trained to aggregate patch-level features based on spatial information. We demonstrate the capability of this framework to predict the likelihood of the patient with NSCLC in two cohorts (n=240) to survive for more than 5 years. The training cohort (n=195) comprised hematoxylin and eosin (H&E)-stained whole slide images (WSI), while the testing cohort (n=45) comprised H&E-stained tumor microarrays (TMA). We show that, with the spatial context of multiple patches encoded as an ordered patch sequence, the performance in the testing cohort of our approach achieves an area under the receiver operating characteristic curve (AUC) of 0.836 (p=0.009; HR=5.62), as opposed to a baseline conventional CNN with an AUC of 0.542 (p=0.105; HR=1.66). The results suggest that the Transformer is a generic spatial information aware MIL framework that can learn the spatial relationship of multiple TIL-enclosing patches from the graph representation of immune cells.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.