Medical image analysis approaches such as data augmentation and domain adaption need huge amounts of realistic medical images. Generating realistic medical images by machine learning is a feasible approach. We propose L-former, a lightweight Transformer for realistic medical image generation. L-former can generate more reliable and realistic medical images than recent generative adversarial networks (GANs). Meanwhile, L-former does not consume as high computational cost as conventional Transformer-based generative models. L-former uses Transformers to generate low-resolution feature vectors at shallow layers, and uses convolutional neural networks to generate high-resolution realistic medical images at deep layers. Experimental results showed that L-former outperformed conventional GANs by FID scores 33.79 and 76.85 on two datasets, respectively. We further conducted a downstream study by using the images generated by L-former to perform a super-resolution task. A high PSNR score of 27.87 proved L-former’s ability to generate reliable images for super-resolution and showed its potential for applications in medical diagnosis.
Purpose: We propose a super-resolution (SR) method, named SR-CycleGAN, for SR of clinical computed tomography (CT) images to the micro-focus x-ray CT CT (μCT) level. Due to the resolution limitations of clinical CT (about 500 × 500 × 500 μm3 / voxel), it is challenging to obtain enough pathological information. On the other hand, μCT scanning allows the imaging of lung specimens with significantly higher resolution (about 50 × 50 × 50 μm3 / voxel or higher), which allows us to obtain and analyze detailed anatomical information. As a way to obtain detailed information such as cancer invasion and bronchioles from preoperative clinical CT images of lung cancer patients, the SR of clinical CT images to the μCT level is desired.
Approach: Typical SR methods require aligned pairs of low-resolution (LR) and high-resolution images for training, but it is infeasible to obtain precisely aligned paired clinical CT and μCT images. To solve this problem, we propose an unpaired SR approach that can perform SR on clinical CT to the μCT level. We modify a conventional image-to-image translation network named CycleGAN to an inter-modality translation network named SR-CycleGAN. The modifications consist of three parts: (1) an innovative loss function named multi-modality super-resolution loss, (2) optimized SR network structures for enlarging the input LR image to k2-times by width and height to obtain the SR output, and (3) sub-pixel shuffling layers for reducing computing time.
Results: Experimental results demonstrated that our method successfully performed SR of lung clinical CT images. SSIM and PSNR scores of our method were 0.54 and 17.71, higher than the conventional CycleGAN’s scores of 0.05 and 13.64, respectively.
Conclusions: The proposed SR-CycleGAN is usable for the SR of a lung clinical CT into μCT scale, while conventional CycleGAN output images with low qualitative and quantitative values. More lung micro-anatomy information could be observed to aid diagnosis, such as the shape of bronchioles walls.
This paper proposes a super-resolution (SR) method, for performing SR of medical images training on a newly-built lung clinical CT / micro CT dataset. Conventional SR methods are always trained on bicubic downsampled images (LR) / original images (HR) image pairs. However, registration precision between LR and HR images is not satisfying for SR. Low precision of registration results in conventional SR methods’ unsatisfactory performance in medical imaging. We propose a coarse-to-fine cascade framework for performing SR of medical images. First, we design a coarse SR network to translate LR medical images into coarse SR images. Next, we utilize a fully convolutional network (FCN) to perform fine SR (translate coarse SR images to fine SR images). We conducted experiments using a newly-built clinical / micro CT lung specimen dataset. Experimental results illustrated that our method obtained PSNR of 27.30 and SSIM of 0.75, outperforming conventional method’s PSNR 19.08 and SSIM 0.63.
This paper newly introduces multi-modality loss function for GAN-based super-resolution that can maintain image structure and intensity on unpaired training dataset of clinical CT and micro CT volumes. Precise non- invasive diagnosis of lung cancer mainly utilizes 3D multidetector computed-tomography (CT) data. On the other hand, we can take μCT images of resected lung specimen in 50 μm or higher resolution. However, μCT scanning cannot be applied to living human imaging. For obtaining highly detailed information such as cancer invasion area from pre-operative clinical CT volumes of lung cancer patients, super-resolution (SR) of clinical CT volumes to μCT level might be one of substitutive solutions. While most SR methods require paired low- and high-resolution images for training, it is infeasible to obtain precisely paired clinical CT and μCT volumes. We aim to propose unpaired SR approaches for clincial CT using micro CT images based on unpaired image translation methods such as CycleGAN or UNIT. Since clinical CT and μCT are very different in structure and intensity, direct appliation of GAN-based unpaired image translation methods in super-resolution tends to generate arbitrary images. Aiming to solve this problem, we propose new loss function called multi-modality loss function to maintain the similarity of input images and corresponding output images in super-resolution task. Experimental results demonstrated that the newly proposed loss function made CycleGAN and UNIT to successfully perform SR of clinical CT images of lung cancer patients into μCT level resolution, while original CycleGAN and UNIT failed in super-resolution.
Micro-CT is a nondestructive scanning device that is capable of capturing three dimensional structures at _m level. With the spread of this device uses in medical fields, it is expected that this device may bring further understanding of the human anatomy by analyzing three-dimensional micro structure from volume of in vivo specimens captured by micro-CT. In the topic of micro structure analysis of lung, the methods for extracting surface structures including the interlobular septa and the visceral pleura were not commonly studied. In this paper, we introduce a method to extract sheet structure such as the interlobular septa and the visceral pleura from micro-CT volumes. The proposed method consists of two steps: Hessian analysis based method for sheet structure extraction and Radial Structure Tensor combined with roundness evaluation for hollow-tube structure extraction. We adopted the proposed method on complex phantom data and a medical lung micro-CT volume. We confirmed the extraction of the interlobular septa from medical volume from experiments.
This paper presents a novel unsupervised segmentation method for the 3D microstructure in micro-computed tomography (micro-CT) images. Micro-CT scanning of resected lung cancer specimens can capture detailed and surrounding anatomical structures of them. However, its segmentation is difficult. Recently, many unsupervised learning methods have become greatly improved, especially in their ability to learn generative models such as variational auto-encoders (VAEs) and generative adversarial networks (GANs). Meanwhile, however, most of the recent segmentation methods using deep neural networks continue to rely on supervised learning. Therefore, it is rather difficult for these segmentation methods to cope with the growing number of unlabeled micro-CT images. In this paper, we develop a generative model that can infer segmentation labels by extending α-GAN, a principled combination that iterates variational inference and adversarial learning. Our method consists of two phases. In the first phase, we train our model by iterating two steps: (1) inferring pairs of continuous and discrete latent variables of image patches randomly extracted from an unlabeled image and (2) generating image patches from the inferred pairs of latent variables. In the second phase, our trained model assigns labels to patches from a target image in order to obtain the segmented image. We evaluated our method using three micro-CT images of a lung cancer specimen. The aim was to automatically divide each image into three regions: invasive carcinoma, noninvasive carcinoma, and normal tissue. Our experiments show promising results both quantitatively and qualitatively.
KEYWORDS: Image segmentation, Medical imaging, 3D image processing, Machine learning, Tissues, 3D acquisition, Convolution, Lung cancer, Information science
This paper presents a novel unsupervised segmentation method for 3D medical images. Convolutional neural networks (CNNs) have brought significant advances in image segmentation. However, most of the recent methods rely on supervised learning, which requires large amounts of manually annotated data. Thus, it is challenging for these methods to cope with the growing amount of medical images. This paper proposes a unified approach to unsupervised deep representation learning and clustering for segmentation. Our proposed method consists of two phases. In the first phase, we learn deep feature representations of training patches from a target image using joint unsupervised learning (JULE) that alternately clusters representations generated by a CNN and updates the CNN parameters using cluster labels as supervisory signals. We extend JULE to 3D medical images by utilizing 3D convolutions throughout the CNN architecture. In the second phase, we apply k-means to the deep representations from the trained CNN and then project cluster labels to the target image in order to obtain the fully segmented image. We evaluated our methods on three images of lung cancer specimens scanned with micro-computed tomography (micro-CT). The automatic segmentation of pathological regions in micro-CT could further contribute to the pathological examination process. Hence, we aim to automatically divide each image into the regions of invasive carcinoma, noninvasive carcinoma, and normal tissue. Our experiments show the potential abilities of unsupervised deep representation learning for medical image segmentation.
This paper presents a novel method for unsupervised segmentation of pathology images. Staging of lung cancer is a major factor of prognosis. Measuring the maximum dimensions of the invasive component in a pathology images is an essential task. Therefore, image segmentation methods for visualizing the extent of invasive and noninvasive components on pathology images could support pathological examination. However, it is challenging for most of the recent segmentation methods that rely on supervised learning to cope with unlabeled pathology images. In this paper, we propose a unified approach to unsupervised representation learning and clustering for pathology image segmentation. Our method consists of two phases. In the first phase, we learn feature representations of training patches from a target image using the spherical k-means. The purpose of this phase is to obtain cluster centroids which could be used as filters for feature extraction. In the second phase, we apply conventional k-means to the representations extracted by the centroids and then project cluster labels to the target images. We evaluated our methods on pathology images of lung cancer specimen. Our experiments showed that the proposed method outperforms traditional k-means segmentation and the multithreshold Otsu method both quantitatively and qualitatively with an improved normalized mutual information (NMI) score of 0.626 compared to 0.168 and 0.167, respectively. Furthermore, we found that the centroids can be applied to the segmentation of other slices from the same sample.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.