Purpose: The purpose of our review paper is to examine many existing works of literature presenting the different methods utilized for diabetic retinopathy (DR) recognition employing deep learning (DL) and machine learning (ML) techniques, and also to address the difficulties faced in various datasets used by DR.

Approach: DR is a progressive illness and may become a reason for vision loss. Early identification of DR lesions is, therefore, helpful and prevents damage to the retina. However, it is a complex job in view of the fact that it is symptomless earlier, and also ophthalmologists have been needed in traditional approaches. Recently, automated identification of DR-based studies has been stated based on image processing, ML, and DL. We analyze the recent literature and provide a comparative study that also includes the limitations of the literature and future work directions.

Results: A relative analysis among the databases used, performance metrics employed, and ML and DL techniques adopted recently in DR detection based on various DR features, is presented.

Conclusions: Our review paper discusses the methods employed in DR detection along with the technical and clinical challenges that are encountered, which is missing in existing reviews, as well as future scopes to assist researchers in the field of retinal imaging.

Purpose: Spatial resolution for current scintillator-based computed tomography (CT) detectors is limited by the pixel size of about 1 mm. Direct conversion photon-counting detector prototypes with silicon- or cadmium-based detector materials have lately demonstrated spatial resolution equivalent to about 0.3 mm. We propose a development of the deep silicon photon-counting detector which will enable a resolution of 1  μm, a substantial improvement compared to the state of the art.

Approach: With the deep silicon sensor, it is possible to integrate CMOS electronics and reduce the pixel size at the same time as significant on-sensor data processing capability is introduced. A Gaussian curve can then be fitted to the charge cloud created in each interaction.We evaluate the feasibility of measuring the charge cloud shape of Compton interactions for deep silicon to increase the spatial resolution. By combining a Monte Carlo photon simulation with a charge transport model, we study the charge cloud distributions and induced currents as functions of the interaction position. For a simulated deep silicon detector with a pixel size of 12  μm, we present a method for estimating the interaction position.

Results: Using estimations for electronic noise and a lowest threshold of 0.88 keV, we obtain a spatial resolution equivalent to 1.37  μm in the direction parallel to the silicon wafer and 78.28  μm in the direction orthogonal to the wafer.

Conclusions: We have presented a simulation study of a deep silicon detector with a pixel size of 12  ×  500  μm² and a method to estimate the x-ray interaction position with ultra-high resolution. Higher spatial resolution can in general be important to detect smaller details in the image. The very high spatial resolution in one dimension could be a path to a practical implementation of phase contrast imaging in CT.

Purpose: Segmentation of the vessel tree from retinal fundus images can be used to track changes in the retina and be an important first step in a diagnosis. Manual segmentation is a time-consuming process that is prone to error; effective and reliable automation can alleviate these problems but one of the difficulties is uneven image background, which may affect segmentation performance.

Approach: We present a patch-based deep learning framework, based on a modified U-Net architecture, that automatically segments the retinal blood vessels from fundus images. In particular, we evaluate how various pre-processing techniques, images with either no processing, N4 bias field correction, contrast limited adaptive histogram equalization (CLAHE), or a combination of N4 and CLAHE, can compensate for uneven image background and impact final segmentation performance.

Results: We achieved competitive results on three publicly available datasets as a benchmark for our comparisons of pre-processing techniques. In addition, we introduce Bayesian statistical testing, which indicates little practical difference (Pr  >  0.99) between pre-processing methods apart from the sensitivity metric. In terms of sensitivity and pre-processing, the combination of N4 correction and CLAHE performs better in comparison to unprocessed and N4 pre-processing (Pr  >  0.87); but compared to CLAHE alone, the differences are not significant (Pr  ≈  0.38 to 0.88).

Conclusions: We conclude that deep learning is an effective method for retinal vessel segmentation and that CLAHE pre-processing has the greatest positive impact on segmentation performance, with N4 correction helping only in images with extremely inhomogeneous background illumination.

Purpose: Robust and accurate segmentation methods for the intracochlear anatomy (ICA) are a critical step in the image-guided cochlear implant programming process. We have proposed an active shape model (ASM)-based method and a deep learning (DL)-based method for this task, and we have observed that the DL method tends to be more accurate than the ASM method while the ASM method tends to be more robust.

Approach: We propose a DL-based U-Net-like architecture that incorporates ASM segmentation into the network. A quantitative analysis is performed on a dataset that consists of 11 cochlea specimens for which a segmentation ground truth is available. To qualitatively evaluate the robustness of the method, an experienced expert is asked to visually inspect and grade the segmentation results on a clinical dataset made of 138 image volumes acquired with conventional CT scanners and of 39 image volumes acquired with cone beam CT (CBCT) scanners. Finally, we compare training the network (1) first with the ASM results, and then fine-tuning it with the ground truth segmentation and (2) directly with the specimens with ground truth segmentation.

Results: Quantitative and qualitative results show that the proposed method increases substantially the robustness of the DL method while having only a minor detrimental effect (though not significant) on its accuracy. Expert evaluation of the clinical dataset shows that by incorporating the ASM segmentation into the DL network, the proportion of good segmentation cases increases from 60/177 to 119/177 when training only with the specimens and increases from 129/177 to 151/177 when pretraining with the ASM results.

Conclusions: A hybrid ASM and DL-based segmentation method is proposed to segment the ICA in CT and CBCT images. Our results show that combining DL and ASM methods leads to a solution that is both robust and accurate.

Purpose: Our goal is to propose a landmark- and contour-matching (LCM) registration method that uses both landmark information and approximate point correspondences to boost the similarity between image pairs with sparse landmark information.

Approach: A model for registering two-dimensional (2D) medical images with landmark information and contour-approximating landmarks was proposed. The model was also extended to accommodate the registration of three-dimensional (3D) cardiac images. We validated the LCM method on 2D hand x-rays and 3D porcine cardiac magnetic resonance images. The following metrics were used to assess the quality of specific aspects of the registered images: Dice similarity coefficient for the overall image overlap, target registration error for pointwise correspondence, and interior angle for local curvature.

Results: Target registrations were reduced from 27.12 to 0.01 mm post-LCM registration. Implementing the proposed algorithm also led to a 112% average improvement in image similarity in terms of Dice coefficients. In addition, interior angle measurements indicate that the proposed method preserved the local curvature at major reference landmarks and mitigated the appearance of deformities in the registered images.

Conclusions: The proposed method addressed several issues associated with purely landmark-based techniques, such as iterative closest point registration and thin plate spline interpolation. Furthermore, it provided accurate registration results even in the presence of landmark localization errors.

Purpose: Accurate classification of COVID-19 in chest radiographs is invaluable to hard-hit pandemic hot spots. Transfer learning techniques for images using well-known convolutional neural networks show promise in addressing this problem. These methods can significantly benefit from supplemental training on similar conditions, considering that there currently exists no widely available chest x-ray dataset on COVID-19. We evaluate whether targeted pretraining for similar tasks in radiography labeling improves classification performance in a sample radiograph dataset containing COVID-19 cases.

Approach: We train a DenseNet121 to classify chest radiographs through six training schemes. Each training scheme is designed to incorporate cases from established datasets for general findings in chest radiography (CXR) and pneumonia, with a control scheme with no pretraining. The resulting six permutations are then trained and evaluated on a dataset of 1060 radiographs collected from 475 patients after March 2020, containing 801 images of laboratory-confirmed COVID-19 cases.

Results: Sequential training phases yielded substantial improvement in classification accuracy compared to a baseline of standard transfer learning with ImageNet parameters. The test set area under the receiver operating characteristic curve for COVID-19 classification improved from 0.757 in the control to 0.857 for the optimal training scheme in the available images.

Conclusions: We achieve COVID-19 classification accuracies comparable to previous benchmarks of pneumonia classification. Deliberate sequential training, rather than pooling datasets, is critical in training effective COVID-19 classifiers within the limitations of early datasets. These findings bring clinical-grade classification through CXR within reach for more regions impacted by COVID-19.

Purpose: Chest x-rays are complex to report accurately. Viral pneumonia is often subtle in its radiological appearance. In the context of the COVID-19 pandemic, rapid triage of cases and exclusion of other pathologies with artificial intelligence (AI) can assist over-stretched radiology departments. We aim to validate three open-source AI models on an external test set.

Approach: We tested three open-source deep learning models, COVID-Net, COVIDNet-S-GEO, and CheXNet for their ability to detect COVID-19 pneumonia and to determine its severity using 129 chest x-rays from two different vendors Phillips and Agfa.

Results: All three models detected COVID-19 pneumonia (AUCs from 0.666 to 0.778). Only the COVID Net-S-GEO and CheXNet models performed well on severity scoring (Pearson’s r 0.927 and 0.833, respectively); COVID-Net only performed well at either task on images taken with a Philips machine (AUC 0.735) and not an Agfa machine (AUC 0.598).

Conclusions: Chest x-ray triage using existing machine learning models for COVID-19 pneumonia can be successfully implemented using open-source AI models. Evaluation of the model using local x-ray machines and protocols is highly recommended before implementation to avoid vendor or protocol dependent bias.

Purpose: Surgery involves modifying anatomy to achieve a goal. Reconstructing anatomy can facilitate surgical care through surgical planning, real-time decision support, or anticipating outcomes. Tool motion is a rich source of data that can be used to quantify anatomy. Our work develops and validates a method for reconstructing the nasal septum from unstructured motion of the Cottle elevator during the elevation phase of septoplasty surgery, without need to explicitly delineate the surface of the septum.

Approach: The proposed method uses iterative closest point registration to initially register a template septum to the tool motion. Subsequently, statistical shape modeling with iterative most likely oriented point registration is used to fit the reconstructed septum to Cottle tip position and orientation during flap elevation. Regularization of the shape model and transformation is incorporated. The proposed methods were validated on 10 septoplasty surgeries performed on cadavers by operators of varying experience level. Preoperative CT images of the cadaver septums were segmented as ground truth.

Results: We estimated reconstruction error as the difference between the projections of the Cottle tip onto the surface of the reconstructed septum and the ground-truth septum segmented from the CT image. We found translational differences of 2.74  (  2.06  −  2.81  )    mm and a rotational differences of 8.95  (  7.11  −  10.55  )    deg between the reconstructed septum and the ground-truth septum [median (interquartile range)], given the optimal regularization parameters.

Conclusions: Accurate reconstruction of the nasal septum can be achieved from tool tracking data during septoplasty surgery on cadavers. This enables understanding of the septal anatomy without need for traditional medical imaging. This result may be used to facilitate surgical planning, intraoperative care, or skills assessment.

Purpose: Deep learning-based image super-resolution (DL-SR) has shown great promise in medical imaging applications. To date, most of the proposed methods for DL-SR have only been assessed using traditional measures of image quality (IQ) that are commonly employed in the field of computer vision. However, the impact of these methods on objective measures of IQ that are relevant to medical imaging tasks remains largely unexplored. We investigate the impact of DL-SR methods on binary signal detection performance.

Approach: Two popular DL-SR methods, the super-resolution convolutional neural network and the super-resolution generative adversarial network, were trained using simulated medical image data. Binary signal-known-exactly with background-known-statistically and signal-known-statistically with background-known-statistically detection tasks were formulated. Numerical observers (NOs), which included a neural network-approximated ideal observer and common linear NOs, were employed to assess the impact of DL-SR on task performance. The impact of the complexity of the DL-SR network architectures on task performance was quantified. In addition, the utility of DL-SR for improving the task performance of suboptimal observers was investigated.

Results: Our numerical experiments confirmed that, as expected, DL-SR improved traditional measures of IQ. However, for many of the study designs considered, the DL-SR methods provided little or no improvement in task performance and even degraded it. It was observed that DL-SR improved the task performance of suboptimal observers under certain conditions.

Conclusions: Our study highlights the urgent need for the objective assessment of DL-SR methods and suggests avenues for improving their efficacy in medical imaging applications.

Purpose: Transvaginal ultrasound (TVUS) is a widely used real-time and non-invasive imaging technique for fetal and maternal care. It can provide structural and functional measurements about the fetal brain, such as blood vessel diameter and blood flow. However, it lacks certain biochemical estimations, such as hemoglobin oxygen saturation (SO₂), which limits its ability to indicate a fetus at risk of birth asphyxia. Photoacoustic (PA) imaging has been steadily growing in recognition as a complement to ultrasound (US). Studies have shown PA imaging is capable of providing such biochemical estimations as SO₂ at relatively high penetration depth (up to 30 mm).

Approach: In this study, we have designed and developed a multi-modal (US, PA, and Doppler) endocavity imaging system (ECUSPA) around a commercialized TVUS probe (Philips ATL C9-5).

Results: The integrated system was evaluated through a set of in-vitro, ex-vivo, and in-vivo studies. Imaging of excised sheep brain tissue demonstrated the system’s utility and penetration depth in transfontanelle imaging conditions. The accuracy of using the spectroscopic PA imaging (sPA) method to estimate SO₂ was validated by comparing sPA oximetry results with the gold standard measurements indicated by a blood gas analyzer. The ability of US and Doppler to measure moving blood volume was evaluated in-vivo. Spectral unmixing capabilities were tested using fluorophores within sheep brains.

Conclusion: The developed system is a high resolution (about 200  μm at 30 mm depth), real-time (at 30 Hz), and quantitative (SO₂ estimation error <10  %  ) imaging tool with a total diameter less than 30 mm, making it suitable for intrapartum applications such as fetal and maternal diagnostics.

Purpose: Electromagnetic acoustic imaging (EMAI) is a hybrid imaging technique using radio-frequency irradiation to induce ultrasound (US), providing an US image in which spatial conductivity differences provide image contrast. The method is potentially clinically important in that the added diagnostic parameter has been shown to be useful in cancer detection and vascular space delineation.

Approach: We report the development of coil configurations and imaging processing techniques designed to address the low signal-to-noise of EMAI and demonstrate achievable resolution and contrast in phantoms along with EMAI signals in excised animal tissue. Experiment results are compared with theoretical calculations.

Results: EMAI signal intensities depend on the square of the ampere-turns in the coil radio frequency coil as predicted theoretically. Resolution is shown to be comparable to conventional US imaging with contrast and signal intensity depending on source conductivity. Optimizing signal-to-noise depends on coil design, orientation of the electromagnetic fields, and coherent processing.

Conclusions: Two-dimensional EMAI images are shown to have the expected resolution of conventional US with image contrast dependent on conductivity. Achievable signal-to-noise is sufficient to form potentially clinically useful images.

Purpose: Recent advances in computational image analysis offer the opportunity to develop automatic quantification of histologic parameters as aid tools for practicing pathologists. We aim to develop deep learning (DL) models to quantify nonsclerotic and sclerotic glomeruli on frozen sections from donor kidney biopsies.

Approach: A total of 258 whole slide images (WSI) from cadaveric donor kidney biopsies performed at our institution (n  =  123) and at external institutions (n  =  135) were used in this study. WSIs from our institution were divided at the patient level into training and validation datasets (ratio: 0.8:0.2), and external WSIs were used as an independent testing dataset. Nonsclerotic (n  =  22767) and sclerotic (n  =  1366) glomeruli were manually annotated by study pathologists on all WSIs. A nine-layer convolutional neural network based on the common U-Net architecture was developed and tested for the segmentation of nonsclerotic and sclerotic glomeruli. DL-derived, manual segmentation, and reported glomerular count (standard of care) were compared.

Results: The average Dice similarity coefficient testing was 0.90 and 0.83. And the F1, recall, and precision scores were 0.93, 0.96, and 0.90, and 0.87, 0.93, and 0.81, for nonsclerotic and sclerotic glomeruli, respectively. DL-derived and manual segmentation-derived glomerular counts were comparable, but statistically different from reported glomerular count.

Conclusions: DL segmentation is a feasible and robust approach for automatic quantification of glomeruli. We represent the first step toward new protocols for the evaluation of donor kidney biopsies.

The erratum corrects the spelling of an author surname.