2.1.

Dataset

The dataset consisted of DCE-MRIs of 994 unique breast lesions (724 malignant and 270 benign) from 689 patients aged 23 to 89 years. The deidentified data were retrospectively collected at the University of Chicago over a span of 8 years (from 2005 to 2013) under Health Insurance Portability and Accountability Act-compliant Institutional Review Board protocols. Routine bilateral breast MRIs were acquired using a Philips Achieva scanner with either 1.5 T ($N=473$) or 3 T ($N=216$) magnet strength. The breast DCE-MRI protocol included a fat-saturated 3D $T1$ weighted spoiled gradient-echo sequence that was used to acquire precontrast and postcontrast images with a temporal resolution of 60 to 75 s (TE = 2.2 to 2.8 ms, TR = 4.5 to 7.5 ms, flip angle = 10 deg to 20 deg, in-plane resolution = 0.5 to 1.0 mm, FOV = 28.0 to 44.1 cm, matrix = 320 to $552\times 256$ to 525, slice thickness = 1 to 3.5 mm, and interslice gap = 0.8 to 2.5 mm). Table 1 contains the clinical characteristics of the data obtained from pathology and radiology reports, including pathological truth (benign or malignant) and lesion type (mass or nonmass enhancement). A subset of 71 lesions was manually selected for radiologist delineations so that the distribution of pathological truth and lesion type within this subset was similar to the distribution in the overall group (Table 1). Table 2 presents size distributions of the lesions.

Table 1

Summary of the DCE-MRI dataset by lesion type. Lesions were categorized by pathological truth and enhancement type. Lesions that were not marked as either mass or nonmass enhancing were labeled “unknown.”

Table 2

Summary of the DCE-MRI dataset by lesion size. Lesions were categorized by effective diameter (mm), defined by 2*√(A/π), where A is the area of the lesion in the center slice of the FCM segmentation in mm2.

2.2.

Establishment of Reference Standards and Preprocessing

Each lesion had previously been segmented using a well-established, in-house, automated 3D FCM approach that yielded, as a surrogate “reference standard,” a 3D binary lesion segmentation.⁴ FCM segmentation was performed within a region defined by a human operator’s selection of a rectangular bounding box about the lesion in a middle slice along with an indication of the first and last slices in which the lesion appeared.⁴ The bounding-box volume of interest (VOI) for the FCM segmentation of each lesion was also used as input for subsequent U-Net segmentations of postcontrast subtraction images. Second postcontrast subtraction images were primarily used as inputs for the U-Net; however, first postcontrast subtraction images were introduced for evaluation in comparisons C and D of this study.

In addition, an expert radiologist (7 years of experience in breast imaging) manually delineated the lesion within the center slice of the second postcontrast subtraction VOI for the subset of 71 lesions. Here the radiologist segmentations were used as the “reference standard” for comparison B of this study. Since radiologist segmentations were only available for a limited set of center slices and FCM segmentations are used in an FDA-approved clinical breast MRI workstation,³ FCM segmentations served as a reasonable surrogate reference standard to train the U-Net architectures.

2.3.

U-Net Architectures

Two different U-Net architectures were evaluated in this study. The first was a 2D U-Net.⁷ We found that the top and bottom slices of lesions were most difficult to segment, so those two slices were excluded from each lesion in training (though they remained in the test set lesions). The image slices of each lesion’s VOI were resized, by interpolation with a preserved pixel value range, to $256\times 256\text{pixels}$ prior to input into the 2D U-Net. The $256\times 256\text{pixels}$ probability map outputs, with values ranging from 0 to 1, were converted to binary segmentation images based on a threshold of 0.25. The 2D U-Net only processes one image slice at a time, so “quasi-3D” lesion segmentations were produced by stacking the 2D slice-by-slice segmentations obtained by the 2D U-Net to form a 3D volume. Hence, in this paper, “quasi-3D U-Net” refers to the volumetric segmentation produced by the 2D U-Net architecture.

The second architecture evaluated in this study was a 3D U-Net.⁹ This network is similar to the structure of the 2D U-Net, but it is modified with the added third dimension. Prior to input into the 3D U-Net, the lesion VOIs were resized, by interpolation with a preserved pixel value range, to $256\times 256\times N\text{voxels}$ ($N$ is the number of slices in the lesion). The network produced $256\times 256\times N\text{voxel}$ probability map outputs, with values ranging from 0 to 1, which were converted to binary segmentation volumes based on a threshold of 0.23. The threshold for the binary conversion was selected from a range of values between 0.14 and 0.30 to produce the greatest mean Dice similarity coefficient (DSC) calculated from the resulting segmentations during training, hence the slight difference in threshold used for 2D U-Net and 3D U-Net.

2.4.

Training and Statistical Analysis of Segmentation Performances

Fivefold cross-validation by lesion ($N=994$ lesions) was conducted to train and evaluate the U-Net models. The folds were partitioned such that each fold contained a similar distribution based on pathological truth (malignant or benign), lesion enhancement type, and lesion size. Additionally, since adjacent slices within the same lesion VOI often are very similar in appearance, all slices belonging to a given lesion were always allocated to the same fold. Training and test folds were allocated by lesion, i.e., not by slice or patient. The base U-Net models were trained using the Adam optimizer and a binary cross-entropy loss function; training was allowed to run for up to 200 epochs.

DSC and Hausdorff distance (HD) were used to evaluate the performances of the different segmentation methods relative to the specific reference standard.^16,17 DSC is a measure of how well the areas of the two regions overlap, and HD is a measure of how well the margins of the two regions agree. Note that, throughout, better segmentation performance is indicated by higher DSCs and lower HDs. Predictions from the quasi-3D and 3D U-Nets were resized to their original lesion VOI dimensions before DSCs and HDs were calculated between the predictions and the reference standards. HDs were calculated for each slice, and the median HD for each lesion was used for 3D performance comparisons. To assess statistical significance of difference in performance, the Wilcoxon signed-rank test was used for matched cases in comparisons A, B, and C, and the Mann–Whitney U-test was used in comparison D due the analysis of unmatched cases.^16–19 The Bonferroni correction was used to correct $p$-values for multiple comparisons in comparisons B, C, and D.²⁰

2.5.

Comparison A: Comparing Quasi-3D U-Net to 3D U-Net Using FCM as the Surrogate Reference Standard

The volumetric segmentations from quasi-3D and 3D U-Nets were compared and as previously noted, FCM segmentations served as the surrogate reference standard for the 994 lesions (Table 1). The Wilcoxon signed-rank test was used to assess statistically significant differences between quasi-3D and 3D U-Net segmentation performances (Fig. 1).

Fig. 1

Flowchart of comparison A of this study ($N=994$). FCM lesion segmentation volumes were used as reference standard to compare quasi-3D U-Net (2D architecture) and 3D U-Net segmentations in a by-lesion fivefold cross-validation process. DSC, Dice similarity coefficient and med HD, median Hausdorff distance of all slices in the lesion. Wilcoxon signed-rank tests were performed on the resulting paired data.

2.6.

Comparison B: Comparing FCM, Quasi-3D U-Net, and 3D U-Net Using Radiologist-Delineations as the Reference Standard

Next, FCM, quasi-3D U-Net, and 3D U-Net center slice segmentations were compared using the radiologist references available for the subset of 71 lesions. For each of the three segmentation methods, DSCs and HDs were calculated on the center slice with respect to the radiologist reference. Statistically significant differences between quasi-3D U-Net, 3D U-Net, and FCM segmentations were assessed using the Wilcoxon signed-rank test including a Bonferroni correction (Fig. 2).

Fig. 2

Flowchart of comparison B of this study ($N=71$). Radiologist segmentations were used as reference standard to compare FCM, quasi-3D U-Net, and 3D U-Net center slice segmentations. DSC, Dice similarity coefficient and HD, Hausdorff distance. Wilcoxon signed-rank tests were performed on the resulting paired data.

2.7.

Comparison C: Comparing Segmentation Across Postcontrast Timepoints (First Versus Second Postcontrast)

The segmentations obtained in comparison A using second postcontrast subtraction images as input were compared to those using the first postcontrast subtraction images for the quasi-3D and 3D U-Nets. Wilcoxon signed-rank tests were used to assess statistical significance between the results after a Bonferroni correction.

2.8.

Comparison D: Comparing Segmentation Across Lesion Enhancement Types (Mass Versus Nonmass Enhancement)

Finally, the segmentation performances on mass and nonmass enhancing lesions were compared. The segmentations resulting from the first and second postcontrast subtraction inputs to the quasi-3D and 3D U-Nets evaluated in comparison C were compared based on lesion enhancement type. For each comparison, a Mann–Whitney U-test including a Bonferroni correction for statistical significance was used to compare the segmentation performances of the set of mass lesions to the set of nonmass lesions.

3.1.

Comparison A: Comparing Quasi-3D U-Net to 3D U-Net Using FCM as the Surrogate Reference Standard

Segmentation performance was assessed by comparing the medians of DSC and HD (Table 3). Note that better segmentation performance is indicated by higher DSCs and lower HDs. Of the 994 lesions in the dataset, the 3D U-Net failed to segment 6 lesions (from 3 unique patients) that were $<9.1\mathrm{mm}$ in effective diameter and had an unknown enhancement type. Without prediction volumes available to compare to the reference standard, DSC was essentially zero and it was impossible to calculate HDs, therefore, these lesions were excluded from HD statistical comparisons for the 3D U-Net. The results of the Wilcoxon signed-rank test show that the quasi-3D U-Net statistically significantly outperformed the 3D U-Net in terms of DSC ($p<0.001$) and HD ($p<0.001$) for lesion segmentation from second postcontrast subtraction VOIs.

Table 3

Comparison A: summary statistics of the performance metrics of quasi-3D and 3D U-Nets as compared to FCM reference standards for volume segmentation. U-Nets were trained and tested using fivefold cross validation by lesion. Minimum, maximum, and median values of DSC and HD metrics of all cases are shown. Parenthetical values represent 95% confidence intervals.

3.2.

Comparison B: Comparing FCM, Quasi-3D U-Net, and 3D U-Net Using Radiologist-Delineations as the Reference Standard

Based on the segmentation results for the subset of 71 lesions, we found that the center slices from each lesion segmentation produced by FCM, quasi-3D U-Net, and 3D U-Net had good agreement with the radiologist-segmented reference standard (Table 4). The statistical comparisons of performance between each segmentation method’s agreement with the reference standard is shown in Table 5. The results indicate that quasi-3D U-Net outperformed both 3D U-Net and FCM for lesion segmentation on second postcontrast subtraction center slices.

Table 4

Comparison B: summary statistics of the performance metrics of FCM, quasi-3D U-Net, and 3D U-Net, as compared to radiologist reference standard for center slice segmentation. U-Nets were trained and tested using fivefold cross validation by lesion. Minimum, maximum, and median DSC and HD metrics of all cases are shown. Parenthetical values represent 95% confidence intervals (N=71).

Table 5

Comparison B: statistical comparisons between the median performance metrics in Table 4 from FCM, quasi-3D U-Net, and 3D U-Net center slice predictions using radiologist-delineations as the reference standard. U-Nets were trained and tested using fivefold cross validation by lesion. Raw, uncorrected p-values from the Wilcoxon signed-rank test are reported; statistical significance was assessed after correcting for three comparisons (N=71).

We observed improved U-Net segmentation agreement with the radiologist reference as lesion size increased, and Fig. 3 shows how quasi-3D U-Net yielded greater DSC values than 3D U-Net, relative to radiologist delineations, across all lesion sizes.

Fig. 3

Difference in DSC calculated from the center slice of the quasi-3D (2D) U-Net or 3D U-Net and the radiologist reference versus lesion size. The majority of lesions yielded greater agreement between the radiologist and the quasi-3D (2D) U-Net than with the 3D U-Net. DSC, Dice similarity coefficient.

3.3.

Comparison C: Comparing Segmentation Across Postcontrast Timepoints (First Versus Second Postcontrast)

An example of the segmentations produced by the 2D U-Net, 3D U-Net, FCM, and radiologist for a mass and nonmass enhancing lesion is shown in Fig. 4. In the second postcontrast subtraction images, more lesion enhancement is provided to the U-Net, which as expected, tended to result in segmentations that more closely resembled FCM than segmentations from the first postcontrast subtraction segmentation inputs. Also as expected, the radiologist delineations acquired on the central slice of the second postcontrast subtraction image tended to resemble the center slice of the second postcontrast subtraction segmentation from the 2D U-Net.

Fig. 4

Example cases showing the center slice U-Net segmentations produced from the first or second postcontrast subtraction images for a (a) mass enhancing lesion and (b) nonmass enhancing lesion. The center slice FCM and radiologist references are also shown.

The performance metrics calculated for the segmentations produced by the U-Nets from first and second postcontrast subtraction inputs are included in Table 6. The statistical comparisons between the resulting DSC and HD metrics for each method are shown in Tables 7 and 8. The results show statistically significantly greater performance from the second postcontrast subtraction inputs than from the first postcontrast subtraction inputs using the quasi-3D and 3D U-Nets, except in the case of nonmass enhancing lesions using the 3D U-Net. The results from both the first and second postcontrast subtraction inputs support the results found in comparisons A and B. The quasi-3D U-Net statistically significantly outperformed the 3D U-Net for the combined lesion types based on DSC, however HD failed to show statistically significant differences between quasi-3D and 3D U-Net for the first postcontrast subtraction input.

Table 6

Comparisons C and D: summary statistics of the performance metrics of quasi-3D U-Net and 3D U-Net, as compared to FCM surrogate reference standard. U-Nets were trained and tested using fivefold cross validation by lesion (Nmass=687 and Nnonmass=224).

Table 7

Comparisons C and D: statistical results for comparisons between input image type and lesion type using the Dice similarity coefficients of U-Net segmentations against FCM reference standard. U-Nets were trained and tested using fivefold cross validation by lesion. Raw, uncorrected p-values from the Wilcoxon signed-rank or Mann–Whitney U-test are reported. However, statistical significance was assessed after including a Bonferroni correction for four comparisons per sample (Nmass=687 and Nnonmass=224).

Table 8

Comparisons C and D: statistical results for comparisons between input image type and lesion type using the Hausdorff distance of U-Net segmentations against FCM reference standard. U-Nets were trained and tested using fivefold cross validation by lesion. Raw, uncorrected p-values from the Wilcoxon signed-rank or Mann–Whitney U-test are reported. However, statistical significance was assessed after including a Bonferroni correction for four comparisons per sample (Nmass=687 and Nnonmass=224).

3.4.

Comparison D: Comparing Segmentation Across Lesion Enhancement Types (Mass Versus Nonmass Enhancement)

The results in Tables 6Table 7–8 demonstrate that, relative to the FCM reference standard, mass lesion segmentation statistically significantly outperformed nonmass enhancing lesion segmentation using first and second postcontrast subtraction image inputs to both the quasi-3D and 3D U-Nets. For nonmass enhancing lesions, quasi-3D U-Net always statistically significantly outperformed the 3D U-Net (as in comparisons A and B). For mass lesions, the DSC results indicate that quasi-3D U-Net statistically significantly outperformed the 3D U-Net (as in comparison B), however, the HD results from the first postcontrast subtraction inputs showed that the 3D U-Net statistically significantly outperformed quasi-3D U-Net.