2.1.

Search Strategy

A literature search was performed to identify relevant research articles written in English and listed in the following databases: the Association for Computing Machinery Digital Library, CENTRAL, Embase, Google Scholar, IEEE Xplore, PubMed, and Web of Science Core Collection (Fig. 3). Results were limited to the timeframe between April 1, 2021, and May 10, 2023. All searches were conducted on May 10, 2023. The search strategy employed specific keywords: [(“VR or virtual reality” or “virtual-reality”) and (“surgery” or “operation” or “surgical”) and (“planning” or “pre-operative” or “preoperative” or “pre-op” or “presurgical” or “pre-surgical” or “preplanning” or “pre-planning”)]. A detailed overview of the search methodology and queries for each database is summarized in Table S1 in the Supplementary Material.

Fig. 3

Search strategy based on the PRISMA flowchart.

2.2.

Eligibility Criteria

Eligible studies included the use of immersive VR with an HMD for preoperative surgical planning with patient-specific medical images. The condition for patient-specific data was fulfilled if the studies employed patient-specific data extracted from the individual’s physical body to generate VR volumes. Studies using generic patient data were not considered. Reviews, editorials, opinion-based articles, abstracts without full-text articles, videos without full-text articles, and tutorials were also excluded from the analysis.

2.3.

Screening and Study Selection

Two independent reviewers (the authors) conducted a screening of titles and abstracts in the bibliography software Zotero.¹³ The initial step was to manually exclude any duplicate entries. Subsequently, the titles and abstracts were independently assessed, followed by an independent evaluation of the full-text articles. Cases of different judgments were resolved through discussion.

2.4.

Data Extraction and Analysis

We used a custom data extraction template. The following data were extracted from the selected research articles: study design, surgical discipline, procedure or indication, VR software utilized, HMD employed, medical imaging technique that provided data for the input of the VR model, visualization modality compared with VR, order of presentation of the different visualization modalities, number and specialization of participants, number of cases studied, outcome variables measured, and whether the research findings favored VR over the compared visualization modalities. All extracted data were noted in a sheet in Table S2 in the Supplementary Material.

3.1.

Search Results

The initial search yielded a total of 1813 studies. After removing duplicates, we reviewed the remaining 1271 studies by scanning their titles and abstracts to assess if they met the criteria, further reducing the number of studies to 150. Those 150 articles were reviewed in full-text, 46 of them fulfilled the eligibility criteria.

3.2.

Study Characteristics: Study Design, Surgical Disciplines, Participants, and Cases

The 46 articles featured 52 studies with the following study designs: experiments (9 of 52), usability studies (9 of 52), retrospective reviews (7 of 52), case series (6 of 52), single-case reports (6 of 52), prospective observational case series (5 of 52), pilot studies (3 of 52), software elaborations (3 of 52), randomized controlled trials (2 of 52), a proof of concept study (1 of 52), and a feasibility study (1 of 52).

The articles covered the following surgical disciplines (Fig. 4): cardiothoracic surgery (20 of 52), general surgery (11 of 52), neurosurgery (10 of 52), oral and maxillofacial surgery (5 of 52), orthopedic surgery (2 of 52), otorhinolaryngologic surgery (2 of 52), plastic surgery (1 of 52), and urology (1 of 52).

Fig. 4

Number of participants per study.

The average number of participants per study was 9.89 ($\pm 12.13$) with a median of 5. Figure 4 indicates the frequencies of participants per study. In 15 instances, the number of participants was not stated. The average number of cases per study was 10.13 ($\pm 11.76$) with a median of 5. Excluding the single-case studies, the average rises to 13.09 ($\pm 12.13$) with a median of 10. Figure 5 shows the frequencies of cases for all studies. Seven studies did not provide the number of cases.

Fig. 5

Number of cases per study.

3.3.

Methods of Comparing Different Visualization Modalities

Comparisons between VR and other visualization modalities were assessed in different ways. One way was to conduct planning based on one visualization modality and then update it after seeing another visualization modality. Other articles assessed planning in VR and with other modalities separately. In 20 cases, no comparisons were made at all. In 23 cases, VR was compared to 2D data displayed on a regular monitor, in seven cases to 3D data displayed on a monitor, and in four cases 2D and 3D data displayed on a monitor. In six cases, VR was compared to 3D printed models. In one instance, VR was compared to the use of a monitor displaying 2D data supplemented with 3D printed models.

3.4.

Image Generation, Segmentation, and Rendering

The following input data and imaging modalities were used in the studies for the generation of VR images (Fig. 5): CT (24 of 52), MRI (11 of 52), CT and MRI (11 of 52), dynamic 3D echocardiography (3 of 52), CT and facial scan (1 of 52), and CT and digital subtraction angiography (1 of 52). One study did not specify the medical imaging technique used to generate the data for the VR model. Eight datasets used for VR were not segmented. In one study, data were segmented in VR. Eight out of 43 segmented datasets were manually segmented, three were automatically segmented, and five were semiautomatically segmented. In the remaining 27 studies, it was not clear if the segmentation was done manually, semiautomatically, or automatically. The deployed rendering techniques were rarely stated explicitly. Therefore, we often needed to identify which technique was used from the images or videos provided by the studies. 19 studies used only meshes, 15 used volumetric rendering, and 18 used a mixture of volumetric rendering with meshes inserted.

3.5.

Virtual Reality Hardware and Software

3D-VR models were displayed with the following commercially available HMDs: HTC Vive (16 of 52), Oculus Rift (5 of 52), HTC Vive Pro (4 of 52), Oculus Quest 1 (3 of 52), Oculus Quest 2 (3 of 52), Oculus Rift S (3 of 52), and Valve Index (2 of 52) (Fig. 6). In 16 instances, the model of the HMD was not specified. In all cases, the HMDs were tethered to a computer; no study stated explicitly that they used an HMD in standalone mode. In 19 of the 52 studies, specifications of the tethered computer were indicated. In 12 of these, the random-access memory (RAM) was specified. In 18 studies, the graphics processing unit (GPU) was named by model series, but in 11 cases, the video random-access memory (VRAM) of the GPU was not provided. In 17 studies, the processor series was mentioned.

Fig. 6

Number of the type of HMDs used in the selected studies.

The studies used the following VR software: custom-made software (18 of 52) Surgical Theater¹⁴ (9 of 52), MedicalVR¹⁵ (7 of 52), Specto VR¹⁶ (3 of 52), DIVA¹⁷ (2 of 52), Enduro¹⁸ (2 of 52), SlicerVR¹⁹ (1 of 52), Adesante SurgeryVision²⁰ (1/52), adapted version of IMHOTEP^21,22 (1/52), BananaVision²³ (1/52), CorFix²⁴ (1 of 52), Elucis (1 of 52), Marion K181 PCNL simulator²⁵ (1 of 52), nonmedical beta test version of software AW Virtual Reality prototype²⁶ (1 of 52), 3D Systems²⁷ (1 of 52), and VisuaMed²⁸ (1 of 52). One study did not indicate the software that was used.

3.6.

Surgical Planning Changes and Outcomes

We distinguished six groups of studies, based on combinations of study type and outcome data: retrospective reviews on preoperative planning changes, retrospective reviews on surgical outcomes, prospective studies on preoperative planning changes, prospective studies on surgical outcomes, case studies and case series, and comparative studies. The following sections summarize the outcomes of the identified groups.

4.1.

Conceptual Clarity and Visualization Modalities

This review focused on VR presented through an HMD. It is possible to create VR using different systems, such as 3D monitors⁵⁴ or cave automatic virtual reality environment systems,⁵⁵ among others. The selected literature often did not specify the visualization modality that was used to display the datasets. Some studies used the term VR to describe 3D models presented on conventional monitors⁵⁶ or smartphone screens.⁵⁷ For illustration, one study stated that surgeons “reviewed the 3D VR models individually or with their surgical team via a mobile application developed by the sponsor and installed on their smartphones.”⁵⁷ Many articles did not report on the technical setups and specifications of the VR visualization modalities that were used. This impedes the comparison between different imaging modalities of VR. Authors should address this ambiguity, by giving a more precise definition of the used visualization modality. This will ensure consistency, improve comparability, and avoid potential misunderstanding.

We observed that in VR, medical images always appeared in a 3D format and were most often compared to medical images in reformatted planes on conventional monitors. This raises a methodological question: is this a fair comparison? The use of 3D models in VR and the use of 2D images for screens could potentially introduce an artificial bias in favor of VR. An alternative approach could be to compare VR images to 3D images on conventional monitors providing similar interaction and manipulation capabilities. We think that a comparison of 2D images on conventional monitors in the form of reformatted planes (axial/sagittal/coronal) and 3D images in VR is nevertheless a viable way to assess the difference between the visualization modalities. This approach corresponds with the scope of most studies that seek to understand the differences between the standard of care and the new visualization modality of VR.

4.2.

Technical Specifications

The reporting of technical equipment was often inconsistent and neglected important information. In 30% of the reviewed studies, the type of HMD was not specified. Technical specifications, such as the field of view and the resolution of HMDs, vary widely. The HTC Vive⁵⁸ has a resolution of $1080\times 1200\text{pixels}$ per eye, whereas the Meta Quest 2⁵⁹ has a resolution of $1832\times 1920\text{pixels}$ per eye. Using different HMDs is likely to result in different outcomes as display quality, tracking performance and usability vary. Authors should specify which HMD was used to enable reproduction and to provide context information.

In 33 out of 52 cases, the literature did not indicate whether HMDs were tethered to a computer and if they provided graphic processing. Even when the information was provided, important details, such as information on VRAM, RAM, and processor, were missing. This information is useful to indicate image quality and frame rate of VR images. A rate of 120 frames per second and more is desirable to prevent simulator sickness.⁶⁰ Lower frame rates force users to adjust their behavior when manipulating images.

4.3.

Imaging Techniques

Regarding the reporting of medical imaging data used to generate the VR models, it is noteworthy that only 6 out of 35 studies using CT reported the parameters of CT image acquisition and processing. Acquisition and processing parameters of CT/MRI data, such as reconstructive interval, affect 3D model resolution and quality.^61,62 The indication of imaging parameters relevant to 3D model quality should be reported when medical imaging data are visualized in VR.

Another technique that determines the planning decision is the rendering method. Some studies rely on polygon meshes, some on volumetric rendering, and others on both. The distribution of the three types is approximately equal. Most studies did not indicate the rendering method, although they have a substantial impact on what the imaging data show. A polygon mesh is a network of interconnected polygons, typically triangles or quadrilaterals, which collectively form the detailed surface of a three-dimensional object [Fig. 7(b)]. The complexity and realism of this object are dependent on the number of polygons used. Polygon meshes achieve high accuracy through precise segmentation, with surface transparency being pivotal for visualizing internal structures.

Fig. 7

Examples of rendering techniques: (a) volumetric rendering⁴ and (b) mesh.⁶³

On the other hand, volumetric rendering uses a transfer function, which applies variable coloring and opacity levels throughout the structures of a volume [Fig. 7(a)]. This volume is composed of voxels, which are the 3D equivalent of pixels. Appropriately chosen transfer functions can offer a comprehensive view of the entire medical imaging dataset, revealing both surface and internal details. Polygon meshes tend to be less computationally intense, they can be generated manually, semiautomatically, or fully automatically allowing for customization of color and transparency values. Volumetric rendering can enhance the accuracy of medical images by capturing the density and attributes within three-dimensional data. Its accuracy largely hinges on the applied transfer functions, which include color mapping and opacity adjustments, mapping data values to visual properties across a spectrum as well as supplementary algorithms that enhance the rendering.

4.4.

Segmentation

In 27 out of 41 cases, the type of segmentation was not explicitly stated. The selection of rendering technique often hinges on the type of segmentation performed on the imaging data. The segmentation strategy plays an important role in medical imaging and directly influences the information derived from visualizations. Therefore, reporting on the segmentation method is essential. This includes specifying whether the segmentation was manual, semiautomatic, or fully automatic. A full segmentation of all surgically relevant structures is typically going to be a mesh. When segmentation is confined to specific targets, such as tumors, most studies combine mesh and volumetric rendering of the surrounding relevant structures. Particularly, when using volumetric rendering, reporting on the details of the process is critical, such as the types of presets and algorithms, which were used for the rendering. Furthermore, understanding the amount of additional labor invested in generating the visualization needs to be considered. A detailed reporting can provide insights into the accuracy, reliability, and reproducibility of the visualizations in surgical planning contexts. It furthermore allows to draw conclusions on the total amount of work required, which is important to understand for the integration into clinical workflows.

4.5.

Software

The functions and scope of software solutions vary widely. In 52 studies, 32 different software solutions were used. They offer different modes of interaction and manipulation and are often tailored for one surgical indication. The large number of custom-made software solutions hinders comparability among the studies. In many cases, the tools to manipulate the VR images were not systematically indicated. This is necessary as the specifications of software impact the individual outcome. Among the most mentioned features are plane cutting and the ability to display the 3D image alongside the 2D counterpart. Direct comparison with 2D images improves assurance of clinicians accustomed to conventional imaging formats, subsequently fostering adoption.³¹

4.6.

Outcomes by Study Type

Retrospective studies on preoperative surgical planning identified advantages of VR compared to other visualization modalities.^26,29–31 They found that actual surgery improved,²⁹ significantly better resection planning predictions and significantly better fitting chest wall substitutes were made²⁶ after additional VR review and that substantial plan modifications were made in 57%³⁰ of cases and 60%³¹ of cases. Retrospective studies proved to be an effective study type to analyze the use of VR in surgery as ground truth may be established and surgical decisions can be compared with each other. Moreover, there is no risk of compromising treatment standards, and it is easier to gather a large number of cases. A retrospective review³² found a statistically significant reduction of surgical procedure time by 80 min on average for the surgeries planned in VR. This is an impressive result, although limited by the number of cases (21). It would be important to replicate this result as it makes a strong case for the use of VR.

Prospective studies using VR to improve decision-making in surgical procedures subsequent to the consultation of initial conventional monitor setups^33–36 observed that the surgery plan was adjusted in 33%,³⁶ 40%,³³ 45%,³⁵ and 52%⁶⁴ of cases. These adjustments improved the quality of the surgical plans.^33,34 These figures represent high rates of surgical plan changes and emphasize the capability of VR to support surgical decision making. Staubli et al.³⁹ found advantages of VR over reformatted slice-based image modalities on conventional monitor setups. Notably, they engaged surgical trainees in performing surgeries using VR exclusively as the preparatory tool. Within this context, a marginal advantage of VR over conventional monitors was observed, albeit without achieving statistical significance in terms of surgical performance measured by the GOALS metric.⁴⁰ These findings are at least a promising result, given that surgical trainees have typically spent more planning time with conventional monitors than with VR. They indicate that performance loss due to VR seems unlikely. It must be noted however that in the study no 3D views were provided on the conventional monitor, which is already a key difference, potentially introducing a bias in favor of VR.

Regarding the reviewed case studies and case series,^38,41–49 we noticed that the indications were mostly highly complex cases with high risks of complications. These surgeries require a high level of spatial understanding and precision due to the critical nature of the target structure and its surroundings. Initial evidence supporting the use of VR in such cases has been presented in a study conducted by Steineke and Barbery³² showing that VR statistically significantly decreased procedure time in microsurgical clipping of middle cerebral artery aneurysms. This finding highlights the importance of conducting controlled studies with objective outcomes. Apparently, VR may be used as an additional tool alongside standard imaging to enhance patient safety while maintaining the current standards of care.

The findings from comparative studies^34,51–53 offer encouraging outcomes, substantiating the efficiency of VR in contrast to other visualization modalities. One study found no statistically significant difference between VR and 3D printed models for the classification accuracy of fractures,⁵³ but stated that 3D printing is more expensive and requires additional time.⁶⁵ Another study found VR to be comparable to 3D printed models but statistically significantly better than the use of a conventional monitor setup for segment assignment of liver tumors.⁵² Two studies reported better performance,⁵¹ faster performance⁵¹ and higher agreement among evaluators³⁴ in performing segment identification for tumors and lesions compared against other visualization methods. These promising results are particularly noteworthy, as they highlight the advantages attributed to VR within a context that permits direct comparison of outcomes to ground truth. Regarding surgical planning and surgical outcomes, the results of the reviewed studies strongly support a positive impact of VR on preoperative surgical decision making and surgical outcomes. Still, a systematic quantitative analysis such as a meta-analysis would be difficult to execute as the outcome data in the reviewed studies is very diverse and often subjective in nature.

4.7.

Comparison with Previous Outcomes

Lan et al.¹¹ provided a systematic review on surgical planning in VR with similar inclusion criteria to this review. Since their cut-off date aligns precisely with our start date, there is no overlap in the studies. Our results substantiate their claim that study designs and outcomes are heterogeneous, hindering comprehensive meta-analyses. The most compelling evidence would stem from clinical outcome data comparing surgeries planned in VR to other visualization modalities. It is evident that most planning decisions were not contested in actual surgeries. Instead, studies in both reviews relied on other variables to assess anatomical understanding and the quality of surgical decision-making. In accordance with Lan et al., we found that most results of the studies favored VR; however, the outcomes remain specific, artificial, and isolated. Consequently, we agree with Lan et al. that future VR research needs to shift toward high-quality studies that measure comparable clinical outcomes for patients and comparable ergonomic outcomes for surgeons.

5.1.

Improving Study Designs

The findings indicate that HMD-based VR research for preoperative surgical planning is still in the early stages of exploring the capabilities of the medium. The field should produce more studies with broader evidence, such as prospective, randomized and controlled multicenter studies focusing on objective clinical outcomes with adequate amounts of participants and cases. This would enable clinical recommendations on the use of VR. Moreover, there should be more comprehensive and detailed reporting on technical implementation to enhance reproducibility and to support clinicians in adoption. We recommend that a comprehensive reporting should encompass details of the employed HMDs and how it was used (standalone and tethered), the software application along with its deployed manipulation tools, specifications of the tethered computer (including RAM, VRAM, and processor specifications), relevant medical imaging parameters, as well as explicit indication of the segmentation and rendering methodologies. Ideally, the frame rate during the use of the HMD should also be included.

5.2.

Enhancing Visual Knowledge and Technical Proficiency

Physicians are trained to study cross sectional or “sliced” images one after another, to render them cognitively, and to ascribe them to the three-dimensional body of a patient during an intervention. 3D models used in VR with HMDs challenge this well-established visual paradigm. The interpretation of VR images requires learning new skills and to work with new tools, including operations such as zoom, rotate, tilt, and navigate in 3D, which will demand training. Although conventional input devices, such as mouse and keyboard, are considered familiar but less intuitive, manipulating and navigating through VR images requires technical skill. This visual knowledge should be considered as a prerequisite for applying VR in surgical planning. In the long run, this knowledge needs to complement the existing skills to interpret cross-sectional images. We expect that additional training for staff will be needed. Consequently, further research needs to address the visual and technical proficiency that physicians need to acquire to cope with this new architecture of image display. This may also positively affect the lack of conceptual clarity of the notion of VR that was found in some research papers. Most importantly, the improved technical proficiency will be a strong driver in the adoption of VR into clinical practices where reliability and efficiency are a must.

5.3.

Implementation of VR into Clinical Settings and Workflows

Integrating VR-based planning into clinical routines was described as one of the major challenges and requires further research. This includes the improvement of collaboration and teamwork as well as technical solutions, such as cloud rendering, integration with hospital information systems, and data interoperability standards. Cloud rendering is a neglected topic to be considered in future research since it facilitates the implementation process and enables the use of sophisticated rendering and segmentation algorithms. Notably, no studies highlighted substantial implementation challenges stemming from simulator sickness.

5.4.

Improving VR Software Usability and Design

The majority of the assessed research articles investigate if VR can support physicians in surgical planning, yet they do not investigate the mechanisms through which VR facilitates this support. Future studies should explore the use of tools for interacting with VR images. Particularly, interaction design concepts, such as tools for navigation, plane cutting, and image manipulation (such as rotating or zooming), of the medical images in VR should be more systematically evaluated. Enhancing the design and usability of VR software is crucial for leveraging its potential.⁶⁶ Design strategies, including the “look and feel” of VR need to adapt to the new visualization modality encompassing visual parameters, such as color, contrast, texture, contour, lighting, or motion. Interaction possibilities, such as eye-tracking and hand-tracking, are evolving and are being integrated into developer frameworks. The automation of image segmentation using AI techniques has the potential to facilitate image manipulation within VR software in the future, particularly with regard to segmentation.^33,64 The application of artificial intelligence (AI) in the form of convolutional neural networks is currently changing the way segmentation of medical images is conducted.⁶⁷ Some researchers are asking for deep-learning models parallel to the large language models of today but for medical image tasks, such as segmentation.⁶⁸ There is a domain-specific, open-source AI framework used by over 1 million researchers for medical image segmentation with a focus on 3D imaging accelerating the development of AI for 3D medical image segmentation.⁶⁹ AI is time efficient and largely accurate when segmenting, enabling the efficient generation of accurate 3D models and thereby enabling VR. However, legal and ethical considerations as well as clinical standards will have to be revisited to make use in clinical practice feasible. Due to advancing technology and the democratization of VR that takes place right now, opportunities for the effective and efficient use of VR in surgical planning becomes more economically viable and plausible even in the short term.