1.

Introduction

During surgery, a surgeon has to remove pathological lesions, replace abnormal tissue, or reconnect or repair damaged tissue structures, all of this while healthy tissue areas have to be maintained and organs at risk have to stay completely untreated and healthy. In order to accomplish this objective, to remove pathological while saving healthy tissue, the surgeon has to differentiate between healthy tissue areas and abnormal or damaged tissue. This manual differentiation process is very complex as the visual occurrence of living tissue, whether healthy or abnormal, often shows no significant optical differences under white light illumination. Furthermore, pathological structures mainly develop between or in the direct proximity to important structures and have most fine extensions. These facts make it hard to distinguish pathological from healthy tissue and, as a consequence, the structure exposition process is of high risk and a significant degree of difficulty, because damage to healthy structures can cause a temporary or permanent loss of functionality of the affected region.

Nowadays, the continuous differentiation during surgery is only based on the surgeon’s perception, experience, and anatomical knowledge, which entails great risk because abnormal tissue in the operating field displaces normal structures and changes the anatomy. Furthermore, since tissue structures are very hard to separate using optical and haptic qualities, this manual differentiation process must be done slowly with the utmost caution because injuring important structures, such as nerves, can cause permanent damage to the patient’s health.

During most surgeries, external tools are used to support the surgeon’s decisions on tissue differentiation. The intraoperative neurophysiological monitoring, e.g., electromyographic monitoring of peripheral nerves, is an external electrophysiological method to monitor the functional integrity of certain neuronal structures.¹ To use this tool, the surgeon has to interrupt dissection and needs to change the instruments, which takes time and disrupts affiliated actions.

The upcoming digitization of information during surgery provides new opportunities of analyzing and visualizing tissue. In order to support the surgeon’s decision by detecting optical tissue characteristics not visible for human eye, we developed and investigated a hyperspectral method for analysis of in vivo tissue characteristics of different tissue types. Hyperspectral imaging (HSI) combines spectroscopy and imaging techniques to acquire spectral and spatial information of a scene.

The aim of this project is to perform first pretrials in noninvasive tissue differentiation using continuous illumination in the visible and near-infrared (IR) spectral range without using any fluorescent dyes. This approach has a number of benefits: there is no risk of side effects or allergies to a fluorescent dye and the treatment can be started without any delay caused by the enhancement of the tracer. Suitable monochromatic light allows one to stimulate endogenous and exogenous dye molecules.² Energy is emitted in the form of light-specific wavelength, resulting in tissue-specific absorption, reflection, and scattering. Attention must be paid to the optical window of human tissue for near-IR illumination, as the absorption of light is minimal in the spectrum of 660 to 1100 nm.³

Tissue consists of various molecular structures, which imply different electromagnetic interactions and, therefore, different optical characteristics in terms of reflection, transmission, and absorption for different wavelength of the individual tissue types.⁴ There have been several attempts to examine optical properties of human tissue, mainly focusing on types that can be clearly separated from surrounding tissue, e.g., brain tissue, lung, and skin.^2,4–7 Other works are focused on tissue differences caused by different blood supply characteristics.² In contrast, the present investigation analyzes the optical properties of human tissue that suffer of large errors. Mostly, the characteristics of specific tissue types are analyzed in vitro.^8,9 Due to the degeneration of human tissue without blood supply as well as changed water and collagen content, the optical properties are assumed to change between in vivo and in vitro analysis.⁷ Therefore, the tissue behavior will be analyzed in vivo in this work.

HSI is established in biomedicine and food science for cell segmentation, skin and meat analysis,^2,10,11 as well as for recording the properties of blood by the detection of differences of oxygen saturation of hemoglobin. However, it has not yet been used in intraoperative medical therapy for image-guided tissue differentiation. Different techniques exist to acquire the hyperspectral data of a scene. To study molecular chromophors as oxyhemoglobin in a patient, liquid crystal tunable filters and digital light processing have been used.^12,13 Furthermore, hyperspectral cameras can be used with linescan or snapshot mode to acquire the dataset in a short amount of time, although with low resolution.^14,15 Alternatively, a filter wheel in front of the sensor or behind the light source can step through the hyperspectral space.^16,17 Since the method using the filter wheel allows higher resolution images, and this work does not aim at real-time results, we decided to use a filter-wheel setup.

The different optical tissue characteristics can be traced back mostly to the different water and collagen content as well as to the fiber development of the tissues.² Water strongly absorbs IR and ultraviolet (UV) light and shows no significant absorption for the visible electromagnetic spectrum.¹⁸ In the spectra of near-IR, visible and near-UV melanin and hemoglobin show high absorption. The differences of the optical behavior of hemoglobin (in the veins) and oxyhemoglobin (in the arteries) are well-known and used in narrow-band-imaging (NBI).¹⁹ Hemoglobin and oxyhemoglobin have the largest difference in absorption at about 415, 560, as well as 660 nm. NBI uses this knowledge to differentiate between arteries and veins.

In this paper, we present a hyperspectral filter-wheel setup, which is used to analyze optical human tissue properties in vivo. The possibility to sequentially distinguish between arteries and veins is examined in the context of this work to verify our method. Furthermore, other tissue types are analyzed to examine whether it is possible to differentiate between different tissue structures. In this study, only healthy tissue structures, well-known by the surgeon and existing in every analyzed patient, are analyzed to minimize uncertainties about different tissue behaviors. Therefore, the optical properties of fat, muscle, connective tissue, nerves, parotid gland, and bone are analyzed and presented.

2.

Materials and Methods

2.1.

Microscopic Setup

We build an imaging setup based on a prototype of the digital surgical microscope ARRISCOPE including a digital camera ARRI Alexa (Arnold und Richter Cine Technik, Munich, Germany) featuring a CMOS sensor as recording device and an illumination unit. In front of the illumination unit, a filter wheel selects a specific wavelength band that reaches the tissue. The use of a filter wheel to achieve multispectral illumination is described by Brauers et al.¹⁶ This approach allows a spectral scanning of the investigated tissue. The complete setup is sketched in Fig. 1.

Fig. 1

Flowchart of the basic setup based on a clinical microscopic setup.

A microcontroller manages the whole illumination and acquisition process. The light source is a white LED, which emits light between 400 and 700 nm. The spectral profile is shown in Fig. 2. The luminous flux is 2400 lm.

Fig. 2

The spectral profiles of the used CMOS sensor (continuous curve) with sensitivity over wavelength and the used light source (dashed curve) with relative intensity over wavelength are plotted.

When the acquisition process is started, the camera back end sends rectangular trigger pulses to the controller. The filter wheel is then synchronously rotated by the motor with the aid of a light barrier and an angle sensor. The filter wheel has a diameter of 21.8 cm and includes 16 bandpass filters with wavelengths starting at 400 up to 700 nm in steps of 20 nm. The bandwidth in terms of full-width half-maximum of every filter is about $20\pm 3\mathrm{nm}$. The light is guided using a fluid light guide to achieve a higher illumination. The optimal light transmission has a spectrum from 340 to 800 nm. In the optical path in front of and behind the filter wheel, a convex lens is needed to widen and narrow the light beam. At the end, different monochromatic light leaves the optical system, and the spectral changes are synchronized to the camera frame rate.

The camera uses a CMOS sensor with a sensitivity range from 380 to 1100 nm, see Fig. 2. The best working spectrum is between 400 and 880 nm. The sensor and camera optics specifications are listed in Table 1. The optimal working distance (WD) to the surgical area is 210 mm, which corresponds to a normal surgical microscope.

Table 1

The sensor and camera optics specifications in the used system.

Sensor specifications
CMOS sensor number of pixels		$3392\times 2200\text{pixels}$
Output resolution	Full HD	$1920\times 1080\text{pixels}$
Mechanical dimension		$23.76\mathrm{mm}\times 13.365\mathrm{mm}$
Diagonal dimension		29.26 mm (1.7 in.)
Camera optics specifications
Beam angle	Wide angle	16.52 deg
	Zoom	6.26 deg
Focal length	Mean	58.237 mm
	Wide angle	64.523 mm
	Zoom	65.220 mm
Field of view at 210 mm WD	Max.	$27\mathrm{mm}\times 48\mathrm{mm}$
	Min.	$13\mathrm{mm}\times 23\mathrm{mm}$

2.4.

Postprocessing

We have analyzed the spectral tissue behavior in the wavelength domain. To analyze the complex information in the images, a few postprocessing steps are needed to minimize the existing movement and luminance intensity differences in the images. First, every acquired dataset is annotated by the surgeon to get a well-known tissue classification. Based on this annotation, every image can be segmented into subimages, each containing one relevant tissue type only (Fig. 3).

Fig. 3

These images give an impression about the annotation process by the surgeon. (a) The original image is labeled with (b) relevant information for further analysis. Based on the annotation image in (b), the subimages are created for the postprocessing. A similar color used for labeling corresponds to similar tissue types.

In addition to the external movement of the imaging system, the cardiac cycle as well as the ventilation lead to internal movement. To transform the measured data into the wavelength domain, the internal movement along the spatial dimensions and external movements along the spectral dimension in the hyperspectral data have to be minimized. Therefore, the images of all wavelengths are divided into subimages according to the annotations of the surgeon so that each subimage includes only one tissue type. The (sub-)images of the corresponding tissue in all wavelengths of a dataset are registered to achieve a 3-D hyperspectral data cube. This registration process of two (sub-)images $f$ and $g$ is done by analyzing local translation of each annotated region only since translation is the main factor of dislocation.^20,21 For registration, we have used two different methods and picked the one with the smallest variation in the resulting 3-D dataset. The first method minimizes mutual information

Eq. (1)

$$I_{\mathrm{MI}}(f,g)=H(f)+H(g)-H(f,g)=\sum _{x\in f}\sum _{y\in g}p(x,y){\log }_2\frac{p(x,y)}{p(x)p(y)},$$

where $H$ is the entropy, $p(x)$ and $p(y)$ are the marginal probability distribution functions of $f$ and $g$, respectively, and $p(x,y)$ is the joint probability function. The second method uses normalized cross correlation as cost function

Eq. (2)

$$I_{\mathrm{NCC}}(u,v)=\frac{\sum _x\sum _{y}f(x,y)g(x+u,y+v)}{\sqrt{\sum _x\sum _{y}f{(x,y)}^2}\sqrt{\sum _x\sum _{y}g{(x+u,y+v)}^2}}.$$

As it is shown in Fig. 2, the intensity of the light source as well as the sensitivity of the sensor vary widely. To compensate this, the measured hyperspectral data are adapted to handle different illumination intensities for every spectral band $n$ using a normalization

Eq. (3)

$$I_x(n)=\frac{I_{\text{measured}}(n)}{\sum _{\lambda }{r}_n(\lambda )·L(\lambda )·F_x(\lambda )},$$

where $r(\lambda )$ is the intensity response of the used filter, $L(\lambda )$ is the light source, and $F_x(\lambda )$ is the specific pixel response. Thus, it becomes possible to analyze the relating tissue properties in a 16-dimensional (16-D) wavelength domain.

3.

Results

To evaluate the accuracy of our hyperspectral analyzer, we have compared the spectral behavior of detected artery and vein data with well-known published oxygenated and deoxygenated blood results. Equal spectral behavior has been observed here.

In Fig. 4, 12 out of the 16 registered monochromatic images of a single scan are shown. A side-by-side comparison of these images already reveals significant differences in several parts of the image and shows at which wavelength the different vascular structures can be differentiated.

Fig. 4

Each subimage shows the same scene illuminated with varying wavelength. Significant differences between the single tissue structures over all wavelength images are visible in terms of absorption and reflection.

During the analysis, one big challenge is the huge variance in the intensity of the light source. Despite intensity normalization, it is still difficult to achieve proper results for all wavelengths because the illumination intensities at 400, 680, and 700 nm are near the limit of detection. Table 2 shows the signal-to-noise ratio (SNR)

Table 2

The signal-to-noise behavior of the acquired images of Fig. 4, for each wavelength. The image at 400 nm contains almost no signal and is thus used as noise reference.

In order to achieve comparable results, we have adjusted the wavelength information according to its corresponding input illumination. For a better comparison of the different optical tissue behavior and presentation in this publication, we additionally normalized every spectrum using

First of all, it is of interest if the same tissue types show the same behavior over different measurement times for one patient as well as between different patients, which includes changes in illumination caused by indirect background lightning. As shown in Figs. 5 and 6 for bone, fat, parotid gland, and nerve, the behavior of the normalized reflection intensity for the analyzed tissue type remains the same for different measurements and individuals in the analyzed spectrum from 400 to 700 nm. The spectrum of bone [Fig. 5(a)] shows its highest intensity at 420 nm with three additional smaller peaks at 460, 520, and 580 nm.

Fig. 5

This plot shows $I_{\mathrm{norm}}$ over wavelength of the analyzed tissues (a) bone and (b) fat. Each curve represents the annotated data in (a) of one dataset of overall two mastoidectomy patients and in (b) of three patients in neck dissection and parotidectomy. Each data point represents the measured average of the annotated region in that specific wavelength.

Fig. 6

These plots show the results of the annotated data of (a) parotid gland and (b) nerve. In (a) two parotidectomy patients each with two measurement sets and in (b) an individual dataset of overall six patients are used.

The spectra of fat [Fig. 5(b)] and parotid gland [Fig. 6(a)] are completely different to bone. Both show highest intensity at 460 nm with a smaller second peak at 580 nm. The differences in these two spectra are the intensity level in the green (480 nm) to red (640 nm) spectral range. The spectrum of the parotid gland shows a small third peak at 520 nm and remains relatively constant in the range of 480 to 560 nm while the spectrum of fat is slightly descending in this range.

The analyzed reflection behavior of nerve tissue is more complex, see Fig. 6(b). At this spectrum, two high peaks at 460 and 580 nm as well as a smaller third peak around 540 nm exist. The variability in the spectrum of the nerve is higher than for the other spectra, which can be clearly seen at 420 and 520 nm.

Figures 5 and 6 clearly show that every single tissue type shows an individual unique trend which allows a precise identification whereas the standard optical appearance allows mostly no accurate identification for the surgeon. During the annotation of the data, the surgeons were not completely sure about the type of tissue for certain areas. Based on their knowledge, the possible tissue type could be narrowed, but not clearly defined. Figures 7(a) and 7(b) show the two well-defined curves of muscle and connective tissue, respectively. Figure 7(c) shows the behavior of tissue that is classified by the surgeons as either muscle, connective tissue, or a mix of both. The curve of patient 4, first measurement (P4/1) can be definitely identified as muscle. The curve of patient 2, first measurement (P2/1) appears to be a mix of both types, but the tissue part contains more muscle than connective tissue. Therefore, the spectrum is more similar to the spectrum of muscle.

Fig. 7

The two upper images plot the results of (a) muscle and (b) connective tissue. (c) Data that have been annotated as undecided tissue type (either muscle or connective tissue) by the surgeon. The curve of P4/1 is similar to muscle while the curve of P2/1 rather refers to connective tissue, but at least a mixture of both types.

The spectrum of connective tissue also reveals a large variability, which can correspond to the different types of connective tissue. This requires further analysis.

Figure 8 combines all unique spectra of the analyzed tissue types. While bone and connective tissue show a high reflection in blue at 420 nm, they differ very much in the range of 500 to 580 nm. In the range of 500 to 580 nm, fat shows a similar trend to bone, but the blue content is similar to muscle or parotid gland. In the green to red range (520 to 580 nm), all other tissues exhibit a higher reflection than bone and fat with high variability and very unique characteristics. Therefore, up to seven spectral points can be identified to differentiate all analyzed tissue types. The higher the difference between two tissue types at these spectral points, the better the different types can be separated.

Fig. 8

The spectral behavior of six tissue types is plotted. Every data point is the normalized average of the measured annotated tissue data. It is clearly visible that every curve shows an individual trend and is unique/independent.

4.

Discussion

We have analyzed the spectral tissue behavior in the wavelength domain. It is shown that our setup is robust and allows us to state reliable conclusions about tissue behavior. As a result, we can present a spectrum that could allow a characterization of the tissues in the wavelength domain and find specific wavelengths to separate the different tissue types in vivo, e.g., bone, muscle, or fat. Interesting wavelengths are in the ranges of 420 to 460 nm and 530 to 590 nm as well as 640 to 680 nm. As indicated in Fig. 8, tissue differences can be identified at the spectral points 420, 440, 520, 540, and 640 nm. In addition, for some tissue types, near-UV and near-IR illumination could bring more significant differentiation options, which need further investigations, e.g., bone in near-UV.

Using these interesting wavelengths and projecting the different measured tissue data, single tissue types can be discriminated from the other tissue types as shown in Fig. 9 for the parotid gland. These fundamental investigations give very promising results and can help to assist or even simplify certain surgical procedures.

Fig. 9

This scatter plot shows a two-dimensional projection of the measured 16-D hyperspectral data. Each 16-D data entry of the different tissue types is obtained from a scan as described in Sec. 2. In this projection view, the measured parotid gland data can be separated from all other tissue types. The included separation line is for better illustration of the tissue classes only.

The main problem in our setup is caused by inhomogeneous illumination in the spectral range. Using a diffuse light source with a wider and homogeneous spectral range as a xenon source could reduce the observed distortion as well as noise parameters and is currently under investigation. On the other side, the used light source should not be too strong in terms of intensity as otherwise distracting reflections at vascular walls and fluids reduce the information output.

Another large source of deviation and error indicator is the existing scattered light in the operating room caused by normal ceiling and surgical lights. Our study showed that the best results could be achieved without any scattering light, which is not possible to put into practice because the physicians and surgical nurses need lighting as well. Therefore, a reasonable compromise between room illumination and optimal darkening has to be found.

To take the approach presented here closer to a practical application, the setup has to be optimized addressing the following points mainly: internal and external movements (pulse beat and breathing as well as camera vibration, in and between the images, respectively) need to be minimized to avoid expensive image postprocessing and speed up the image analysis. In addition, the existing time misalignment, caused by the cycling through the different bandpass filters, which is too slow and therefore not applicable for clinical routine, has to be prevented in an optimized setup. Caused by these facts, setup changes will be investigated in future. Further, the position of the wavelength filters will be analyzed in more detail since there are different possibilities of placing the filter in the setup. In this work, the filters are placed directly behind the light source, but it is also possible to place them in front of the sensor (hyperspectral cameras). Further, a possible approach is to combine both possibilities: an illumination unit consisting of different filters with a wider spectrum and a sensor also with filters in front, e.g., an RGB sensor.

5.

Conclusion

We developed and investigated a hyperspectral analyzer to detect optical tissue characteristics not visible for human eyes. These characteristics have the potential to assist the surgical procedure by providing additional information about the visible tissues to the surgeon. This would induce a faster treatment and has the potential to reduce the risks of postoperative damage.

In future work, we will optimize the setup as described and plan to adapt the light source to achieve homogeneous illumination as well as to extend the spectral range to near-UV and near-IR in order to potentially detect further interesting tissue behavior and thereby make a potential tissue differentiation more robust. Further, it is planned to use a snapshot hyperspectral camera despite the disadvantage of the lower spatial resolution and analyze the tissue behavior using white light illumination to check if the results are comparable to the ones presented here. Following, an optimal setup will be developed using information convolution. Thus, several narrow-band signals can be achieved in a shorter time and fewer images, which would balance the setup in terms of spatial resolution and acquisition time for real-time capability. Using an optimized setup that acquires the data in less time allows to optimize the postprocessing, mainly the registration process can be extended to more comprehensive models, e.g., as presented by Furch et al.²² In addition, we will investigate other interesting tissue structures, healthy as much as diseased tissue, in further surgical procedures.

Once the tissue differentiation is robust, the visualization options will be discussed and analyzed. The visualization should occur directly in sight of the operator without distorting the environmental truth. Therefore, our idea has the potential to optimize surgery time and costs.