Open Access
11 January 2017 Real-time structured light-based otoscopy for quantitative measurement of eardrum deformation
Sam Van der Jeught, Joris J. J. Dirckx
Author Affiliations +
Abstract
An otological profilometry device based on real-time structured light triangulation is presented. A clinical otoscope head is mounted onto a custom-handheld unit containing both a small digital light projector and a high-speed digital camera. Digital fringe patterns are projected onto the eardrum surface and are recorded at a rate of 120 unique frames per second. The relative angle between projection and camera axes causes the projected patterns to appear deformed by the eardrum shape, allowing its full-field three-dimensional (3-D) surface map to be reconstructed. By combining hardware triggering between projector and camera with a dedicated parallel processing pipeline, the proposed system is capable of acquiring a live stream of point clouds of over 300,000 data points per frame at a rate of 40 Hz. Real-time eardrum profilometry adds an additional dimension of depth to the standard two-dimensional otoscopy image and provides a noninvasive tool to enhance the qualitative depth perception of the clinical operator with quantitative 3-D data. Visualization of the eardrum from different perspectives can improve the diagnosis of existing and the detection of impending middle ear pathology. The capability of the device to detect small middle ear pressure changes by monitoring eardrum deformation in real time is demonstrated.

1.

Introduction

The tympanic membrane (TM) or eardrum is a thin membrane that separates the middle ear from the external ear. Besides conducting sound wave vibrations from air to the ossicles, the TM protects the middle ear from external influences that may cause infections. The three-dimensional (3-D) conical shape of the TM plays a crucial role in this process and any structural change to its topography can be an indicator for existing or impending pathology. In the following, we will refer to the measurement of the TM surface shape as “tympano-topography.”

Currently, primary clinical diagnosis of middle ear inflammation or otitis media is carried out using a standard otoscope. In the otoscope head, a magnifying lens is placed directly behind a disposable speculum, allowing for visual inspection of the ear canal and the external side of the TM. Although this generally suffices for the detection of visually observable defects, such as TM ruptures, retraction pockets, or inflammation of the cutaneous layer, otoscopic examination is limited to two-dimensional (2-D) visualization of the eardrum structure only, and no information on structurally less discernible abnormalities can be generated. By extending the 2-D qualitative view of the TM with real-time 3-D surface shape information, a new dimension of quantitative data is made available to the operator, which may support correct diagnosis of currently manifested pathology and open up a new area of predictive otological diagnostics.

Due to the relatively small dimensions of the human TM (8 to 10 mm diameter, 1 to 2 mm apex height), measuring the dynamic deformation of its 3-D surface shape in vivo with sufficiently high resolution is not a trivial task. Recent reports of optical coherence tomography (OCT) systems have demonstrated the capability of the technique to measure human TM thickness in vivo and in real time by delivering and collecting the light through a handheld probe.1,2 Although additional data on the internal structure of the membrane is generated this way, extracting the 3-D surface shape model from the axial depth scans requires full-field segmentation which is typically done in postprocessing. In addition, the required broadband light source reduces portability of the otoscopic OCT system. Alternatively, digital holography has been used to measure microdeformations and 3-D vibrational (sound-induced) motion of TMs3,4 using double exposure, multiple wavelength techniques, but subwavelength stability of the measurement sample is required between subsequent exposures.

Recently, we have developed a real-time microscopic profilometry system5 capable of acquiring 30 microscopic height maps per second with depth resolutions in the micrometer range by projecting structured light patterns onto the object surface through one optical pathway of a stereoscopic operation microscope and by recording the deformed patterns through the second pathway. Although this resulted in high-quality 3-D maps of the object surface with depth resolution in the order of 10  μm, a rather large working distance of 200 to 415 mm from object lens to target surface was required. Unfortunately, the human ear canal has a sigmoid shape and straight optical passage to the TM cannot always be guaranteed. Similarly, a number of other medical procedures suffer to extract 3-D information directly from optically inaccessible human tissue due to anatomical limitations. These include the family of minimally invasive laparoscopic surgery techniques employed during gallbladder removal,6 hernia repair,7 removal of part of the colon or small intestine,8 removal of the spleen,9 removal of the adrenal glands,10 and removal of the kidneys.11 Although correct 3-D orientation is critical in these procedures, the operator is typically provided only with a live feed of 2-D images. To provide the laparoscopic surgeon with better depth perception, marker-less tracking systems have been designed to align the real-time video feed to preoperatively recorded 3-D CT scans of the tissue surface as an overlaid augmented display.12 Depth accuracy of these supplementing tracking systems is limited since they rely on prerecorded measurements and provide no direct, real-time feedback on the current topology of the observed tissue.

To this end, recent efforts have been made to integrate optical profilometry systems into miniaturized 3-D otoscopic and endoscopic devices. The first stereoscopic endoscopy systems illuminated the object surface with uniform light and imaged the scene using a dual sensor setup. This was achieved by employing either two imaging sensor chips with two distinct lenses13 or a single lens behind two pupil openings combined with a lenticular array and a single imaging sensor.14,15 Either way, 3-D data were extracted from the dual-viewpoint images by solving the correspondence problem. The quality of this technique is known to depend heavily on the optical structure of the imaged scene and suffers from featureless areas or shadow effects. Alternatively, the correspondence problem can be solved by replacing one of the two imaging sensors with a light source that projects predefined structured light patterns onto the scene. This way, triangulation can be performed directly by virtually intersecting the projected and the intercepted light rays using custom reconstruction algorithms.

In endoscopy, the narrow size of the sensor head and the small diameter of the attached flexible shaft complicate the design of the device optics that are required for projection and recording of the fringe patterns. As size constraints prohibit implementation of a digital light projector (DLP) unit into the sensor head itself, light is typically generated remotely outside of the endoscope and supplied through glass fibers. To solve the correspondence problem, the projected light must be provided with position dependent and uniquely identifiable markers. One way to landmark the object surface is to illuminate the scene with a grid of dots of multiple wavelengths.16 After identification of these dots in the camera image, the correspondence problem can be solved and 3-D reconstruction of the object surface can be performed. This technique is highly sensitive to the dynamic range and color resolution of the employed CCD chip and produces 3-D point clouds of limited density. Another way to install codification into the projected light beams is to pass the light through a ring-shaped slide placed in the sensor head, generating colored cones around the central endoscope axis in a circular pattern.17 Although this design is optimized for imaging of tubular cavities, no 3-D information of the object scene in front of the endoscope head is generated. Alternatively, dual-barrel systems with separate endoscopes for illumination and observation have been proposed,18 but these result in large and impractical setups and are generally unsuited for common medical applications. A detailed overview of 3-D endoscopic systems can be found in Refs. 19 and 20.

In otoscopy, the limited penetration depth of the scanner head imposes fewer dimensional restrictions onto the otoscopic device than is the case in standard endoscopy. As the human ear canal has an average length of about 25 mm and a diameter of 7 mm, the eardrum can generally be monitored in its entirety by inserting a funnel-shaped speculum into the canal. Recently, a light field otoscope has been reported that uses a microlens array in combination with a main imaging lens placed directly behind the speculum to capture both the intensity and the angular distribution of light rays coming from the eardrum surface.21 The plenoptic camera-based setup allows submillimeter resolution height maps of the TM to be constructed. Although this technique provides the operator with shape estimation of the eardrum under static conditions, the depth precision is insufficient to quantify local deformations of the membrane or to measure dynamic effects. More recently, Das et al.22 proposed a structured light-based otoscopic device capable of achieving depth resolutions in the order of 20 to 30  μm. Their prototype contains a portable LED projector for the projection of structured light patterns onto the eardrum and a telephoto lens arrangement whose output is coupled to a webcam sensor. The conical shape of the speculum enables a small angle between projection and recording axis to be made and the 3-D otoscopic device optics to be redesigned into a new, compact device. Since Das et al. employed multiphase structured light profilometry in combination with a low-end projector-camera system, recording and visualization of height maps required 1.5 s per measurement. This relatively long acquisition time not only reduces the height measurement precision due to patient or operator movement but also inhibits monitoring of dynamic eardrum deformation.

Both the detection of weak spots on the eardrum and the evaluation of Eustachian tube (ET) functioning require 3-D shape data of the TM to be gathered when varying pressures are applied over the membrane. As pressure stabilization is a dynamic process,23 multiple intermediate measurements are required, and real-time feedback of the TM surface shape change is needed. In the following, we present a new 3-D otoscopic handheld device containing a real-time structured light triangulation engine with custom triggering between projection and camera hardware and a dedicated digital signal processing pipeline based on graphics processor unit programming. By using a DLP with configurable digital micromirror device (DMD) displays, any single- or multi-shot structured light profilometry technique can be employed. This way, the pixel period of projected fringes or the number of total phase-shifted frames per 3-D measurement can easily be modified, depending on the reflectivity profile of the object surface, the required depth resolution, and the minimum 3-D frame rate. The 3-D otoscope generates a live stream of 40 full-field surface maps per second, each containing a point cloud of 300,000 data points per frame. For a comprehensive overview of current state-of-the-art real-time structured light profilometry techniques, we direct the reader to Zhang24 and Van der Jeught and Dirckx.25

This paper is organized as follows: the next section presents the experimental setup and outlines the technical aspects of the device. A brief overview of the parallel digital signal processing pipeline and the employed structured light profilometry technique is given. Section 3 demonstrates the experimental results of the 3-D otoscopic device, including measurements of calibration samples with known topographies and an ex vivo TM. Section 4 discusses important aspects of the hardware setup and the employed algorithms and explores the transition of the device into the clinical setup. Finally, Sec. 5 concludes the paper.

2.

Methods

2.1.

Experimental Setup

A standard clinical otoscope was used as the base optical setup for our real-time 3-D scanning device. In its standard configuration, the otoscope consists of a handle and a head with at its distal end an attachment for disposable plastic specula. Uniform white light is generated in an LED module located inside the otoscope handle and is injected into the ear canal via a plastic mirror. The light that is reflected from the TM is focused onto the observer eye by a 3× magnifying lens placed at the rear of the head.

To provide the otoscopic device with digital fringe projection and 3-D data extraction capabilities, several modifications to its optical configuration were implemented. First, the LED module and the entire otoscope handle were removed from the otoscope head. The plastic mirror was replaced by a semitransparent mirror (SM) that was installed inside the otoscope head at a 40 deg to 45 deg angle between central speculum axis and former handle shaft. Next, a custom-handheld aluminum frame was designed to align a high-speed CCD sensor (Imperx IGV-B0620M, 210 frames per second, 640×480  pixels, 12-bit grayscale images) perpendicularly to the prior otoscope handle axis. Finally, the magnifying lens was removed and a Texas Instruments (TI) lightcrafter pico projector evaluation module was installed at the rear of the otoscope head. The modified otoscopic unit and a schematic overview of its optical layout are shown in Fig. 1. In its entirety, the prototype weighs around 800 g and can easily be held in one hand. The device is powered and connected to the PC only through flexible cables and can easily be maneuvered under any angle. Nevertheless, future iterations of the hardware design will include more detail to ergonomics and form factor.

Fig. 1

(a) Modified otoscopic unit and (b) schematic overview of its optical layout with enlarged detail of otoscope head.

JBO_22_1_016008_f001.png

Uniform red, green, or blue light is generated by the RGB-LED light engine inside the TI lightcrafter module and reaches the DMD under a fixed angle. The light is reflected by the individually rotating micromirrors into a pixelated (608×684  pixel) grayscale fringe pattern that is projected onto the object surface after passing the projector lens and the SM. The deformed fringe pattern is observed by the CCD sensor after being reflected on the SM and after passing the observation lens. The coaxial bayonet neill–concelman cable enables triggering between CCD and DMD, and the recorded fringe patterns are transferred from CCD memory to RAM memory using the gigabit ethernet interface.

The TI lightcrafter module is a compact (116  mm×65  mm) DLP-based light projection system containing flash memory for on-board pattern and sequence storage. It generates 8-bit grayscale patterns at a rate of 120 Hz and binary patterns at a rate of 4 kHz. The pico projector and its focusing lens system was oriented so that the central projection axis of the projected images enters the head under an angle of α=20  deg relative to the observation axis. This angle between projection and observation axes causes the projected fringe patterns to appear deformed by the object shape and allows the 3-D object surface data to be extracted.

The duty cycle of the rapidly rotating micromirrors determines the amount of light that is reflected toward the output and therefore sets the grayscale intensity per pixel. In contrast to commercial video projectors, which adopt nonlinear duty cycles for better subjective visual effect, the lightcrafter module is designed to maintain a linear relation between input and output grayscale intensity or “gamma curve.” The micromirror duty cycle is divided discretely into 8-bit planes, each representing a fraction between 1/2th and 1/256th of the total illumination time. Therefore, to avoid visual artifacts, it is important that the projector-camera system is synchronized to record integer multiples of the total illumination time only. To ensure proper triggering, custom-made electronics were built to connect the trigger output of the lightcrafter logic board to the high-speed camera, and the camera exposure time was set to the interval between successive trigger pulses, corresponding to 1/120th of a second or 8.33 ms per image capture.

2.2.

Real-Time Digital Signal Processing Pipeline

The digital nature of the DLP unit allows us to implement any structured light projection technique in the otoscopic profilometer. Without loss of generality, in the following we describe the implementation of the standard three-phase 2+1 technique.26 This technique requires only three unique intensity images per 3-D measurement and therefore allows a high 3-D frame rate to be obtained. For a similar description of four-phase fringe pattern pipelines, we direct the reader to Van der Jeught et al.5

Before acquisition, three preconstructed 8-bit digital patterns are uploaded to the on-board flash memory of the lightcrafter module and are displayed in a loop: two line patterns with sinusoidally varying intensity distribution and a relative phase shift of π/2 and a single uniform white image. When observed by the camera under a relative angle with the projection axis, these line patterns appear deformed by the object shape and can be described as follows:

Eq. (1)

I1(i,j)=I(i,j)+I(i,j)sin{ϕ(i,j)},

Eq. (2)

I2(i,j)=I(i,j)+I(i,j)cos{ϕ(i,j)},

Eq. (3)

I3(i,j)=I(i,j),
with I13(i,j) the three deformed intensity patterns at pixel indices i and j, I the average or background intensity, I the intensity profile modulation or fringe pattern amplitude, and ϕ(i,j) the phase map of the deformed fringe pattern sequence. The phase map can be obtained by solving Eqs. (1)–(3)

Eq. (4)

ϕ(i,j)=  tan1(I1I3I2I3).
The phase ϕ is now extracted from the three input intensity images and can be linked directly to the local object height z through proper calibration and scaling and by removing the artificial 2π discontinuity jumps that originate from the limited principle value domain [π,π] of the arctangent function.

To achieve real-time display of 3-D height maps, several optimization techniques were employed in the proposed setup. First, acquisition, digital signal processing, and visualization of height maps are executed simultaneously in an asynchronous workload organization. Second, all digital signal processing components themselves are designed to operate in parallel on the graphics processing unit (GPU) of a standard graphics card. Third, GPU interoperability between parallel data processing interface compute unified device architecture (CUDA) and visualization interfaces, OpenGL and GLSL, were established to minimize data transfers across the PCI bus and to relieve the CPU from most computationally intensive tasks. The entire software library, from data acquisition to onscreen rendering of 3-D measurement results, was written in custom C++-based code to ensure optimal data flow control and memory management. The digital signal processing pipeline was executed on a commercial GTX980 graphics card with 3072 CUDA cores and 4096MB of on-board GDDR5 RAM memory.

A more detailed schematic overview of the real-time 2+1-phase profilometry pipeline is presented in Fig. 2. Input fringe patterns   I13 are acquired continuously and are downloaded from the camera data buffer in blocks of three. This process is supported by hardware triggering and controlled by the CPU. Simultaneously, the previous block of fringe patterns is transferred from host (CPU) to device (GPU) memory, where the 12-bit grayscale input images are stored as 2-byte unsigned integer texture arrays. Next, control over the data processing pipeline is passed over to the GPU and the CPU returns to data acquisition control. The first step in the digital signal processing pipeline is the extraction of the wrapped phase ϕW from the three input fringe patterns using Eq. (4). As the phase of neighboring pixels can be calculated independently, this process is highly suited for parallel implementation. Next, the original, continuous phase signal ϕU is recovered by adding an integer multiple of 2π to every phase value of the wrapped phase grid ϕW in a process commonly referred to as 2-D phase unwrapping.27 A 3×3 Gaussian filter kernel is applied to the unwrapped phase array to reduce spikes caused by motion or system noise, and the filtered phase values ϕf are converted to actual 3-D Euclidian coordinates by applying the reference-plane-based calibration approach.28 After recording the phase grid of a reference plane ϕR, where the base plane depth is set to z=0, the difference between ϕR and subsequent phase measurements is proportional to the object depth z by some constant scaling value. This scaling value zS can then easily be obtained from calibration measurements on an object of known dimensions. Finally, point normals to the vertex geometry zN are calculated by taking the cross products of neighboring phase value gradients so that Phong29 lighting may be applied in 3-D visualization mode. Note that the digital signal processing blocks as shown in Fig. 2 are not scaled to represent the respective time they take but are in fact expanded to fill the maximum time slot they could take up without obstructing the real-time pipeline. In reality, the entire digital signal processing of the three images, including memory transfer from host to device, takes up only 11.2 ms in total.

Fig. 2

(Color online) Real-time three-step 2+1-phase shifting profilometry timeline. Horizontally aligned blocks occur sequentially; vertically aligned blocks occur simultaneously. At all times, fringe pattern acquisition, digital signal processing, and on-screen rendering of successive measurements are executed asynchronously. Following the data-processing cycle of one 3-D-measurement (blue, diagonally hatched), three input fringe patterns are first captured by the high-speed camera and transferred from CPU to GPU memory. Next, the digital signal processing pipeline performs successive wrapped phase extraction (ϕW), phase unwrapping (ϕU), Gaussian filtering (ϕf), phase-to-height conversion and scaling (zS), and normal calculation (zN). Finally, the 3-D height map is rendered and displayed.

JBO_22_1_016008_f002.png

The GPU-calculated height measurements are visualized from GPU memory and no additional device-to-host memory transfers are needed. For final rendering of the result, two graphics application programming interfaces are employed: OpenGL and GLSL. To allow off-screen rendering, i.e., the drawing of scenes into buffers other than the frame or screen buffer, a standard double buffering technique is employed. To this end, an additional OpenGL frame buffer object is created as a backbuffer and its output is bound to GPU texture memory. Only when the backbuffer is fully rendered, it is swapped with the screen buffer or front buffer, effectively hiding the rendering process from the user while minimizing latency. For final display, three small shader programs are written in the OpenGL shading language GLSL. The first one incorporates the previously calculated point normals in a 3-D Phong light reflection model, computing the virtual illumination brightness and color of the object surface pixel per pixel, depending on user-defined variables such as object surface reflectance and 3-D position of light source and observer. The second one converts the height values to a pixel color using a predefined color map, generating a 2-D color-coded height map that is projected onto the object surface for quick assessment of object topography. The third one projects the uniform bright image that was gathered in the acquisition step of the 2+1-phase technique onto the object surface. This shader program enhances operator orientation as object texture features that cannot be resolved as height differences are incorporated into the final 3-D result, also.

3.

Results

3.1.

Measurement Precision

To determine the depth resolution of the proposed otoscopic profilometry setup, the surface profile of a flat glass plate was measured at various discrete distance intervals around the central focal plane. Pregenerated structured light patterns with a sinusoidal intensity profile of 24 pixels per fringe period were projected onto the glass plate surface. The glass plate itself was mounted onto a translation stage, which allowed displacement along the projection (or Z-) axis. The projected fringe patterns spanned a field of view of 8.15×6.11  mm across the glass plate surface, which was coated with magnesium oxide to ensure good diffuse reflectivity. Cross sections of these measurements along the central pixel lines in both X- and Y-dimensions are shown in the top row of Fig. 3. The numerical difference between these cross sections at depths of Z=+4  mm (red), Z=0  mm (black), and Z=4  mm (blue) around the central focal plane are included in the second row. No Gaussian smoothing was applied to the 3-D data.

Fig. 3

The surface profile of a flat glass plate was measured at discrete intervals around the central focal plane Z=0 to determine the measurement precision of the proposed otoscopic system. Cross sections of these depth measurements along (a) the X-axis and (b) the Y-axis are included in the first row, and the corresponding depth measurement errors (dZ) at object depths of Z=+4  mm (red), Z=0  mm (black), and Z=4  mm (blue) are included in the second row. (c) Plots the 2-D-RMS values of the difference between the full flat glass plate measurements and their respective best planar fits in function of object depth.

JBO_22_1_016008_f003.png

It can be seen that measurement precision decreases with increasing distance of the object to the focal plane. This effect is standard for structured light projection techniques due to the fact that the contrast of the projected fringe patterns deteriorates away from the focal plane due to defocusing effects. With decreasing fringe pattern contrast, the intensity values I13(i,j) in Eq. (4) converge toward each other. This reduces the numerical digital range of the intensity ratio formula used to calculate the phase ϕ(i,j) and limits the measurement resolution of the technique. Furthermore, it should be noted that the projection optics of the TI lightcrafter module are designed for asymmetric upward projection. When the projected fringe patterns reach the object surface outside of the focal plane, they contain a certain amount of geometric distortion relative to the angle between the normal of the object surface and the central projection axis. The accumulated measurement errors due to fringe pattern defocus and geometric distortion artifacts largely remain within a region of dZ=±200  μm.

To quantize the system’s mean measurement error as a function of object depth, the full-field two-dimensional root-mean-square (2D-RMS) values of the differences between the flat plate measurements at each depth interval and their respective best 2-D plane fits are plotted in part of Fig. 3(c). From this plot, axial resolution of 3-D measurements can be determined, given the total object depth span and assuming that the object is placed centrally around the focal plane. The measurement error in depth (dZ) increases with distance away from the central focal plane, though remains below an RMS of 60  μm within a region of ±4  mm around Z=0.

3.2.

Calibration Standard and Live Visualization Software

To demonstrate the real-time digital signal processing and visualization engine of the proposed otoscopic profilometry setup, a calibration standard with right-angled triangular ridges of 2.2-mm deep was measured. Again, the object was coated with magnesium oxide to ensure maximum diffuse reflection and its height profile was distributed symmetrically around the focal plane. A still frame from the live-recorded measurement video (Video 1) is shown in part of Fig. 4(a). The central pixel line of the resulting 3-D measurement is highlighted in red. The cross section of the height measurement along this central pixel line is illustrated in blue in Fig. 4(b) with its segmented linear fit overlaid in red. The numerical difference between height measurement and linear fit is included in Fig. 4(c).

Fig. 4

(a) Video 1 still frame from live measurement video demonstrating the real-time otoscopic profilometry technique, including a cross section of the object’s height profile Z along (b) the central pixel line, and (c) its measurement error dZ. (Video 1, 5.65 MB, mp4 [URL: http://dx.doi.org/10.1117/1.JBO.22.1.016008.1]).

JBO_22_1_016008_f004.png

As can be seen in the measurement video, the software allows the user to rotate the object in 3-D perspective view and to control visualization effects, such as colormap range, position of virtual light source relative to the observer, and quality threshold of rendered height map data. The video stream was recorded directly from the computer screen using frame grabber software and was resampled from 40 fps to 29.97 fps in accordance with standard H.264 codec protocol. During live measurement, the user is able to switch between custom shader programs designed to emphasize either the height profile or the texture of the object surface. For fast assessment of the object’s height profile, the object vertices are each assigned a color ranging from blue (low Z-value) to red (high Z-value). Alternatively, the shader can superimpose the uniformly lit input intensity image onto the object surface to incorporate texture features in the 3-D model. Finally, a blend or a combination of these two shader programs can be selected, which is shown in the first part of Video 1 and Fig. 4(a).

To reduce high-frequency phase noise during live measurements, a 3×3 Gaussian filter is applied to the unwrapped 3-D data. As a result, the 3-D geometry is smoothed out, and sharp corners become rounded. This effect can be seen clearly in Figs. 4(b) and 4(c) when measuring the right-angled indentations of the calibration standard. The measurement noise stays well within dZ=±50  μm, except at the borders between successive segmented lines, where the height measurements deviate up to dZ=±100  μm from the linear fit. Since the otoscopic profilometer was designed mainly to measure the surface profile of human TMs that typically possess a smooth 3-D tent-like shape and generally do not contain features with edged transitions, rounding artifacts of sharp angles do not impede the intended 3-D TM shape measurements. Nevertheless, the Gaussian filter is optional and can easily be omitted from the digital signal processing pipeline by the user during live measurement.

3.3.

Dynamic Pressurization of Tympanic Membrane Ex Vivo

As a first step toward implementation of real-time tympano-topography in the clinical ENT office, the otoscopic profilometer was used to measure the 3-D shape distortion of an ex vivo TM caused by dynamic pressurization. A fresh (frozen) temporal bone was obtained from the Life Legacy Foundation, and the TM remained in situ during the measurements. To mimic an in vivo measurement setup, the ear canal was left intact, and the TM was optically accessible only by inserting a plastic speculum into the ear canal. A small drop of water base cosmetic dye was used to coat the TM with a diffusely reflecting white layer. The middle ear cavity was (de-)pressurized by inserting a small tube through one of the mastoid cells and by sealing the remainder of the mastoid cells airtight using two-component silicon paste. The air tube was connected to a pressure generator that was controlled manually. The measurement setup is shown in Fig. 5.

Fig. 5

Dynamic tympano-topography measurement setup. The otoscopic profilometer head contains a disposable speculum, which is inserted into the ear canal of an airtight sealed temporal bone. The middle ear cavity is pressurized by connecting a small tube through one of the mastoid cells to a pressure generator (not shown here).

JBO_22_1_016008_f005.png

Dynamic pressurization of the TM was established by alternating the middle ear cavity pressure between 0.4  kPa and +0.4  kPa several times over the course of a 7.1-s measurement period. The 2+1 phase-shifting profilometry technique was used to monitor the 3-D deformation of the TM shape in real time, and the measurement results were downloaded to CPU RAM memory buffers and stored in computer memory for postprocessing and analysis purposes. Figure 6 shows the various steps in the reconstruction process as it is applied to one of the 284 sets of three input intensity images [Figs. 6(a) to 6(c)]. Two of these images contain sinusoidally varying intensity profiles with a relative phase shift of π/2 and the third one has a uniform intensity profile. Together, they can be combined to form a phase map of the object [Fig. 6(d)] from which the height profile is constructed after removal of the 2π-discontinuities [Fig. 6(e)]. Next, the phase values are converted to absolute height values, Gaussian filtering is applied to the 3-D geometry, and a reference map is deducted from the resulting height map. As we are interested in monitoring the 3-D shape deformation of the membrane when pressure is applied over it, we have chosen to assign the first unwrapped phase map of the measurement stream (at pressure 0 kPa) as the default reference geometry to be subtracted from the subsequent phase maps. The resulting height deformation maps [Fig. 6(e)] are therefore to be interpreted relative to zero pressurization.

Fig. 6

Various steps of the 2+1 phase-shifting profilometry pipeline during measurement of TM (left ear, medial view) shape deformation. (a)–(c) Input intensity images, (d) wrapped phase map, (e) unwrapped phase map, and (d) height deformation map.

JBO_22_1_016008_f006.png

It should be noted that a quality map is constructed in real time for every trio of input intensity patterns by calculating the intensity modulation [the I-term in Eqs. (1) and (2)] pixel per pixel and by performing binary thresholding accordingly. This way, parts of the image that are out of focus, contain shadows, or are under- or overexposed can be excluded from the phase map and do not affect the resulting 3-D measurement. However, the user is able to set outer limits to the region of interest by marking the region with an interactive polygon, should manual intervention be required. The quality map threshold value and the boundaries of the binary mask can be modified in real time during acquisition. Incorporation of the dynamic quality map in the measurement pipeline increases the robustness of the technique during live measurement when local structural irregularities (e.g., the tissue of the ear canal partially blocking the view of the TM in the upper right side of the measurement sequence in Fig. 6) might otherwise induce measurement artifacts in the reconstruction of the 3-D phase map.

As shown in Fig. 7, the pressure change in the middle ear cavity results in height deformation of the TM. Part (I) shows the height deformation map of the TM when the applied pressure was near maximum at +0.4  kPa overpressure in the middle ear cavity. The manubrium border of the malleus is marked by a dashed black line, and three distinct locations on the membrane surface along the manubrium (black), near the center of the posterior part of the pars tensa (blue), and near the side of the pars tensa toward the annulus (red) are indicated by asterisks. The corresponding height deformation profiles of the membrane at these locations are plotted in function of time in part (II) and reflect the pressure change cycles clearly. It can be seen that the TM surface along the manubrium (black line) hardly moves during pressurization and that maximum membrane deformation occurs toward the center of the pars tensa (blue line) and to a lesser degree near the edge of the pars tensa (red line). This observation is confirmed by the 2-D [part (III), (a) to (e)] and 3-D [part (IV), (f) to (j)] visualizations of the full-field membrane height deformation maps, plotted here at discrete intervals [(a) to (e)] during a pressurization cycle. A full video of the eardrum deformation response is included in Video 2. The (semitransparent) colored height deformation map is plotted on top of the input intensity image to illustrate the deformation of the projected line patterns on the moving object surface.

Fig. 7

Video 2 TM height deformation response to dynamic middle ear pressure variation. Height deformation profiles and full-field 2-D and 3-D deformation maps are included in parts (II), (III) and (IV), respectively. The positive z-axis is defined toward the observer and the colorbar legend defined in part I applies to parts (III) and (IV), also. (Video 2, 4.74 MB, mp4 [URL: http://dx.doi.org/10.1117/1.JBO.22.1.016008.2]).

JBO_22_1_016008_f007.png

Since full-field phase maps are recorded, the user is able to monitor membrane deformation at any desired point on the observable TM surface. This way, the sensitivity of the technique to detect middle ear pressure variation can be maximized by monitoring the response curves of local areas on the TM that deform more than others. As the TM expands up to 1 mm when realistic pressure variations of 0.4 kPa are applied to it, the system’s measurement precision is more than sufficiently high enough to resolve the resulting deformation. Note the asymmetric response curve of the TM topography as a function of induced pressure. The TM displacement induced by middle ear overpressure is larger than for the corresponding underpressure by a factor of almost 2. This effect was previously described by Dirckx and Decraemer30 and can be attributed to the conical shape of the membrane.

4.

Discussion

The ex vivo TM measurements demonstrate the ability of the 3-D otoscope to detect pressure variations in the middle ear cavity by monitoring the 3-D shape variation of the eardrum in real time. By asking patients to perform middle ear cavity (de)pressurization maneuvers, such as the Valsalva maneuver (forceful exhalation while closing one’s mouth and nose), TM displacement can be measured and eardrum mobility can be quantified. Therefore, the technique may have applications in the diagnosis of different pathologies encountered in the ENT office.

For example, impeded mobility at the level of the eardrum may be used to detect certain cases of otosclerosis or otospongiosis where fixation of the incudomallear or incudostapedial joint reduces the mobility of the eardrum.31

Another application could be the detection of local elasticity variations in the TM. As aberrant local deformations or weak spots due to loss of stiffness of the TM could indicate structural damage and be an important precursor for cholesteatoma formation,32 a need for quantitative evaluation of the full-field TM shape in vivo has been expressed.

To link the pathophysiological mechanisms of chronic middle ear disease, such as otitis media with effusion (OME) to loss of TM stiffness, Dirckx et al.33 and von Unge et al.34 have conducted a series of ex vivo tympano-topography experiments on gerbil ears. By measuring the 3-D pressure-induced shape deformations of TMs with varying degrees of otitis media when increasing levels of pressure were applied, a correlation between TM volume displacement and stage of OME could be made. Therefore, as it has been shown that eardrum topography is a valuable technique to quantify the stage of the disease, a real-time handheld device to measure eardrum deformation may also find application in this field.

In addition to the detection of pathologies that result directly in aberrant eardrum mobility, tympano-topography could also be employed as a noninvasive indicator for deficient ET functioning. Patients who suffer from obstructive ET dysfunction lack a mechanism to automatically equalize the air pressure in their middle ear cavity to the outside atmospheric pressure. This leads to inadequate aeration of the middle ear and causes the eardrum to retract inward with possible destruction of the ossicular chain. Recently, a surgical technique to treat obstructive ET dysfunction using a balloon catheter to stretch the ET was proposed.35 Promising short- and intermediate-term results were confirmed by a study of Van Rompaey et al.,36 indicating no complications within the first year after 41 catheter insertions in 26 patients. Again, however, the authors stated that the first priority is to have an objective evidence-based measurement technique to validate the long-term success of the proposed surgical procedure. To this end, dynamic tympano-topography may provide a solution for early and quantitative detection. By measuring the surface shape variation of the TM while patients are asked to perform the Valsalva maneuver, pressure stabilization of the middle ear cavity and therefore functioning of the ET can be evaluated objectively and quantitatively by monitoring the resulting changes in TM surface volume.

Due to the digital nature of the projector-camera system, any structured light-based profilometry technique can be implemented in the otoscopic profilometer. In this paper, we have selected the 2+1 phase-shifting technique for two main reasons. First, it requires only three distinct input intensity images. This limits the acquisition time of a single 3-D image to 25 ms for our system, sufficiently brief to minimize motion artifacts due to patient or operator movement. Second, the uniformly lit intensity image can be used in real time during live measurements to aid operator orientation when selecting the desired field of view on the TM surface.

Although the presented results provide a successful proof-of-principle of tympano-topography as a noninvasive tool to monitor middle ear pressure change, several optimizations to the otoscopic profilometry prototype system may be necessary before validation in the clinical ENT office is in order. Currently, measurement resolution is limited by the 12-bit grayscale range of the CCD camera. Due to the semitransparent nature of the TM structure, only a fraction of the light projected onto the TM surface is reflected back. To improve the measurement sensitivity, the CCD range needs to be divided into as many discrete grayscale bins as possible. This can be achieved either by employing a camera sensor with a higher bit-range or coating the TM surface with a diffusely reflecting dye or both.

Finally, it is worth mentioning that since the object-to-lens distance in our system is much larger than the observed depth variations themselves, the resulting geometric distortion artifacts between object points closer to and farther away from the otoscope head are very small when compared with systems that employ wide-angle lenses close to the object surface (e.g., dual-barrel endoscopic setups).

5.

Conclusion

An otoscopic profilometry system based on structured light projection was developed and experimentally tested. The handheld system is able to detect and quantify 3-D eardrum deformation caused by small middle ear pressure changes in real time. Full-field height maps of 3-D objects can be recorded at 40 frames per second with a measurement precision varying between 10 and 100  μm, depending on local surface reflection parameters and total object depth span. The device has a form factor similar to existing otoscopes, is easy to use, and provides a first step toward implementation of quantitative tympano-topography in the clinical ENT office.

Disclosures

The authors have no relevant financial interests in the manuscript and no other potential conflicts of interest to disclose.

Acknowledgments

The authors thank the Research Foundation—Flanders (FWO) for their financial support.

References

1. 

R. L. Shelton et al., “Optical coherence tomography for advanced screening in the primary care office,” J. Biophotonics, 7 (7), 525 –533 (2014). http://dx.doi.org/10.1002/jbio.201200243 Google Scholar

2. 

Z. Hubler et al., “Real-time automated thickness measurement of the in vivo human tympanic membrane using optical coherence tomography,” Quantum Imaging Med. Surg., 5 (1), 69 –77 (2015). http://dx.doi.org/10.3978/j.issn.2223-4292.2014.11.32 Google Scholar

3. 

S. M. Solís, F. M. Santoyo and M. D. S. Hernández-Montes, “3D displacement measurements of the tympanic membrane with digital holographic interferometry,” Opt. Express, 20 (5), 5613 (2012). http://dx.doi.org/10.1364/OE.20.005613 OPEXFF 1094-4087 Google Scholar

4. 

M. Khaleghi et al., “Digital holographic measurements of shape and 3D sound-induced displacements of tympanic membrane,” Opt. Eng., 52 (10), 101916 (2013). http://dx.doi.org/10.1117/1.OE.52.10.101916 Google Scholar

5. 

S. Van der Jeught, J. Soons and J. J. J. Dirckx, “Real-time microscopic phase-shifting profilometry,” Appl. Opt., 54 (15), 4953 –4959 (2015). http://dx.doi.org/10.1364/AO.54.004953 APOPAI 0003-6935 Google Scholar

6. 

K. Gurusamy, K. Samraj and B. R. Davidson, “Three dimensional versus two dimensional imaging for laparoscopic cholecystectomy,” Cochrane Database Syst. Rev., (1), CD006882 (2008). http://dx.doi.org/10.1002/14651858.CD006882.pub2 Google Scholar

7. 

R. C. W. Bell and J. G. Price, “Laparoscopic inguinal hernia repair using an anatomically contoured three-dimensional mesh,” Surg. Endosc. Other Interv. Tech., 17 (11), 1784 –1788 (2003). http://dx.doi.org/10.1007/s00464-002-8763-4 Google Scholar

8. 

N. J. Durr, G. Gonzalez and V. Parot, “3D imaging techniques for improved colonoscopy,” Expert Rev. Med. Dev., 11 (2), 105 –107 (2014). http://dx.doi.org/10.1586/17434440.2013.868303 1743-4440 Google Scholar

9. 

S. Ieiri et al., “Augmented reality navigation system for laparoscopic splenectomy in children based on preoperative CT image using optical tracking device,” Pediatr. Surg. Int., 28 (4), 341 –346 (2012). http://dx.doi.org/10.1007/s00383-011-3034-x PSUIED 1437-9813 Google Scholar

10. 

M. E. Hurley et al., “Three-dimensional volume-rendered helical CT before laparoscopic adrenalectomy,” Radiology, 229 (2), 581 –586 (2003). http://dx.doi.org/10.1148/radiol.2292021390 RADLAX 0033-8419 Google Scholar

11. 

A. Schneider et al., “Augmented reality assisted laparoscopic partial nephrectomy,” in Int. Conf. on Medical Image Computing and Computer-Assisted Intervention, 357 –364 (2014). Google Scholar

12. 

L. M. Su et al., “Augmented reality during robot-assisted laparoscopic partial nephrectomy: toward real-time 3D-CT to stereoscopic video registration,” Urology, 73 (4), 896 –900 (2009). http://dx.doi.org/10.1016/j.urology.2008.11.040 Google Scholar

13. 

A. F. Durrani and G. M. Preminger, “Three-dimensional video imaging for endoscopic surgery,” Comput. Biol. Med., 25 (2), 237 –247 (1995). http://dx.doi.org/10.1016/0010-4825(95)00001-K CBMDAW 0010-4825 Google Scholar

14. 

A. Tabaee et al., “Three-dimensional endoscopic pituitary surgery,” Neurosurgery, 64 (Suppl. 5), ons288 –ons295 (2009). http://dx.doi.org/10.1227/01.NEU.0000338069.51023.3C NEQUEB Google Scholar

15. 

D. Stoyanov et al., “Real-time stereo reconstruction in robotically assisted minimally invasive surgery,” Lect. Not. Comput. Sci., 6361 275 –282 (2010). http://dx.doi.org/10.1007/978-3-642-15705-9_34 LNCSD9 0302-9743 Google Scholar

16. 

N. T. Clancy et al., “Spectrally encoded fiber-based structured lighting probe for intraoperative 3D imaging,” Biomed. Opt. Express, 2 (11), 3119 –3128 (2011). http://dx.doi.org/10.1364/BOE.2.003119 BOEICL 2156-7085 Google Scholar

17. 

C. Schmalz et al., “An endoscopic 3D scanner based on structured light,” Med. Image Anal., 16 (5), 1063 –1072 (2012). http://dx.doi.org/10.1016/j.media.2012.04.001 Google Scholar

18. 

M. Hayashibe, N. Suzuki and Y. Nakamura, “Laser-scan endoscope system for intraoperative geometry acquisition and surgical robot safety management,” Med. Image Anal., 10 (4), 509 –519 (2006). http://dx.doi.org/10.1016/j.media.2006.03.001 Google Scholar

19. 

P. Mountney, D. Stoyanov and G. Z. Yang, “Three-dimensional tissue deformation recovery and tracking,” IEEE Signal Process Mag., 27 (4), 14 –24 (2010). http://dx.doi.org/10.1109/MSP.2010.936728 ISPRE6 1053-5888 Google Scholar

20. 

D. J. Mirota, M. Ishii and G. D. Hager, “Vision-based navigation in image-guided interventions,” Annu. Rev. Biomed. Eng., 13 297 –319 (2011). http://dx.doi.org/10.1146/annurev-bioeng-071910-124757 ARBEF7 1523-9829 Google Scholar

21. 

N. Bedard et al., “Light field otoscope,” in Imaging Systems and Applications, IM3C-6 (2014). Google Scholar

22. 

A. J. Das et al., “A compact structured light based otoscope for three dimensional imaging of the tympanic membrane,” Proc. SPIE, 9303 93031F (2015). http://dx.doi.org/10.1117/12.2079761 PSISDG 0277-786X Google Scholar

23. 

M. H. McDonald et al., “New insights into mechanism of Eustachian tube ventilation based on cine computed tomography images,” Eur. Arch. Oto-Rhino-Laryngol., 269 (8), 1901 –1907 (2012). http://dx.doi.org/10.1007/s00405-011-1829-y EAOTE7 Google Scholar

24. 

S. Zhang, “Recent progresses on real-time 3D shape measurement using digital fringe projection techniques,” Opt. Lasers Eng., 48 (2), 149 –158 (2010). http://dx.doi.org/10.1016/j.optlaseng.2009.03.008 Google Scholar

25. 

S. Van der Jeught and J. J. J. Dirckx, “Real-time structured light profilometry: a review,” Opt. Lasers Eng., 87 18 –31 (2016). http://dx.doi.org/10.1016/j.optlaseng.2016.01.011 Google Scholar

26. 

S. Zhang and S.-T. Yau, “High-speed three-dimensional shape measurement system using a modified two-plus-one phase-shifting algorithm,” Opt. Eng., 46 (11), 113603 (2007). http://dx.doi.org/10.1117/1.2802546 Google Scholar

27. 

S. Van der Jeught, J. Sijbers and J. Dirckx, “Fast Fourier-based phase unwrapping on the graphics processing unit in real-time imaging applications,” J. Imaging, 1 (1), 31 –44 (2015). http://dx.doi.org/10.3390/jimaging1010031 Google Scholar

28. 

Y. Xu et al., “Phase error compensation for three-dimensional shape measurement with projector defocusing,” Appl. Opt., 50 (17), 2572 –2581 (2011). http://dx.doi.org/10.1364/AO.50.002572 APOPAI 0003-6935 Google Scholar

29. 

B. T. Phong, “Illumination for computer generated pictures,” Commun. ACM, 18 (6), 311 –317 (1975). http://dx.doi.org/10.1145/360825.360839 CACMA2 0001-0782 Google Scholar

30. 

J. J. J. Dirckx and W. F. Decraemer, “Human tympanic membrane deformation under static pressure,” Hear. Res., 51 (1), 93 –105 (1991). http://dx.doi.org/10.1016/0378-5955(91)90009-X HERED3 0378-5955 Google Scholar

31. 

W. Alian et al., “The effect of increased stiffness of the incudostapedial joint on the transmission of air-conducted sound by the human middle ear,” Otol. Neurotol., 34 (8), 1503 –1509 (2013). http://dx.doi.org/10.1097/MAO.0b013e3182923ed0 Google Scholar

32. 

M. Von Unge et al., “Evaluation of a model for studies on sequelae after acute otitis media in the Mongolian gerbil,” Acta Otolaryngol., 129 (3), 261 –267 (2009). http://dx.doi.org/10.1080/00016480802239091 Google Scholar

33. 

J. J. J. Dirckx et al., “Volume displacement of the gerbil eardrum pars flaccida as a function of middle ear pressure,” Hear. Res., 118 (1–2), 35 –46 (1998). http://dx.doi.org/10.1016/S0378-5955(98)00025-2 Google Scholar

34. 

M. Von Unge et al., “Tympanic membrane displacement patterns in experimental cholesteatoma,” Hear. Res., 128 (1–2), 1 –15 (1999). http://dx.doi.org/10.1016/S0378-5955(98)00183-X HERED3 0378-5955 Google Scholar

35. 

T. Ockermann et al., “Balloon dilatation eustachian tuboplasty: a clinical study,” Laryngoscope, 120 (7), 1411 –1416 (2010). http://dx.doi.org/10.1002/lary.20950 Google Scholar

36. 

V. Van Rompaey et al., “Eustachian tube balloon dilation surgery: current state of the art,” Tijdschr. Geneeskd., 70 (18), 1052 –1059 (2014). http://dx.doi.org/10.2143/TVG.70.00.2000000 Google Scholar

Biography

Sam Van der Jeught received his PhD in physics in 2015 from the University of Antwerp for his dissertation entitled “Optical techniques for real-time morphology measurement of the tympanic membrane.” He is currently researching new optical methods for measuring human eardrum shape in real-time and in vivo as a postdoctoral fellow with funding from the Research Foundation—Flanders.

Joris J. J. Dirckx received his PhD in physics in 1991 from the University of Antwerp for his dissertation entitled “Automated moiré topography and its use for shape and deformation measurements of the eardrum.” He is currently director of the Laboratory of Biomedical Physics and full professor in the Department of Physics. He teaches courses in general physics, optical microscopy, and biomedical imaging.

© 2017 Society of Photo-Optical Instrumentation Engineers (SPIE) 1083-3668/2017/$25.00 © 2017 SPIE
Sam Van der Jeught and Joris J. J. Dirckx "Real-time structured light-based otoscopy for quantitative measurement of eardrum deformation," Journal of Biomedical Optics 22(1), 016008 (11 January 2017). https://doi.org/10.1117/1.JBO.22.1.016008
Received: 17 October 2016; Accepted: 20 December 2016; Published: 11 January 2017
Lens.org Logo
CITATIONS
Cited by 7 scholarly publications and 5 patents.
Advertisement
Advertisement
RIGHTS & PERMISSIONS
Get copyright permission  Get copyright permission on Copyright Marketplace
KEYWORDS
Ear

3D metrology

Fringe analysis

Visualization

Head

Digital signal processing

Structured light

RELATED CONTENT

Reaching into Pictorial Spaces
Proceedings of SPIE (February 25 2014)
3D real-time measurement system of seam with laser
Proceedings of SPIE (February 21 2014)
Presence capture cameras a new challenge to the...
Proceedings of SPIE (April 29 2016)
Video-image-based photogrammetry and applications
Proceedings of SPIE (October 06 1994)

Back to Top