1.

Introduction

Glasses-free three-dimensional display has experienced an unprecedented development in recent years. This is mainly because three-dimensional display provides more perceptual depth cues than conventional two-dimensional display.^1,2 Two perceptual cues are crucial for humans to observe objects: the stereo parallax (seeing a different image with each eye-pupil) and the motion parallax (seeing different images when viewer moves). Providing high quality three-dimensional experience for viewers with both stereo and motion parallax is ideal but challenging.

At present, the auto-stereoscopic display based on multi-view technique is attainable without need of any special aids, and has been widely used in many commercial applications.^3–6 Multi-view three-dimensional display generates a number of individual images, with each of them visible only in a particular viewing zone. These viewing zones are sophisticatedly designed to abut one another. Thus, at an optimal viewing distance and for certain viewpoints, different images are seen by different eyes to achieve stereoscopy. However, one of the main limitations of multi-view displays is the limited size of viewing zones. Perfect stereo motion parallax can only be achieved at a small number (usually several or decades) of viewing positions. This makes these three-dimensional displays unable to provide required continuous motion parallax. Other aspects related to imaging quality factors of conventional auto-stereoscopic displays are ray source resolution, angular sampling resolution, and viewing angle, despite its easier implementation.⁷

To enable motion parallax and conform to the way in which humans observe objects in actual visual space, displays with large (even 360-deg all-around) viewing zones and smooth motion parallax have been a major advancement of three-dimensional display technology during the past few years, and undoubtedly they suggest tremendous application potentials in the future. Among diverse kinds of reported methods, volumetric display and light field display are two representative large viewing zone three-dimensional display technologies and considered as the natural three-dimensional displays.^8,9 In terms of conventional volumetric displays such as static volume and swept volume displays, research on computational photography and electronic manufacturing development, while the products are still in experimental phase, great potential is indicated.^10–12 However, the massive data and high-performance modulation devices that holographic display requires make its impossible to be manufactured soon.⁸

The light field display describes rays of the actual light field of virtual objects. Therefore, it produces the best visual qualities. Zhejiang University proposed two systems achieving omni-directional view three-dimensional display effect using high-speed projector in 2009 and 2010.^13,14 With this method, a high-speed projector projected, in a time-sequential fashion, images of different views onto a rotating selective-diffusing screen. Thus, observers will get delicate three-dimensional images all around the display system. Another approach to achieve all-around three-dimensional display is to create a cylindrical display area by rotating LED array or other display devices.¹⁵ However, most existing large viewing zone three-dimensional displays require high-speed projectors, rotation mechanism, precise optoelectronic modulators, and huge data to transfer and display.⁸ These factors have until now resulted in small-sized but inordinately high-cost systems.

An effective alternative method for achieving large-scale display is using the multi-projector light field display approach. Some researches have already provided continuous horizontal parallax to enhance volumetric understanding in this method.^16–18 This multi-projector configuration succeeds in achieving large-scale display but existing systems, like Holografika systems,^16,17 have not yet performed 360-deg display effect. Moreover, there is far less discussion in the literature about specific light field reconstruction image synthesis to achieve large-scale cylindrical, even 360-deg display and viewing area.

In this paper, a large-sized light field three-dimensional display system using multi-projectors and directional diffuser is proposed. Stereo parallax and smooth horizontal motion parallax are achieved by this approach. The viewing area of this system is much larger than that of conventional auto-stereoscopic displays and can be easily extended to 360-deg display performance. In the rest of the paper, we first introduce the system that generates the light field of a three-dimensional scene by projecting processed images from projectors onto a cylindrical directional diffuser screen. Both optical characteristic of directional diffuser and specific image synthesis algorithm to produce 360-deg light field are discussed to help carry out the prototype. Far less in the literature has reported these together to make general readers understand the light field display clearly and better. In the third section, a prototype with 100 mini-projectors and a nearly 2-meter-diameter cylindrical directional diffuser screen is developed to validate this approach. Finally, results are presented to demonstrate that the three-dimensional scene reconstructed by the prototype can be observed in quite a wide range around the diffuser screen by any number of observers.

2.

Proposed System

The display system proposed in this paper is a light field display with horizontal parallax that employs a large number of projectors and a directional diffuser screen to generate rays of light in the light field. In this section, the light field reconstruction principle is introduced first, followed by description of the hardware configuration and parameters. The algorithm designed for 360-deg display to create the images projected by projectors is given at the end of the section.

2.1.

Principle of Light Field Display

The light field, a mapping from rays to radiance, is an important tool in optical system simulation and analysis.^19,20 Rays in the light field can be parameterized in different formations. In this paper, we use a concentric light field parameterization that a ray is represented by two spatial points, $P$ and $A$. The point $P$ is on a circle and the other point $A$ is on virtual three-dimensional scene, which the light field displays, as is shown in Fig. 1.

Fig. 1

Schematic of the three-dimensional display system (a) using multi-projectors and a special directional diffuser; display principle (b).

To display these rays in a light field, the proposed system discretizes the circle into spatial points and uses an array of projectors, consisting of $N$ projectors aligned in the same horizontal plane with lens pupil positioned at $P_1$ to $P_N$. However, due to a limited number of projectors, rays are projected discontinuously and horizontally from these discretely positioned projectors. That is to say, without a cylindrical directional diffuser, screen observers would only obtain a series of discontinuous emitting exit-pupils of the projectors. To smooth the discontinuity of rays, a cylindrical directional diffuser screen is set in front of these projectors, as illustrated in Fig. 1(a). Rays in the light field are generated from projectors and projected on the diffuser screen. Any arbitrary point on the diffuser screen within the viewing zone is illuminated by a set of projectors. The directional diffuser screen here ensures that these rays coming from different directions are emitted in carefully controlled angular sections, so that each ray is correctly emitted to its direction, forming the light field. Such a scattering smooths the gaps between rays and produces continuous imagery.

Given a virtual three-dimensional object in the light field, lights emitted from the virtual three-dimensional object are represented in rays projected from projectors. The point is to figure out the direction and magnitude of each possible ray, represented as Ray. Then the light field distribution could be described using the mathematical vector set, represented as $S$, defined by Eq. (1):

Eq. (1)

$$S=[\mathrm{Ray}|\overrightarrow{P_{ij}{A}_k},i=(1,2,3,\cdots ,N),j=(1,2,3,\cdots ,M),k=(1,2,3,\cdots ,L)|],$$

where $i$ stands for the sequence number of the projector, $j$ stands for a certain pixel in the projector it belongs to, and $k$ stands for the sequence number of the reconstructed point in three-dimensional scene. $N$, $M$, and $L$ stand for the total number of projectors, pixels of each projector, and points of the three-dimensional scene, respectively.

Take any two viewing positions $V_1$, $V_2$ in the viewing area, for example. Due to the light field reconstruction principle, rays that make the point $A_1$, $A_2$, or $A_3$ on the virtual three-dimensional object observed by the viewing positions $V_1$ and $V_2$ are emitted from different projecting points to form the light field, as illustrated in Fig. 1(b). This produces the stereo parallax.

In the system, the distribution of light rays is not specially designed for certain viewing positions as that of multi-view three-dimensional displays. It means, ideally, by distributing rays as uniformly as possible, the adjacent light rays could be made close sufficiently. Then the viewed imagery can be more continuous without the jumping when the viewer moves. This produces the smooth motion parallax. The specific image mapping algorithm is given in Sec. 2.3.

2.2.

Characteristics of the Diffuser Screen

Images of projectors are projected onto the directional diffuser screen from different directions simultaneously. The function of the special directional diffuser is similar to that of the holographic functional materials, which is applied to control the diffuse angle.^21,22 Because in this system the projectors are arranged in horizontal arc and consider horizontal parallax only, the special directional diffuser is designed to have a small certain diffuse angle in horizontal and a large diffuse angle in vertical. Hence, one projector gives a narrow vertical stripe of image for a certain viewing position [Fig. 2(a)]. The diffuse angle in horizontal is an important characteristic of the diffuser screen to achieve smoother images.

Fig. 2

Schematic of the function of directional diffuser (a); and illustration of the relation between the interval angle of adjacent projectors and diffuse angle of the diffuser (b).

Based on the geometrical relation in Fig. 2(b), the two principal rays of adjacent projectors, represented in different colors, should be diffused in a certain angle so as to meet accurately, as the shadow regions show. Then we can attain the relation between interval angle of adjacent projectors $\delta$ and diffuse angle of the special diffuser $\epsilon$, which is defined by Eq. (2):

Eq. (2)

$$\frac{\sin (\delta /2-\epsilon /2)}{R}=\frac{\sin (\epsilon /2)}{D},$$

where $R$ is the radius of the directional diffuser, and $D$ is the distance from the projector to the center of the system $O$. Since $\delta$ and $\epsilon$ are rather small, approximation can be obtained to come out as follows in Eq. (3):

Eq. (3)

$$\frac{\delta -\epsilon }{2R}=\frac{\epsilon }{2D}.$$

Finally, the optimum diffuse angle of the special diffuser $\epsilon$ is defined by Eq. (4):

Eq. (4)

$$\epsilon =\frac{D\delta }{R+D}.$$

In addition, the boundary condition of $\delta$, which should be satisfied to attain stereo parallax and motion parallax when the head moves across different viewing positions, is given by Eq. (5):

Eq. (5)

$$\delta <2\arctan \frac{d}{2R_V},$$

where $d$ is the distance between two eye- pupils, and $R_V$ is the optimum viewing distance from the center of the system $O$.

2.3.

Image Synthesis Algorithm

The task of image synthesis algorithm is to produce images projected from projectors so as to correctly represent the lights emitted from the virtual three-dimensional object. In contrast to multi-view three-dimensional displays, our system does not generate view images for certain viewing positions. It makes our system unable to directly use the standard multiple-center-of-projection (MCOP) algorithm that has been generally used in previous three-dimensional displays.²³ And the key feature of our image synthesis lies in its ability to generate 360-deg projecting images with more feasible and flexible algorithm. Based on the adaption proposed in Ref. 21, we present our modified image synthesis algorithm here to generate projecting images.

The scene point $A_k$ is first projected to the projectors’ array to find its projected pixels $P_{ij}$, as well as which projector it belongs to. Then the intersection of ray $\overrightarrow{P_{ij}{A}_k}$ and the arc diffuser screen is computed and regarded as the shading point for observing. Such a process is widely adapted in computer graphics.

Specifically, take a central projector in the conventional $xyz$ coordinate system first, and deduce the algorithm in $xy$ plane and $yz$ plane individually. The image synthesis algorithm can be illustrated by Fig. 3. Based on the divergence angle $\omega$ of every projector, assume an image plane located on $yz$ plane, which is also the projected image, defined as $Q_1{Q}_2$ in the figure. Actually, choice of the location of this computing image plane will not affect the final mapping results. Consider a certain pixel in a projector owns horizontal and vertical position, defined as $P_{\mathrm{ist}}$ here to present clearly, wherein $s$ and $t$ stand for the horizontal and vertical sequential number of this pixel in the projector, respectively. The product of maximum $s$ and $t$ equals $M$ mentioned above. Then the intersection of vector $\overrightarrow{P_{\mathrm{ist}}{A}_k}$ and $Q_1{Q}_2$ is the pixel $P_{\mathrm{ist}}$ that needs to be projected. In the given $xyz$ coordinate system, the pixel $P_{\mathrm{ist}}$ $(0,y,z)$ can be inferred with the known $A_k$ $(x_1,y_1,z_1)$ and $P_m$ $(-D,0,0)$, which is given by Eq. (6):

Eq. (6)

$$(y,z)=(\frac{y_1·D}{x_1+D},\frac{z_1·D}{x_1+D}).$$

Fig. 3

Illustration of the image synthesis algorithm.

Then we further get the value of $s$ and $t$, which can map a certain pixel in the projector it belongs to, defined by Eq. (7):

Eq. (7)

$$(s,t)=\mathit{round}\left[\right(\frac{y}{Q_1{Q}_2}+1)·\frac{M_1}{2},(\frac{z}{Q_1{Q}_2}+1)·\frac{M_2}{2}],$$

where the function round rounds the number to the nearest integer, and $M_1$ and $M_2$ are the total pixel numbers of the projector in horizontal and vertical mentioned above. Hence we get the final equation [Eq. (8)] by substituting all the known parameters:

Eq. (8)

$$(s,t)=\mathit{round}\left\{\right[\frac{y_1·\cot (\omega /2)}{x_1+D}+1]·\frac{M_1}{2},[\frac{z_1·\cot (\omega /2)}{x_1+D}+1]·\frac{M_2}{2}\}.$$

Traverse all the scene point $A_k$, then compute all the intersection point $P_{\mathrm{ist}}$ for this projector, which is chosen as the basic reference. After attaining the mapping relation for this projector, for arbitrary projector $P_i$, it is easy to carry out the similar relation by rotating the coordinate system from $xyz$ to $uvz$ system around the original point $O$ in an angle of $\frac{(N-i)·\delta }{2}$, where $\delta$ is the interval angle of adjacent projectors mentioned above. At this time, $A_k$ $(x_1,y_1,z_1)$ turns to $A_k$’ $(u_1,v_1,z_1)$, and mapping results can be attained by putting the $A_k$’ into Eq. (8). Hence, at last we obtain a series of projecting image sources, even 360-deg display requirements.

3.

Prototype Equipment and Experimental Results

To verify this display principle and system design, a piece of prototype equipment is developed with 100 synchro-control L-CoS mini-projectors with a nearly 2-meter-diameter arc directional diffuser. The prototype is shown in Fig. 4. The 100 mini-projectors are set in four rows staggered horizontally to make the adjacent exit-pupils closer in the horizontal direction. Generally, it displays three-dimensional images with numerous continuous viewing positions in horizontal around the diffuser within a large vertical observing area. The interval angle between the adjacent projectors is designed as 0.8 deg, which is related to the horizontal diffuse angle of the special directional diffuser. The present resolution used for two-dimensional images from each projector is $320\times 240\text{pixels}$, while the size of three-dimensional images is as large as 110 cm in width and 90 cm in height. The specifications of our prototype are listed in Table 1. Computer graphics real-time image synthesis algorithm is applied to generate the projected source images for light field reconstruction as introduced in Sec. 2.3.

Fig. 4

Experimental device of the mini-projectors’ array (a); and the arc directional diffuser screen (b).

Table 1

Specifications of the prototype equipment.

As analyzed in Sec. 2.2, the directional diffuser plays a crucial part in achieving continuous images constituted by many element images. Thus, the luminance angular distribution of the light rays emitted through the special directional diffuser might have great influence on the uniformity of three-dimensional images, as well as the splicing accuracy. Here we use two different designed perpendicular lenticular sheets assembled together to attain the required diffuse angle in horizontal and vertical. (The supplier of these sheets is an ordinary Chinese company.) The characteristics are measured with a narrow parallel beam of incoherent rays (LED source), which passes through the directional diffuser and then illuminates a band area on a lambertian screen.

The image of the illuminated band area is then captured by a camera, as illustrated by Fig. 5(a). After processing the image with engineering software, the luminance angular distribution is obtained, as illustrated by Fig. 5(b). The curve demonstrates the characteristic of diffusing the light to a controlled angular distribution in both horizontal and vertical directions. In the figure, the horizontal axis represents the angle of the diffused light, while the vertical axis represents the relative luminance. The horizontal and vertical angular distribution of diffused light is presented by dotted and solid lines, respectively. From the distribution, it can be inferred that the small diffuse angle in horizontal is matched to the interval angles of adjacent viewing positions, taking the half-width of the angle distribution into consideration, which is about 0.9 deg. In the meantime, from the vertical perspective, the diffuse angle is sufficiently large to create enough comfortable observing area. Because of this characteristic, when an element image is projected onto the special diffuser screen, only a narrow stripe image would be observed.

Fig. 5

Characteristics of the directional diffuser. (a) Measuring setup; (b) horizontal and vertical luminance angular distribution.

Figure 6(a) to 6(c) shows the designed results of a three-dimensional scene with a rotation angle of 2.4 deg each. A three-dimensional scene of a color cartoon tiger and a color cartoon dragon is designed for display, respectively. Figure 7(a) to 7(c) shows the photos of a three-dimensional scene displayed by the prototype equipment, and the photos are captured at different horizontal viewing positions. Obviously, bright and large-sized vivid three-dimensional scenes displayed by the system can be observed at different horizontal viewing positions around the cylindrical display area. The observers will get depth perception from delicate stereo parallax and horizontal motion parallax.

Fig. 6

Designed results of the three-dimensional scene from three different views in horizontal (a), (b), and (c).

Fig. 7

Experimental results of the reconstructed three-dimensional scene from three different views in horizontal (a), (b), and (c) (Video 1, MOV, 2.5 MB) [URL: http://dx.doi.org/10.1117/1.OE.52.1.017402.1].

This method will also achieve floating effect for interaction to some degree. Figure 8(a) and 8(b) present the photos of an observer interacting, or “playing,” with the vivid virtual characters from two different perspectives. Moreover, other interaction devices can be attached to build future three-dimensional human–machine interaction systems, which could be widely applied in many areas, including medicine, advertising, public exhibition, and so on.

Fig. 8

Photos of the viewer playing with virtual cartoon characters from the left view (a) and right view (b).

The number of views and the image resolution are two crucial problems for natural three-dimensional display. This is mainly because the resolution of the underlying light field display is shared between views. As mentioned above, this light field display approach is capable of producing smooth motion parallax, which might result in distortion between two exit-pupils of projectors since approximation is introduced. It is known as a distinct disadvantage compared with multi-view auto-stereoscopic displays. Thus, multi-projection style is verified to compensate for such problems and to potentially achieve large-sized all-around display.

In terms of the imaging performance, flat images projected onto such a large curvature of the arc screen might tend to decrease the fineness of the pixels resulting from being out of focus, and may cause image distortion. Image distortion will cause problems such as stripe images not splicing accurately. Appropriate and fast geometric calibration is the following task of our research. In addition, the occurrence of dark stripes is another noticeable problem, as could be seen from the experimental results. Instable performance of experimental mini-projectors and the aberration of system configuration are the main reasons for this. Meanwhile, the optimum match between the hardware parameters and the diffuser characteristic should be designed considering the splicing accuracy and images’ uniformity of luminance in further research.

4.

Conclusions

In conclusion, our research demonstrates an approach to achieve large-sized light field three-dimensional displays, which has potential to be used in future natural three-dimensional cinema. The multi-projection style is verified in the prototype equipment to provide large-sized, even 360-deg natural imaging performance with motion parallax and stereo parallax. The directional diffuser screen has great performance in diffusing the light within different ranges in both horizontal and vertical directions. Meanwhile, its characteristics in terms of the uniformity of the luminance and acceptable transmit-rate demonstrate commercial feasible to some degree. More refined and delicate 360-deg all-around reconstructed imaging performance can be expected with utilizing more mini-projectors or pico-projectors. Therefore, bright and large-sized floating three-dimensional scenes reconstructed by this approach will be observed at continuous horizontal viewing positions around the arc diffuser screen with less fatigue. This approach owns great engineering contribution on providing 360-deg displays with motion parallax in the near future.