1.

Introduction

Imaging is inherently a two-dimensional (2D) process; however, many biological samples of interest are three-dimensional (3D). This presents a challenge for traditional microscopy when fast, dynamic processes such as neuronal activity are the object of study. With these methods, volumetric imaging rates are hampered by the relatively slow movement of high inertia focusing mechanisms.^1–3 Recent technological advances have improved the speed of these focal plane shifts through rapid movement of the objective using piezo drives, “remotely” focusing the microscope with low-inertia devices or completely inertia-less defocusing methods.^4,5 Still, volumetric rates over relevant tissue volumes remain inadequate for most fast dynamical processes.⁵ Together with the need for imaging large numbers of neurons over large fields of view (FOV), dynamic volumetric imaging is a challenge that is not fully met using these “serial” acquisition methods.

To further advance volumetric imaging rates, other recent light sheet^6–8 and light field-based methods leverage the incredible advances in imaging sensor technology for large-scale parallelism in the spatial domain and fast acquisition speeds. Light field microscopy (LFM)^9,10 is one such spatially parallelized volumetric imaging technique that enables volumetric imaging of fluorescent and non-fluorescent samples. By measuring the light pattern at the focal plane of a lenslet array, both spatial and angular information about the object are gathered, and 3D information can be reconstructed.¹⁰ However, because the quality of the volumetric reconstruction depends on the number of lenslets, there is a fundamental trade-off between resolution and depth: As fewer camera pixels are capturing light from the object at a specific angular position when the lenslets are small, the spatial resolution of the LFM image is compromised relative to conventional 2D imaging and is not uniform along all depths.¹⁰ Although benefiting from high-resolution camera sensors, LFM does not optimally use the high pixel counts of modern sCMOS sensors, and the time-consuming and computation-intensive reconstruction process^10–12 severely underutilizes real-time visualization capabilities theoretically afforded by high frame rate cameras.

Multifocus microscopy (MFM) techniques offer a possible solution to these primary limitations of LFM. These methods reassign depth information from multiple focal planes into a tiled 2D array that can be imaged simultaneously on a camera.¹³ Sub-cellular resolution can be easily maintained even over large volumes by leveraging modern, large-format imaging sensors, in a more efficient manner than LFM, with speeds limited only by the camera frame rate. These techniques have been used for decades in various forms, including using beam-splitting strategies with single^14,15 or multiple cameras,^15,16 multiple imaging lenses of varied focal lengths,¹⁷ and diffractive optical elements (DOEs).^18–21 These methods also have the advantage of directly acquiring adjacent focal planes without the need for computational reconstruction, enabling real-time volumetric visualization. The DOE approach in particular is promising due to its simplicity, scalability to many simultaneous focal planes, and long history of use.¹³ Nevertheless, these methods are typically used to image tiny volumes for particle tracking applications.^18–21 Although they have been demonstrated for fluorescence functional neuroimaging¹⁹ of a small volume ($40\times 40\times 16\mu \mathrm{m}$), they have not found widespread use in large FOV volumetric fluorescence imaging typically used in systems neuroscience.⁵

Here, we present a powerful adaptation of the multifocal DOE approach for real-time acquisition of a large number of focal planes simultaneously, with diffraction-limited spatial resolution and importantly, over large (mm scale) volumetric FOVs. Similar to previous work,¹⁸ our technique achromatically diverts multiple focal plane images into different offset positions on the camera sensor, so they can be acquired in a single, fast (up to 10 ms) camera exposure [Fig. 1(a)]. The uniqueness of this work is its extension to low magnification and low NA configurations, enabling the extension of MFM to the rapid, 3D functional imaging of neural activity across FOVs containing hundreds of neurons. The novel versatility in our implementation also enables other important applications such as volumetric visualization of single blood cell flow in Zebrafish larvae and one-shot rendering of microscopic 3D objects.

Fig. 1

MFG-based multifocal imaging. (a) Imaging concept. 3D imaging is achieved by transforming the different focal plane images into a mosaic of images on the camera sensor. (b) Schematic of the optical design. The multiple depths optical imaging setup is based on three customized optical elements. The first element (MFG) is a diffraction grating that is responsible for splitting the microscope image into $N\times N$ different images (diffraction orders) while adding to each order a phase curvature to refocus different focal depths. The CCG and prism are responsible for correcting the chromatic dispersion caused by MFG. (c) Zoomed view of the different customized elements described in panel (b). Both CCG and prism are made of nine different panels, each corresponding to a different direction of the MFG diffraction light. The central panel is a plain optical window. (d) The mosaic of images on the camera sensor for the case of nine focal planes, with their associated diffraction orders from the grating. (e) Simulation validation of the nine PSFs obtained by a single point source at the focal plane. (f) Experimental validation of the method for the $10\times$ microscope objective. The two extreme countercases of defocus are shown. (g) The stage shift distance versus each sub-image best focus. The experimental depth separation (graph slope, $16.46\mu \mathrm{m}$) corresponds well to the original design ($17\mu \mathrm{m}$) and is well within the error bars range (the objective depth of field). (h) Zoom of the USAF resolution target verifying that the spatial resolution was not reduced, allowing for a resolution better than $2\mu \mathrm{m}/\mathrm{cycle}$.

2.

Materials and Methods

Similar to earlier work in this field, our multifocal system is added to a Nikon TE2000-U inverted microscope and consists of three customized elements that form $N\times N$ different images on the camera sensor (Andor Zyla 4.2 sCMOS), each of a different depth in the sample. The first element, a multifocal grating (MFG), is a 2D diffractive grating that has two roles: splitting the light into $N\times N$ different orders and adding an inverse phase curvature to each order to compensate for the defocus of each object’s depth. The splitting of light into $N^2$ directions is achieved with a binary phase grating that was designed by a pixel flip phase retrieval algorithm adapted from previous work¹⁸ for an object of $N\times N$ point objects. To obtain large FOV images at each depth, we designed our element to have nine different sub-images ($N=3$) with a light efficiency of 60% after optimizing for multifocal performance factors such as the focal plane spacing and uniform brightness across sub-images [Fig. 1(b)]. This gives three orders of diffraction in both the $x$ and $y$ axes ($m_x$, $m_y=0$, $\pm 1$). The inverse defocus phase compensation to each order of diffraction was done by a geometrical distortion at each axis of the grating¹³ according to

MFGs serve as an excellent solution for the case of multiple-depth imaging of monochromatic light. However, for polychromatic light, such as fluorescence emission from common fluorophores, the linear dependence of the wavelength on the diffraction angle in a diffraction grating²² will form chromatic dispersion that distorts the image. To overcome this problem, we designed a chromatic correction grating (CCG) consisting of a customized blazed grating and prism. The MFG and CCG were custom-manufactured by Holo/Or. The blazed grating is made of $N\times N$ different panels, each designed to compensate for the chromatic dispersion of an individual MFG order of diffraction. This is done by deflecting the angle of diffraction of each MFG order (for the central wavelength) by its inverse angle. Correcting the chromatic dispersion of the MFG inverts the angular separation of the focal planes, leading to overlapping images at the camera sensor. To compensate for this effect, a customized prism was added to deflect each order back to the inverse angular direction that was formed by the MFG.¹⁸ Both the chromatic correction grating and the prism are made of $N^2$ panels, each oriented according to the expected diffraction angle from the MFG (the central panel that corresponds to light that was not diffracted by the MFG is a plane window). Figure 1(c) shows the drawing of each of the customized elements.

Unlike previous multifocal microscopes that were aimed at high magnification via a specific high-power objective, our goal was to image large numbers of depths at large FOVs with the flexibility to use different objectives with various magnifications. The condition of using different objectives with the same setup is found by calculating the MFG size:

3.

Results

We first validated the physical characteristics of our method with a series of optical simulations. We verified the design of our MFG by simulating nine different point sources at the depths that correspond to the desired depth corrections. The obtained point spread function at each depth demonstrates the depth correction [Fig. 1(e)]. The validation of the chromatic correction as well as the agreement with the camera sensor dimensions (to avoid overlapping between the sub-images) was done by ray transfer matrix analysis simulation, which gave us both the transverse and angular positions of each ray at each element position (see Fig. S1 in the Supplementary Material for the chromatic correction validation). To experimentally validate our multiple-depth ability, we focused each of the sub-images using a 2D USAF resolution target as an object and gradually taking the microscope out of focus [Fig. 1(f)]. By measuring the vertical shift of the microscope stage from its focus plane, we can measure the focal depth separation between the planes of each sub-image. This analysis shows excellent agreement with our design [Fig. 1(g)], as well as the method’s excellent spatial resolution [Fig. 1(h)].

To test our system’s ability to visualize large-scale neural networks, we first imaged 3D neural “Optonet” cultures,^23–25 an optically accessible and sensitive bioengineered cortical neural network exhibiting spontaneous neural activity. Imaging of these samples was done using a 10× objective at a 5 Hz frame rate. To further improve the in-plane resolution, we deconvolved each of the frames with the compatible 2D PSF, which was measured in advance with a sub-diffraction-limit bead.²⁶ The application of the multiple-depth technique to the Optonet cultures enables the observation of different cells over the entire volume while preserving effective resolution, further allowing for the visualization of axons between cells at different depths [Fig. 2(a)]. We also measured the spontaneous neural activity of an Optonet culture that expresses the genetically encoded Ca²⁺ indicators (GECI) GCaMP6m [Fig. 2(b)], demonstrating that sub-cellularly resolved, independent neural activity can be observed throughout a large volume. Figure 2(c) shows the spontaneous activity of two cells in the volume, having both synchronized and independent ${\mathrm{Ca}}^{2+}$ traces.

Fig. 2

Experimental results for the neural Optonet culture. (a) The different depths images of the Optonet as taken by a single image of the multiple-depth microscope. The different cells in focus at different layers demonstrate the optical sectioning of this method. Notice the axon in panel 6 that connects two cells at different depths (denoted by an arrow). $\mathrm{Scale}\mathrm{bar}=50\mu \mathrm{m}$. (b) An example of two different depth images that were captured simultaneously for measurement of fast neural activity. (c) Spontaneous neural activity of two different cells [denoted by corresponding colors in panel (b)]. The results show correlated and independent activity of the cells. $\mathrm{Scale}\mathrm{bar}=30\mu \mathrm{m}$.

We next used our multifocal strategy in Tg(HuC:GCaMP5G)²⁷ Zebrafish larvae expressing GCaMP5G pan-neuronally, in which network hyper-activity was induced by the application of 0.25 mM 4-AP, a potassium channel inhibitor. The larval zebrafish is a popular model for linking large-scale neural activity to simple behaviors. To better differentiate the neural activity from the background image, we subtracted the average image from each of the frames and visualized the ${\mathrm{Ca}}^{2+}$ events by pseudocolor [Fig. 3(a) and Video 1]. To further reduce the crosstalk effect of the scattered light across depths, we weakly thresholded the activity pattern (passing 95% of the detected signal at each depth, after first applying Gaussian spatial smoothing). The obtained neuronal activity pattern demonstrates peaks at different sub-regions in different planes as well as the time progress of the network along the volume at a resolution of single-cell activations [Figs. 3(a)–3(d) and Video 1]. Figure 3(b) shows the activity of two different areas inside the volume, demonstrating both synchronized and non-synchronized activity in those areas.

Fig. 3

Simultaneous multiple depths calcium imaging in vivo. (a) Demonstration of the neural activity imaging at different depths of the larval zebrafish. Spatially and temporally distributed neural activity is shown across the midbrain and part of the forebrain with single-cell resolution. Scale bar: $100\mu \mathrm{m}$, $\mathrm{\Delta }z=17\mu \mathrm{m}$. (b) Neural activity of two single subregions [denoted by colored rectangles in panel (b)], demonstrating both independent and correlated neural activity. Inset, Image showing putative single-cell-resolved functional activity. Scale bar: $50\mu \mathrm{m}$.

In addition to its application in multiple-depth visualizations of neural networks and their activity, the fast acquisition rate of this technique, limited only by the camera’s frame rate, can potentially be used in other interesting applications, such as visualizing 3D blood flow. To test this possibility, we imaged an agarose-embedded Zebrafish larva’s tail at 100 frames per second (camera readout limit), using brightfield imaging with the $4\times$ objective. We confirmed that this enabled us to discern the flow of single blood cells at different depths, including the observation of high-velocity flow rates in veins as well as slower ones in capillaries (Video 2) at almost 1 mm depth. Finally, we also used our technique to surface-render a continuous 3D outline of a mesoscopic object from the nine independent focal depths. This was done by removing the out-of-focus information of each depth by a semi-automatic image processing algorithm that seeks sharp edges to estimate the focused region. For instance, we demonstrate this for an ant head [Figs. 4(a) and 4(b), see also Video 3]. The 3D rendering was done using a commercial software package (Imaris, Oxford Instruments, UK).

Fig. 4

3D reconstruction by a single multiple-depth image. (a) An ant head at nine different focal planes, taken by a single camera exposure. (b) Zoomed view of four different sub-images showing how different parts of the ant head are in focus at different depths. Scale bar, $75\mu \mathrm{m}$. (c) 3D reconstruction of the head after removal of the out-of-focus data from (b).

4.

Discussion and Conclusion

In this study, we demonstrated a technique that enables rapid imaging of large volumes at microscopic resolution, using both fluorescence and bright-field modalities. Inspired by previous work on diffractive multi-focus microscopes,^18–21 the main advance in this work is the transition to the low magnification/large FOV domain, which now enables covering large areas (millimeter-scale) in each sub-image. This development moves multifocal techniques away from the small FOV, high NA, particle tracking regime into the realm of biological systems and large-scale functional neurophotonics, both of which benefit from volumetric imaging. This application domain requires both high spatial resolution and a large FOV, in addition to excellent temporal resolution. As an additional improvement, we also show the versatility of this approach to different microscope objectives by setting a selection rule for the objectives and demonstrating our approach with two different magnifications. These capabilities are becoming increasingly desirable in neuroscience to take advantage of next-generation GECIs and voltage indicators,^28,29 which require extremely high volumetric rates to accurately sample.

In comparison with LFM, which also enables scan-less volumetric imaging, at the price of a lower spatial resolution, the trade-off in our approach is the limited field of view rather than the resolution. However, this limitation is generally much easier to overcome as it only requires a larger imaging sensor. In addition, this multiple-depth technique has other distinct advantages: the extraction of the volume information is immediate, all depths have the same resolution (which is the microscope resolution), there is no need for prior calibration of the setup, and it is simple to use. At the theoretical level, one of the major advantages of our method over the microscopic light field is that the point spread function is space invariant, as in most imaging techniques. This allows us to use conventional 3D deconvolution algorithms to reduce the out-of-focus crosstalk between the different planes. In future studies, we intend to investigate different algorithms for this goal.³⁰ Although our method in its current implementation is most suitable for transparent or semi-transparent samples, in many cases, the effect of out-of-focus crosstalk can be reduced computationally (as described) or optically by designing a setup with depth separation that is much larger than the depth of field of the microscope objective. Moreover, further development of near-infrared functional indicators will further mitigate these issues,³¹ especially in shallow neocortical structures.

In addition to the important application of this approach in the field of neuroscience,²⁹ the ability to measure large volumes rapidly without degradation of resolution has promising applications in the study of fluidics (see Video 2). It can also be advantageous in essentially all cases in which scan-based microscopic techniques are not fast enough to capture a dynamic process in all three dimensions. Moreover, the simplicity of this method, which enables scanless 3D microscopic imaging at a single camera exposure, may promote it as a favorable alternative even in cases in which high temporal resolution is not essential (e.g., Fig. 4).

5.

Appendix: Supplemental Videos