1. INTRODUCTION

Multidimensional information sensing is one of the actively researched themes in both imaging science and industry. Multidimensional imaging has been applied to observe realistic scenes of remote locations, structures with invisible fields of view, such as microscopic and nanoscopic regions, and invisible distributions such as acoustic and electromagnetic waves at all wavelength bands. Acquisition of three-dimensional (3D) information is important, especially when a person and a machine perceive and observe 3D structures of materials and samples. Microscopy has enabled the visualization of fine structures and regions invisible to human eyes and has provided a wealth of physical information through the combination of physical theory and digital signal processing. Computational microscopy has realized 3D sensing with a single camera and without the need for any mechanical scanning by applying mathematics to physical models [1,2].

Digital holographic microscopy (DHM) [3–5] is becoming a promising computational 3D microscopy that is based on the combination of holography [6] and digital signal processing. Quantitative 3D information is recorded as a digital hologram and reconstructed with a computer by calculations based on diffraction optics. Computational 3D image sensing based on digital holography (DH) [5,7–9] has enabled 3D microscopy with a single-pixel detector [3], quantitative phase microscopy [5], real-time 3D measurement [10], tomographic imaging with computational optics [11], and multidimensional imaging with holographic multiplexing [12]. Holography and DH have also been applied to incoherent-light hologram sensing, called incoherent holography [13,14] and incoherent DH [15–17], respectively. Incoherent DH has led to the realization of quantitative holographic 3D microscopy for spatially incoherent light and has yielded applications to fluorescence 3D microscopy [15,17], single-shot 3D imaging with a light-emitting diode [18], full-color holographic image sensing with sunlight [19], and multispectral 3D sensing based on the principle of Fourier spectroscopy [20]. One of the merits of incoherent DH is the sensing of 3D information of spatially incoherent light without the need for a microlens array or multiple image sensors. Therefore, 3D image sensing with a compact setup and a single image sensor can be achieved. Incoherent DH is suitable for application to 3D microscopy with spatially incoherent light such as sunlight, thermal light, fluorescence including auto-fluorescence, spontaneous Raman scattering, and other nonlinear light and chemically activated light.

Multiwavelength information in incoherent DHM is useful for clarifying 3D distributions of various compositions separately. Color 3D sensing of stained cells is useful for analyzing both structures and composition distributions simultaneously. Holographic recording of spatially incoherent color light enables both 3D sensing and identification of self-luminous objects. Therefore, it is important to record both 3D spatial and wavelength information in incoherent DHM. Conventional multiwavelength incoherent DH and DHM systems require sequential changes of illumination-light wavelengths and/or wavelength filters [21], or a large number of recordings of holograms [20,22]. The technique to change recorded wavelengths sequentially is termed temporal division [21], and the technique to record a large number of wavelength-multiplexed holograms and to adopt Fourier spectroscopy is termed temporal frequency-division multiplexing [22]. In the former, spatially incoherent color self-interference holograms are recorded by changing the wavelengths of illumination light and/or wavelength filters sequentially. Sensing with high stability can be achieved by a single-path self-interference interferometer, but it is difficult to record a multicolor image simultaneously because wavelength information is temporally divided. When recording color fluorescence 3D information, color images cannot be obtained without mechanical movements to change wavelength filters. When the number of fluorescence wavelength bands recorded is $N$, recording of ${3}N$ holograms and changing the wavelengths of excitation light and/or wavelength filters are required. In the latter, a two-arm interferometer is set and a series of wavelength-multiplexed images with a carrier wave is recorded while moving optical elements that are set in an arm of the interferometer. From the series of recorded wavelength-multiplexed images, wavelength information on each pixel of the monochrome image sensor is separated by applying the principle of Fourier spectroscopy. This wavelength separation technique is called superheterodyne [23] and Doppler phase shifting [24] in holography. This technique is effective for conducting hyperspectral 3D imaging. However, recording of more than 250 holograms was required when 3D sensing of fluorescence particles with two compositions was conducted [25]. As another problem, the stability of the system using a two-arm interferometer should be improved. Michelson and Mach–Zehnder interferometers with mechanical scanning are adopted for Fourier spectroscopy [20,22–25]. The stability of the system against external noise sources such as vibration is low in a general case, although the problem has been successfully solved by adopting a single-path setup [26,27]. Owing to progress in research, another incoherent DH technique with the use of point spread function (PSF) differences between wavelengths has been proposed in the last five years [28,29]. Single-path interferometers without mechanical scanning have been successfully constructed and demonstrated, but a color-synthesized image of a multicolored object has not been obtained because of the appearance of severe crosstalk between wavelengths. This is because it is difficult to generate a large PSF difference with a Fresnel lens and a binary diffractive lens, resulting in a decrease in the wavelength resolution. Furthermore, diffractive lenses such as Fresnel and binary-pattern lenses generate multiple diffraction waves, and, therefore, undesired higher-order diffraction waves appear if a general diffractive lens is used [30].

Computational coherent superposition (CCS) [31–39] is one of the multidimensional multiplexed sensing schemes with a holographic multiplexing technique that is based on phase-shifting interferometry. In CCS, multidimensional information is multiplexed in recording and then separated in a computer utilizing the principle of phase-shifting interferometry. Wavelength-dependent phase shifts are introduced to the recorded holographic images, and wavelength information is retrieved from the wavelength-multiplexed holographic images by a phase-shifting interferometry technique, which is termed phase-shifting interferometry selectively extracting wavelength information [31–34]. The initially presented optical implementations of CCS resembled that of temporal frequency-division multiplexing, but only ${2} N + {1}$ wavelength-multiplexed holograms were required [31–33]. After that, an $N$-wavelength 3D image was reconstructed from ${2}N$ holograms by making the best use of the ${2}\pi$ phase ambiguity and two-step phase-shifting interferometry [34]. This interferometry has been applied to multicolor DHM with red-, green-, and blue-laser light sources [12,35], phase imaging with temporally incoherent light [36], multiwavelength-multiplexed holographic image sensing with a single-pixel detector [37], and color DHM with a temporally and spatially incoherent light source [38]. Then, single-shot color-multiplexed 3D sensing with CCS for multiple fluorescent particles with two compositions was achieved by adopting a Fresnel incoherent correlation holography (FINCH) system [39]. A FINCH system is composed of a single-path in-line interferometer with self-interference and has high stability. Furthermore, PSF improvement of an incoherent imaging system is expected in principle [40–43]. However, owing to the small number of pixels and low pixel density, in Ref. [39], there were no experimental verifications of either color fluorescence imaging (not fluorescent sensing of particles) or PSF improvement in color-multiplexed fluorescence imaging. Furthermore, the resolution of the system in Ref. [39] was low because spatial information was sacrificed owing to space-division multiplexing.

In this paper, we present color fluorescence imaging with a self-interference incoherent color DHM system utilizing CCS. We demonstrate color 3D microscopic sensing of self-luminous objects including particles whose diameters are less than 10 µm. Improvement of the PSF with a FINCH system in color-multiplexed fluorescence imaging is also presented. Then, we conduct an experiment to show its applicability to color fluorescence cell imaging with a commercially available monochrome image sensor.

2. COLOR DIGITAL HOLOGRAPHY WITH COMPUTATIONAL COHERENT SUPERPOSITION

CCS is a multidimension-multiplexed in-line holographic technique with phase-shifting interferometry and is categorized as multidimension-multiplexed phase-shifting interferometry. Multidimensional information is multiplexed on the time, space, temporal frequency, and spatial frequency domains, while multiplexing of multidimensional information has been conducted conventionally with spatial frequency-division multiplexing [44–47] and temporal frequency-division multiplexing [22–25]. Figure 1(a) illustrates the recording process of incoherent color DH with CCS. In Fig. 1(a), CCS is merged into a FINCH system with polarization multiplexing. The basic configuration of a CCS-FINCH system is similar to a FINCH system, but wavelength filters to separate multiwavelength object waves to be recorded are not required in CCS, while a conventional color FINCH system needs these filters. Two polarizers and a birefringent lens are set to generate a self-interference wavelength-multiplexed hologram from multiwavelength object waves. A phase modulator changes phases at multiple wavelengths in a wavelength-multiplexed hologram simultaneously. A monochrome image sensor records self-interference, wavelength-multiplexed, phase-shifted, and incoherent holograms by changing the phases at multiple wavelengths with a phase modulator. In the multiplexed recording of multiwavelength object waves, wavelength-dependent phase shifts ${\alpha _1}, \ldots , {\alpha _N}$ at central wavelengths of multiple wavelength bands ${\lambda _1}, \ldots , {\lambda _N}$ are added to wavelength-multiplexed holograms $I(x, y;{\alpha _1}, \ldots , {\alpha _N})$ with a phase modulator. This results in the phase encoding for object waves at respective wavelengths.

 

Fig. 1. Basic concept of DH with CCS of wavelengths. (a) Recording and (b) reconstruction processes.

Download Full Size | PDF

Figure 1(b) illustrates the reconstruction process of DH with CCS. Decoding is conducted by phase-shifting interferometry selectively extracting wavelength information [31–35]. In general phase-shifting interferometry, only the desired wave is extracted by calculating the superposition of light waves generated from the phase-shifted holograms. In the same manner, only the object wave at the desired wavelength is selectively extracted from the multiplexed holograms because other light waves are canceled by the interference between the waves because of the superposition of light waves. Interference means that the coherent superposition of light waves and the calculation of interference in a computer is termed computational coherent superposition (CCS). As described above, this concept of extracting only the desired wave from the holograms is based on phase-shifting interferometry. In a mathematical expression of CCS, where ${A_{\textit{ok}}}(x,y)$ and ${\phi _k}(x,y)$, respectively, are the amplitude and phase distributions at wavelength ${\lambda _k}, k = {1}, \ldots ,\;N$ is an integer, ${\alpha _{\textit{kl}}}$ is the $l$th phase shift at a wavelength ${\lambda _k}$, ${C_k}$ is a coefficient, $j$ is the imaginary unit, and ${I_{0th}}(x, y)$ is the sum of zeroth-order diffraction waves, the relationship between $I(x, y;{\alpha _{1 l}}, \ldots ,{\alpha _{\textit{kl}}}, \ldots , {\alpha _{\textit{Nl}}})$ and complex amplitude distributions at wavelengths ${U_k}(x, y) = {C_k}{A_{\textit{ok}}}(x, y)$ [${\cos} {\phi _k}(x, y) + j {\sin} {\phi _k}(x, y)$] is expressed as follows:

Then, complex amplitude distributions at multiple wavelengths are derived by

Equation (5) means that object waves at multiple wavelengths are selectively extracted from multiplexed holograms on the basis of phase-shifting interferometry. It is noted that phase shifts were added to respective wavelengths simultaneously. Arbitrary phase shifts can be set to ${\alpha _{1l}}, \ldots , {\alpha _{\textit{Nl}}}$ in Eq. (3). ${\alpha _{1l}}, \ldots , {\alpha _{\textit{Nl}}}$ have not to be set as an integral multiple of ${2}\pi$ as described in Ref. [35]. When ${\boldsymbol P}$ is a regular matrix, $N$-wavelength object waves are generally derived from ${2} N + {1}$ wavelength-multiplexed phase-shifted holograms [35]. A small condition number of ${\boldsymbol P}$ should be selected for a CCS algorithm, and regular phase shifts to set a small condition number should be designed. The relationship between the quality of the reconstructed image and regular phase shifts was investigated numerically in Ref. [35]. From the numerical results, it was clarified that a high-quality image with little noise was obtained by setting regular phase shifts that make the condition number of ${\boldsymbol P}$ small. This is owing to the numerical stability of a CCS algorithm and the finite signal-to-noise ratio and finite bit depth of the recorded digital images. Therefore, it is important to design regular phase shifts to set a small condition number in a CCS algorithm for high-quality imaging.

Figure 2 shows an optical implementation of self-interference incoherent color DHM with CCS. A polarization-based self-interference incoherent DH system is adopted to implement a single-path interferometer. A commercially available optical microscope can be exploited in the application to incoherent DHM. The magnified image that is obtained with an optical microscope is introduced to a self-interference single-path interferometer. A polarizer is placed to generate linear polarization from the incident light diffracted from the specimens. A wavelength-dependent polarization-sensitive phase modulator such as a polarization-sensitive variable phase retarder or a liquid crystal on silicon spatial light modulator (LCoS-SLM) is set to introduce wavelength-dependent phase shifts to the wave polarized along the horizontal direction. Two object waves whose curvature radii are different and polarization directions are orthogonal are generated by birefringent lenses. After passing through the phase modulator and birefringent lenses, the polarization directions of the two object waves are aligned by a polarizer placed in front of a monochrome image sensor. The phase modulator introduces wavelength-dependent phase shifts, and the monochrome image sensor records wavelength-multiplexed phase-shifted incoherent holograms. After recording, object waves at multiple wavelengths are selectively extracted by using Eqs. (1)–(5). Then, diffraction integrals are calculated for the extracted object waves at multiple wavelengths and a multiwavelength 3D image is reconstructed.

 

Fig. 2. Optical implementation of self-interference incoherent color DHM with CCS.

Download Full Size | PDF

3. COLOR FLUORESCENCE DHM WITH CCS

We present experimental demonstrations of color fluorescence imaging with improved resolution by using DHM with CCS in the case of $N = {2}$.

A. COLOR 3D SENSING OF MULTIPLE SELF-LUMINOUS OBJECTS

We constructed the self-interference color fluorescence DHM system shown in Fig. 2 to demonstrate color fluorescence 3D sensing. The system was composed of a commercially available optical microscope, a self-interference DH system, and the system based on CCS. We set metal complex molecules of europium and terbium as the red and green luminescent materials, respectively. The central wavelengths of these materials were 618 and 545 nm, respectively, and the full widths at half maximum of the luminescence were within 10 nm. These complex molecules were set between a cover glass and a slide glass and were sparsely distributed in a 3D space. Here, we adopted a FINCH system as a single-path self-interference in-line DH system and merged it into the CCS system to obtain wavelength information with a commercially available monochrome image sensor. We used an optical microscope (IX-73, Olympus) with an excitation light source (U-HGLGPS, Olympus) to obtain the magnified images of the fluorescent specimens. Excitation light at wavelengths of less than 500 nm was reflected by a dichroic mirror, and complex molecules were excited by the light. Green and red fluorescence light at continuous wavelength bands passed through the dichroic mirror and the other dichroic filter whereas the excitation light was filtered out by these dichroic optical elements. Fluorescence light irradiated from the specimens passed through the magnification system, and the magnified image of the specimens was introduced from the output port of the microscope to the self-interference DH system. The light wave was linearly polarized by a polarizer and optically Fourier transformed by a lens. In the Fresnel domain, an LCoS-SLM (X10468-01, Hamamatsu Photonics K.K.) was set to generate wavelength-dependent phase shifts sequentially. The regular phase shifts at 618 and 545 nm were ${24}\pi / {55}$ and $\pi /{2}$, respectively. After that, in the Fourier transform (FT) plane, a birefringent crystal lens was set to generate two object waves whose curvature radii and polarization directions were different. Then, another crystal lens was used to conduct an inverse FT optically on the two generated object waves. The focal lengths of the birefringent lenses for ordinary and extraordinary rays were 179 and 182 mm, respectively. These birefringent crystal lenses were fabricated by Sigma KOKI Co., Ltd. The magnification of the whole system was set to 13. After the color incoherent light wave generated from the complex molecules passed through these birefringent lenses, two light waves with orthogonal polarization directions and different curvature radii or wavefronts were generated. Then, the polarizer placed in front of the monochrome image sensor aligned the polarization directions of the waves to generate interference of these light waves. A scientific complementary metal–oxide semiconductor (sCMOS) monochrome image sensor (Zyla 4.2 plus, Andor Technology) recorded five wavelength-multiplexed phase-shifted holograms by changing the phases of one of the multiwavelength object waves with the LCoS-SLM. After the recording of holograms, object waves at two wavelength bands were selectively extracted from the five holograms, and focused images of the specimens at multiple wavelengths were obtained by calculating diffraction integrals. Figure 3 shows that color 3D sensing of fluorescent objects was successfully demonstrated experimentally. The two types of complex molecules were selectively identified by color sensing, and 3D information of the multiple specimens was simultaneously recorded and reconstructed. As shown in Figs. 3(e)–3(g), depth-resolved imaging of complex molecules was demonstrated, and the red specimen whose diameter was less than 10 µm was successfully reconstructed. Thus, color 3D sensing of multiple self-luminous objects was performed.

 

Fig. 3. Experimental results for multiple metal complex molecules of europium and terbium. (a) One of the holograms recorded with a monochrome CMOS image sensor and (b)–(d) color-synthesized images reconstructed from the holograms numerically focused on depths of (b) 0 µm, (c) 15 µm, and (d) 26 µm from the image sensor plane. (e)–(g) Magnified images inside the rectangles in (b)–(d), respectively. The scale bar of (e) is 10 µm. The brightness of the images was enhanced.

Download Full Size | PDF

B. PSF IMPROVEMENT FOR A COLOR FLUORESCENCE OBJECT

We conducted an additional experiment on color fluorescence imaging with CCS. We set a negative USAF1951 test target in the constructed system shown in Fig. 2 to investigate whether the PSF was improved in the in-plane direction. It was expected that the in-plane PSF of an incoherent DH system would be improved in a certain depth range by adopting a FINCH system. The magnification of the whole system was increased to 30 to observe the improvement of the PSF clearly. To obtain a fluorescence image of the test target, we placed clusters of metal complex molecules of europium and terbium on the test target. As a result, a 2D yellow-fluorescence image was generated, and the color fluorescence imaging ability of CCS was investigated. Figure 4 shows the experimental results. For comparison, photographs of the specimen were obtained using color and monochrome CMOS image sensors, as shown in Figs. 4(a) and 4(b), respectively. The color photograph was taken using a color CMOS image sensor with a Bayer color-filter array and an optical microscope, and the monochrome photograph was taken with a monochrome CMOS image sensor and the system shown in Fig. 2 after setting the transmission axis of the polarizer in front of the image sensor perpendicular to the horizontal axis. By self-interference DHM with CCS, not only the focused image but also wavelength information was retrieved from five wavelength-multiplexed phase-shifted holograms of the specimen, as shown in Fig. 4(c). Figures 4(d) and 4(e) show the plots of group 9, lines 1–3 of Figs. 4(b) and 4(c), respectively. The graphs indicate that the visibility of the image was improved with the constructed CCS-FINCH system. This was because the PSF in the high-spatial-frequency components was improved by FINCH even though an incoherent imaging system was constructed [40–43]. It is noted that PSF improvement is limited in a certain depth range, as described in Refs. [41,42]. Thus, by constructing a CCS-FINCH system, color fluorescence imaging with an improved PSF was successfully performed.

 

Fig. 4. Experimental results for a USAF1951 test target with fluorescence light. (a) Photograph of the specimen taken using a color CMOS image sensor with a Bayer color-filter array and an optical microscope. (b) Photograph taken with a monochrome CMOS image sensor and the system shown in Fig. 2. (c) Color-synthesized image reconstructed from five wavelength-multiplexed holograms. (d), (e) Plots for the border line pairs in group 9 of (b) and (c), respectively.

Download Full Size | PDF

C. COLOR FLUORESCENCE CELL IMAGING

In biology, multicolor information is essential for the mapping of multiple molecular compositions and for analyzing cells in detail. However, multicolor fluorescence imaging has confronted a difficulty to simultaneously obtain color images without a time interval. Live biological specimens can move while changing filter sets for sequential color acquisition, resulting in misregistration of the resulting color images. Simultaneous acquisition of multicolor fluorescence light is important, particularly when applying to microscopy with ultimately weak light such as multiphoton-excitation fluorescence microscopic imaging [48,49]. Our CCS-FINCH system has the potential to quickly obtain multicolor images simultaneously, suggesting an advantage in multicolor fluorescence cell imaging. To apply CCS-FINCH to biological samples, however, the current CCS-FINCH system needs to demonstrate its functionality with biological specimens that have a limited photon budget and remarkable molecular complexity. For this reason, we investigated experimentally the applicability of the constructed DHM system to color fluorescence cell imaging. HeLa cells grown on a 0.17 mm coverslip (Matsunami) were fixed with a solution containing 3.7% formaldehyde, and then indirectly immunostained, first by anti-tubulin monoclonal antibody (TAT1 [50]) at a dilution of 1/50 and second by anti-mouse IgG with Alexa Fluor 555 (Thermo Fisher Scientific) at a dilution of 1/200, and counterstained with 0.5 µg/ml 4^′,6-diamidino-2-phenylindole (DAPI). The specimen was mounted in Prolong Gold (Thermo Fisher Scientific) and immediately sealed with nail polish. A water-immersion microscope objective lens with a magnification of 40 and a numerical aperture of 1.15 (UAPON40XW340, Olympus) was used for imaging. The magnification of the whole optical system was set to 30. Multiple multiband-pass filters (DA/FI/TR-A-OFF, OPTO-LINE, Inc.) were inserted to generate the desired wavelength bands of excitation light and introduce the generated color fluorescence light to the self-interference DH system. By using the filters, excitation light at the wavelength bands of 382–393, 466–490, and 546–565 nm illuminated the HeLa cells, and fluorescence light only at the wavelength bands of 414–450 and 584–645 nm passed through the output port of the optical microscope. The regular phase shifts for the fluorescence light of DAPI and Alexa Fluor 555 were ${2}\pi / {3}$ and $\pi /{2}$, respectively. Figure 5 shows the experimental results for color fluorescence cell imaging with CCS. The areas in Figs. 5(a)–5(d) were ${239}\;\unicode{x00B5}{\rm m} \times {270}\;\unicode{x00B5}{\rm m}$. Color information of fluorescence-stained cells was successfully retrieved from wavelength-multiplexed phase-shifted monochrome holograms, as shown in Figs. 5(a)–5(d). Then, another area was recorded as wavelength-multiplexed phase-shifted holograms and its reconstructed images were reconstructed, as shown in Figs. 5(e)–5(g). As can be seen from Figs. 5(e)–5(g), a cell cytoskeleton was resolved by the constructed CCS-FINCH system. Thus, the capability of DH with CCS for visualizing cell structures was experimentally demonstrated.

 

Fig. 5. Experimental results from fluorescence-stained cells. (a) One of the holograms recorded with a monochrome CMOS image sensor. (b), (c) Images reconstructed from fluorescence light of DAPI and Alexa Fluor 555, respectively. (d) Color-merged image of (b) and (c). (e), (f) Reconstructed images in another area, showing the fluorescence of DAPI and Alexa Fluor 555, respectively. (g) Color-merged image of (e) and (f). The brightness of (e)–(g) is enhanced.

Download Full Size | PDF

4. CONCLUSION

We have presented color fluorescence imaging using CCS and a FINCH system. Color holographic 3D information of self-luminous objects with improved resolution was successfully obtained with a monochrome image sensor. Color fluorescence imaging with an improved PSF was also demonstrated successfully by merging CCS into a FINCH system. Furthermore, color imaging of fluorescence-stained cells was experimentally performed. Incoherent DH with CCS does not require a digital hologram of laser light, and, therefore, the presented technique is suitable for applications to color fluorescence 3D microscopy, color 3D sensing of nonlinear light, color 3D microscopy with light-emitting diodes, and other holographic color 3D sensing applications. We believe that the presented microscopy system will contribute to observations of phenomena in physics and life sciences.