1. Introduction

Optics is an exceptional playground for the development of analog computing, offering unique advantages over conventional approaches, such as broad bandwidth operation, low cross-talk, ultrafast speeds, and high power efficiency [1]. However, tailoring and optimizing all these features in a single device is an open challenge. Early attempts were focused on mimicking digital processing architectures to replace electronic and opto-electronic switches with all-optical equivalents [2]. The limitation of this strategy is that all-optical switching requires devices with high activation power and large volumes. On the other hand, traditional optical techniques based on lenses and filters, such as the 4f Fourier optics setup, can perform analog image processing at low power, but they are bulky, sensitive to perturbations, and not compatible with integrated systems. A paradigm shift in analog image processing has recently emerged thanks to the advancements of flat-optics photonic nanostructures [3]. Integro-differential operations have been demonstrated in a variety of nanophotonic configurations [4–14], including rather simple resonant systems, such as surface plasmons, Mie and grating resonances [15–19].

Until now, investigations on flat-optics image processing have mostly focused on the linear response of nanophotonic structures. Key ingredients to realize spatial filtering with linear flat-optics are resonances and dispersion engineering. Indeed, the image processing abilities of nanophotonic structures stem from the sensitivity to variations of the spatial frequency when these systems are excited near any kind of photonic resonance [19]. The main limitations of linear flat-optics are the limited-bandwidth operation, related to the presence of resonances, and the restricted complexity and variety of the mathematical operations that can be synthesized. The use of nonlinear optics for imaging devices has given important results using the nonlinear scattering of the object to be imaged [20,21]. However the use of nonlinear optics for image manipulations has been limited to the cases of nonlinear lenses that combine the function of an ultrathin planar lens with simultaneous frequency conversion [22–24], and to intensity-dependent imaging from metalenses based on the large Kerr nonlinearity of metallic quantum wells [25].

Here we bring nonlinearities into the field of engineered nonlocality for optical analog computing. We show that nonlinear flat optics offers a great potential to enable new functionalities in image processing and adds important degrees of freedom in the design of optical analog computing devices.

After outlining the general concept of nonlinear flat-optics image processing, we focus on a particularly illustrative system, consisting of a flat sheet with second-order optical nonlinear response. We show that even this simple system can lead to edge detection at the second-harmonic frequency via nonlinear spatial filtering of an image at the fundamental frequency. In the proposed nonlinear flat-optics solution, the non-resonant nature of the nonlinear interaction involved in this device allows edge detection for a broadband spectrum of fundamental frequencies, outperforming analog edge detection systems based on linear optics. We show that the edge-detection abilities, and the achievable types of nonlinear image processing, are ruled by the shape and symmetry of the susceptibility tensor, its spatial dispersion and the polarization state of the input light.

2. Nonlinear flat-optics image processing

The concept of image processing with nonlinear flat optics is illustrated in Fig. 1. In the linear regime (Fig. 1(a)), a flat-optics structure has a transfer function $\widehat {H}_{\ell m}^{(1)}(\mathbf {f})$ that depends on the linear susceptibility. Given a bi-dimensional input vector function $i_m^{\omega _0}(\mathbf {x})$, where $\mathbf {x} = (x,y)$ is the coordinate vector in the input plane, $m$ is one of the coordinates of the reference system, the spatial-frequency content of this input is given by the Fourier transform:

The input function represents an electric-field vector component of light scattered by some object at an angular frequency $\omega _0$. A linear flat-optics filter (Fig. 1(a)) provides the output:

 

Fig. 1. Linear (a) vs nonlinear (b) flat-optics imaging. (c) Analog image processing with nonlinear flat optics.

Download Full Size | PDF

In the nonlinear regime, in addition to the linear term, the flat-optics structure supports a nonlinear contribution that can be described by a Volterra filter (Fig. 1(b)); limiting ourselves to second-order optical nonlinear effects, and to sum-frequency processes, this filter is described by its $2^{\rm nd}$ order frequency-domain Volterra kernel $\widehat H_{\ell mn}^{(2)}(\mathbf {f}_1,\mathbf {f}_2)$. It receives as input two bi-dimensional vector functions, $i_m^{\omega _1}(\mathbf {x})$ at $\omega _1$ and $i_n^{\omega _2}(\mathbf {x})$ at $\omega _2$. The two functions represent electric-field components in the $m$ and $n$ directions of the reference system.

The nonlinear response is then obtained by applying the Volterra kernel to the inputs, providing the following output at $\omega _3=\omega _1+\omega _2$:

In the language of image processing, the Volterra kernel operates on two input images: in the non-degenerate case ($i_m^{\omega _1}(\mathbf {x}) \ne i_n^{\omega _2}(\mathbf {x})$) one of them may be thought of as the image to be processed and the other as a reference image; in the degenerate case, ($i_m^{\omega _1}(\mathbf {x})=i_n^{\omega _2}(\mathbf {x})$=$i^{\omega _0}(\mathbf {x})$), the filter receives at its input the square of the image to be processed. The nonlinear filter is described by $\hat H_{\ell mn}^{(2)}(\mathbf {f}_1,\mathbf {f}-\mathbf {f}_1)$, a function of four variables which is sampled by the spectrum of two inputs. Whenever $\hat H _{\ell mn}^{(2)}(\mathbf {f}_1,\mathbf {f}-\mathbf {f}_1)$ is not constant, the system behaves in a nonlocal way and since $\ell,m,n$ can be exploited in all possible permutations, the filter is completely described by a third-rank tensor with 27 elements.

3. Image processing with second-order nonlinear thin films

In the context of nonlinear optics, Eq. (3) describes the relation between the projections of electric fields at input ($i_{m,n}^{\omega _1, \omega _2}(\mathbf {x})=E_{m,n}^{\omega _1, \omega _2}(\mathbf {x})$) and induced polarization ($o_{\ell }^{\omega _3}(\mathbf {x})=P_\ell ^{(2) \, \omega _3}(\mathbf {x})$) in a sum-frequency experiment. Noteworthy, the projections of the electric fields at input on the directions specified by $m,n$ introduces an extra dependence of the Volterra kernel on the in-plane spatial frequencies. We consider an effective second-order nonlinearity determined by the material’s local permittivity (third-rank tensor $\chi _{\ell mn}^{(2)}$), by the propagation direction and polarization of the electric fields at input ($E^{\omega _1, \omega _2}(\mathbf {x})$) [28]. Remarkably, in the context of flat optics, one may further engineer the Volterra kernel by a proper ad-hoc design of meta-atoms, metasurfaces, multilayers and gratings. In these structures, several properties of the effective nonlinearity can be controlled: the tensor shape, the efficiency of the nonlinear interaction, the spatial-frequency and time-frequency dependence [29,30].

In the degenerate case, in which $i_m^{\omega _1}(\mathbf {x})$=$i_n^{\omega _2}(\mathbf {x})$=$i^{\omega _0}(\mathbf {x})$, we obtain the scenario illustrated in Fig. 1(c): light at $\omega _0$ impinges on a screen where the aperture is the image of interest; the light diffracted by the aperture feeds a nonlinear device performing second-order Volterra filtering, producing at the output the induced polarization $o^{2\omega _0}(\mathbf {x}) = P_\ell ^{(2) \, 2 \omega _0}(\mathbf {x})$, which is the source of the second-harmonic electric field imaged on a screen. As an illustrative example, we are assuming that the Volterra kernel produces the edges of the image. To illustrate the procedure adopted to compute the second-harmonic signal on the screen, we refer to a simplified situation as sketched in Fig. 2.

 

Fig. 2. (a) Setup for image processing with a nonlinear flat-optics element. P (polarizer); A (aperture); M (flat-optics element/metasurface); O (objective); L (lens); C (camera). (b) Action of a Volterra kernel based on the only $\chi ^{(2)}_{zzz}$: the two input vector functions are first projected into the direction of the two input spatial frequencies and then mixed into the nonlinear susceptibility block. (c) Amplitude of the electric field of an input image formed by light at $\omega _0$ impinging on a square aperture (A) with side length $a = 40 \lambda _0$. (d) Output image at $2 \omega _0$ collected by the camera (C) when the input light is polarized (P) linearly in the horizontal direction, as indicated by the horizontal red arrow. (e) Same as (d), for vertically-polarized input light. (f) Same as (d), for circularly-polarized input light.

Download Full Size | PDF

The setup shown in Fig. 2 is an ideal reference situation for which the nonlinear flat-optics element has zero thickness, and therefore is concentrated at $z=0$. The nonlinear response of the sheet can be then written as $\chi _{\ell mn}^{(2)} (x,y,z) = \chi _{\ell mn}^{(2)} (x,y) \delta (z)$. The second-harmonic light emitted by this nonlinear sheet can be derived analytically, and the general results can be found in Supplement 1. Here, for the sake of clarity, we further simplify the discussion considering a homogeneous (or homogenized [30]) sheet ($\chi _{\ell mn}^{(2)}$ does not depend on $x$ and $y$); we also assume that only one of the 27 elements of the second-order nonlinear tensor is different from zero ($\chi _{zzz}^{(2)}$), where $z$ is the optical axis of the system (see Fig. 2). This is a scenario that may occur, for example, at a metal surface [31,32], where the tensor element orthogonal to the surface generates most of the second-harmonic light [33]; or in multi-quantum well slabs with wells grown in the $z$ direction [34]; or in properly designed nanolaminates of centrosymmetric materials [35,36] and metasurfaces [30].

In Fig. 2(a), a plane wave at $\omega _0$ impinges on the nonlinear sheet with propagation vector parallel to $z$. The polarization direction vector $\hat e$ of the input light is controlled with a polarizer. The nonlinear sheet is then illuminated with a packet of plane waves, $\widehat {\mathbf {E}}^{\omega _0}_{0}(\mathbf {f})=\widehat {E}^{\omega _0}_{0}(\mathbf {f}) \; \hat {e}$, containing a continuum of directions, corresponding to all the in-plane spatial frequencies $\mathbf {f}= (f_x,f_y)$ excited by the image defined by the aperture. The nonlinear sheet induces a second-order nonlinear (surface) polarization that emits second-harmonic light at $2 \omega _0$. The efficiency and the radiation pattern of the generated second-harmonic is a function of the impinging plane wave spectrum at the fundamental frequency and is modulated by the properties of the susceptibility.

According to Eq. (3), the nonlinear sheet will induce a second-harmonic polarization density

The second-harmonic field produced by this source can be easily calculated enforcing phase matching at the nonlinear boundary and from the spectral Green’s function of a polarized sheet emitting in a homogeneous medium. This expression has been reported by Sipe [38] in terms of $p$- and $s$-polarization components of the emitted light. Omitting details that are discussed in Supplement 1, the result in $z=0^+$ reads:

To clarify the meaning of Eqs. (5), let us consider a scenario in which the field at $\omega _0$ impinging on the aperture is linearly polarized along the $x$ axis ($\hat e = \hat x$). Eqs. (5) then become:

It is remarkable that the simple nonlinear sheet with the only $\chi ^{(2)}_{zzz}$ element features a nonlinear nonlocal response with an output image that is proportional to derivatives of the input image along the direction specified by the polarization of the fundamental field illuminating the aperture. In particular, an $x$-polarized fundamental-frequency light can be used to detect edges parallel to the $y$ axis: indeed, the partial derivative along the $x$ direction (parallel to the input-light linear polarization) peaks along these edges, while the partial derivative along the $y$ direction (orthogonal to the input-light linear polarization) is vanishing, and therefore edges aligned with the $y$ axis will induce strong second-harmonic electric fields along the $x$ direction. In general, if the input light is linearly polarized and aligned along a direction $\hat {e}$, the output image will be bright along the edges of the input image that are orthogonal to $\hat {e}$. The general action of a Volterra kernel of type $H^{(2)}_{zzz}$ and its relation to the susceptibility of the flat-optics element is clarified in Fig. 2(b). In this system, the two inputs are a pair of spatial frequency components, $\mathbf {f}_1$ and $\mathbf {f}-\mathbf {f}_1$, originated from an arbitrary input image. The two inputs represent electric field amplitude distributions with angular frequencies $\omega _1$ and $\omega _2$ and polarization states $\hat e _1$ and $\hat e _2$, respectively. The two inputs can be written as $\widehat E _0^{\omega _1}(\mathbf {f}_1)$ and $\widehat E _0^{\omega _2}(\mathbf {f}-\mathbf {f}_1)$. The first operation of the Volterra kernel is a projection of each input along the direction of its in-plane spatial frequency, which provides the $z$ component of each input. The $z$ components can be written as $\widehat E _z^{\omega _1} = \widehat E _0^{\omega _1} \lambda _1 \hat e _1 \cdot \mathbf {f}_1$ and $\widehat E _z^{\omega _2} = \widehat E _0^{\omega _2} \lambda _2 \hat e _2 \cdot (\mathbf {f}-\mathbf {f}_1)$.

Next the $\chi ^{(2)}_{zzz}$ susceptibility mixes the two $z$ components of the inputs and gives the output polarization at $\omega _3=\omega _1+\omega _2$. In summary, the overall filtering action of the Volterra kernel is summarized by the second-order transfer function

Figures 2(c-f) summarize our findings for an input object given by a rectangular planar aperture at $z=0$. We assume that the rectangular shape of the aperture is given, at $z=0$, by the electric field amplitude distribution $E_0(x,y) = e^{-\frac {x^6+y^6}{a^6}}$, where the aperture size is $a = 40 \lambda _0$, shown in Fig. 2(c). When the input light is $x$-polarized, the output image shows only edges aligned with the $y$-axis – see Fig. 2(d); the dual effect is visible when the input light is $y$-polarized – see Fig. 2(e). All the edges, including the corners of the apertures, are well discernible when the input light is circularly polarized ($\hat {e} =(\hat {x} \pm j\hat {y})/\sqrt {2}$) – see Fig. 2(f).

Other operations are possible when a different nonlinear tensor element is selected as a Volterra kernel. Here, we only highlight two examples of kernels that act as differentiators for edge detection: (i) a homogeneous (or homogenized) film with a $\chi ^{(2)}_{xzz}$ response; (ii) a homogeneous (or homogenized) film with a $\chi ^{(2)}_{xyz}$ response. Using the procedure outlined in Supplement 1 for large apertures (small angular divergence), it is straightforward to prove that for $x$-polarized input light at $\omega _0$, the $x-$component of the $2 \omega _0$ field produced by the $\chi ^{(2)}_{xzz}$ kernel is:

Edge detection is also possible for more complex objects.

As an example, in Fig. 3, we consider the case in which the input function of the $\chi ^{(2)}_{zzz}$-kernel is the gray-scale image obtained from a picture of the Vittoria alata statue in Brescia, Italy. We assume that circularly-polarized light at $\omega _0$ is modulated in amplitude by this object, which we assume with a size of $900 \lambda _0 \times 900 \lambda _0$. The circularly-polarized input allows to detect edges with arbitrary orientation, as well as corners. The output image at $2 \omega _0$ shows the most prominent edges of the picture with great contrast. While the resolution of this edge detection is limited by diffraction since it is solely determined by the Green’s function at $2 \omega _0$, the contrast in edge detection is boosted both by the nonlinearity of the Volterra kernel and by the natural suppression of the input image provided by second-harmonic generation. More sophisticated designs with additional degrees of freedom, for example nonlinear metasurfaces, may be able to achieve enhanced contrast and resolution in a single flat-optics device. It is worth mentioning that while the derivative operations described in Eqs. (7), (9) and (10) are valid in the small-divergence approximation, edges from input objects that produce larger angular spectra can still be detected with good precision by carefully selecting the nonlinear kernel. In Supplement 1, a qualitative assessment of our edge detection system’s numerical aperture is provided, particularly in scenarios involving objects of sizes comparable to the wavelength and exhibiting extremely sharp edges.

 

Fig. 3. Edge detection with a nonlinear sheet having only the $\chi ^{(2)}_{zzz}$ tensor element. (a) The input image is a picture of the statue of Vittoria alata, one of the symbols of the city of Brescia. The input light is circularly polarized at a fundamental frequency $\omega _0$ (wavelength $\lambda _0 = 1 \mu m$). (b) The output image at the second harmonic evaluated as the intensity of the field. The brightest red and green in the color bars correspond to maxima of the field intensity at $\omega _0$ and $2 \omega _0$, respectively.

Download Full Size | PDF

4. Conclusion

We have shown that nonlocal nonlinear flat optics can realize analog image processing with previously not accessible functionalities. By exploring the simple scenario of a uniform $\chi ^{(2)}$ sheet, we have demonstrated edge detection operation with exciting potential. In particular, in our proposed nonlinear flat-optics solution, the non-resonant nature of the nonlinear interaction involved in image processing allows edge detection over a broadband spectrum with ultra–high contrast. Although we have reported a first step towards the merge of analog computing and nonlinear optics, our results indicate that Volterra kernels of nonlinear nonlocal flat optics can open new opportunities in applications such as image processing, item recognition for computer vision, high-contrast and high-resolution microscopy, implementation of nonlinear functions for analog deep learning.