ThreeDimensional Automatic Target Recognition System Based on Optical Integral Imaging Reconstruction
 Author: Lee MinChul, Inoue Kotaro, Cho Myungjin
 Publish: Journal of information and communication convergence engineering Volume 14, Issue1, p51~56, 31 March 2016

ABSTRACT
In this paper, we present a threedimensional (3D) automatic target recognition system based on optical integral imaging reconstruction. In integral imaging, elemental images of the reference and target 3D objects are obtained through a lenslet array or a camera array. Then, reconstructed 3D images at various reconstruction depths can be optically generated on the output plane by backprojecting these elemental images onto a display panel. 3D automatic target recognition can be implemented using computational integral imaging reconstruction and digital nonlinear correlation filters. However, these methods require nontrivial computation time for reconstruction and recognition. Instead, we implement 3D automatic target recognition using optical crosscorrelation between the reconstructed 3D reference and target images at the same reconstruction depth. Our method depends on an alloptical structure to realize a realtime 3D automatic target recognition system. In addition, we use a nonlinear correlation filter to improve recognition performance. To prove our proposed method, we carry out the optical experiments and report recognition results.

KEYWORD
Integral imaging , Nonlinear correlation , Threedimensional automatic target recognition

I. INTRODUCTION
Object recognition’s many applications include its use in autonomous vehicles, artificial intelligence, and defense. Within this field, threedimensional (3D) automatic target recognition using integral imaging techniques has been an important topic of research [19].
Integral imaging, first proposed by G. Lippmann in 1908 [10], employees lenslet or camera arrays to capture and display 3D images. Each lenslet provides a different perspective on the subject, generating an elemental image for capture. The resulting array of 2D images, illuminated from behind and projected through another lenslet array, displays a 3D reconstruction. The technique provides full parallax, continuous viewing points, and relatively high depth resolution without the need for special glasses [1015], all desirable characteristics.
The mechanism for image reconstruction, itself, can be used to implement forms of auto target recognition. For example, in [4], the authors propose a method for locating the position of target objects in 3D space. They use the fact that computational integral imaging reconstruction (CIIR) can output a range of 3D reconstructions, each at a different reconstruction depth [14]. Then, computers apply advanced correlation filters to the range of reconstructed images created by CIIR and complete the automatic target recognition task.
In practice, such methods are difficult to apply in realtime applications. The computation time and resources required for 3D reconstruction limit its usefulness [6]. To address the limitation imposed by the computational cost of 3D reconstruction, an alloptical structure for 3D automatic target recognition offers an alternative.
This paper proposes such an alloptical integral imaging approach. In our method, we optically reconstruct 3D reference and target images. We then perform automatic target recognition, applying a nonlinear correlation filter to the reference and target images. To assess this proposed method, we discuss optical experiments performed and present recognition results obtained in these experiments.
II. REVIEW OF CIIRTYPE 3D OBJECT CORRELATION METHOD
Fig. 1 illustrates the principle of the conventional CIIRtype 3D object correlation method. The process is separated into two main stages: pickup and recognition. In the pickup stage, we assume that
z_{r} , the distance between the reference object and the lenslet array, is known, and that the distance between the target and the lenslet array,z_{o} , is unknown or arbitrary. For both reference and target objects, elemental images created by the lenslet array are recorded by the pickup device. In the recognition stage, a CIIR technique reconstructs images of the reference object at known reconstruction depths and the image of the target object at a range of reconstruction depths [59]. CIIR can be implemented by computational simulation of ray optics.Fig. 1(a) depicts the reference object method, and Fig. 1(b) the steps in the target object method. Fig. 1(b) also depicts the automatic target recognition step, implemented using crosscorrelation between a single, reference 3D image and the various target 3D images.
When recognition is implemented with this method, a high correlation peak can be obtained only at
z =z_{r} . Thus, this method can extract the depth information of a reconstructed target image that matches the depth of the reconstructed reference image.While the method works, CIIR is very slow, computationally, and requires substantial computing resources. Therefore, if realtime processing is required, an alternative method may be superior.
III. PROPOSED 3D CORRELATION METHOD BASED ON OPTICAL INTEGRAL IMAGING RECONSTRUCTION
Fig. 2 illustrates our proposed 3D object correlation method. In the conventional CIIRtype 3D object correlation method presented above, 3D images are reconstructed by computer simulation. In our proposed method, an optical display system reconstructs the 3D images. The alloptical structure allows us to implement automatic target recognition in realtime.
Our system has three main parts: pickup, display, and recognition. For pickup, let us assume that the reference object has known position
r (x_{r}, y_{r}, z_{r} ) at a distance ofz_{r} from the lenslet array and that the target object att (x_{o}, y_{o}, z_{o} ) stands at an unknown (or arbitrary) distance,z_{o} , from the lenslet array. Elemental images for reference and target objects are obtained by the first image sensor (Sensor 1) by way of the lenslet array.In the display part of the process, 3D image reconstructtion begins when these elemental images are reproduced on a liquid crystal display (LCD). Light illuminating the LCD passes through the lenslet array, generating a 3D optical image for capture by the second image sensor (Sensor 2) at the speed of light. This design choice produces a system that is much faster than those used to implement the conventional CIIRtype correlation method. The 3D, opticallycreated images have relatively high 3D resolution (lateral and depth) since all elemental images are superimposed by optical magnification.
For the reference object, Sensor 2 records its reconstructed 3D image at the known distance
z_{r} . By locating this sensor at the focal plane, we can obtain a highquality image. We call the reconstructed 3D image of the reference object at knownz_{r} planeR (x_{r}, y_{r}, z_{r} ). This image is stored in the computer for use as a reference image.To record the reconstructed image of the target object, Sensor 2 is positioned at
z_{r} , the identical position from which the reference object was recorded, as shown in Fig. 2(a). Recall that, in the pickup stage, the target object was located at thez_{o} plane. Images reconstructed at the other various reconstruction depths are noticeably outoffocus or blurred. Only whenz_{o} =z_{r} is the best, clearly reconstructed 3D image obtained. This is a main feature of our 3D object correlation method. LetT (x_{o}, y_{o}, z_{r} ) be the reconstructed 3D image of the target object as captured by Sensor 2 atz_{r} .Recognition proceeds by means of a correlation process comparing
T (x_{o}, y_{o}, z_{r} ) andR (x_{r}, y_{r}, z_{r} ) , the 3D target and reference images reconstructed at a distance ofz_{r} from the lenslet array. Correlation results are obtained by means of the followingk thlaw nonlinear correlation filterwhere are the Fourier transform and phase of the reconstructed reference image,
R , at the distancez_{r} , are the Fourier transform and phase of the reconstructed target image,T , at the distancez_{r} , andk is the nonlinearity parameter for the correlation filter. Whenk = 0, Eq. (1) becomes a phaseonly filter (POF). Whenk = 1, Eq. (1) becomes a linear correlation filter.IV. EXPERIMENTAL RESULTS
We carried out optical experiments to assess our proposed 3D object recognition method. A ‘die’, shown in Fig. 3(a), was used as the reference and target object in this experiment. Its size was 2 cm × 2 cm × 2 cm (W × H × D). When serving as the reference object, the ‘die’ was positioned at (
x_{r}, y_{r}, z_{r} ) = (0 cm, 0 cm, 6 cm). The experimental setup is depicted in Fig. 3(b). Here, we used a 30 × 30 array of lenslets. Each lenslet measured 1.08 mm × 1.08 mm and was square in shape. The lenslet focal length was 3 mm. Total resolution of the elemental images was 900 × 900 pixels (H × V). Each elemental image had 30 × 30 pixels as the pixel pitch of the LCD panel shown in Fig. 3(c) was 36 μm.A set of recorded elemental images is shown in Fig. 4(a). We used a paper screen and a digital camera to form Sensor 2. The paper screen was placed
z = 6 cm from the lenslet array. The digital camera captured images from the paper screen. An example of the images captured in this manner is shown in Fig. 4(b). This reconstructed 3D reference image was taken atz = 6 cm.When used as a target object, the ‘die’ was shifted along the zaxis or rotated by an arbitrary angle on the yaxis. Elemental images for each case were recorded using the optical pickup apparatus shown in Fig. 3(b). Then, with a set of elemental images for each case, reconstructed 3D images at various reconstruction depths could be obtained using the optical display system shown in Fig. 3(c). Examples of reconstructed 3D images at different reconstruction depths are shown in Fig. 5(a). The figure clearly shows that the 3D image reconstructed at
z = 6 cm exhibits the sharpest focus. Fig. 5(b) provides examples of the impact of target rotation angle on the reconstructed 3D image. The figures demonstrate that reconstructed 3D images of target objects depend on object distance and rotation angle.Finally, we present the crosscorrelation between the reference image and the reconstructed 3D images of the target object at various depths and rotation angles. We computed correlation using the
k thlaw nonlinear correlation filter described by Eq. (1). Correlation results, withk = 0.5,k = 0.7, andk = 1, are shown in Fig. 6. The highest peaks (Fig. 6(a), (c), and (e)) correspond toz = 6 cm. In this case, the reconstruction plane matches the zplane in which the target object was located in the pickup phase. Lower peaks were observed when the reconstruction plane and the zplane during pickup did not match. Fig. 6(b), (d), and (f) display correlation results forz = 10 cm.We repeated the analysis for the other reconstructions of the target object image shown in Fig. 5. Applying the correlation test to the reference and each reconstructed target image produced the results shown in Fig. 7. Thus, we conclude that our proposed method is robust when the reconstruction plane shifts along the zaxis and when the target object is rotated.
V. CONCLUSIONS
In this paper, we presented a 3D automatic target recognition system based on integral imaging and using an alloptical structure to provide realtime processing.
Fig. 8 shows a practical application of our method. The two main processing stages in Fig. 2 may be combined into one imaging system using an autocollimating screen [16]. In this system, the reconstructed 3D images are generated at light speed and displayed on a spatial light modulator (SLM) to convert these images into coherent images. Then, a joint transform correlator (JTC) could serve as a correlation filter. This design may produce a realtime, compact 3D object recognition system.
In future work, we will investigate practical realtime systems that use our proposed method.

[Fig. 1.] CIIRtype 3D object correlation method. (a) Storing of reconstructed 3D image for reference, (b) reconstruction of target images at various depths, followed by crosscorrelation with reference.

[Fig. 2.] Structure of the proposed 3D object correlation method. (a) Storing of reconstructed 3D image for reference object, (b) reconstruction of target object at all depth planes.

[]

[Fig. 3.] (a) 3D object, (b) optical pickup setup, and (c) optical display setup.

[Fig. 4.] Reference 3D object. (a) Elemental images, (b) reconstructed 3D image at z = 6 cm.

[Fig. 5.] Reconstructed 3D images with (a) various reconstruction depths, (b) various rotation angles.

[Fig. 6.] kthlaw nonlinear correlation results at z = 6 cm when (a) k = 1, (c) k = 0.7, (e) k = 0.5 at z = 10 cm when (b) k = 1, (d) k = 0.7, (f) k = 0.5.

[Fig. 7.] Experimental results of nonlinear correlation under (a) various reconstruction depths, (b) various rotation angles.

[Fig. 8.] Practical system concept of realtime 3D automatic target recognition using our proposed method.