A Novel Omnidirectional Stereo Vision System with a Single Camera

A new method for the catadioptric omnidirectional stereo vision with single camera is presented in this paper. The proposed method uses a concave lens with a convex mirror. Since the optical part of the proposed method is simple and commercially available, the resulting omnidirectional stereo system is compact and cost-effective. The closed-form solution for 3D distance is derived based on the simple optics including the reflection and the refraction of the convex mirror and the concave lens. The compactness of the system and the simplicity of the image processing make the omnidirectional stereo system appropriate for real-time applications such as autonomous navigation of a mobile robot or the object manipulation. In order to verify the feasibility of the proposed method, an experimental prototype is implemented.

Scene Reconstruction, Pose Estimation and Tracking 456 the omnidirectional stereo vision system obtains the distance information for all directions in one shot, it is especially useful for a mobile robot application. K. Koyasu et al. developed an omnidirectional stereo vision system with two pairs of cameras and convex mirrors for the map-making and the autonomous navigation of a mobile robot (Koyasu et al., 2002). For high resolution, the multiview panoramic cameras have been developed using a mirror pyramid (Nalwa, 1996). The single camera approach is also an important issue in the omnidirectional stereo vision area. A. Basu and D. Southwell proposed a double lobed mirror for the single camera stereo vision system (Southwell et al., 1996) and developed the required image processing algorithm (Fiala & Basu, 2005). E. Cabral et al. also designed the similar double lobed hyperbolic mirror for the single camera omnidirectional stereo vision (Cabral et al., 2004). Recently, another single camera approach using two pieces of hyperbolic mirrors is reported to improve the accuracy in 3D distance computation (Jang et al., 2005). The main advantages of the single camera omnidirectional stereo vision system are the reduced system complexity and the simple image processing due to the consistent intrinsic camera parameters. A main aim of this paper is to present a new approach for the single camera omnidirectional stereo vision system. Eliminating the costly two pieces of mirrors or the double lobed mirrors, the proposed method uses a simple combination of the off-the-shelf convex mirror and concave lens. Thus, the resulting omnidirectional stereo vision system becomes compact and cost-effective. This paper is organized as follows: In Sec. 2, the principles of the proposed omnidirectional stereo system are briefly described. In Sec. 3, the closed-form depth estimate is addressed based on the simple optics for a convex mirror and a concave lens. A prototype of the proposed system and some preliminary experiments are described in Sec. 4. Sec. 5 presents some concluding remarks.

The Proposed Omnidirectional Stereo System
The optical part of the proposed omnidirectional stereo system consists of a convex mirror and a concave lens. A hyperbolic omnidirectional mirror is used as the convex mirror in this paper. However, a parabolic mirror could also be used instead. Fig. 1 illustrates the principles of the proposed system, where (, ) rz O denotes an object point. Light ray I from the object is reflected on the hyperbolic mirror, and the reflected ray passes through the pinhole. There is an image point on the sensor plane corresponding to the ray I as shown in Fig. 1 (a). The real object emits the light rays for all directions, not only the ray I of course. However, the other light rays having different directions from the ray I, e.g., the ray II in Fig. 1 (a) cannot reach the pinhole after the reflection, thereby, cannot have any image on the sensor plane. On the contrary, the reflected ray from the ray II in Fig. 1 (b) can reach the pinhole owing to the refraction through the concave lens. The amount of refraction depends on the refraction index of the lens material, curvature, position, and thickness of the concave lens. It should be noted that the concave lens does not affect on the ray I. Since both the rays I and II come from the same object point, the image points, 1 ρ and 2 ρ on the sensor plane constitute the stereo pair with disparity. Therefore, it is possible to compute 3D distance to the object point based on the simple optics composed of the reflection on the hyperbolic mirror and the refraction through the concave lens.

Refraction Through a Concave Lens
As passing through a concave lens, a light ray experiences the refraction as illustrated in Fig.  2, which is given by Snell's law (Jenkins & White, 1976). First order optics with Gaussian approximation gives the relationships between the incident ray and the refracted ray as follows: where 2 θ ′′ and 2 p′′ denote the incidence angle and the cross point with the vertical axis, and 2 θ and 2 p represent the angle of the refracted ray and the lens position as shown in Fig. 2. Derivation in detail is described in Appendix. It is assumed that the coordinate system is assigned at the pinhole position in this sequel. In (1) and (2), c , d , and n imply the curvature, thickness, and the refraction index of the lens material respectively. Here, a plano-concave lens with flat surface on one side is used without loss of generality. It is also possible to get the similar expressions for a biconcave lens.

Reflection on the Hyperbolic Mirror Surface
The analysis in this paper is described on rz − plane rather than in the whole 3D space. It is easy to extend this analysis to 3D space by rotating about the vertical z axis. Given the reflected ray with angle θ and cross point p as shown in Fig where a and b denote the parameters of the hyperbolic function, and f represents its focal point given by The reflected ray is then described by the given angle θ and the cross point p as: The intersection point, C between the hyperbolic function (3) and the reflected ray (5) where φ and o z represent vertical angle and cross point with z axis as shown in Fig. 3. By using the simple law of reflection on a specular surface and the geometry given in Fig. 3, it is possible to get the first parameter, φ for (6) as follows: Since the incident ray (6) In other words, given angle θ and cross point p of the reflected ray, the corresponding incident ray toward O z is described in (6), where the parameters, φ and O z can be obtained by (7) through (9).
It is assumed in this paper that the first focal point of the hyperbolic mirror is located at 2 hf = , so that the pinhole position of a camera coincides with the second focal point of the hyperbolic mirror. According to the property of the hyperbolic function, all incident rays toward the first focal point, i.e., 2 o zf = , reach the pinhole at the origin after the reflection.
Thus, the reflected ray always has 0 p = without regard to θ .

Depth Computation
As shown in Fig. 4, the position of the object point, is the solution of the simultaneous equations for the rays I and II . Thus, it is necessary to get the expressions for the rays based on the measurement data in the system. Given measured stereo pairs, 1 ρ and 2 ρ on the image plane in Fig. 4, the vertical angles of two rays toward the pinhole are described as follows: where λ denotes the distance from the pinhole position to the image plane. In order to get the equation for the ray II , the refraction through the concave lens should be taken into consideration. As described in Sec. 3.1, it is possible to get 2 θ ′′ and 2 p′′ by (1) and (2) with given 2 θ and 2 p , where

Experimental Results
A prototype of the system is implemented as shown in Fig. 5, and some preliminary experiments are carried out with the prototype. The parameters of the experimental system are tabulated in Table 1. Fig. 6 (a) and Fig. 6 (b) show the omnidirectional images acquired by the proposed system without and with the concave lens respectively. As shown in Fig. 4, the side of the concave lens blocks the light rays incident from sufficiently horizontal directions, which causes the opaque ring at the boundary between the outer and the inner stereo image as shown in Fig.  6. By making the lens side inclined rather than vertical, it is possible to minimize the blocked angle, thereby the thickness of the opaque ring in the image. However, the vertical side of the lens is preferable for the case of a parabolic mirror and the orthographic imaging.  Since the epipolar line is radial in the omnidirectional stereo image, it is relatively easy to match the stereo pair. Recently, many corresponding algorithms for the omnidirectional stereo image have been developed (Fiala & Basu, 2005) (Jang et al., 2005) (Zhu, 2001). The depth resolution in rz − plane is depicted in Fig. 7, where each point represents the depth computed using the correspondences of all pixels along an epipolar line. It is possible to get longer range of sight by using a modified lens. The lens used in Fig. 8 (a) has the convex part in its outer side as well as the concave part in the inner side. As illustrated in Fig. 8 (a), the convex part and the concave part of the lens introduce the refractions in the opposite directions to a pair of light rays, II and III, thereby gives the large disparity in the image points. Fig. 8 (b) shows a simulation result for the depth resolution of the imaging system, which has the longer range of sight than Fig. 7.
(a) The omnidirectional stereo imaging system (b) Depth resolution Figure 8. The omnidirectional stereo imaging system with the modified concave lens and the corresponding depth resolution

Conclusion
Wide field of view is the most attractive feature of the omnidirectional vision. There exist two approaches to omnidirectional stereo imaging with a single camera. They use: (1) a double lobed mirror (Fiala & Basu, 2005) (Cabral et al., 2004), or (2) two mirrors (Jang et al., 2005). In this paper, a third approach is described using a mirror and a concave lens. By adjusting the position of the concave lens, it is possible to control the disparity between two stereo images and the accuracy of the 3D distance computation. Since the optical components adopted in the proposed system are commercially available, the proposed omnidirectional stereo imaging system is compact and cost-effective. Based on the simple optics composed of the reflection and the refraction on the convex mirror and the concave lens, an expression for the 3D distance is derived. The proposed method is versatile in the sense that it is also applicable to different types of convex mirrors, e.g., the parabolic mirror. The second approach (2) mentioned above involves a relatively lengthy baseline, and therefore a longer depth range than (1) and the approach proposed in this paper. Two simple ways of getting longer range of sight with (1) and the approach in this paper are to use a larger mirror or a camera with higher resolution.

Appendix: Refraction through Concave Lens
When passing through a concave lens, a light ray experiences the refraction according to the surface shape and the refraction index of the lens material. The refraction equations, (1) and (2) for θ ′′ and p′′ are derived here in terms of θ and p of the light lay. The overall refraction through the lens consists of two stages: (1) Free space to lens material and (2) Lens material to free space.