A Hemispherical Imaging Camera
Abstract
We have developed a camera which is capable of acquiring very large field of view (FOV) images at high and uniform resolution, from a single viewpoint, at video rates. The FOV can range from being nearly hemispherical, to being nearly omni-directional, barring some small scene parts being obstructed by image sensors themselves. The camera consists of multiple imaging sensors and a hexagonal prism made of planar mirror faces. Each sensor is paired with a planar face of the prism. The sensors are positioned in such a way that they image different parts of the scene from a single virtual viewpoint, either directly or after reflections off the prism. A panoramic image is constructed by concatenating the images taken by different sensors. The resolution of the panoramic image is proportional to the number of sensors used and therefore a multiple of that of an individual sensor. Further, the resolution is substantially uniform across the entire panoramic image.
1. Introduction
A panoramic camera is an imaging device capable of capturing a very large field of view (FOV). Like any other camera, it is desirable that such cameras acquire the entire FOV from a single viewpoint, in real time, at high resolution which is uniform across the FOV, with large dynamic range, and over a large depth of field. Such devices find applications in many areas including tele-conferencing, surveillance and robot navigation. Many efforts have been made to achieve various subsets of these properties (i.e. wide FOV, high and uniform resolution, large depth of field, high dynamic range, a single viewpoint, and real-time acquisition. These methods of capturing panoramic or omni-directional images fall into two categories: dioptric methods, where only refractive elements (lenses) are employed, and catadioptric methods, where a combination of reflective and refractive components is used.
Typical dioptric systems include camera clusters, panning cameras, and fisheye lenses. Catadioptric methods include curved mirror systems where a conventional camera captures the scene reflected off a single non-planar mirror (e.g. parabolic or hyperbolic mirror), and planar mirror systems such as mirror pyramid systems where multiple conventional cameras image the scene reflected off the faces of a mirror-pyramid. The cameras that use a parabolic- or a hyperbolic-mirror to map an omni-directional view onto a single sensor are able to capture a large FOV from a single viewpoint at video rate. However, the FOV shape is hemispherical minus a central cone which is blocked by self-occlusion. The overall resolution of the acquired images is limited to that of the sensor used, and further it varies with the viewing direction across the ring-like FOV, e.g., from a high just outside the central blind spot to a low in the periphery. The cameras using a spherical or conical mirror have similar properties as those using parabolic or hyperbolic mirrors except that they do not possess a single viewpoint.
Many of the aforementioned systems provide a cylindrical shape FOV which is 360° wide in azimuth, but has limited height in elevation (Fig. 1a). In certain applications such as robot navigation and surveillance, however, a hemispherical shape FOV is highly desirable (Fig. 1b). Here we present a system which is capable of acquiring hemispherical panoramic images in real time, with high and substantially uniform resolution, and from a single viewpoint. By substantially uniform resolution we mean the same level of uniformity as delivered by a conventional, non-panoramic camera, and we will refer to it simply as “uniform resolution.”
2. Proposed Hemispherical Camera
Our approach uses a system of conventional cameras as component imagers, also called cameras.
Large FOV from Regular FOV Imagers: Constructing a hemispherical FOV from smaller FOV’s (of conventional imagers) is similar to the problem of tessellating the surface of a hemisphere centered at the viewpoint with cells, each corresponding to the FOV of an imager. Adjacency of the cells would ensure that the imagers together define a contiguous, large FOV. Since real imagers are rectangular, the problem more specifically amounts to juxtaposing multiple rectangular cones, having a common vertex at the viewpoint, such that together they cover as much of the hemisphere as possible.
Co-location Through Mirrors: Clearly, multiple cameras cannot be physically placed in the same space around the viewpoint to achieve the above FOV configuration. A straightforward solution is to have a central cell/camera surrounded by a ring of side cells/cameras collocated with the central camera by mirrors so the image of each side camera coincides with the center camera. This amounts to surrounding the center camera with a ring of planar mirrors and placing the side cameras at the images of the center camera’s viewpoint into the mirrors. The planar mirrors would form an upright pyramid, a prism or an inverse pyramid, according to whether the camera viewing directions converge, are parallel, or diverge. Thus, there are three possible mirror-camera configurations according to whether the mirror planes form an acute, right, or obtuse angle with the ground plane. It can be seen that a right angle would suffice for FOV’s less than 180˚, and has the advantage that the corresponding mirror prism is easier to fabricate.
Uniformity of Resolution: In addition to forming a tessellation, we choose the individual FOV cells to be as close to being identical in size as possible, to yield uniform resolution across the hemispherical FOV.
3. Prototype
We have implemented the proposed hemispherical camera design using a hexagonal mirror prism with a slope angle of α=90º. The height of the prism is 74.27mm and the width of the cross section is 69.54mm. The wedge angle β is 25º. A total of 7 Dragonfly color cameras from Point Grey, each with a resolution of 640x480 pixels, are used in the system. Six of them equipped with 4mm lenses are placed around the mirror prism with a tilt angle 47.5º. The seventh camera is equipped with a 3.5mm lens and placed on the axis of the prism. Each side camera effectively covers 45º vertically, while the center camera covers 50º vertically. The overall FOV of the panoramic system is 360ºx140º. Thus, the side cameras have a total horizontal FOV of 360º and the sides plus center camera have a total vertical FOV of 50º+2x45º=140º. The value of 140º can be increased by controlling the orientations of the side cameras. Figure 2 shows our prototype implementation.
To construct the hemispherical FOV image, each of the 7 images is warped to the viewing direction (image plane)of the central camera.
Near the edges between images obtained by adjacent cameras, the finite size of the camera apertures results in mixing of the images of scene parts visible in the direct and reflected fields of view. We have developed a stereo matching algorithm that filters out the image parts (interference) due to the directly visible parts.
Fig. 3 shows a sample image acquired by the prototype. The first row shows two input images taken from one of the side cameras and the center camera. The second row shows the images after the warping process. The third row shows the images after compensating for mixing artifacts. The fourth row shows a panoramic image around the view direction of the center camera. The effective FOV from the top boundary to the bottom boundary is exactly 140º.
Fig. 4 shows two panoramic images with and without compensating for mixing artifacts. In Fig 4a, note that the boundaries between the image parts corresponding to the side camera FOV’s, and the much darker strips at the boundary of the center camera caused by the 0.5mm thickness of the prism’s top edges. Fig 4b shows that the compensation process has removed both of the mixing region artifacts and the images in those areas are well recovered.
