| Home | Projects | People | Publications | Links |

INTEGRATED 3D ANALYSIS & SYNTHESIS

Sanghoon Sull, Narenda Ahuja

INTRODUCTION

This research is concerned with three-dimensional (3D) analysis, and analysis-guided syntheses of images showing 3D motion of an observer relative to a scene. There are two objectives of the research. First, 3D motion and structure parameters are recovered from multiple features present in a monocular image sequence, such as point features, optical flow, regions, lines, texture gradient, and vanishing lines. Second, image sequences are synthesized by using the intermediate outputs of 3D image analysis. A new notion is introduced according to the cues that contribute the most to 3D interpretation, also the ones that would contribute the most to realistic synthesis. This suggests an approach to analysis-guided 3D compression. It should be noted that 3D interpretation here is intended to communicate to the observer certain chosen 3D characteristics of the scene, such as those that may be useful for navigation. A key feature of the approach presented that helps meet both objectives is an integrated use of multiple image attributes or cues. These cues carry the motion and structure information of interest to different degrees and have different, often complementary, strengths and shortcomings. The goal is to estimate motion and structure parameters such that the estimates best explain the presence of all the observed image cues throughout the image sequence. The framework for integration used is one of optimization. The objective function to be minimized is based on the differences between the observed image features and those corresponding to motion and structure parameters. The focus is on the images obtained by an observer moving above a planar, textured surface, such as a flying aircraft.

OVERVIEW OF APPROACH

The main steps of the approach are illustrated here through real flying image sequence - Desert Sequence. [Movie Clip (1.38M)]

Step 1:Feature Extraction

Features are extracted from two images [Frame 1, Frame 2]. Optical flow is also computed. Two alternating frames [Frame1, Frame 2] show the extracted regions from two images. The extracted line features are shown in [Frame 1, Frame 2]. The computed flow between the two images is shown Here. Flow vectors are used at only those locations where point feature detector responds.

Step 2: Integrated Matching and Segmentation

Correspondences of multiple features are established in each pair of adjacent images and features are segmented into local planar surface patches. For flight images, the largest plane segment corresponding to ground is identified. Matched and segmented ground features are shown here for two frames. Integrated interpretation leads to these features which are diverse but mutually compatible. Note that the part corresponding to the sky and bottom of the aircraft are successfully segmented out.

Step 3: Recognition of Vanishing Line

The vanishing line is identified from the set of detected lines in each frame, are using in two-view estimation. The identified vanishing lines are used as constraints in estimating motion and structure parameters. For this desert sequence, vanishing lines were successfully recognized in all frames.

Step 4: Integrated Nonlinear Batch Estimation and Sequential Update

Multiple frames in a batch are used to interactively estimate motion and structure parameters from various features. This step also updates motion parameters derived from each overlapping batch. Then these motion parameters are used to compute globally compatible structure parameters.

Step 5: Synthesis

This step synthesizes an image sequence to depict the motion and scene structures. The synthesis incorporates features used to obtain the motion and structure estimates, as well as artificial image features, which are consistent with the motion and structure estimates, but not present in the original images.

The following movie clips illustrate the input images and synthesis images:

ANOTHER EXAMPLE

Now we give another example- Runaway Sequence. This sequence of 34 frames was derived from a commercially available laserdisc of films shot from flying aircraft. This is a challenging sequence since the images contain the partially or completely occluded vanishing lines, and there is a reflection of the ground on the bottom of the airplane. For this runaway sequence, vanishing lines were not identified in all frames due to an occlusion by the bottom parts of the airplane.

A flying sequence containing a runaway is an enhance in three ways by using the integrated 3D analysis, first, the recognized runaway edges are repainted with yellow discs simulating lights, second, the parts of the images occluded by the bottom of the aircraft are filled in. third, the seen parts above the estimated vanishing line are colored blue to simulate sky.

The following movie clips illustrate the input images and synthesis images: