Mashhour Solh

Hierarchical Hole-Filling For Depth-Based View Synthesis in FTV and 3D Video

Three-dimensional television (3DTV) is believed to be the future of television broadcasting that would replace current 2D HDTV technology. Future 3DTV would bring a more life-like and visually immersive home entertainment experience, in which users will have the freedom to navigate through the scene to choose a different viewpoint. A desired view can be synthesized at the receiver side using depth image-based rendering (DIBR). While this approach has many advantages, one of the key challenges in DIBR is how to fill the holes caused by disocclusion regions and wrong depth values. In this paper, we propose two new approaches for disocclusion removal in DIBR. Both approaches namely hierarchical hole-filling (HHF) and depth adaptive hierarchical hole-filling eliminate the need for any smoothing or filtering of the depth map. Both techniques use a pyramid-like approach to estimate the hole pixels from lower resolution estimates of the 3D wrapped image. The lower resolution estimates involve a pseudo zero canceling plus Gaussian filtering of the wrapped image. The depth adaptive HHF incorporates the depth information to produce a higher resolution rendering around previously occluded areas. Experimental results show that HHF and depth adaptive HHF yield virtual images and stereoscopic videos that are free of geometric distortions and a better rendering quality both subjectively and objectively than traditional hole-filling approaches.

3VQM: A vision-based quality measure for DIBR-based 3D videos

In this paper, we present a new method for objectively evaluating the quality of stereoscopic 3D videos generated by depth-image-based rendering (DIBR). First we show how to derive an ideal depth estimate at each pixel value that would constitute a distortion-free rendered video. The ideal depth estimate will then be used to derive three distortion measures to objectify the visual discomfort in the stereoscopic videos. The three measures are temporal outliers (TO), temporal inconsistencies (TI), and spatial outliers (SO). The combination of the three measures will constitute a vision-based quality measure for 3D DIBR-based videos, 3VQM. Finally, 3VQM will be presented and verified against a fully conducted subjective evaluation. The results show that our proposed measure is significantly accurate, coherent and consistent with the subjective scores.

A no-reference quality measure for DIBR-based 3D videos

In this paper we present a no-reference objective quality measure for stereoscopic 3D videos generated by depth image-based rendering (DIBR). At first we will derive an ideal depth estimate for each pixel value. The ideal depth estimate will then be used to calculate three distortion measures: temporal outliers (TO), temporal inconsistencies (TI), and spatial outliers (SO). The combination of the three measures constitute the proposed no-reference measure. Its performance is verified using subjective DMOS scores and compared to the full reference version of the proposed algorithm. The subjective results show that the proposed measure highly correlates with subjective scores and is close in performance to the full reference version of the measure.

Hierarchical Hole-Filling(HHF): Depth image based rendering without depth map filtering for 3D-TV

In this paper we propose a new approach for disocclusion removal in depth image-based rendering (DIBR) for 3D-TV. The new approach, Hierarchical Hole-Filling (HHF), eliminates the need for any preprocessing of the depth map. HHF uses a pyramid like approach to estimate the hole pixels from lower resolution estimates of the 3D wrapped image. The lower resolution estimates involves a pseudo zero canceling plus Gaussian filtering of the wrapped image. Then starting backwards from the lowest resolution hole-free estimate in the pyramid, we interpolate and use the pixel values to fill in the hole in the higher up resolution image. The procedure is repeated until the estimated image is hole-free. Experimental results show that HHF yields virtual images that are free of any geometric distortions, which is not the case in other algorithms that preprocess the depth map. Experiments has also shown that unlike previous DIBR techniques, HHF is not sensitive to depth maps with high percentage of bad matching pixels.

MIQM: A Multicamera Image Quality Measure

Although several subjective and objective quality assessment methods have been proposed in the literature for images and videos from single cameras, no comparable effort has been devoted to the quality assessment of multicamera images. With the increasing popularity of multiview applications, quality assessment of multicamera images and videos is becoming fundamental to the development of these applications. Image quality is affected by several factors, such as camera configuration, number of cameras, and the calibration process. In order to develop an objective metric specifically designed for multicamera systems, we identified and quantified two types of visual distortions in multicamera images: photometric distortions and geometric distortions. The relative distortion between individual camera scenes is a major factor in determining the overall perceived quality. In this paper, we show that such distortions can be translated into luminance, contrast, spatial motion, and edge-based structure components. We propose three different indices that can quantify these components. We provide examples to demonstrate the correlation among these components and the corresponding indices. Then, we combine these indices into one multicamera image quality measure (MIQM). Results and comparisons with other measures, such as peak signal-to noise ratio, mean structural similarity, and visual information fidelity show that MIQM outperforms other measures in capturing the perceptual fidelity of multicamera images. Finally, we verify the results against subjective evaluation.

Depth adaptive hierarchical hole filling for DIBR-based 3D videos

In this paper we introduce a depth adaptive approach for disocclusion removal in depth image-based rendering (DIBR). This approach extends the hierarchical hole-filling (HHF) presented in an earlier work. Similar to HHF, the depth adaptive approach results in synthesized 3D videos that are free of geometric distortions. Furthermore, the edges and texture around the disoccluded areas can be sharpened and enhanced through adding the depth adaptive preprocessing step before applying the hierarchal hole-filling. The subjective and objective results show a significant improvement in quality for the synthesized views using the depth adaptive approach.

MIQM: A novel Multi-view Images Quality Measure

Although several subjective and objective quality assessment methods have been proposed in the literature for images and videos from single cameras, no comparable effort has been devoted to the quality assessment for multi-camera images. With the increasing popularity of multi-view applications, quality assessment for multi-camera images and videos is becoming fundamental to the development of these applications. The quality of images, which are captured by a multi-view system, are affected by multiple factors such as camera configuration, number of cameras, and the calibration process. In the process of developing an objective metric specifically designed for multi-camera systems, we identified two types of visual distortions in multi-view images: photometric distortions and geometric distortions. In this paper, we show that in the presence of well defined reference these distortions can be translated into luminance, contrast, spatial motion and edge-based structure components. Then, we propose different index values that can quantify these components. We provide several examples to demonstrate the correlation between each of these components and the corresponding index metric. Then, we combine these indexes into one Multi-view Image Quality Measure (MIQM). The results and examples show that not only MIQM can capture the perceptual quality of multi-view images it also outperforms the Structural SIMilarity(SSIM) measure for single-view images quality assessment.

Depth map estimation in DIBR stereoscopic 3d videos using a combination of monocular cues

We propose a method to reconstruct the depth map from multiple estimated depth maps relying on monocular cues. Based on extracted depth cues from luminance, chrominance, motion and texture, we obtain an optimal depth estimation by analytically deriving the best combinations. We first analyze a ground truth depth map to extract a set of depth cues. Then, using these depth cues, we process the colored reference video to reconstruct the depth map. We tested this approach on different video sequences with different monocular properties. The results show that the extracted depth maps generate a 3D video with quality close to the video rendered using the ground truth depth map. We report subjective and objective results using 3VQM.

The Mosaic Camera: Streaming, Coding and Compositing Experiments

The HP Fan Camera is a panoramic mosaic king camera that is a composite of 24-imager array system. Streaming the captured video is a challenging problem due to several factors such as the large bandwidth requirements, the limited capabilities of the clients machines, and our desire to provide independent viewing controls for users. In the process of developing an optimal rate controller for the HP Fan Camera we developed a client-server framework for multi-camera streaming and performed a set of experiments using various bandwidth allocation schemes. From our preliminary research, we found that sending individual streams of the cameras over the network provides more interactivity to the end users and requires less bandwidth in case the behavior of the end users is aggressive in scene selection. In this paper we present this framework and share the results of our conducted experiments.

Depth-less 3D rendering

We propose a new view synthesis without using a depth map for stereoscopic images with small baseline. The new technique utilizes the linear relationship between local luminance changes and small horizontal shifts in an image. In this paper, we prove the linear relationship between local luminance changes and small horizontal shifts. Then, we will use this relationship to derive a 3D wrapping to render 3D images in absence of a depth map. The rendered images using the proposed technique will be evaluated subjectively and objectively and compared to rendered images using depth maps.