Mohammad Rafiq Swash

Immersive 3D Holoscopic Video System

We demonstrated a 3D holoscopic video system for 3DTV application. We showed that using a field lens and a square aperture significantly reduces the vignetting problem associated with a relay system and achieves over 95 percent fill factor. The main problem for such a relay system is the nonlinear distortion during the 3D image capturing, which can seriously affect the reconstruction process for a 3D display. The nonlinear distortion mainly includes lens radial distortion (intrinsic) and microlens array perspective distortion (extrinsic). This is the task of future work. Our results also show that the SS coding approach performs better than the standard HEVC scheme. Furthermore, we show that search and retrieval performance relies on the depth maps quality and that the multimodal fusion boosts the retrieval performance.

Feature extraction and clustering for dynamic video summarisation

In this paper an effective dynamic video summarisation algorithm is presented using audio-visual features extracted from videos. Audio, colour and motion features are dynamically fused using an adaptively weighting mechanism. Dissimilarities of temporal video segments are formulated using the extracted features before these segments are clustered using a fuzzy c-means algorithm with an optimally determined cluster number. The experimental results demonstrate the ability of the proposed algorithm to automatically summarise the videos with good performance.

Adaptive depth map estimation from 3D integral image

Integral Imaging (InIm) is one of the most promising technologies for producing full color 3-D images with full parallax. InIm requires only one recording in obtaining 3D information and therefore no calibration is necessary to acquire depth values. The compactness of using InIm in depth measurement has been attracting attention as a novel depth extraction technique. In this paper, an algorithm for depth extraction that builds on previous work by the authors is presented. Three main problems in depth map estimation from InIm have been solved; the uncertainty and region homogeneity at image location where errors commonly appear in disparity process, dissimilar displacements within the matching block around object borders, object segmentation. This method is based on the distribution of the sample variance in sub-dividing non-overlapping blocks. A descriptor which is unique and distinctive for each feature on InIm has been achieved. Comparing to state-of-the-art techniques, it is shown that the proposed algorithm has improvements on two aspects: depth map extraction level, computational complexity.

RUSHES--an annotation and retrieval engine for multimedia semantic units

Multimedia analysis and reuse of raw un-edited audio visual content known as rushes is gaining acceptance by a large number of research labs and companies. A set of research projects are considering multimedia indexing, annotation, search and retrieval in the context of European funded research, but only the FP6 project RUSHES is focusing on automatic semantic annotation, indexing and retrieval of raw and un-edited audio-visual content. Even professional content creators and providers as well as home-users are dealing with this type of content and therefore novel technologies for semantic search and retrieval are required. In this paper, we present a summary of the most relevant achievements of the RUSHES project, focusing on specific approaches for automatic annotation as well as the main features of the final RUSHES search engine.

Reference based holoscopic 3D camera aperture stitching for widening the overall viewing angle

Holoscopic 3D imaging also known as Integral imaging is a promising technique for creating full color 3D optical models that exist in space independently of the viewer. The images exhibit continuous parallax throughout the viewing zone. In order to achieve depth control, robust and real-time, a single aperture holoscopic 3D imaging camera is used for recording holoscopic 3D image using a regularly spaced array of small lenslets, which view the scene at a slightly different angle to its neighbor. However, the main problem the holoscopic 3D camera aperture faces is that it is not big enough for recording larger scene with existing 2D camera sensors. This paper proposes a novel reference based holoscopic 3D camera aperture stitching method that enlarges overall viewing angle of the holoscopic 3D camera in post-production after the capture.

Pre-processing of holoscopic 3D image for autostereoscopic 3D displays

Holoscopic 3D imaging also known as Integral imaging is an attractive technique for creating full color 3D optical models that exist in space independently of the viewer. The constructed 3D scene exhibits continuous parallax throughout the viewing zone. In order to achieve depth control, robust and real-time, a single aperture holoscopic 3D imaging camera is used for recording holoscopic 3D image using a regularly spaced array of microlens arrays, which view the scene at a slightly different angle to its neighbor. However, the main problem is that the microlens array introduces a dark borders in the recorded image and this causes errors at playback on the holoscopic 3D Display. This paper proposes a reference based pre-processing of holoscopic 3D image for autostereoscopic holoscopic 3D displays. The proposed method takes advantages of microlens as reference point to detect amount of introduced dark borders and reduce/remove them from the holoscopic 3D image.

Super depth-map rendering by converting holoscopic viewpoint to perspective projection

The expansion of 3D technology will enable observers to perceive 3D without any eye-wear devices. Holoscopic 3D imaging technology offers natural 3D visualisation of real 3D scenes that can be viewed by multiple viewers independently of their position. However, the creation of a super depth-map and reconstruction of the 3D object from a holoscopic 3D image is still in its infancy. The aim of this work is to build a high-quality depth map of a real 3D scene from a holoscopic 3D image through extraction of multi-view high resolution Viewpoint Images (VPIs) to compensate for the poor features of VPIs. To manage this, we propose a reconstruction method based on the perspective formula to convert sets of directional orthographic low resolution VPIs into perspective projection geometry. Following that, we implement an Auto-Feature point algorithm for synthesizing VPIs to distinctive Feature-Edge (FE) blocks to localize and provide an individual feature detector that is responsible for integration of 3D information. Detailed experiments proved the reliability and efficiency of the proposed method, which outperforms state-of-the-art methods for depth map creation.

Distributed pixel mapping for refining dark area in parallax barriers based holoscopic 3D Display

Autostereoscopic 3D Display is robustly developed and available in the market for both home and professional users. However 3D resolution with acceptable 3D image quality remains a great challenge. This paper proposes a novel pixel mapping method for refining dark areas between two pinholes by distributing it into 3 times smaller dark areas and creating micro-pinholes in parallax barriers based holoscopic 3D displays. The proposed method allows to project RED, GREEN, BLUE subpixels separately from 3 different pinholes and it distributes the dark spaces into 3 times smaller dark spaces, which become unnoticeable and improves quality of the constructed holoscopic 3D scene significantly. Parallax barrier technology refers to a pinhole sheet or device placed in front or back of a liquid crystal display, allowing to project viewpoint pixels into space that reconstructs a holoscopic 3D scene in space. The holoscopic technology mimics the imaging system of insects, such as the fly, utilizing a single camera, equipped with a large number of micro-lenses or pinholes, to capture a scene, offering rich parallax information and enhanced 3D feeling without the need of wearing specific eyewear.

Three-dimensional integral image reconstruction based on viewpoint interpolation

This paper presents new algorithm for improving the visual definition quality of real integral images computationally throughimage reconstructing. The proposed algorithm takes advantage of true 3D “Integral imaging”. A real world scene is recorded based on the flys eye technique, which is simulated by an array of microlenses.. The proposed method works on orthographic viewpoint images, where shift-and-integration of the neighboring viewpoints are used with quadratic interpolation to increase the visual quality on the final image. This process returns a standard photographic image with enhanced image quality. Detailed experiments have been conducted to demonstrate the effectiveness of proposed method and results are offered.

Holoscopic 3D image rendering for Autostereoscopic Multiview 3D Display

The Autostereoscopic Multiview 3D Display is robustly developed and widely adopted by both home and professional users however Multiview 3D content generation remains a great challenge. This paper proposes a novel method for Multiview 3D content generation and it describes the necessary steps for Holoscopic 3D image rendering for autostereoscopic Multiview 3D display. The Holoscopic imaging technology mimics the imaging system of insects, such as the fly, utilizing a single camera, equipped with a large number of micro-lenses, to capture a scene, offering rich parallax information and enhanced 3D feeling without the need of wearing specific eyewear. In addition a 3D pixel mapping/conversion algorithm has been developed that is supported by the Multiview 3D display.