Konstantinos Rapantzikos

Dense saliency-based spatiotemporal feature points for action recognition

Several spatiotemporal feature point detectors have been used in video analysis for action recognition. Feature points are detected using a number of measures, namely saliency, cornerness, periodicity, motion activity etc. Each of these measures is usually intensity-based and provides a different trade-off between density and informativeness. In this paper, we use saliency for feature point detection in videos and incorporate color and motion apart from intensity. Our method uses a multi-scale volumetric representation of the video and involves spatiotemporal operations at the voxel level. Saliency is computed by a global minimization process constrained by pure volumetric constraints, each of them being related to an informative visual aspect, namely spatial proximity, scale and feature similarity (intensity, color, motion). Points are selected as the extrema of the saliency response and prove to balance well between density and informativeness. We provide an intuitive view of the detected points and visual comparisons against state-of-the-art space-time detectors. Our detector outperforms them on the KTH dataset using nearest-neighbor classifiers and ranks among the top using different classification frameworks. Statistics and comparisons are also performed on the more difficult Hollywood human actions (HOHA) dataset increasing the performance compared to current published results.

Detection and segmentation of drusen deposits on human retina: potential in the diagnosis of age-related macular degeneration.

Assessment of the risk for the development of age-related macular degeneration requires reliable detection and quantitative mapping of retinal abnormalities that are considered as precursors of the disease. Typical signs for the latter are the so-called drusen that appear as abnormal white-yellow deposits on the retina. Segmentation of these features using conventional image analysis methods is quite complicated mainly due to the non-uniform illumination and the variability of the pigmentation of the background tissue. This paper presents a novel segmentation algorithm for the automatic detection and mapping of drusen in retina images acquired with the aid of a digital Fundus camera. We employ a modified adaptive histogram equalization, namely the multilevel histogram equalization (MLE) scheme, for enhancing local intensity structures. For the detection of drusen in retina images, we develop a novel segmentation technique, the histogram-based adaptive local thresholding (HALT), which extracts the useful information from an image without being affected by the presence of other structures. We provide experimental results from the application of our technique to real images, where certain abnormalities (drusen) have slightly different characteristics from the background. The performance of the algorithm is established through statistical analysis of the results. This analysis indicates that the proposed drusen detector gives reliable detection accuracy in both position and mass size.

Multimodal Saliency and Fusion for Movie Summarization Based on Aural, Visual, and Textual Attention

Multimodal streams of sensory information are naturally parsed and integrated by humans using signal-level feature extraction and higher level cognitive processes. Detection of attention-invoking audiovisual segments is formulated in this work on the basis of saliency models for the audio, visual, and textual information conveyed in a video stream. Aural or auditory saliency is assessed by cues that quantify multifrequency waveform modulations, extracted through nonlinear operators and energy tracking. Visual saliency is measured through a spatiotemporal attention model driven by intensity, color, and orientation. Textual or linguistic saliency is extracted from part-of-speech tagging on the subtitles information available with most movie distributions. The individual saliency streams, obtained from modality-depended cues, are integrated in a multimodal saliency curve, modeling the time-varying perceptual importance of the composite video stream and signifying prevailing sensory events. The multimodal saliency representation forms the basis of a generic, bottom-up video summarization algorithm. Different fusion schemes are evaluated on a movie database of multimodal saliency annotations with comparative results provided across modalities. The produced summaries, based on low-level features and content-independent fusion and selection, are of subjectively high aesthetic and informative quality.

Video event detection and summarization using audio, visual and text saliency

Detection of perceptually important video events is formulated here on the basis of saliency models for the audio, visual and textual information conveyed in a video stream. Audio saliency is assessed by cues that quantify multifrequency waveform modulations, extracted through nonlinear operators and energy tracking. Visual saliency is measured through a spatiotemporal attention model driven by intensity, color and motion. Text saliency is extracted from part-of-speech tagging on the subtitles information available with most movie distributions. The various modality curves are integrated in a single attention curve, where the presence of an event may be signified in one or multiple domains. This multimodal saliency curve is the basis of a bottom-up video summarization algorithm, that refines results from unimodal or audiovisual-based skimming. The algorithm performs favorably for video summarization in terms of informativeness and enjoyability.

Hyperspectral imaging: potential in non-destructive analysis of palimpsests

Palimpsests -twice written manuscripts- are of great interest since they may contain important hidden text underneath the visible one. Hyperspectral imaging may aid the expert to read the old script by enhancing the contrast between the under- and -overwriting. We present a hyperspectral imager (MUSIS), capable of acquiring 34 calibrated spectral bands in the range of 360-1150 nm (extended to 1550 nm when coupled with a photocathode tube). The potential of spectral imaging to improve the readability of manuscripts by employing common spectral analysis techniques is explored. Results on different manuscripts obtained by principal component analysis (PCA) and linear spectral mixture analysis (LSMA) prove that hyperspectral imaging tools has the potential to become an indispensable tool for the analysis of old manuscripts.

A snake model for object tracking in natural sequences

Abstract Tracking moving objects in video sequences is a task that emerges in various fields of study: video analysis, computer vision, biomedical systems, etc. In the last decade, special attention has been drawn to problems concerning tracking in real-world environments, where moving objects do not obey any afore-known constraints about their nature and motion or the scenes they are moving in. Apart from the existence of noise and environmental changes, many problems are also concerned, due to background texture, complicated object motion, and deformable and/or articulated objects, changing their shape while moving along time. Another phenomenon in natural sequences is the appearance of occlusions between different objects, whose handling requires motion information and, in some cases, additional constraints. In this work, we revisit one of the most known active contours, the Snakes, and we propose a motion-based utilization of it, aiming at successful handling of the previously mentioned problems. The use of the object motion history and first order statistical measurements of it, provide us with information for the extraction of uncertainty regions, a kind of shape prior knowledge w.r.t. the allowed object deformations. This constraining also makes the proposed method efficient, handling the trade-off between accuracy and computation complexity. The energy minimization is approximated by a force-based approach inside the extracted uncertainty regions, and the weights of the total snake energy function are automatically estimated as respective weights in the resulting evolution force. Finally, in order to handle background complexity and partial occlusion cases, we introduce two rules, according to which the moving object region is correctly separated from the background, whereas the occluded boundaries are estimated according to the objects expected shape. To verify the performance of the proposed method, some experimental results are included, concerning different cases of object tracking, indoors and outdoors, with rigid and deformable objects, noisy and textured backgrounds, as well as appearance of occlusions.

AN EMBEDDED SALIENCY MAP ESTIMATOR SCHEME: APPLICATION TO VIDEO ENCODING

In this paper we propose a novel saliency-based computational model for visual attention. This model processes both top-down (goal directed) and bottom-up information. Processing in the top-down channel creates the so called skin conspicuity map and emulates the visual search for human faces performed by humans. This is clearly a goal directed task but is generic enough to be context independent. Processing in the bottom-up information channel follows the principles set by Itti et al. but it deviates from them by computing the orientation, intensity and color conspicuity maps within a unified multi-resolution framework based on wavelet subband analysis. In particular, we apply a wavelet based approach for efficient computation of the topographic feature maps. Given that wavelets and multiresolution theory are naturally connected the usage of wavelet decomposition for mimicking the center surround process in humans is an obvious choice. However, our implementation goes further. We utilize the wavelet decomposition for inline computation of the features (such as orientation angles) that are used to create the topographic feature maps. The bottom-up topographic feature maps and the top-down skin conspicuity map are then combined through a sigmoid function to produce the final saliency map. A prototype of the proposed model was realized through the TMDSDMK642-0E DSP platform as an embedded system allowing real-time operation. For evaluation purposes, in terms of perceived visual quality and video compression improvement, a ROI-based video compression setup was followed. Extended experiments concerning both MPEG-1 as well as low bit-rate MPEG-4 video encoding were conducted showing significant improvement in video compression efficiency without perceived deterioration in visual quality.

Movie summarization based on audiovisual saliency detection

Based on perceptual and computational attention modeling studies, we formulate measures of saliency for an audiovisual stream. Audio saliency is captured by signal modulations and related multi-frequency band features, extracted through nonlinear operators and energy tracking. Visual saliency is measured by means of a spatiotemporal attention model driven by various feature cues (intensity, color, motion). Audio and video curves are integrated in a single attention curve, where events may be enhanced, suppressed or vanished. The presence of salient events is signified on this audiovisual curve by geometrical features such as local extrema, sharp transition points and level sets. An audiovisual saliency-based movie summarization algorithm is proposed and evaluated. The algorithm is shown to perform very well in terms of summary informativeness and enjoyability for movie clips of various genres.

Spatiotemporal saliency for event detection and representation in the 3D wavelet domain: potential in human action recognition

Event detection and recognition is still one of the most active fields in computer vision, since the complexity of the dynamic events and the need for computational efficient solutions pose several difficulties. This paper addresses detection and representation of spatiotemporal salient regions using the 3D Discrete Wavelet Transform (DWT). We propose a framework to measure saliency based on the orientation selective bands of the 3D DWT and represent events using simple features of salient regions. We apply this method to human action recognition, test it on a large public video database consisting of six human actions and compare the results against an established method in the literature. Qualitative and quantitative evaluation indicates the potential of the proposed method to localize and represent human actions.

Nonlinear enhancement and segmentation algorithm for the detection of age-related macular degeneration (AMD) in human eye's retina

Assessment of the risk for the development of age related macular degeneration requires reliable detection of retinal abnormalities that are considered as precursors of the disease. A typical sign for the latter are the so-called drusen, which appear as abnormal white-yellow deposits on the retina. This paper presents a novel segmentation algorithm for automatic detection of abnormalities in images of the human eyes retina, acquired from a depth-vision camera. Conventional image processing techniques are sensitive to non-uniform illumination and nonhomogeneous background, which obstructs the derivation of reliable results for a large set of different images. Homomorphic filtering and a multilevel variant of histogram equalization are used for non-uniform illumination compensation and enhancement. We develop a novel segmentation technique, the histogram-teased adaptive local thresholding (HALT), to detect drusen in retina images by extracting the useful information without being affected by the presence of other structures. We provide experimental results from the application of our technique to real images, where certain abnormalities (drusen) have slightly different characteristics from the background and are hard to be segmented by other conventional techniques.