Dragan Sekulovski

Synchronization of Multiple Camera Videos Using Audio-Visual Features

Digital video capturing is getting popular with the decreasing price of camcorders and the increasing availability of devices with embedded video cameras such as digital-still cameras, mobile phones and PDAs. While a raw home video is considered as visually non-appealing, having multiple recordings of the same event provides the opportunity to combine audio and video segments from different cameras for improving quality and aesthetics. Mixing content from different recordings requires precise synchronization among the recordings. In most present applications, synchronization is done manually and considered as a very tedious task. In this paper, we propose a novel automated synchronization approach based on detecting and matching audio and video features extracted from the recorded content. We assess experimentally three realizations of this approach on a common data set and make recommendations on the usability of the different realizations in practical use cases. The realizations have no limitations on the number and movement of the cameras. Moreover, they are robust against various ambient noises and audio-visual artifacts occurring during the recordings.

Synchronization of multiple video recordings based on still camera flashes

Combining audiovisual sequences from different cameras requires precise alignment in time. Current synchronization techniques involve using geometrical properties such as camera positions and object features. In this paper, we present a synchronization method based on detecting flashes present in the video content. Such flashes, generated by still cameras, cause a sharp bright frame in the videos. They are detected using an adaptive threshold on luminance variation across the frames. The resulting flash sequences are searched for matches. The matching flashes indicate overlapping content, and allow determining the offset time between cameras. The experimental results from flash detection and flash sequence matching show perfect synchronization in all examined cases.

Enriching music with synchronized lyrics, images and colored lights

We present a method to synchronize popular music with its lyrics at the stanza level. First we apply an algorithm to segment audio content into harmonically similar and/or contrasting progressions, i.e. the stanzas. We map the stanzas found to a sequence of labels, where stanzas with a similar progression are mapped to the same label. The lyrics are analyzed as well to compute a second sequence of labels. Using dynamic programming, an optimal match is found between the two sequences, resulting in a stanza-level synchronization of the lyrics and the audio. The synchronized lyrics can be used to compute a synchronized slide show to accompany the music, where the images are retrieved using the lyrics. For an additional enrichment of the experience, colored light effects are synchronized with the music that are computed from the sets of images. The song segmentation can be done reliably, while the mapping of the audio segments and lyrics gives encouraging results.

Modeling the visibility of the stroboscopic effect occurring in temporally modulated light systems

Three perception experiments were conducted to develop a measure for predicting the visibility of the stroboscopic effect occurring in temporally modulated light systems. In the first experiment, different methodologies were evaluated for their measurement error. In the second experiment, the visibilities of the stroboscopic effect for square wave and sine wave light modulations were measured and the results were found to be consistent with previous findings for flicker perception. In the third experiment, specifically crafted, complex waveforms were used to test the theory of frequency summation. Based on the results of these three experiments, a new measure for the visibility of the stroboscopic effect was developed, consisting of a summation of the energy in all frequency components, normalized for human sensitivity.

Enriching text with images and colored light

We present an unsupervised method to enrich textual applications with relevant images and colors. The images are collected by querying large image repositories and subsequently the colors are computed using image processing. A prototype system based on this method is presented where the method is applied to song lyrics. In combination with a lyrics synchronization algorithm the system produces a rich multimedia experience. In order to identify terms within the text that may be associated with images and colors, we select noun phrases using a part of speech tagger. Large image repositories are queried with these terms. Per term representative colors are extracted using the collected images. Hereto, we either use a histogram-based or a mean shift-based algorithm. The representative color extraction uses the non-uniform distribution of the colors found in the large repositories. The images that are ranked best by the search engine are displayed on a screen, while the extracted representative colors are rendered on controllable lighting devices in the living room. We evaluate our method by comparing the computed colors to standard color representations of a set of English color terms. A second evaluation focuses on the distance in color between a queried term in English and its translation in a foreign language. Based on results from three sets of terms, a measure of suitability of a term for color extraction based on KL Divergence is proposed. Finally, we compare the performance of the algorithm using either the automatically indexed repository of Google Images and the manually annotated Flickr.com. Based on the results of these experiments, we conclude that using the presented method we can compute the relevant color for a term using a large image repository and image processing.

Quantifying the Visibility of Periodic Flicker

ABSTRACT Three experiments that measure the visibility of periodic flicker are presented. Temporal light modulations were presented to a large visual field to make the results valid for general lighting applications. In addition, the experiments were designed to control for flicker adaptation. In the first experiment, the sensitivity of human observers to light modulations with a sinusoidal waveform at several temporal frequencies up to 80 Hz was measured. The results showed that the sensitivity to flicker (that is, the inverse of the Michelson contrast) is as high as 500 for frequencies between 10 and 20 Hz, which is more than twice the maximum sensitivity reported in the literature. In the second experiment, the sensitivity to flicker for light modulations with complex waveforms, composed of two or three frequency components, was measured. Sensitivity to flicker was found to be higher than the sum of the sensitivities of the individual frequency components of the complex waveform. Based on these results, we defined the flicker visibility measure (FVM), predicting flicker visibility by a weighted summation of the relative energy of the frequency components of the waveform. In the third experiment, sensitivity to realistic waveforms (that is, waveforms of light emitting diode [LED] light sources available on the market) was measured. The flicker predictions of FVM showed a high correlation with the experimental data, in contrast to some other existing flicker measures, including flicker index and percent flicker, demonstrating the usefulness of the measure to objectively assess the visibility of periodic flicker for lighting applications.

Retail lighting and textiles: Designing a lighting probe set

This paper investigates the interaction between retail lighting and textiles to provide retailers and lighting designers with a set of physical textile objects that can be used to test the visual effect of a lighting setting. Since the relationships between optical aspects of lighting–textiles interactions and subjective qualities associated with them have not been systematically investigated, we conducted two experiments to study those relationships. The first experiment concerned photometric measurements of textiles in order to categorize the reflectance types. The second experiment examined human observers’ judgments of a range of material-expressing qualities, such as shininess and softness, in two canonical types of lighting. The textiles for which the differences in those qualities were maximal were used in designing a lighting probe set.

Changing color over time

The revolution in lighting we are experiencing goes beyond the basic capabilities of the light sources used and has enabled new ways of improving the overall experience of both lighting and displays. However, specifics of LEDs, the technical driving force behind the revolution, also introduce new challenges. One of those challenges is the temporal control of full-color light systems. In this work we explore the properties of human color vision relevant to the generation of pleasant dynamic light effects. We show that the spatial models of color are unsuitable for predicting temporal phenomena and give steps towards building a new, temporal model.

Attributes underlying involvement with video material

This paper looks at attributes helpful for a definition of involvement, because the authors believe that involvement is one of the key cognitive mediating factors between content and viewing experience. In the end, it should lead to an operational definition, and a valid and reliable measurement tool.

Spatio-chromatic sensitivity explained by post-receptoral contrast

We measured and modeled visibility thresholds of spatial chromatic sine-wave gratings at isoluminance. In two experiments we manipulated the base color, direction of chromatic modulation, spatial frequency, the number of cycles in the grating, and grating orientation. In Experiment 1 (18 participants) we studied four chromatic modulation directions around three base colors, for spatial frequencies 0.15-5 cycles/deg. Results show that the location, size and orientation of fitted ellipses through the observer-averaged thresholds varied with spatial frequency and base color. As expected, visibility threshold decreased with decreasing spatial frequency, except for the lowest spatial frequency, for which the number of cycles was only three. In Experiment 2 (27 participants) we investigated the effect of the number of cycles at spatial frequencies down to 0.025 cycles/deg. This showed that the threshold elevation at 0.15 cycles/deg in Experiment 1 was at least partly explained by the small number of cycles. We developed two types of chromatic detection models and fitted these to the threshold data. Both models incorporate probability summation across spatially weighted chromatic contrast signals, but differ in the stage at which the contrast signal is calculated. In one, chromatic contrast is determined at the cone receptor level, the dominant procedure in literature. In the other model, it is determined at a postreceptoral level, that is, after cone signals have been transformed into chromatic-opponent channels. We applied Akaikes Information Criterion to compare the performance of the models and calculated their relative probabilities and evidence ratios. We found evidence in favor of the second model and conclude that postreceptoral contrast is the most accurate determinant for chromatic contrast sensitivity.