Namunu Chinthaka Maddage

Creating audio keywords for event detection in soccer video

This paper presents a novel framework called audio keywords to assist event detection in soccer video. Audio keyword is a middle-level representation that can bridge the gap between low-level features and high-level semantics. Audio keywords are created from low-level audio features by using support vector machine learning. The created audio keywords can be used to detect semantic events in soccer video by applying a heuristic mapping. Experiments of audio keywords creation and event detection based on audio keywords have illustrated promising results. According to the experimental results, we believe that audio keyword is an effective representation that is able to achieve more intuitionistic result for event detection in sports video compared with the method of event detection directly based on low-level features.

Audio Based Event Detection for Multimedia Surveillance

With the increasing use of audio sensors in surveillance and monitoring applications, event detection using audio streams has emerged as an important research problem. This paper presents a hierarchical approach for audio based event detection for surveillance. The proposed approach first classifies a given audio frame into vocal and nonvocal events, and then performs further classification into normal and excited events. We model the events using a Gaussian mixture model and optimize the parameters for four different audio features ZCR, LPC, LPCC and LFCC. Experiments have been performed to evaluate the effectiveness of the features for detecting various normal and the excited state human activities. The results show that the proposed top-down event detection approach works significantly better than the single level approach

Content-based music structure analysis with applications to music semantics understanding

In this paper, we present a novel approach for music structure analysis. A new segmentation method, beat space segmentation, is proposed and used for music chord detection and vocal/instrumental boundary detection. The wrongly detected chords in the chord pattern sequence and the misclassified vocal/instrumental frames are corrected using heuristics derived from the domain knowledge of music composition. Melody-based similarity regions are detected by matching sub-chord patterns using dynamic programming. The vocal content of the melody-based similarity regions is further analyzed to detect the content-based similarity regions. Based on melody-based and content-based similarity regions, the music structure is identified. Experimental results are encouraging and indicate that the performance of the proposed approach is superior to that of the existing methods. We believe that music structure analysis can greatly help music semantics understanding which can aid music transcription, summarization, retrieval and streaming.

Automatic music classification and summarization

Automatic music classification and summarization are very useful to music indexing, content-based music retrieval and on-line music distribution, but it is a challenge to extract the most common and salient themes from unstructured raw music data. In this paper, we propose effective algorithms to automatically classify and summarize music content. Support vector machines are applied to classify music into pure music and vocal music by learning from training data. For pure music and vocal music, a number of features are extracted to characterize the music content, respectively. Based on calculated features, a clustering algorithm is applied to structure the music content. Finally, a music summary is created based on the clustering results and domain knowledge related to pure and vocal music. Support vector machine learning shows a better performance in music classification than traditional Euclidean distance methods and hidden Markov model methods. Listening tests are conducted to evaluate the quality of summarization. The experiments on different genres of pure and vocal music illustrate the results of summarization are significant and effective.

Musical genre classification using support vector machines

Automatic musical genre classification is very useful for music indexing and retrieval. In this paper, an efficient and effective automatic musical genre classification approach is presented. A set of features is extracted and used to characterize music content. A multi-layer classifier based on support vector machines is applied to musical genre classification. Support vector machines are used to obtain the optimal class boundaries between different genres of music by learning from training data. Experimental results of multi-layer support vector machines illustrate good performance in musical genre classification and are more advantageous than traditional Euclidean distance based method and other statistic learning methods.

Study of empirical mode decomposition and spectral analysis for stress and emotion classification in natural speech

Two new approaches to the feature extraction process for automatic stress and emotion classification in speech are proposed and examined. The first method uses the empirical model decomposition (EMD) of speech into intrinsic mode functions (IMF) and calculates the average Renyi entropy for the IMF channels. The second method calculates the average spectral energy in the sub-bands of speech spectrograms and can be enhanced by anisotropic diffusion filtering of spectrograms. In the second method, three types of sub-bands were examined: critical, Bark and ERB. The performance of the new features was compared with the conventional mel frequency cepstral coefficients (MFCC) method. The modeling and classification process applied the classical GMM and KNN algorithms. The experiments used two databases containing natural speech, SUSAS (annotated with three different levels of stress) and the Oregon Research Institute (ORI) data (annotated with five different emotions: neutral, angry, anxious, dysphoric, and happy). For the SUSAS data, the best average recognition rates of 77% were obtained when using spectrogram features calculated within ERB bands and combined with anisotropic filtering. For the ORI data, the best result of 53% was obtained with the same method but without anisotropic filtering. Both the GMM and KNN classifiers showed similar performance. These results indicated that the spectrogram patterns provide promising results in the case of stress recognition, however further improvements are needed in the case of emotion recognition.

Content-adaptive digital music watermarking based on music structure analysis

A novel content-adaptive music watermarking technique is proposed in this article. To optimally balance inaudibility and robustness when embedding and extracting watermarks, the embedding scheme is highly related to the music structure and human auditory system (HAS). A note-based segmentation method is proposed and used for music vocal/instrumental boundary detection. A multiple bit hopping and hiding scheme with different embedding parameters is applied to vocal and instrumental frames of the music. The experimental results in inaudibility and robustness are provided to support all novel features in the proposed watermarking scheme.

Influence of acoustic low-level descriptors in the detection of clinical depression in adolescents

In this paper, we report the influence that classification accuracies have in speech analysis from a clinical dataset by adding acoustic low-level descriptors (LLD) belonging to prosodic (i.e. pitch, formants, energy, jitter, shimmer) and spectral features (i.e. spectral flux, centroid, entropy and roll-off) along with their delta (Δ) and delta-delta (Δ-Δ) coefficients to two baseline features of Mel frequency cepstral coefficients and Teager energy critical-band based autocorrelation envelope. Extracted acoustic low-level descriptors (LLD) that display an increase in accuracy after being added to these baseline features were finally modeled together using Gaussian mixture models and tested. A clinical data set of speech from 139 adolescents, including 68 (49 girls and 19 boys) diagnosed as clinically depressed, was used in the classification experiments. For male subjects, the combination of (TEO-CB-Auto-Env + Δ + Δ-Δ) + F0 + (LogE + Δ + Δ-Δ) + (Shimmer + Δ) + Spectral Flux + Spectral Roll-off gave the highest classification rate of 77.82% while for the female subjects, using TEO-CB-Auto-Env gave an accuracy of 74.74%.

Music structure based vector space retrieval

This paper proposes a novel framework for music content indexing and retrieval. The music structure information, i.e., timing, harmony and music region content, is represented by the layers of the music structure pyramid. We begin by extracting this layered structure information. We analyze the rhythm of the music and then segment the signal proportional to the inter-beat intervals. Thus, the timing information is incorporated in the segmentation process, which we call Beat Space Segmentation. To describe Harmony Events, we propose a two-layer hierarchical approach to model the music chords. We also model the progression of instrumental and vocal content as Acoustic Events. After information extraction, we propose a vector space modeling approach which uses these events as the indexing terms. In query-by-example music retrieval, a query is represented by a vector of the statistics of the n-gram events. We then propose two effective retrieval models, a hard-indexing scheme and a soft-indexing scheme. Experiments show that the vector space modeling is effective in representing the layered music information, achieving 82.5% top-5 retrieval accuracy using 15-sec music clips as the queries. The soft-indexing outperforms hard-indexing in general.

Singing voice detection using twice-iterated composite Fourier transform

In this paper, we propose a twice-iterated composite Fourier transform (TICFT) technique to detect the singing voice boundaries from acoustical polyphonic music signals. We show that the cumulative TICFT energy in the lower coefficients is capable of differentiating the harmonic structures of vocal and instrumental music in higher octaves. The musical signal is first segmented into frames based on quarter-notes. Then TICFT is used to measure the harmonic structure of each frame. Finally, the vocal and instrumental frames are classified by applying music domain knowledge. Experimental results show over 80% frame level accuracy can be achieved