Jieping Xu

Speech emotion classification using acoustic features

Emotion recognition from speech is a challenging research area with wide applications. In this paper we explore one of the key aspects of building an emotion recognition system: generating suitable feature representation. We extract features from four angles: (1) low-level acoustic features such as intensity, F0, jitter, shimmer and spectral contours etc. and statistical functions over these features, (2) a set of features derived from segmental cepstral-based features scored against emotion-dependent Gaussian mixture models, (3) a set of features derived from a set of low-level acoustic codewords and (4) GMM supervectors constructed by stacking the means or covariance or weights of the adapted mixture components on each utterance. We apply these features for emotion recognition independently and jointly and compare their performance within this task. We build a support vector machine (SVM) classifier based on these features on the IEMOCAP database. The four-class emotion recognition accuracy of 71.9% of our system outperforms the previously reported best results on this dataset.

Source Separation Improves Music Emotion Recognition

Despite the impressive progress in music emotion recognition, it remains unclear what aspect of a song, i.e., singing voice and accompanied music, carries more emotional information. As an initial attempt to answer the question, we introduce source separation into a standard music emotion recognition system. This allows us to compare systems with and without source separation, and consequently reveal the influence of singing voice and accompanied music on emotion recognition. Classification experiments on a set of 267 songs with last.fm annotations verify the new finding that source separation improves song music emotion recognition.

A hybrid approach based on stochastic competitive Hopfield neural network and efficient genetic algorithm for frequency assignment problem

Graphical abstractDisplay Omitted HighlightsOur algorithm owns good adaptability to deal with some practical problems.Our algorithm obtains better or comparable performance than other algorithms.The solutions are helpful for hybrid between neural nets and evolutionary algorithms. This paper presents a hybrid efficient genetic algorithm (EGA) for the stochastic competitive Hopfield (SCH) neural network, which is named SCH-EGA. This approach aims to tackle the frequency assignment problem (FAP). The objective of the FAP in satellite communication system is to minimize the co-channel interference between satellite communication systems by rearranging the frequency assignment so that they can accommodate increasing demands. Our hybrid algorithm involves a stochastic competitive Hopfield neural network (SCHNN) which manages the problem constraints, when a genetic algorithm searches for high quality solutions with the minimum possible cost. Our hybrid algorithm, reflecting a special type of algorithm hybrid thought, owns good adaptability which cannot only deal with the FAP, but also cope with other problems including the clustering, classification, and the maximum clique problem, etc. In this paper, we first propose five optimal strategies to build an efficient genetic algorithm. Then we explore three hybridizations between SCHNN and EGA to discover the best hybrid algorithm. We believe that the comparison can also be helpful for hybridizations between neural networks and other evolutionary algorithms such as the particle swarm optimization algorithm, the artificial bee colony algorithm, etc. In the experiments, our hybrid algorithm obtains better or comparable performance than other algorithms on 5 benchmark problems and 12 large problems randomly generated. Finally, we show that our hybrid algorithm can obtain good results with a small size population.

Detecting Violence in Video using Subclasses

This paper attacks the challenging problem of violence detection in videos. Different from existing works focusing on combining multi-modal features, we go one step further by adding and exploiting subclasses visually related to violence. We enrich the MediaEval 2015 violence dataset by manually labeling violence videos with respect to the subclasses. Such fine-grained annotations not only help understand what have impeded previous efforts on learning to fuse the multi-modal features, but also enhance the generalization ability of the learned fusion to novel test data. The new subclass based solution, with AP of 0.303 and P100 of 0.55 on the MediaEval 2015 test set, outperforms the state-of-the-art. Notice that our solution does not require fine-grained annotations on the test set, so it can be directly applied on novel and fully unlabeled videos. Interestingly, our study shows that motion related features (MBH, HOG and HOF), though being essential part in previous systems, are seemingly dispensable. Data is available at http://lixirong.net/datasets/mm2016vsd

Adaptive Tag Selection for Image Annotation

Not all tags are relevant to an image, and the number of relevant tags is image-dependent. Although many methods have been proposed for image auto-annotation, the question of how to determine the number of tags to be selected per image remains open. The main challenge is that for a large tag vocabulary, there is often a lack of ground truth data for acquiring optimal cutoff thresholds per tag. In contrast to previous works that pre-specify the number of tags to be selected, we propose in this paper adaptive tag selection. The key insight is to divide the vocabulary into two disjoint subsets, namely a seen set consisting of tags having ground truth available for optimizing their thresholds and a novel set consisting of tags without any ground truth. Such a division allows us to estimate how many tags shall be selected from the novel set according to the tags that have been selected from the seen set. The effectiveness of the proposed method is justified by our participation in the ImageCLEF 2014 image annotation task. On a set of 2,065 test images with ground truth available for 207 tags, the benchmark evaluation shows that compared to the popular top-k strategy which obtains an F-score of 0.122, adaptive tag selection achieves a higher F-score of 0.223. Moreover, by treating the underlying image annotation system as a black box, the new method can be used as an easy plug-in to boost the performance of existing systems.

Semantic Concept Annotation For User Generated Videos Using Soundtracks

With the increasing use of audio sensors in user generated content (UGC) collections, semantic concept annotation from video soundtracks has become an important research problem. In this paper, we investigate reducing the semantic gap of the traditional data-driven bag-of-audio-words based audio annotation approach by utilizing the large-amount of wild audio data and their rich user tags, from which we propose a new feature representation based on semantic class model distance. We conduct experiments on the data collection from HUAWEI Accurate and Fast Mobile Video Annotation Grand Challenge 2014. We also fuse the audio-only annotation system with a visual-only system. The experimental results show that our audio-only concept annotation system can detect semantic concepts significantly better than does random guessing. The new feature representation achieves comparable annotation performance with the bag-of-audio-words feature. In addition, it can provide more semantic interpretation in the output. The experimental results also prove that the audio-only system can provide significant complementary information to the visual-only concept annotation system for performance boost and for better interpretation of semantic concepts both visually and acoustically.

Semantic Concept Annotation of Consumer Videos at Frame-Level Using Audio

With the increasing use of audio sensors in user generated content UGC collection, semantic concept annotation using audio streams has become an important research problem. Huawei initiates a grand challenge in the International Conference on Multimedia & Expo ICME 2014: Huawei Accurate and Fast Mobile Video Annotation Challenge. In this paper, we present our semantic concept annotation system using audio stream only for the Huawei challenge. The system extracts audio stream from the video data and low-level acoustic features from the audio stream. Bag-of-feature representation is generated based on the low-level features and is used as input feature to train the support vector machine SVM concept classifier. The experimental results show that our audio-only concept annotation system can detect semantic concepts significantly better than random guess. It can also provide important complementary information to the visual-based concept annotation system for performance boost.

Detecting semantic concepts in consumer videos using audio

With the increasing use of audio sensors in user generated content collection, how to detect semantic concepts using audio streams has become an important research problem. In this paper, we present a semantic concept annotation system using soundtracks/ audio of the video. We investigate three different acoustic feature representations for audio semantic concept annotation and explore fusion of audio annotation with visual annotation systems. We test our system on the data collection from HUAWEI Accurate and Fast Mobile Video Annotation Grand Challenge 2014. The experimental results show that our audio-only concept annotation system can detect semantic concepts significantly better than random guess. It can also provide significant complementary information to the visual-based concept annotation system for performance boost. Further detailed analysis shows that for interpreting a semantic concept both visually and acoustically, it is better to train concept models for the visual system and audio system using visual-driven and audio-driven ground truth separately.

A Novel Hybrid SCH-ABC Approach for the Frequency Assignment Problem

This paper proposes an efficient hybrid approach based on the stochastic competitive Hopfield neural networkSCHNN and artificial bee colony ABC, which named SCH-ABC. The hybrid algorithm aims to cope with the frequency assignment problem FAP. The objective of FAP is to minimize the cochannel interference between satellite communication systems by rearranging the frequency assignments so that they can accommodate the increasing demands. In fact, as our SCH-ABC algorithm owns good adaptability, it can not only deal with the frequency assignment problem, but also cope with other problems including the clustering, classification, the maximum clique problem etc. With the help of hybridization, SCH-ABC makes up for the defects in the Hopfield neural network and ABC while fully utilizing the advantages of the two algorithms.

SCH-EGA: An Efficient Hybrid Algorithm for the Frequency Assignment Problem

This paper proposes a hybrid stochastic competitive Hopfield neural network-efficient genetic algorithm (SCH-EGA) approach to tackle the frequency assignment problem (FAP). The objective of FAP is to minimize the cochannel interference between satellite communication systems by rearranging the frequency assignments so that they can accommodate the increasing demands. In fact, as SCH-EGA algorithm owns the good adaptability, it can not only deal with the frequency assignment problem, but also cope with the problems of clustering, classification, the maximum clique problem and so on. In this paper, we first propose five optimal strategies to build an efficient genetic algorithm(EGA) which is the component of our hybrid algorithm. Then we explore different hybridizations between the Hopfield neural network and EGA. With the help of hybridization, SCH-EGA makes up for the defects in the Hopfield neural network and EGA while fully using the advantages of the two algorithms.