Sanghoon Jun

Music emotion classification and context-based music recommendation

Context-based music recommendation is one of rapidly emerging applications in the advent of ubiquitous era and requires multidisciplinary efforts including low level feature extraction and music classification, human emotion description and prediction, ontology-based representation and recommendation, and the establishment of connections among them. In this paper, we contributed in three distinctive ways to take into account the idea of context awareness in the music recommendation field. Firstly, we propose a novel emotion state transition model (ESTM) to model human emotional states and their transitions by music. ESTM acts like a bridge between user situation information along with his/her emotion and low-level music features. With ESTM, we can recommend the most appropriate music to the user for transiting to the desired emotional state. Secondly, we present context-based music recommendation (COMUS) ontology for modeling user’s musical preferences and context, and for supporting reasoning about the user’s desired emotion and preferences. The COMUS is music-dedicated ontology in OWL constructed by incorporating domain-specific classes for music recommendation into the Music Ontology, which includes situation, mood, and musical features. Thirdly, for mapping low-level features to ESTM, we collected various high-dimensional music feature data and applied nonnegative matrix factorization (NMF) for their dimension reduction. We also used support vector machine (SVM) as emotional state transition classifier. We constructed a prototype music recommendation system based on these features and carried out various experiments to measure its performance. We report some of the experimental results.

Hashtag Recommendation Based on User Tweet and Hashtag Classification on Twitter

With the explosive popularity of various social network services (SNSs), an enormous number of user documents are generated and shared daily by users. Considering the volume of user documents, efficient methods for grouping or searching relevant user documents are required. In the case of Twitter, self-defined metadata called hashtags are attached to tweets for that purpose. However, due to the wide scope of hashtags, users are having difficulty in finding out appropriate hashtags for their tweets. In this paper, we propose a new hashtag recommendation scheme for user tweets based on user tweet analysis and hashtag classification. More specifically, we extract keywords from user tweets using TF-IDF and classify their hashtags into pre-defined classes using Naive Bayes classifier. Next, we select a user interest class based on keywords of user tweets to reflect user interest. To recommend appropriate hashtags to users, we calculate the ranks of candidate hashtags by considering similar tweets, user interest and popularity of hashtags. To show the performance of our scheme, we developed an Android application named “TWITH” and evaluate its recommendation accuracy. Through various experiments, we show that our scheme is quite effective in the hashtag recommendation.

Music Retrieval and Recommendation Scheme Based on Varying Mood Sequences

A typical music clip consists of one or more segments with different moods and such mood information could be a crucial clue for determining the similarity between music clips. One representative mood has been selected for music clip for retrieval, recommendation or classification purposes, which often gives unsatisfactory result. In this paper, the authors propose a new music retrieval and recommendation scheme based on the mood sequence of music clips. The authors first divide each music clip into segments through beat structure analysis, then, apply the k-medoids clustering algorithm for grouping all the segments into clusters with similar features. By assigning a unique mood symbol for each cluster, one can transform each music clip into a musical mood sequence. For music retrieval, the authors use the Smith-Waterman SW algorithm to measure the similarity between mood sequences. However, for music recommendation, user preferences are retrieved from a recent music playlist or user interaction through the interface, which generates a music recommendation list based on the mood sequence similarity. The authors demonstrate that the proposed scheme achieves excellent performance in terms of retrieval accuracy and user satisfaction in music recommendation.

Classification and indexing scheme of large-scale image repository for spatio-temporal landmark recognition

In this paper, we propose a classification and indexing scheme of large-scale image repository for spatio-temporal landmark recognition using the local features, GPS data and user tags of images. For spatio-temporal landmark image classification, we first divide Earths entire surface into unit grid cells and collect pictures taken in each cell through Flickr. The collected images contain information such as location, titles and other user tags. Usually, the titles or user tags of landmark images include landmark names. Hence, by analyzing such tags, we can identify promising landmark names in the region and create a collection of images for each landmark using Flickr API. Even though each landmark class contains images of the same landmark, their spatio-temporal features could be different depending on shooting time, distance or angle. Therefore, we further divide the images in each landmark class into several subclasses according to their spatio-temporal characteristics using their color and local features. Especially, we detect the interest points of the images in the class, construct their feature descriptors using SURF and perform statistical analysis to select their representative points. Similar representative points are merged for fast comparison. Finally, we construct an index on the representative points using k-d tree. To identify the landmark in a user query image, we extract its SURF features and search for them in the index. Most similar matches are returned, along with descriptive text and GPS information. We implemented a prototype system based on a client-server architecture and performed various experiments to demonstrate that our scheme can achieve reasonable precision and scalability and provide a new browsing experience to the user.

TrendsSummary: a platform for retrieving and summarizing trendy multimedia contents

With the flood and popularity of various multimedia contents on the Internet, searching for appropriate contents and representing them effectively has become an essential part for user satisfaction. So far, many contents recommendation systems have been proposed for this purpose. A popular approach is to select hot or popular contents for recommendation using some popularity metric. Recently, various social network services (SNSs) such as Facebook and Twitter have become a widespread social phenomenon owing to the smartphone boom. Considering the popularity and user participation, SNS can be a good source for finding social interests or trends. In this study, we propose a platform called TrendsSummary for retrieving trendy multimedia contents and summarizing them. To identify trendy multimedia contents, we select candidate keywords from raw data collected from Twitter using a syntactic feature-based filtering method. Then, we merge various keyword variants based on several heuristics. Next, we select trend keywords and their related keywords from the merged candidate keywords based on term frequency and expand them semantically by referencing portal sites such as Wikipedia and Google. Based on the expanded trend keywords, we collect four types of relevant multimedia contents—TV programs, videos, news articles, and images—from various websites. The most appropriate media type for the trend keywords is determined based on a naïve Bayes classifier. After classification, appropriate contents are selected from among the contents of the selected media type. Finally, both trend keywords and their related multimedia contents are displayed for effective browsing. We implemented a prototype system and experimentally demonstrated that our scheme provides satisfactory results.

A Similar Music Retrieval Scheme Based on Musical Mood Variation

Music evokes various human emotions or creates music moods through low level musical features. In fact, typical music consists of one or more moods and this can be used as an important factor for determining the similarity between music. In this paper, we propose a new music retrieval scheme based on the mood change pattern. For this, we first divide music clips into segments based on low level musical features. Then, we apply K-means clustering algorithm for grouping them into clusters with similar features. By assigning a unique mood symbol for each group, each music clip can be represented into a sequence of mood symbols. Then, we estimate the similarity of music based on the similarity of their musical mood sequence using the Longest Common Subsequence (LCS) algorithm. To evaluate the performance of our scheme, we carried out various experiments and measured the user evaluation. We report some of the results.

Music structure analysis using self-similarity matrix and two-stage categorization

Music tends to have a distinct structure consisting of repetition and variation of components such as verse and chorus. Understanding such a music structure and its pattern has become increasingly important for music information retrieval (MIR). Thus far, many different methods for music segmentation and structure analysis have been proposed; however, each method has its advantages and disadvantages. By considering the significant variations in timbre, articulation and tempo of music, this is still a challenging task. In this paper, we propose a novel method for music segmentation and its structure analysis. For this, we first extract the timbre feature from the acoustic music signal and construct a self-similarity matrix that shows the similarities among the features within the music clip. Further, we determine the candidate boundaries for music segmentation by tracking the standard deviation in the matrix. Furthermore, we perform two-stage categorization: (i) categorization of the segments in a music clip on the basis of the timbre feature and (ii) categorization of segments in the same category on the basis of the successive chromagram features. In this way, each music clip is represented by a sequence of states where each state represents a certain category defined by two-stage categorization. We show the performance of our proposed method through experiments.

Social mix: automatic music recommendation and mixing scheme based on social network analysis

General preferences for music change over time. Moreover, music preferences depend on diverse factors, such as language, people, location, and culture. This dependency should be carefully considered to provide satisfactory music recommendations. Presently, typical music recommendations simply involve providing a list of songs that are then played sequentially or randomly. Recently, there has been an increasing demand for new music recommendation and playback methods. In this paper, we propose a scheme for recommending music automatically by considering both personal and general musical predilections, and for blending such music into a mixed clip for seamless playback. For automatic music recommendations, we first analyze social networks to identify a general predilection for certain music genres that depends on time and location. Songs that are generally preferred within a certain time period and location are identified through statistical analysis. This is done by analyzing, filtering, and storing massive social network streams into our own database in real time. In addition, a personal predilection for certain music genres can be inferred by analyzing similar user relationships in social network services. We selected such music based on instant graphs that are generated by user relationships and underlying music information. After the songs are selected, an automatic music mixing method is used to blend those songs into a continuous music clip. We implemented a prototype system and experimentally confirmed that our scheme provides satisfactory results.

Forecasting power consumption for higher educational institutions based on machine learning

Electric power consumption is affected by diverse factors. In particular, a university campus, which is one of the highest power consuming institutions, shows a wide variation of electric load depending on time and environment. For stable operation of such institution, reliable electric power supply should be guaranteed. One of possible methods to do that is to forecast the electric load accurately and supply power accordingly. Even though various influencing factors of power consumption have been discovered for educational institutions by analyzing power consumption patterns and usage cases, further studies are required for the quantitative forecasting of their electric load. In this paper, we build a power consumption forecasting model using various machine learning algorithms. To evaluate their effectiveness, we consider four building clusters in a university and collect their power consumption data of 15-min interval over more than one year. For the data, we first extract features based on the periodic characteristic and then perform the principal component analysis and factor analysis for the features. We build two electric load forecasting models using artificial neural network and support vector regression. We evaluate the prediction performance of each forecasting model by 5-fold cross-validation and compare the prediction result to actual electric load. The experimental results show that the two forecasting models can achieve average error rate of 3.46–10 % for all clusters.

Music segmentation and summarization based on self-similarity matrix

In this paper, we propose a new method for segmenting and summarizing music based on its structure analysis. To do that, we first extract timbre feature from acoustic music signal and construct a self-similarity matrix that shows similarities among the features within music clip. We then determine candidate boundaries for music segmentation by tracking standard deviation in the matrix. Similar segments such as repetition in music clip are clustered and merged. In this way, each music clip can be represented by a sequence of states where each state represents a music segment with similar feature. We assume that the longest segment of a music clip represents the music and hence use it as a summary of the music clip. We show the performance of our proposed method through experiments.