Ching-Hua Chuan

American Sign Language Recognition Using Leap Motion Sensor

In this paper, we present an American Sign Language recognition system using a compact and affordable 3D motion sensor. The palm-sized Leap Motion sensor provides a much more portable and economical solution than Cyblerglove or Microsoft kinect used in existing studies. We apply k-nearest neighbor and support vector machine to classify the 26 letters of the English alphabet in American Sign Language using the derived features from the sensory data. The experiment result shows that the highest average classification rate of 72.78% and 79.83% was achieved by k-nearest neighbor and support vector machine respectively. We also provide detailed discussions on the parameter setting in machine learning methods and accuracy of specific alphabet letters in this paper.

Polyphonic Audio Key Finding Using the Spiral Array CEG Algorithm

Key finding is an integral step in content-based music indexing and retrieval. In this paper, we present an O(n) real-time algorithm for determining key from polyphonic audio. We use the standard Fast Fourier Transform with a local maximum detection scheme to extract pitches and pitch strengths from polyphonic audio. Next, we use Chews Spiral Array Center of Effect Generator (CEG) algorithm to determine the key from pitch strength information. We test the proposed system using Mozarts Symphonies. The test data is audio generated from MIDI source. The algorithm achieves a maximum correct key recognition rate of 96% within the first fifteen seconds, and exceeds 90% within the first three seconds. Starting from the extracted pitch strength information, we compare the CEG algorithms performance to the classic Krumhansl-Schmuckler (K-S) probe tone profile method and Temperleys modified version of the K-S method. Correct key recognition rates for the K-S and modified K-S methods remain under 50% in the first three seconds, with maximum values of 80% and 87% respectively within the first fifteen seconds for the same test set. The CEG method consistently scores higher throughout the fifteen-second selections.

Generating and evaluating musical harmonizations that emulate style

In this article we present a hybrid approach to the design of an automatic, style-specific accompaniment system that combines statistical learning with a music-theoretic framework, and we propose quantitative methods for evaluating the results of machine-generated accompaniment. The system is capable of learning accompaniment style from sparse input information, and of capturing style over a variety of musical genres. Generating accompaniments involves several aspects, including choosing chords, determining the bass line, arranging chords for voicing, instrumentation, etc. In this article we focus on harmonization: selecting chords for melodies, with an emphasis on style. Given exemplar songs as MIDI melodies with corresponding chords labeled as text, the system uses decision trees to learn the melody–chord relations shared among the songs. Markov chains on the neo-Riemannian framework are applied to model the likelihood of chord patterns. Harmonization is generated in a divide-and-conquer manner: Melody fragments that strongly imply certain triads are designated as checkpoints that are in turn connected by chord progressions generated using the Markov model. Chord subsequences are then refined and combined to form the final sequence. We propose two types of measures to quantify the degree to which a machine-generated accompaniment achieves its style emulation goal: one based on chord distance, and the other based on the statistical metrics entropy and perplexity. Using these measures, we conduct two sets of experiments using Western popular songs. Two albums by each of three artists (Green Day, Keane, and Radiohead), for a total of six albums, are used to evaluate the

Audio Properties of Perceived Boundaries in Music

Data mining tasks such as music indexing, information retrieval, and similarity search, require an understanding of how listeners process music internally. Many algorithms for automatically analyzing the structure of recorded music assume that a large change in one or another musical feature suggests a section boundary. However, this assumption has not been tested: while our understanding of how listeners segment melodies has advanced greatly in the past decades, little is known about how this process works with more complex, full-textured pieces of music, or how stable this process is across genres. Knowing how these factors affect how boundaries are perceived will help researchers to judge the viability of algorithmic approaches with different corpora of music. We present a statistical analysis of a large corpus of recordings whose formal structure was annotated by expert listeners. We find that the acoustic properties of boundaries in these recordings corroborate findings of previous perceptual experiments. Nearly all boundaries correspond to peaks in novelty functions, which measure the rate of change of a musical feature at a particular time scale. Moreover, most of these boundaries match peaks in novelty for several features at several time scales. We observe that the boundary-novelty relationship can vary with listener, time scale, genre, and musical feature. Finally, we show that a boundary profile derived from a collection of novelty functions correlates with the estimated salience of boundaries indicated by listeners.

Cache management of dynamic source routing for fault tolerance in mobile ad hoc networks

Mobile ad hoc networks have gained more and more research attention. They provide wireless communications without location limitations and pre-built fixed infrastructures. Because of the absence of any static support structure, ad hoc networks are prone to link failure. This has become the most serious cause of throughput degradation when using TCP over ad hoc networks. Some researchers chose dynamic source routing (DSR) as the routing protocol and showed that disabling the assignment of a route directly from cache gives better performance. We introduce an efficient cache management mechanism to increase the TCP throughput by replying with a route directly from the cache of DSR and perform cache recovery when a host failure has occurred. We use simulations to compare the performance of our algorithm with the original DSR under the link failure prone environment due to mobility. We also provide the simulation results when host failures are considered in the ad hoc networks.

Indexing and Retrieving Continuations in Musical Time Series Data Using Relational Databases

This paper proposed and tested a model that provides quick search and retrieval of continuations for time series, particularly musical data, using relational databases. The model extends an existing interactive music-generation system by focusing on large input sequences. Experiments using textural and musical data provided satisfactory performance results for the model.

A Multi-modal Platform for Semantic Music Analysis: Visualizing Audio-and Score-Based Tension

Musicologists, music cognition scientists and others have long studied music in all of its facets. During the last few decades, research in both score and audio technology has opened the doors for automated, or (in many cases) semi-automated analysis. There remains a big gap, however, between the field of audio (performance) and score-based systems. In this research, we propose a web-based Interactive system for Multi-modal Music Analysis (IMMA), that provides musicologists with an intuitive interface for a joint analysis of performance and score. As an initial use-case, we implemented a tension analysis module in the system. Tension is a semantic characteristic of music that directly shapes the music experience and thus forms a crucial topic for researchers in musicology and music cognition. The module includes methods for calculating tonal tension (from the score) and timbral tension (from the performance). An audio-to-score alignment algorithm based on dynamic time warping was implemented to automate the synchronization between the audio and score analysis. The resulting system was tested on three performances (violin, flute, and guitar) of Paganinis Caprice No. 24 and four piano performances of Beethovens Moonlight Sonata. We statistically analyzed the results of tonal and timbral tension and found correlations between them. A clustering algorithm was implemented to find segments of music (both within and between performances) with similar shape in their tension curve. These similar segments are visualized in IMMA. By displaying selected audio and score characteristics together with musical score following in sync with the performance playback, IMMA offers a user-friendly intuitive interface to bridge the gap between audio and score analysis.

Designing SmartSignPlay: An Interactive and Intelligent American Sign Language App for Children who are Deaf or Hard of Hearing and their Families

This paper describes an interactive mobile application that aims to assist children who are deaf or hard of hearing (D/HH) and their families to learn and practice American Sign Language (ASL). Approximately 95% of D/HH children are born to hearing parents. Research indicates that the lack of common communication tools between the parent and child often results in delayed development in the childs language and social skills. Benefiting from the interactive advantages and popularity of touchscreen mobile devices, we created SmartSignPlay, an app to teach D/HH children and their families everyday ASL vocabulary and phrases. Vocabulary is arranged into context-based lessons where the vocabulary is frequently used. After watching the sign demonstrated by an animated avatar, the user performed the sign by drawing the trajectory of the hand movement and selecting the correct handshape. While the app is still under iterative development, preliminary results on the usability are provided.

Audio Classification and Retrieval Using Wavelets and Gaussian Mixture Models

This paper presents an audio classification and retrieval system using wavelets for extracting low-level acoustic features. The author performed multiple-level decomposition using discrete wavelet transform to extract acoustic features from audio recordings at different scales and times. The extracted features are then translated into a compact vector representation. Gaussian mixture models with expectation maximization algorithm are used to build models for audio classes and individual audio examples. The system is evaluated using three audio classification tasks: speech/music, male/female speech, and music genre. They also show how wavelets and Gaussian mixture models are used for class-based audio retrieval in two approaches: indexing using only wavelets versus indexing by Gaussian components. By evaluating the system through 10-fold cross-validation, the author shows the promising capability of wavelets and Gaussian mixture models for audio classification and retrieval. They also compare how parameters including frame size, wavelet level, Gaussian components, and sampling size affect performance in Gaussian models.

Harmonic style-based song retrieval using N-gram

N-gram models have been successfully applied to harmonic analysis for differentiating a composers style based on all the pieces in a large corpus of the composer. In this paper, we focus on each individual song and explore the effectiveness of the n-gram model when it is applied to a different but equally important musical task: harmonic style-based song retrieval. A chord profile is generated for a song by using the n-gram model as the descriptor of the songs harmonic features. The system retrieves songs based on the similarity between the chord profile in the query and that of the songs in the database. The retrieval result is evaluated in terms of a style-based retrieval score as well as traditional information retrieval metrics such as precision and recall. Finally, we list the most common pop-rock chord patterns from the most frequently retrieved songs, and compare the patterns with those described in previous works.