Simon Lui

JoggAR: a mixed-modality AR approach for technology-augmented jogging

JoggAR demonstrates a novel combination of wearable visual, audio and sensing technology to realize a game-like persistent augmented reality (AR) environment to enhance jogging and other exertion experiences that involves changing attention intensities in the course of the activities. In particular we developed a method to perform an audio-first exploration of 3D virtual spaces so as to achieve our experiential goal of supporting exertion-focused activities.

Robust phoneme classification for automatic speech recognition using hybrid features and an amalgamated learning model

Phoneme recognition is an important aspect of speech processing and recognition. Research on phoneme recognition is several years old and numerous algorithms have been developed over the years to improve its accuracy. In this paper, a quantitative analysis of phoneme recognition using supervised learning is investigated. Most approaches to phoneme recognition rely on using mel frequency cepstrum based features for identification of the phoneme class. In our approach, we take into consideration the vocal tract area function along with mel frequency cepstrum coefficients and analyze the change in accuracy obtained by its introduction in the feature set. Support Vector Machines have been an attractive approach to pattern recognition and its usage as a supervised learning model has been popular in the speech processing community. We compare Support Vector Machines to other supervised learning models like the Naïve Bayes, the k-Nearest Neighbors and the linear discriminant analysis classifiers, for our feature set. We impose a soft voting rule between the three best classifiers to produce our variation of a voting classifier. We enhance the accuracy of our classifier by using a priority based approach to estimate the three most likely phonemes, after the predicted phoneme. Through a figurative and quantitative approach, we show that our modified algorithm outperforms other traditional methods. Experiments were conducted on the WSJCAM0 corpus, a British English corpus.

Perceptual evaluation of measures of spectral variance

In many applications, it is desirable to achieve a signal that is as close as possible to ideal white noise. One example is in the design of an artificial reverberator, whereby there is a need for its lossless prototype output from an impulse input to be perceptually white as much as possible. The Ljung-Box test, the Drouiche test, and the Wiener Entropy-also called the Spectral Flatness Measure-are three well-known methods for quantifying the similarity of a given signal to ideal white noise. In this paper, listening tests are conducted to measure the Just Noticeable Difference (JND) on the perception of white noise, which is the JND between ideal Gaussian white noise and noise with a specified deviation from the flat spectrum. This paper reports the JND values using one of these measures of whiteness, which is the Ljung-Box test. This paper finds considerable disagreement between the Ljung-Box test and the other two methods and shows that none of the methods is a significantly better predictor of listeners perception of whiteness. This suggests a need for a whiteness test that is more closely correlated to human perception.

Design exploration for the "squeezable" interaction

We explore hand squeezing as a possible alternative way in which a user could engage in as the primary interaction modality to interact with everyday IT devices today. In doing so, we work on constructing a conceptual design space as a practical tool to orient ourselves and systematically brainstorm design possibilities for squeezable interaction. The constructed design space abstracts essential characteristics of squeezing interaction including the intensity, frequency and areas of squeezing in a structured way, helping explain existing examples of squeezing interactions we engage in our daily lives. By offering a set of new concepts and vocabulary to express different aspects of this modality, the design space can be effectively used to help brainstorm, discuss and create novel applications and usage situations with squeezing as the main interaction modality, resulting in facilitating pioneering new lines of interactive IT applications. The construction and refinement of the design space was in part supported by instrumenting and prototyping squeezable applications and testing with twenty participants one by one.

Modeling Affective Responses to Music Using Audio Signal Analysis and Physiology

A key issue in designing personalized music affective applications is to find effective ways to direct emotion by music selection with appropriate combination of acoustic features. The aim of this study is to understand the dynamic relationships between acoustic features, physiology and affective states. To model these relationships we used a multivariate approach including continuous measures of emotions from behavioral, subjective and physiological responses. Classical music excerpts taken from opera overtures were used as stimuli to induce emotional variations across time between neutral and intense emotional states. Continuous ratings of arousal and valence along with cardiovascular, respiratory, skin conductance and facial expressive activity were recorded simultaneously. Results show that parts of the music with higher loudness and pulse clarity induced higher ratings of arousal, sympathetic activation and increased cardiorespiratory synchronization. In contrast, pleasant and calming parts with major mode and prominent key strength induced higher ratings of valence, parasympathetic activation and increased facial activity.

Sensor-enabled yo-yos as new musical instruments

An interactive and reprogrammable musical yo-yo system is designed. The aim of it is to demonstrate the feasibility of converting any sensor-enabled objects into potential musical instruments. This involves three design phases. First, the physical yo-yo is designed to house Iris sensors. The software is developed to sense the movement of yo-yo and transmit its measurements to Max/MSP for corresponding music generation. Finally, aurally pleasing and real-time musical sounds are designed and generated in effect of yo-yo by the computer music composer.

Demo abstract: Sensor-enabled Yo-yos as new musical instruments

ABSTRACT An interactive and reprogrammable musical yo-yo system is designed. The aim of it is to demonstrate the feasibility of converting any sensor-enabled objects into potential musical instruments. This involves three design phases. First, the physical yo-yo is designed to house Iris sensors. The software is developed to sense the movement of yo-yo and transmit its measurements to Max/MSP for corresponding music generation. Finally, aurally pleasing and real-time musical sounds are designed and generated in effect of yo-yo by the computer music composer.