Ilkka Juuso

MORE --- a multimodal observation and analysis system for social interaction research

The MORE system is designed for observation and machine-aided analysis of social interaction in real life situations, such as classroom teaching scenarios and business meetings. The system utilizes a multichannel approach to collect data whereby multiple streams of data in a number of different modalities are obtained from each situation. Typically the system collects a 360-degree video and audio feed from multiple microphones set up in the space. The system includes an advanced server backend component that is capable of performing video processing, feature extraction and archiving operations on behalf of the user. The feature extraction services form a key part of the system and rely on advanced signal analysis techniques, such as speech processing, motion activity detection and facial expression recognition in order to speed up the analysis of large data sets. The provided web interface weaves the multiple streams of information together, utilizes the extracted features as metadata on the audio and video data and lets the user dive into analyzing the recorded events. The objective of the system is to facilitate easy navigation of multimodal data and enable the analysis of the recorded situations for the purposes of, for example, behavioral studies, teacher training and business development. A further unique feature of the system is its low setup overhead and high portability as the lightest MORE setup only requires a laptop computer and the selected set of sensors on site.

Simulation of the Complex System of Speech Interaction: Digital Visualizations

Computer simulation is the only practical way to model diffusion of cultural features, including speech. We describe the use of a cellular automaton to model feature diffusion as the adaptive aspect of the complex system of speech. Throughout hundreds of iterations that correspond to the daily interaction of speakers across time, we can watch regional distributional patterns emerge as a consequence of simple update rules. A key feature of our simulations is validation with respect to distributions known to occur in survey data. We focus on the importance of appropriate visualizations to observe what is happening during the process of diffusion, with comparison between visualizations of actual survey data and visualizations applied to our simulation. In this way we believe that we are breaking new ground in simulation of cultural interactions as complex systems. The study of speech as a complex system addresses language as an aspect of culture that emerges from human interaction. We believe that successful simulation of speech in cultural interaction as a complex system can suggest how other aspects of the humanities, such as sites or artifacts or styles in archaeology, can diffuse and change across space and time. Our successful simulation confirms our complex systems approach, and indicates how appropriate use of visualizations makes this possible.

Defining an architecture for evolving environments

The architecture of a system specifies how the system should be designed and built. However, shortcomings are identified in current architecture process frameworks concerning evolving domains like healthcare. We claim that an iterative architecture process is required, where the technical concerns are separated from the non-technical ones. Furthermore, a strong guiding vision is required. Based on our experiences from a biobank IT infrastructure process, we present an architecture process that is modular, interoperable, controlled and abstracted, thus being capable of handling complex systems with large uncertainties.

Creation of Regions for Dialect Features Using a Cellular Automaton

An issue in dialect research has been how to make generalizations from survey data about where some dialect feature might be found. Precomputational methods included drawing isoglosses or using shadings to indicate areas where an analyst expected a feature to be found. The use of computers allowed for faster plotting of locations where any given feature had been elicited, and also allowed for the use of statistical techniques from technical geography to estimate regions where particular features might be found. However, using the computer did not make the analysis less subjective than isoglosses, and statistical methods from technical geography have turned out to be limited in use. We have prepared a cellular automaton (CA) for use with data collected for the Linguistic Atlas Project that can address the problems involved in this type of data visualization. The CA plots the locations where survey data were elicited, and then through the application of rules creates an estimate of the spatial distributions of selected features. The application of simple rules allows the CA to create objective and reproducible estimates based on the data it was given, without the use of statistical methods.

Cellular Automata for Modeling Language Change

This paper describes the use of cellular automata to model dialect feature diffusion as the adaptive aspect of the complex system of speech. We show how a feature, once established, can spread across an area, and how the distribution of a dialect feature as it stands in Linguistic Atlas data could either spread or diminish. Throughout hundreds of iterations, we can watch regional and social distribution patterns emerge as a consequence of update rules. We validate patterns with respect to the linguistic distributions known to occur in the Linguistic Atlas Project.