Kia Ng

Automated reconstruction of 3D models from real environments

Abstract This paper describes an integrated approach to the construction of textured 3D scene models of building interiors from laser range data and visual images. This approach has been implemented in a collection of algorithms and sensors within a prototype device for 3D reconstruction, known as the EST (Environmental Sensor for Telepresence). The EST can take the form of a push trolley or of an autonomous mobile platform. The Autonomous EST (AEST) has been designed to provide an integrated solution for automating the creation of complete models. Embedded software performs several functions, including triangulation of the range data, registration of video texture, registration and integration of data acquired from different capture points. Potential applications include facilities management for the construction industry and creating reality models to be used in general areas of virtual reality, for example, virtual studios, virtualised reality for content-related applications (e.g., CD-ROMs), social telepresence, architecture and others. The paper presents the main components of the EST/AEST, and presents some example results obtained from the prototypes. The reconstructed model is encoded in VRML format so that it is possible to access and view the model via the World Wide Web.

Tracking Conductors Hand Movements Using Multiple Wiimotes

In this paper, we present the design and validation of a 3D tracking system using a commercially available game controller. The system uses the wiimote to track and analyze a conductorpsilas hand movement and the Wii Balance Board to track conductorpsilas balance shifts. The wiimote provides an easily attainable controller that has both accelerometer motion sensors and infrared tracking capabilities. We describe the multiple wiimotes to track an infrared marker in 3D using triangulation. The gesture acquisition system is designed to capture and analyze ldquoconductingrdquo gesture which may also be used in the study of musical expression and in an educational environment.

Preservation of interactive multimedia performances

Preservation of Interactive Multimedia Performances (IMPs) is becoming important, as they are getting more and more popular in the performing arts communities. Preservation requires keeping not only all the necessary components for the production of a performance but also the knowledge about these components so that the original production process can be repeated at any given time. Ontologies provide the means for semantic processing, cataloguing and querying on preserved digital objects. In this paper, a framework for digital preservation of IMPs is introduced. The framework is based on the use of ontologies for describing the complex relationships among different components of a performance to support its preservation.

Music Manuscript Tracing

This paper presents an ongoing project working on an optical handwritten music manuscript recognition system. A brief background of Optical Music Recognition (OMR) is presented, together with a discussion on some of the main obstacles in this domain. An earlier OMR prototype for printed music scores is described, with illustrations of the low-level pre-processing and segmentation routines, followed by a discussion on its limitations for handwritten manuscripts processing, which led to the development of a stroke-based segmentation approach using mathematical morphology. The pre-processing sub-systems consist of a list of automated processes, including thresholding, deskewing, basic layout analysis and general normalization parameters such as the stave line thickness and spacing. High-level domain knowledge enhancements, output format and future directions are outlined.

Sensing and mapping for interactive performance

This paper describes a trans-domain mapping (TDM) framework for translating meaningful activities from one creative domain onto another. The multi-disciplinary framework is designed to facilitate an intuitive and non-intrusive interactive multimedia performance interface that offers the users or performers real-time control of multimedia events using their physical movements. It is intended to be a highly dynamic real-time performance tool, sensing and tracking activities and changes, in order to provide interactive multimedia performances.From a straightforward definition of the TDM framework, this paper reports several implementations and multi-disciplinary collaborative projects using the proposed framework, including a motion and colour-sensitive system, a sensor-based system for triggering musical events, and a distributed multimedia server for audio mapping of a real-time face tracker, and discusses different aspects of mapping strategies in their context.Plausible future directions, developments and exploration with the proposed framework, including stage augmenta tion, virtual and augmented reality, which involve sensing and mapping of physical and non-physical changes onto multimedia control events, are discussed.

Automated 3D reconstruction of interiors with multiple scan-views

This paper presents two integrated solutions for realistic 3D model acquisition and reconstruction; an early prototype, in the form of a push trolley, and a later prototype in the form of an autonomous robot. The systems encompass all hardware and software required, from laser and video data acquisition, processing and output of texture-mapped 3D models in VRML format, to batteries for power supply and wireless network communications. The autonomous version is also equipped with a mobile platform and other sensors for the purpose of automatic navigation. The applications for such a system range from real estate and tourism (e.g., showing a 3D computer model of a property to a potential buyer or tenant) or as tool for content creation (e.g., creating 3D models of heritage buildings or producing broadcast quality virtual studios). The system can also be used in industrial environments as a reverse engineering tool to update the design of a plant, or as a 3D photo-archive for insurance purposes. The system is Internet compatible: the photo-realistic models can be accessed via the Internet and manipulated interactively in 3D using a common Web browser with a VRML plug-in. Further information and example reconstructed models are available on- line via the RESOLV web-page at http://www.scs.leeds.ac.uk/resolv/.

Musical Robots and Interactive Multimodal Systems

Musical robotics is a multi- and trans-disciplinary research area involving a wide range of different domains that contribute to its development, including: computer science, multimodal interfaces and processing, artificial intelligence, electronics, robotics, mechatronics and more. A musical robot requires many different complex systems to work together; integrating musical representation, techniques, expressions, detailed analysis and controls, for both playing and listening. The development of interactive multimodal systems provides advancements which enable enhanced human-machine interaction and novel possibilities for embodied robotic platforms. This volume is focused on this highly exciting interdisciplinary field. This book consists of 14 chapters highlighting different aspects of musical activities and interactions, discussing cutting edge research related to interactive multimodal systems and their integration with robots to further enhance musical understanding, interpretation, performance, education and enjoyment. It is dichotomized into two sections: Section I focuses on understanding elements of musical performance and expression while Section II concentrates on musical robots and automated instruments. Musical Robots and Interactive Multimodal Systems provides an introduction and foundation for researchers, students and practitioners to key achievements and current research trends on interactive multimodal systems and musical robotics.

A Monophonic Pitch-Tracking Algorithm Based on Waveform Periodicity Determinations Using Landmark Points

In many musical situations, currently available MIDI controllers prove inflexible or artificial to musicians unfamiliar with their ergonomic eccentricities. While the range of such devices has been extended beyond the piano-style keyboard to ones modeled on wind, string, and percussion instruments, they can still be frustrating to use because of their generalized nature. Our aim is to provide a means whereby conventional instruments can be used to control musical processes in a way that is transparent to the performer. Such a system must have a fast response time, be non-computationally intensive, and operate effectively on a lowspecification hardware platform. In this article, we propose an approach to monophonic pitch tracking that uses the pattern and shape of digitized sound waves to isolate the fundamental frequency, rather than the more common frequencyand time-domain methods. We discuss the strategy of segmentation of the sampled sound wave and a procedure for finding the shortest distance between two shape-similar repeating segments. A new composition that makes use of the algorithm, Philip Wilbys Dance Before the Lord for brass band and electronics, is presented as an illustration of the practical application of the algorithm.

Metadata Editing and Mapping within the AXMEDIS Framework

Metadata is increasingly used in the digital industry for content description. In line with this emerging trend, the AXMEDIS framework has employed several kinds of metadata for describing digital items embedded in AXMEDIS Objects. This paper introduces the metadata utilised inside AXMEDIS Objects and presents the AXMEDIS Metadata Editor and Mapper which provide graphical user interface and tools for metadata manipulation to accelerate the production and automated transcoding of metadata.

Big Data Optical Music Recognition with Multi Images and Multi Recognisers

In this paper we describe work in progress towards Multi-OMR, an approach to Optical Music Recognition (OMR) which aims to significantly improve the accuracy of musical score digitisation. There are a large number of scores available in public databases, as well as a range of different commercial and open source OMR tools. Using these resources, we are exploring a Big Data approach to harnessing datasets by aligning and combining the results of multiple versions of the same score, processed with multiple technologies. It is anticipated that this approach will yield high quality results, opening up large datasets to researchers in the field of digital musicology.