Herng-Yow Chen

MultiSync: a synchronization model for multimedia systems

Synchronization among various media sources is one of the most important issues in multimedia communications and various audio/video (A/V) applications. For continuous playback (such as lip synchronization) under a time-sharing multiprocessing operating system (such as UNIX), the synchronization quality of traditional synchronization mechanisms employed on single processes may vary according to the workload of the system. When the system encounters an overload situation, the synchronization usually fails and, even worse, results in two fatal defects in human perception: the audio discontinuity (audio break) and the out-of-synchronization (synchronization anomaly). In order to overcome these problems, a novel media synchronization model employed on multiple processes (or threads) in a multiprocessing environment is proposed. The problem of asynchronism due to system overload is solved by assigning a higher priority to more important media and adopting a delay-or-drop policy to treat the lower priority ones. Some experimental results are presented to show the effectiveness of the proposed model and the implementation mechanisms under a UNIX, X-Windows environment. On the basis of the proposed model, a continuous media playback (CMP) module, which acted as the key component of some popular multimedia systems such as multimedia authoring system, multimedia E-mail system, multimedia bulletin board system (BBS), and video-on-demand (VoD) System, was implemented.

The MOS multimedia E-mail system

We present the architecture of a new multimedia E-mail system, which has been successfully developed in the Communication and Multimedia Laboratory of National Taiwan University. This prototype E-mail system is fully compatible with traditional Internet E-mails in that the multimedia E-mails can be sent through Internet. For allowing one to compose multimedia E-mails, the system provides various editors for editing video, audio, image, graphics, and multilingual (English and Chinese) text. It is currently implemented on SUN workstations with X-windows and OpenLook as its standard graphical user interface. In order to display video, the workstations are equipped with either an add-on video board or a software-based codecs. Since its development, the system have been widely used in the National Taiwan University. Experience of users showed that the system is convenient to use and is indeed fully compatible with the traditional Internet E-mails.<<ETX>>

A digital museum of Taiwanese butterflies

Taiwan is renown for its great variety of butterflies. There are about 400 species, a number of which unique to Taiwan, over its 36,500 sq km land. Last year we built a comprehensive digital collection of Taiwans butterflies to provide a modern research environment on butterflies for academic institutions, as well as an interactive butterfly educational environment for the general public. Our digital museum emphasizes on the ease to use, and provides a number of innovative features to help the user fully utilize the information provided by the system. The digital museum is accessible through the Web at http://digimuse.nmns.edu.tw.

A multi-layer video browsing system

The paper proposes an effective and efficient browsing system which is capable of representing the contents of a video sequence in a multi-layer manner. In this system, users can rapidly preview some preprocessed video shots, based on some simple scene change detection algorithm, to find out the particular video segments which are of interest. Because users just preview some video shots instead of examining all the video frames, this shot-based browsing method is superior to the conventional frame-based browsing one from the view points of the network bandwidth, user friendliness, and/or browsing efficiency. Based on the proposed browsing tool, a video-on-demand (VOD) prototype system has been implemented in the Communication and Multimedia Laboratory of National Taiwan University in an Ethernet LAN environment. >

Exploring media correlation and synchronization for navigated hypermedia documents

This paper is devoted to explore media correlation and media synchronization in a composite multimedia document, the so-called navigated hypermedia document in our language learning system, to facilitate the multimedia authoring, presentation, and access. Two levels of media correlation in temporal, spatial, and content domains are investigated: syntactic level correlation and semantic level correlation. We devise a capturing mechanism to record all the media streams and relations between them, including voice and event streams, for replaying the lecturing in a form as close as possible to the original classroom experience. The syntactic level correlation is based on specific timestamps of the media stream and used to reconstruct the recorded lecture for synchronized presentation. Furthermore, to integrate media objects with specific segments within the media stream, some computed synchronization processes are required to discover semantic content of the media. The proposed computed synchronization techniques, including speech-event binding process for temporal domain, tele-pointer (i.e. cursor) movement interpolation and adaptable handwriting presentation for spatial domain, and erasing handling for content domain, will be addressed. Experimental results show that in the speech-event binding process 74% of speech access entries for accessible visualized events are found. The acceptable rate of human perception on tele-pointer movement is higher than 85% if time interval is selected carefully. Finally, the accuracy of erasing handling for content removing is about 71%.

Toward better retrieval and presentation by exploring cross-media correlations

Abstract.This paper addresses explicit correlation and implicit correlation between various media streams in a composite multimedia document, the so-called navigated hypermedia document in our language learning system, in order to facilitate document retrieval and synchronized presentation. For replaying a recorded lecture in a form as close as possible to the original classroom experience, we devised a capturing mechanism to explicitly record all the lecturing media streams and relations between them, including instructor’s voice, slide change of the HTML lectures, and various guiding actions (e.g., tele-pointers, pen strokes, document scrolling, keyword highlighting, and text annotations) on HTML-based slides. In addition, for more effective learning, we study three different aspects - temporal, spatial, and content relation - of the implicit correlations that are inherently hidden between the media involved. The implicit relations are discovered by three designed processes: the speech-text alignment process for temporally synchronized speech-text presentation, the automatic scrolling process for the viewing window’s spatial synchronization, and the content dependency checking process to ensure consistency of the content processed and the relations involved. The experimental results show that exploring cross-media correlations is helpful for system development in document presentation and retrieving. Users are allowed to replay a vivid and learning-effective multimedia lecture and to access the desired part of the document very easily via cross-media indexing. Hence the results have been applied to the development of online multimedia language learning systems aimed at improving students’ English and Chinese language capabilities.

Cross-media correlation: a case study of navigated hypermedia documents

The research issues on multiple media correlation have arisen with more and more integrated multimedia applications. The multimedia correlation is used to coordinate different media and facilitate cross-media access. This paper presents our work on two types of multimedia correlation: explicit and implicit relations. We develop a system to carefully capture explicit relations and devise some computed synchronization processes to discover implicit relations between media objects. The proposed computed synchronization techniques, including speech-text alignment process in temporal domain, automatic scrolling process in spatial domain, and content dependency check process in content domain, will be addressed. Experimental results show that in the speech-text alignment process 80% of forced alignment are in-sync even the speech recognition accuracy is as low as 25%. The automatic scrolling process effectively maintains a resynchronization mechanism in different displaying environments.

An RTP-based synchronized hypermedia live lecture system for distance education

In this article, we have introduced a “Live Synchronized Hypermedia Live Lecture (SHLL) System” using RTP to synchronize the live presentation of streaming video lecture, HTML-based lecture notes, and HTML page Navigation Events. The SHLL framework consists of three major modules: (1) SHLL Recorder- for recording the temporal information of the AV lecture and the HTML-based lecture notes navigation processes. (2) SHLL Event Server- for receiving, depositing, and multicasting SHLL events. (3) SHLL Browser- for presentation of the synchronized AV lecture and HTML-based lecture notes navigation. To manage the synchronization presentation of different media, we have proposed an RTP-based Multi-Sync synchronization model, which account for the human perception factors. To evaluate the performance of the proposed SHLL framework and synchronization model, a RealSystem-based prototype Synchronized HTML-AV Distance Lecture System has been implemented using Java/JavaScript and C. The prototype system certifies the feasibility of the proposed framework for synchronized hypermedia live multicasting.

Real-Time Pitch Training System for Violin Learners

This paper specifically targets violin learners who are working on their pitch accuracy. We employ a pitch tracking algorithm to extract the pitch played. Through volume thresholding and region detection, only parts of frames are processed. So our system can provide real-time feedback to show violin learners whether they played the right pitch. The system also provides the major scales and arpeggio scores as teaching materials, and violin learners can choose different tempos to practice, depending on their level. The user-friendly system interface allows violin learners to easily perceive the pitch differential between the pitch of the target note and the pitch played, allowing users to precisely adjust their playing. The statistical feedback records progress and analyzes error patterns, enabling violin teachers to evaluate student progress precisely, and correct common error patterns effectively.

The WSML system: web-based synchronization multimedia lecture system

This demonstration presents a web-based multimedia lecture system that perfectly integrates multimedia lecturing with static HTML pages and enables their presentation in synchronization. We develop key techniques to reproduce vivid web-teaching scenarios for on-demand access, including events capturing scheme and synchronized mechanism. To facilitate multimedia lectures creation, the easy-to-use authoring and managing tools specially designed for teachers have been also included in the development. In addition, integrated presentation and efficient access issues are also discussed for the students aspect.