Markus Waltl

Improving the Quality of multimedia Experience through sensory effects

In previous and related work sensory effects are presented as a tool for increasing the user experience of multimedia presentations by stimulating also other senses than vision or audition. In this paper we primarily investigated the relationship of the Quality of Experience (QoE) due to various video bit-rates of multimedia contents annotated with sensory effects (e.g., wind, vibration, light). Therefore, we defined a subjective quality assessment methodology based on standardized methods. The paper describes the test environment, its setup, and conditions in detail. Furthermore, we experimented with a novel voting device that allows for continuous voting feedback during a sequence in addition to the overall quality voting at the end of each sequence. The results obtained from the subjective quality assessment are presented and discussed thoroughly. In anticipation of the results we can report an improvement of the quality of the multimedia experience thanks to the sensory effects.

A test-bed for quality of multimedia experience evaluation of Sensory Effects

This paper introduces a prototype test-bed for triggering sensory effects like light, wind, or vibration when presenting audiovisual resources, e.g., a video, to users. The ISO/IEC MPEG is currently standardizing the Sensory Effect Description Language (SEDL) for describing such effects. This language is briefly described in the paper and the testbed that is destined to evaluate the quality of the multimedia experience of users is presented. It consists of a video annotation tool for sensory effects, a corresponding simulation tool, and a real test system. Initial experiments and results on determining the color of light effects from the video content are reported.

Assessing the quality of sensory experience for multimedia presentations

This paper introduces the concept of sensory experience by utilizing sensory effects such as wind or lighting as another dimension which contributes to the quality of the user experience. In particular, we utilize a representation format for sensory effects that are attached to traditional multimedia resources such as audio, video, and image contents. Sensory effects (e.g., wind, lighting, explosion, heat, cold) are rendered on special devices (e.g., fans, ambient lights, motion chair, air condition) in synchronization with the traditional multimedia resources and shall stimulate other senses than audition and vision (e.g., mechanoreception, equilibrioception, thermoreception), with the intention to increase the users Quality of Experience (QoE). In particular, the paper provides a comprehensive introduction into the concept of sensory experience, its assessment in terms of the QoE, and related standardization and implementation efforts. Finally, we will highlight open issues and research challenges including future work.

Sensory Experience: Quality of Experience Beyond Audio-Visual

This chapter introduces the concept of Sensory Experience which aims to define the Quality of Experience (QoE) going beyond audio-visual content. In particular, we show how to utilize sensory effects such as ambient light, scent, wind, or vibration as additional dimensions contributing to the quality of the user experience. Therefore, we utilize a standardized representation format for sensory effects that are attached to traditional multimedia resources such as audio, video, and image contents. Sensory effects are rendered on special devices (e.g., fans, lights, motion chair, scent emitter) in synchronization with the traditional multimedia resources and shall stimulate also other senses than hearing and seeing with the intention to increase the Quality of Experience (QoE), in this context referred to as Sensory Experience.

A web based subjective evaluation platform

Preparing and conducting subjective quality assessments is a time consuming and expensive task. Therefore, we present a Web-based evaluation framework which aims on reducing the time needed for planning and designing a subjective quality assessment. The presented framework can be used for both crowdsourced and laboratory experiments. It should ease the task of designing a subjective quality assessment by providing a flexible framework. The framework has proven its applicability and flexibility to design and conduct assessments in the past and is available as open source.

A toolset for the authoring, simulation, and rendering of sensory experiences

This paper describes a toolset for the authoring, simulating, and rendering of multimedia content annotated with Sensory Effect Metadata (SEM) descriptions as specified in Part 3 of the MPEG V standard. This part of MPEG-V standardizes the description of sensory effects (e.g., light, wind) in order to be rendered on sensory devices (e.g., fans, vibration chairs) aiming at generating a sensory experience stimulating possibly all human senses. Our implementation comprises a toolset to author sensory effects associated with multimedia content and the simulation thereof. Furthermore, it includes a library, a standalone player, and a Web browser plug-in which enables the playback and rendering of sensory effects on off-the-shelf rendering devices and in various contexts. All software modules are available under the GNU General Public License (GPL) v3 and the GNU Lesser General Public License (LGPL) v3 respectively.

Sensory effect dataset and test setups

Additional constituents for the representation of multimedia content gained more and more attention. For example, the amount of cinemas equipped with additional devices (e.g., ambient light, vibrating seats, wind generators, water sprayers, heater/coolers) that stimulate senses going beyond audition and vision increases. On the content side the MPEG-V standard specifies - among others - Sensory Effect Metadata (SEM) which provides means to describe sensory effects such as wind, vibration, light, etc. to be attached to audio-visual content and, thus, offering an enhanced and immersive experience for the user. However, there is a lack of a common set of test content allowing for various subjective user studies and verification across different test sites. In this paper we provide our dataset comprising a number of videos from different genres enriched with MPEG-V compliant Sensory Effect Metadata descriptions. Furthermore, we describe possible test setups using off-the-shelf hardware for conducting subjective quality assessments.

Sensory effects for ambient experiences in the World Wide Web

More and more content in various formats becomes available via the World Wide Web (WWW). Currently available Web browsers are able to access and interpret these contents (i.e., Web videos, text, image, and audio). These contents stimulate only senses like audition or vision. Recently, it has been proposed to stimulate also other senses while consuming multimedia content, through so-called sensory effects. These sensory effects aim to enhance the ambient experience by providing effects such as light, wind, vibration, etc. The effects are represented as Sensory Effect Metadata (SEM) description which is associated to multimedia content and is rendered on devices like fans, vibration chairs, or lamps. In this paper we present two subjective quality assessments which comprise sensory effects, such as light, in the area of the WWW and their results achieved. The first assessment evaluates the influence of light effects on the Quality of Experience (QoE). The second assessment measures the impact of different settings for the color calculation on the viewing experience. Furthermore, we describe a Web browser plug-in for Mozilla Firefox which is able to render such sensory effects that are provided via the WWW.

Sensory Experience for Videos on the Web

More and more multimedia content is becoming available via the World Wide Web (WWW). These contents stimulate only senses like hearing or vision. Recently, it has been proposed to stimulate also other senses while consuming multimedia content, through so-called sensory effects. These sensory effects aim at enhancing the users viewing experience by providing effects such as light, wind, vibration, etc. These effects are represented as Sensory Effect Metadata (SEM) which is associated to multimedia content and is rendered on devices like fans, lamps, or vibration chairs. In this paper, we present a plug-in for Web browsers which is able to render such sensory effects provided via Web content, and we describe a demonstrator that uses this plug-in to control an amBX system.

A utility model for sensory experience

Enriching multimedia with additional effects such as olfaction, light, wind, or vibration is gaining more and more momentum in both research and industry. Hence, there is the need to determine the influence of individual effects on the Quality of Experience (QoE). In this paper, we present a subjective quality assessment using the MPEG-V standard to annotate video sequences with individual sensory effects (i.e., wind, light, and vibration) and all combinations thereof. Based on the results we derive a utility model for sensory experience that accounts for the assessed sensory effects. Finally, we provide an example instantiation of the utility model and validate it against current and past results of our subjective quality assessments conducted so far.