Jonas Etzold

Context-aware querying for multimodal search engines

Multimodal interaction provides the user with multiple modes of interacting with a system, such as gestures, speech, text, video, audio, etc. A multimodal system allows for several distinct means for input and output of data. In this paper, we present our work in the context of the I-SEARCH project, which aims at enabling context-aware querying of a multimodal search framework including real-world data such as user location or temperature. We introduce the concepts of MuSeBag for multimodal query interfaces, UIIFace for multimodal interaction handling, and CoFind for collaborative search as the core components behind the I-SEARCH multimodal user interface, which we evaluate via a user study.

I-SEARCH: a unified framework for multimodal search and retrieval

In this article, a unified framework for multimodal search and retrieval is introduced. The framework is an outcome of the research that took place within the I-SEARCH European Project. The proposed system covers all aspects of a search and retrieval process, namely low-level descriptor extraction, indexing, query formulation, retrieval and visualisation of the search results. All I-SEARCH components advance the state of the art in the corresponding scientific fields. The I-SEARCH multimodal search engine is dynamically adapted to end-users devices, which can vary from a simple mobile phone to a high-performance PC.

I-SEARCH: a multimodal search engine based on rich unified content description (RUCoD)

In this paper, we report on work around the I-SEARCH EU (FP7 ICT STREP) project whose objective is the development of a multimodal search engine. We present the projects objectives, and detail the achieved results, amongst which a Rich Unified Content Description format.

A streaming framework for instant 3D rendering and interaction

We present a framework for optimizing the streaming of large 3D applications to allow an instant interaction independently from network bandwidth or available processing power of the rendering unit. The instantaneous display of imperfect content that gets continuously refined is state of the art for many application areas on the web, why we propose to utilize this idea for interactive 3D web applications. To achieve this, two new approaches were chosen. In addition to a progressive transmission of geometry and texture data, we firstly propose an overall rating function to control the priorities between different media transmission blocks. Secondly, we use a similarity metric to use already loaded scene parts as a replacement of parts with lower priority. Our approach utilizes a client-server architecture to collect, convert, control, and transmit the scene data at runtime to allow an instant distribution of adapted data, while also reducing the CPU load and network traffic.

Model-based development of accessible, personalized web forms for ICF-based assessment

Tools, such as interactive forms, for International Classification of Functioning, Disability and Health (ICF) assessment are expensive to develop, because they are used in different domains and have to be fitted to the special needs of the user groups. This paper focuses on ICF assessment for and by people with dyslexia caused by cognitive impairments. It presents a cost-effective approach to semi-automatically generated, accessible web-forms. Based on a developed meta-model enriched with so called clarifying communication elements and state information, a system was developed, which transforms the model at runtime into an interactive web application. Every text element of the form can on the one hand be personalized and on the other hand can be enhanced with available content like icons, images, videos showing explanations.

Instant texture transmission using bandwidth-optimized progressive interlacing images

We present an adaptive bandwidth-optimized approach for progressive image transmission to allow instant textured rendering in web-based 3D applications. Applications utilizing a lot of image data require an adaptive technology to build responsive user interfaces. This applies especially for the use in mobile networks. While several approaches provide a progressive geometry streaming, they do not focus on a fast and simple texture streaming for 3D web applications. However, standard 2D image transmission technologies are usually inappropriate within a 3D context. In 2D, image size as well as resolution are often set during the authoring phase, whereas in 3D applications size and displayed resolution of textured 3D objects depend on the virtual camera. An adaptive texture transmission mechanism has to consider this as part of its control function. Our approach thus combines progressive texture streaming with an adaptive number of refinement levels without any pixel retransmission. It enables a fast user feedback and reduces the transmitted data to a minimum without loss of visual quality. Moreover, our GPUII approach (GPU-based Image Interlacing) allows an easy integration into existing applications while also reducing the CPU load.

MIPos: towards mobile image positioning in mixed reality web applications based on mobile sensors

In this paper we propose a position aware, user controlled and real time enabled visual approach to arrange real world images within virtual scenes by leveraging modern web APIs to get access to location, mobile sensors and camera. Enhancing a virtual scene with images of its real substitute to create Mixed Reality scenes involves the accurate positioning of real world images within their corresponding location in the virtual scene. To achieve that in 3D web applications, which are able to run on mobile devices, the approach needs to overcome web and mobile specific limitations. Instead of relying on heavy image processing in JavaScript we describe the detection of movement and orientation of a users device by leveraging sensor fusion and filters to compensate sensor noise along with a pattern-based step detection scheme. We reflect the acquired data to the device camera stream within a corresponding web 3D scene. This allows the instant capturing of precisely positioned real world images within the virtual scene. In addition, our approach achieves an indoor positioning system along with the possibility to directly embed user media within Mixed and Augmented Reality scenes by using pure web APIs and avoiding the extensive use of computational resources through feature detection algorithms.

Efficient Image Distribution on the Web

In this paper, we present a browser-based Web 3D application that allows an instant distribution of image data even over mobile networks as well as textured rendering of large image collections on mobile devices with restricted processing power. Applications utilizing a lot of image data require an adaptive technology to build responsive user interfaces. This applies especially for the use in mobile networks. Furthermore the up- and download of the massive amount of image data should be transmitted in a progressive manner to get an instant feedback. While people are used to instant reaction of web applications and do not care about the amount of data that has to be transferred, the instantaneous display of imperfect content that gets continuously refined is state of the art for many application areas on the web. However, standard 2D image transmission technologies are usually inappropriate within a 3D context. In 2D, image size as well as resolution are often set during the authoring phase, whereas in 3D applications size and displayed resolution of textured 3D objects depend on the virtual camera. Our GPUII approach (GPU-based Image Interlacing) follows a client-server architecture, which allows an instant distribution of new data while also reducing the CPU load and network traffic.

kARbon: a collaborative MR web application for communicationsupport in construction scenarios

kARbon demonstrates a web-based mixed reality (MR) support and collaboration tool for a wide range of construction planning and supervising scenarios. We describe how the construction process can be supported for locally distributed workers, experts and decision makers by leveraging MR methods in combination with effective and pure web-based collaboration, interaction and presentation concepts. kARbon therefore combines classical CAD planning data with photo collections representing temporal snapshots of the associated construction site in a precise MR scene. We focus on our collaborative use case and describe how involved people in different locations and on different device types can collaborate and interact through our tool based on the power of modern web standards.