Maria Husmann

Interactive development of cross-device user interfaces

Current GUI builders provide a design environment for user interfaces that target either a single type or fixed set of devices, and provide little support for scenarios in which the user interface, or parts of it, are distributed over multiple devices. Distributed user interfaces have received increasing attention over the past years. There are different, often model-based, approaches that focus on technical issues. This paper presents XDStudio--a new GUI builder designed to support interactive development of cross-device web interfaces. XDStudio implements two complementary authoring modes with a focus on the design process of distributed user interfaces. First, simulated authoring allows designing for a multi-device environment on a single device by simulating other target devices. Second, on-device authoring allows the design process itself to be distributed over multiple devices, as design and development take place on the target devices themselves. To support interactive development for multi-device environments, where not all devices may be present at design and run-time, XDStudio supports switching between the two authoring modes, as well as between design and use modes, as required. This paper focuses on the design of XDStudio, and evaluates its support for two distribution scenarios.

XDKinect: development framework for cross-device interaction using kinect

Interactive systems set in multi-device environments continue to attract increasing attention, prompting researchers to experiment with emerging technologies. This paper presents XDKinect--a lightweight framework that facilitates development of cross-device applications using Kinect to mediate user interactions. The main benefits of XDKinect include its simplicity, adaptability and extensibility based on a flexible client-server architecture. Our framework features a time-based API to handle full-body interactions, a multi-modal API to capture gesture and speech commands, an API to utilise proxemic awareness information, a cross-device communication API, and a settings API to optimise for particular application requirements. A study with developers was conducted to investigate the potential of these features in terms of ease of use, effectiveness and possible use in the future. We show several example applications of XDKinect, as well as discussing advantages and limitations of our framework as revealed by our user study and experiments.

MultiMasher: Providing Architectural Support and Visual Tools for Multi-device Mashups

The vast majority of web applications still assume a single user on a single device and provide fairly limited means for interaction across multiple devices. In particular, developing applications for multi-device environments is a challenging task for which there is little tool support. We present the architecture and tools of MultiMasher, a system for the development of multi-device web applications based on the reuse of existing web sites created for single device usage. Web sites and devices can be mashed up and accessed by multiple users simultaneously, with our tools ensuring a consistent state across all devices. MultiMasher supports the composition of arbitrary elements from any web site, inter-widget communication across devices, and awareness of connected devices. We present both conceptual and technical evaluations of MultiMasher including a study on 50 popular web sites demonstrating high compatibility in terms of browsing, distribution and linking of web site components.

MultiMasher: A Visual Tool for Multi-device Mashups

The proliferation of a wide range of computing devices from tablets to large displays has created many situations where we no longer use a single device only. Rather, multiple devices are commonly used together to achieve a task. However, there is still little tool support for such scenarios in which different devices need to be combined to control an interface. Our goal is to enable multiple devices to view and interact with multiple web resources in a coordinated manner based on our new idea of multi-device mashups. In this paper, we present a first, visual tool for mashing up devices to access web sites, and discuss how we addressed the challenges as well as interesting issues for further research.

Surveying personal device ecosystems with cross-device applications in mind

To inform the design of cross-device applications, it is important to know what devices users typically have within reach in various settings in the home, in the workplace and on the move. We report on a survey carried out using an online questionnaire and discuss the most significant findings that should be taken into account in cross-device scenarios. The results are based on 293 participants covering a wide range of ages, nationalities, professions and living arrangements.

XDSession: integrated development and testing of cross-device applications

Despite the recent proliferation of new cross-device application frameworks, there is still a lack of sophisticated tools for testing new applications during their development. This paper presents XDSession---a framework for cross-device application development based on a concept of cross-device sessions, not only useful for managing distribution and synchronisation, but also for logging and debugging. Integrated with XDSession are two new tools specifically designed for cross-device testing. First, the session controller supports management and testing of cross-device sessions with connected or simulated devices at run-time. Second, the session inspector enables inspection and analysis of multi-device/multi-user sessions with support for deterministic record/replay of cross-device sessions. We show the utility of XDSession based on a case study of a semester-long course project in which our tools were used by students to reimplement an existing application and extend it with cross-device capabilities.

CTAT: Tilt-and-Tap Across Devices

Motion gestures have been proposed as an interaction para-digm for pairing, and sharing data between, mobile devices. They have also been used for interaction with large screens such as semi-public displays where a mobile phone can be used as a form of remote control in an eyes-free manner. Yet, so far, little attention has been paid to their potential use in cross-device web applications. We therefore decided to develop a framework that would support investigations into the use of a combination of touch and tilt interactions in cross-device scenarios. We first report on a study that motivated the development of the framework and informed its design. We then present the resulting Cross-Tilt-and-Tap (CTAT) framework for the rapid development of applications that make use of various motion gestures for communication between two or more devices. We conclude by describing an applications developed using CTAT.

Is a framework enough?: cross-device testing and debugging

Although numerous cross-device frameworks have been proposed in recent years, we know of only a few cross-device applications that have been implemented and many of these are prototypes to showcase the frameworks. Fewer still have been deployed and are accessible to the public. To go beyond prototyping, applications need to be carefully tested and bugs eliminated but current cross-device frameworks provide little support for this and it therefore remains a challenge. Tied to the lack of real-world applications, there is also a lack of cross-device developers who could be studied to gather requirements for better support. However, based on principles of responsive design, there are many tools available and widely used in practice to support the development of web applications that can be accessed from diverse devices, including ones to facilitate testing and debugging. Inspired by these tools, we have created XDTools -- a new integrated set of tools for testing and debugging cross-device applications. A preliminary qualitative evaluation with 12 developers produced promising results.

Eyes-Free Touch Command Support for Pen-Based Digital Whiteboards via Handheld Devices

Digital whiteboards that only sense pen input are limited in their interactive capabilities. One way to artificially add touch support is through personal mobile devices, which people carry with them. This work investigates how smartphones can be used as portable quick-access toolboxes held by the non-dominant hand to provide assistive touch commands for pen-driven whiteboard tasks. We developed two interface designs, one based on a classic remote with standard GUI controls and another optimised for eyes-free operation to eliminate gaze shifts between the two devices. In a controlled evaluation based on an established mode-switching study protocol, we compare the two phone interfaces and a baseline technique consisting of a pen-triggered popup menu on the whiteboard. Our results show a superior efficiency of the phone UIs over the popup. The eyes-free UI only partially performed better than the classic interface at the subtask level after subtracting the costs of errors.

Orchestrating multi-device presentations with OmniPresent

Meeting rooms and classrooms are increasingly equipped with more than one screen that could be used for presentations. However, current presentation software provides little support for arranging content across multiple screens beyond mirroring slides. We introduce OmniPresent, a system for orchestrating presentations on multiple devices. OmniPresent offers two modes for authoring these orchestrations. The first mode is suited in particular for legacy, single-device presentations and allows a presenter to orchestrate the presentation on the fly by pinning slides to extra screens. The second mode offers complete freedom in allocating slides to available screens by authoring an orchestration script beforehand. We have evaluated the two modes qualitatively in a user study and report on patterns in content and motivation for distributing presentations across multiple screens.