Justin Matejka

Chronicle: capture, exploration, and playback of document workflow histories

We describe Chronicle, a new system that allows users to explore document workflow histories. Chronicle captures the entire video history of a graphical document, and provides links between the content and the relevant areas of the history. Users can indicate specific content of interest, and see the workflows, tools, and settings needed to reproduce the associated results, or to better understand how it was constructed to allow for informed modification. Thus, by storing the rich information regarding the documents history workflow, Chronicle makes any working document a potentially powerful learning tool. We outline some of the challenges surrounding the development of such a system, and then describe our implementation within an image editing application. A qualitative user study produced extremely encouraging results, as users unanimously found the system both useful and easy to use.

YouMove: enhancing movement training with an augmented reality mirror

YouMove is a novel system that allows users to record and learn physical movement sequences. The recording system is designed to be simple, allowing anyone to create and share training content. The training system uses recorded data to train the user using a large-scale augmented reality mirror. The system trains the user through a series of stages that gradually reduce the users reliance on guidance and feedback. This paper discusses the design and implementation of YouMove and its interactive mirror. We also present a user study in which YouMove was shown to improve learning and short-term retention by a factor of 2 compared to a traditional video demonstration.

CommunityCommands: command recommendations for software applications

We explore the use of modern recommender system technology to address the problem of learning software applications. Before describing our new command recommender system, we first define relevant design considerations. We then discuss a 3 month user study we conducted with professional users to evaluate our algorithms which generated customized recommendations for each user. Analysis shows that our item-based collaborative filtering algorithm generates 2.1 times as many good suggestions as existing techniques. In addition we present a prototype user interface to ambiently present command recommendations to users, which has received promising initial user feedback.

Spotlight: directing users' attention on large displays

We describe a new interaction technique, called a spotlight, for directing the visual attention of an audience when viewing data or presentations on large wall-sized displays. A spotlight is simply a region of the display where the contents are displayed normally while the remainder of the display is somewhat darkened. In this paper we define the behavior of spotlights, show unique affordances of the technique, and discuss design characteristics. We also report on experiments that show the benefit of using the spotlight a large display and standard desktop configuration. Our results suggest that the spotlight is preferred over the standard cursor and outperforms it by a factor of 3.4 on a wall-sized display.

Magic desk: bringing multi-touch surfaces into desktop work

Despite the prominence of multi-touch technologies, there has been little work investigating its integration into the desktop environment. Bringing multi-touch into desktop computing would give users an additional input channel to leverage, enriching the current interaction paradigm dominated by a mouse and keyboard. We provide two main contributions in this domain. First, we describe the results from a study we performed, which systematically evaluates the various potential regions within the traditional desktop configuration that could become multi-touch enabled. The study sheds light on good or bad regions for multi-touch, and also the type of input most appropriate for each of these regions. Second, guided by the results from our study, we explore the design space of multi-touch-integrated desktop experiences. A set of new interaction techniques are coherently integrated into a desktop prototype, called Magic Desk, demonstrating potential uses for multi-touch enabled desktop configurations.

The design and evaluation of multi-finger mouse emulation techniques

We explore the use of multi-finger input to emulate full mouse functionality, such as the tracking state, three buttons, and chording. We first present the design space for such techniques, which serves as a guide for the systematic investigation of possible solutions. We then perform a series of pilot studies to come up with recommendations for the various aspects of the design space. These pilot studies allow us to arrive at a recommended technique, the SDMouse. In a formal study, the SDMouse was shown to significantly improve performance in comparison to previously developed mouse emulation techniques.

Patina: dynamic heatmaps for visualizing application usage

We present Patina, an application independent system for collecting and visualizing software application usage data. Patina requires no instrumentation of the target application, all data is collected through standard window metrics and accessibility APIs. The primary visualization is a dynamic heatmap overlay which adapts to match the content, location, and shape of the user interface controls visible in the active application. We discuss a set of design guidelines for the Patina system, describe our implementation of the system, and report on an initial evaluation based on a short-term deployment of the system.

Design and evaluation of a command recommendation system for software applications

We examine the use of modern recommender system technology to aid command awareness in complex software applications. We first describe our adaptation of traditional recommender system algorithms to meet the unique requirements presented by the domain of software commands. A user study showed that our item-based collaborative filtering algorithm generates 2.1 times as many good suggestions as existing techniques. Motivated by these positive results, we propose a design space framework and its associated algorithms to support both global and contextual recommendations. To evaluate the algorithms, we developed the CommunityCommands plug-in for AutoCAD. This plug-in enabled us to perform a 6-week user study of real-time, within-application command recommendations in actual working environments. We report and visualize command usage behaviors during the study, and discuss how the recommendations affected users behaviors. In particular, we found that the plug-in successfully exposed users to new commands, as unique commands issued significantly increased.

ViewCube: a 3D orientation indicator and controller

Literally hundreds of thousands of users of 2D computer-aided design (CAD) tools are in the difficult process of transitioning to 3D CAD tools. A common problem for these users is disorientation in the abstract virtual 3D environments that occur while developing new 3D scenes. To help address this problem, we present a novel in-scene 3D widget called the ViewCube as a 3D orientation indicator and controller. The ViewCube is a cube-shaped widget placed in a corner of the window. When acting as an orientation indicator, the ViewCube turns to reflect the current view direction as the user re-orients the scene using other tools. When used as an orientation controller, the ViewCube can be dragged, or the faces, edges, or corners can be clicked on, to easily orient the scene to the corresponding view. We conducted a formal experiment to measure the performance of the ViewCube comparing: (1) ArcBall-style dragging using the ViewCube for manual view switching, (2) clicking on face/edge/corner elements of the ViewCube for automated view switching and (3) clicking on a dedicated row of buttons for automated view switching. The results indicate that users prefer and are almost twice as fast at using the ViewCube with dragging compared to clicking techniques, independent of a number of ViewCube representations that we examined.

Waken: reverse engineering usage information and interface structure from software videos

We present Waken, an application-independent system that recognizes UI components and activities from screen captured videos, without any prior knowledge of that application. Waken can identify the cursors, icons, menus, and tooltips that an application contains, and when those items are used. Waken uses frame differencing to identify occurrences of behaviors that are common across graphical user interfaces. Candidate templates are built, and then other occurrences of those templates are identified using a multi-phase algorithm. An evaluation demonstrates that the system can successfully reconstruct many aspects of a UI without any prior application-dependant knowledge. To showcase the design opportunities that are introduced by having this additional meta-data, we present the Waken Video Player, which allows users to directly interact with UI components that are displayed in the video.