Christian P. Janssen

Generating user interfaces from data models and dialogue net specifications

A method and a set of supporting tools have been developed for an improved integration of user interface design with software engineering methods and tools. Animated user interfaces for database-oriented applications are generated from an extended data model and a new graphical technique for specifying dialogues. Based on views defined for the data model, an expert system uses explicit design rules derived from existing guidelines for producing the static layout of the user interface. A petri net based technique called dialogue nets is used for specifying the dynamic behaviour. Output is generated for an existing user interface management system. The approach supports rapid prototyping while using the advantages of standard software engineering methods.

Strategic Adaptation to Performance Objectives in a Dual­Task Setting

How do people interleave attention when multitasking? One dominant account is that the completion of a subtask serves as a cue to switch tasks. But what happens if switching solely at subtask boundaries led to poor performance? We report a study in which participants manually dialed a UK-style telephone number while driving a simulated vehicle. If the driver were to exclusively return his or her attention to driving after completing a subtask (i.e., using the single break in the xxxxx-xxxxxx representational structure of the number), then we would expect to see a relatively poor driving performance. In contrast, our results show that drivers choose to return attention to steering control before the natural subtask boundary. A computational modeling analysis shows that drivers had to adopt this strategy to meet the required performance objective of maintaining an acceptable lateral position in the road while dialing. Taken together these results support the idea that people can strategically control the allocation of attention in multitask settings to meet specific performance criteria.

Natural Break Points The Influence of Priorities and Cognitive and Motor Cues on Dual-Task Interleaving

What factors determine when people interleave tasks when multitasking? Here the authors look at the role of priorities and cognitive and motor cues. A study was conducted in which participants steered a simulated vehicle while also dialing two phone numbers that contained sets of repeating digits. Participants tended to interleave tasks after typing in a complete set of repeating digits and sometimes also at the cognitive chunk boundary. The exact pattern of how participants interleaved these tasks depended on their priority objective. A modeling analysis that explored performance for a series of alternative strategies for task interleaving, given the cognitive and task constraints, suggested why participants avoided interleaving at other points: Such strategies tend to move performance away from a trade-off curve that strikes an optimal balance between dialing and driving performance. The study highlights the role that cognitive and motor cues can play in dual-task performance and the importance of being aware, and acting on, priorities. Further implications and limitations are discussed.

Fast or safe?: how performance objectives determine modality output choices while interacting on the move

In-car devices that use audio output have been shown to be less distracting than traditional graphical user interfaces, but can be cumbersome and slow to use. In this paper, we report an experiment that demonstrates how these performance characteristics impact whether people will elect to use an audio interface in a multitasking situation. While steering a simulated vehicle, participants had to locate a source of information in a short passage of text. The text was presented either on a visual interface, or using a text-to-speech audio interface. The relative importance of each task was varied. A no-choice/choice paradigm was used in which participants first gained experience with each of the two interfaces, before being given a choice on which interface to use on later trials. The characteristics of the interaction with the interfaces, as measured in the no-choice phase, and the relative importance of each task, had an impact on which output modality was chosen in the choice phase. Participants that prioritized the secondary task tended to select the (faster yet more distracting) visual interface over the audio interface, and as a result had poorer lane keeping performance. This work demonstrates how a users task objective will influence modality choices with multimodal devices in multitask environments.

Integrating knowledge of multitasking and interruptions across different perspectives and research methods

Multitasking and interruptions have been studied using a variety of methods in multiple fields (e.g., HCI, cognitive science, computer science, and social sciences). This diversity brings many complementary insights. However, it also challenges researchers to understand how seemingly disparate ideas can best be integrated to further theory and to inform the design of interactive systems. There is therefore a need for a platform to discuss how different approaches to understanding multitasking and interruptions can be combined to provide insights that are more than the sum of their parts. In this article we argue for the necessity of an integrative approach. As part of this argument we provide an overview of articles in this special issue on multitasking and interruptions. These articles showcase the variety of methods currently used to study multitasking and interruptions. It is clear that there are many challenges to studying multitasking and interruptions from different perspectives and using different techniques. We advance a six-point research agenda for the future of multi-method research on this important and timely topic.

When, What, and How Much to Reward in Reinforcement Learning-Based Models of Cognition.

Reinforcement learning approaches to cognitive modeling represent task acquisition as learning to choose the sequence of steps that accomplishes the task while maximizing a reward. However, an apparently unrecognized problem for modelers is choosing when, what, and how much to reward; that is, when (the moment: end of trial, subtask, or some other interval of task performance), what (the objective function: e.g., performance time or performance accuracy), and how much (the magnitude: with binary, categorical, or continuous values). In this article, we explore the problem space of these three parameters in the context of a task whose completion entails some combination of 36 state-action pairs, where all intermediate states (i.e., after the initial state and prior to the end state) represent progressive but partial completion of the task. Different choices produce profoundly different learning paths and outcomes, with the strongest effect for moment. Unfortunately, there is little discussion in the literature of the effect of such choices. This absence is disappointing, as the choice of when, what, and how much needs to be made by a modeler for every learning model.

Integrating Object-Oriented Analysis and Graphical User Interface Design

A methodology is described for the derivation of graphical user interfaces (GUIs) from the object model of information systems applications. User interface design starts with a conceptual user interface model consisting of data objects as well as objects related to the interactive computer application. User views of the conceptual object model are constructed according to the users tasks. An explicit dialogue model defines the dynamics of the user interface. The coarse grain dialogue is described by a petri-net-based notation called Dialogue Nets. Our approach integrates methods and notations from software-engineering with a methodology for graphical user interface design.

Priming Drivers before Handover in Semi-Autonomous Cars

Semi-autonomous vehicles occasionally require control to be handed over to the driver in situations where the vehicle is unable to operate safely. Currently, such handover requests require the driver to take control almost instantaneously. We investigate how auditory pre-alerts that occur well before the handover request impact the success of the handover in a dual task scenario. In a study with a driving simulator, drivers perform tasks on their phone while the car is in an autonomous mode. They receive a repeated burst audio pre-alert or an increasing pulse audio pre-alert preceding the standard warning for immediate handover. Results show that pre-alerts caused people to look more at the road before the handover occurred, and to disengage from the secondary task earlier, compared to when there was no pre-alert. This resulted in safer handover situations. Increasing pulse pre-alerts show particular promise due to their communication of urgency. Our detailed analysis informs the design and evaluation of alerts in safety-critical systems with automation.

Sharing a driver's context with a caller via continuous audio cues to increase awareness about driver state

In an experiment using a driving simulator we investigated whether sharing information of a drivers context with a remote caller via continuous audio cues can make callers more aware of the driving situation. Increased awareness could potentially help in making the conversation less distracting. Prior research has shown that although sharing context using video can create such beneficial effects, it also has some practical disadvantages. It is an open question whether other modalities might also provide sufficient context for a caller. In particular, the effects of sharing audio, a cheaper, more salient, and perhaps more practical alternative than video, are not well understood. We investigated sharing context using direct cues in the form of realistic driving sounds (e.g., car honks, sirens) and indirect cues in the form of elevated heartbeats. Sound sharing affected the callers perception of the drivers busyness. However, this had at most a modest effect on conversation and driving performance. An implication of these results is that although sharing sounds can increase a callers awareness of changes in the drivers busyness, they need more training or information on how to leverage such context information to reduce disruption to driving. Limitations and implications are discussed.

Strategic Adaptation to Task Characteristics, Incentives, and Individual Differences in Dual-Tasking

We investigate how good people are at multitasking by comparing behavior to a prediction of the optimal strategy for dividing attention between two concurrent tasks. In our experiment, 24 participants had to interleave entering digits on a keyboard with controlling a randomly moving cursor with a joystick. The difficulty of the tracking task was systematically varied as a within-subjects factor. Participants were also exposed to different explicit reward functions that varied the relative importance of the tracking task relative to the typing task (between-subjects). Results demonstrate that these changes in task characteristics and monetary incentives, together with individual differences in typing ability, influenced how participants choose to interleave tasks. This change in strategy then affected their performance on each task. A computational cognitive model was used to predict performance for a wide set of alternative strategies for how participants might have possibly interleaved tasks. This allowed for predictions of optimal performance to be derived, given the constraints placed on performance by the task and cognition. A comparison of human behavior with the predicted optimal strategy shows that participants behaved near optimally. Our findings have implications for the design and evaluation of technology for multitasking situations, as consideration should be given to the characteristics of the task, but also to how different users might use technology depending on their individual characteristics and their priorities.