Baris Serim

Pointing while Looking Elsewhere: Designing for Varying Degrees of Visual Guidance during Manual Input

We propose using eye tracking to support interface use with decreased reliance on visual guidance. While the design of most graphical user interfaces take visual guidance during manual input for granted, eye tracking allows distinguishing between the cases when the manual input is conducted with or without guidance. We conceptualize the latter cases as input with uncertainty that require separate handling. We describe the design space of input handling by utilizing input resources available to the system, possible actions the system can realize and various feedback techniques for informing the user. We demonstrate the particular action mechanisms and feedback techniques through three applications we developed for touch interaction on a large screen. We conducted a two stage study of positional accuracy during target acquisition with varying visual guidance, to determine the selection range around a touch point due to positional uncertainty. We also conducted a qualitative evaluation of example applications with participants to identify perceived utility and hand eye coordination challenges while using interfaces with decreased visual guidance.

Extreme Co-design: Prototyping with and by the User for Appropriation of Web-connected Tags

We describe a field prototyping project where open-ended prototype tools for web-connected tags are weekly co-designed and programmed with and by the user. We call this approach Extreme Co-design to denote how design is inscribed in Extreme Programming sessions with rapid cycles of use, design and development that allow extensive exploration and experiencing of appropriation scenarios. Such an approach is particularly suited for repurposing malleable technologies such as RFID/NFC, which can take a variety of affordances and be applied for many uses, in particular acknowledging trends such as composition at end-user level of web functionality. We analyse the results of a one-month field work highlighting how to document explored ideas, appropriation scenarios, use try-outs, developed features and gained insights. We discuss this successful approach as a design tactic for unfinished products to foster end-users’ creativity through situated use and show how Extreme Programming and in-situ deployment supported meaningful designer-user interactions that resulted in the advancement of the initial design.

Gaze-Adaptive Above and On-Surface Interaction

We explore the combination of above-surface sensing with eye tracking to facilitate concurrent interaction with multiple regions on touch screens. Conventional touch input relies on positional accuracy, thereby requiring tight visual monitoring of ones own motor action. In contrast, above-surface sensing and eye tracking provides information about how users hands and gaze are distributed across the interface. In these situations we facilitate interaction by 1) showing the visual feedback of the hand hover near users gaze point and 2) decrease the requisite of positional accuracy by employing gestural information. We contribute input and visual feedback techniques that combine these modalities and demonstrate their use in example applications. A controlled study showed the effectiveness of our techniques for manipulation tasks against conventional touch, while the effectiveness in acquisition tasks depended on the amount of mid-air motion, leading to our conclusion that the techniques can benefit interacting with multiple interface regions.

Querying and Display of Information: Symbiosis in Exploratory Search Interaction Scenarios

This paper examines potential interaction aspects related to querying and the display of information in exploratory search scenarios with a particular focus on user state and interactive visualization. Exploratory search refers to a specific type of information seeking that is open-ended, continuous and evolving. The evolving nature of exploratory search also provides the computer with sequential data that can be used to estimate user state and intention as the search unfolds. In this setting, the system supports querying by relying on user’s pointing actions, sequential organization of user interaction and query metadata. The system also adapts the display of information by determining the timing and visual representation. The paper illustrates potential interactions that employ new input modalities such as eye gaze and physiological signals. The paper concludes by discussing the possible functions of interactive visualization regarding querying and the display of information.

Developing a Symbiotic System for Scientific Information Seeking: The Mindsee Project

This paper describes an approach for improving the current systems supporting the exploration and research of scientific literature, which generally adopt a query-based information-seeking paradigm. Our approach is to use a symbiotic system paradigm, exploiting central and peripheral physiological data along with eye-tracking data to adapt to users’ ongoing subjective relevance and satisfaction with search results. The system described, along with the interdisciplinary theoretical work underpinning it, could serve as a stepping stone for the development and diffusion of next-generation symbiotic systems, enabling a productive interdependence between humans and machines. After introducing the concept and evidence informing the development of symbiotic systems over a wide range of application domains, we describe the rationale of the MindSee project, emphasizing its BCI component and pinpointing the criteria around which users’ evaluations can gravitate. We conclude by summarizing the main contribution that MindSee is expected to make.

Improving genomics-based predictions for precision medicine through active elicitation of expert knowledge

Predicting the efficacy of a drug for a given individual, using highdimensional genomic measurements, is at the core of precision medicine. However, identifying features on which to base the predictions remains a challenge, especially when the sample size is small. Incorporating expert knowledge offers a promising alternative to improve a prediction model, but collecting such knowledge is laborious to the expert if the number of candidate features is very large. We introduce a probabilistic model that can incorporate expert feedback about the impact of genomic measurements on the sensitivity of a cancer cell for a given drug. We also present two methods to intelligently collect this feedback from the expert, using experimental design and multi-armed bandit models. In a multiple myeloma blood cancer data set (n=51), expert knowledge decreased the prediction error by 8%. Furthermore, the intelligent approaches can be used to reduce the workload of feedback collection to less than 30% on average compared to a naive approach.Motivation Precision medicine requires the ability to predict the efficacies of different treatments for a given individual using high‐dimensional genomic measurements. However, identifying predictive features remains a challenge when the sample size is small. Incorporating expert knowledge offers a promising approach to improve predictions, but collecting such knowledge is laborious if the number of candidate features is very large. Results We introduce a probabilistic framework to incorporate expert feedback about the impact of genomic measurements on the outcome of interest and present a novel approach to collect the feedback efficiently, based on Bayesian experimental design. The new approach outperformed other recent alternatives in two medical applications: prediction of metabolic traits and prediction of sensitivity of cancer cells to different drugs, both using genomic features as predictors. Furthermore, the intelligent approach to collect feedback reduced the workload of the expert to approximately 11%, compared to a baseline approach. Availability and implementation Source code implementing the introduced computational methods is freely available at https://github.com/AaltoPML/knowledge‐elicitation‐for‐precision‐medicine.

Integrating Neurophysiological Relevance Feedback in Intent Modeling for Information Retrieval

The use of implicit relevance feedback from neurophysiology could deliver effortless information retrieval. However, both computing neurophysiologic responses and retrieving documents are characterized by uncertainty because of noisy signals and incomplete or inconsistent representations of the data. We present the first‐of‐its‐kind, fully integrated information retrieval system that makes use of online implicit relevance feedback generated from brain activity as measured through electroencephalography (EEG), and eye movements. The findings of the evaluation experiment (N = 16) show that we are able to compute online neurophysiology‐based relevance feedback with performance significantly better than chance in complex data domains and realistic search tasks. We contribute by demonstrating how to integrate in interactive intent modeling this inherently noisy implicit relevance feedback combined with scarce explicit feedback. Although experimental measures of task performance did not allow us to demonstrate how the classification outcomes translated into search task performance, the experiment proved that our approach is able to generate relevance feedback from brain signals and eye movements in a realistic scenario, thus providing promising implications for future work in neuroadaptive information retrieval (IR).

Exploring gaze-adaptive features for interacting with multi-document visualizations

We present a preliminary design study for utilizing eye tracking to support interacting with a multi-document visualization. Complex information seeking tasks can involve collection and comparison of multiple documents, resulting in long and sustained search sessions. The sustained and evolving nature of the session also provides the search interface the opportunity to gather information on the user state and interaction history, which can be used to adapt the information content and representation. We designed a system to evaluate how eye tracking information can be used for adapting the visual salience of information entities. The interface features documents and related keywords that are arranged in a radial layout configuration called the intent radar. Reading history and visual attention, as registered by eye tracking data, are respectively used to trace read items and for visual cueing. We evaluated the interface with 16 participants to gather subjective feedback about specific components and features of the interface. The overall results show that the interface and gaze-related appearance of keywords was positively received by the users.

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

This paper describes an approach for improving the current systems supporting the exploration and research of scientific literature, which generally adopt a query-based information-seeking paradigm. Our approach is to use a symbiotic system paradigm, exploiting central and peripheral physiological data along with eye-tracking data to adapt to users’ ongoing subjective relevance and satisfaction with search results. The system described, along with the interdisciplinary theoretical work underpinning it, could serve as a stepping stone for the development and diffusion of next-generation symbiotic systems, enabling a productive interdependence between humans and machines. After introducing the concept and evidence informing the development of symbiotic systems over a wide range of application domains, we describe the rationale of the MindSee project, emphasizing its BCI component and pinpointing the criteria around which users’ evaluations can gravitate. We conclude by summarizing the main contribution that MindSee is expected to make.

mailVis: Visualizing emailbox for re-finding emails

We present the design and implementation of mailVis, an interactive visual interface for email boxes that facilitates re-finding of emails. Email re-finding tasks can be challenging, involving scanning of many emails and modifying the query as the search progresses. We designed mailVis for such tasks in which the user would benefit from having memory clues and multiple options to direct the search. During the design process, we devised a novel interaction technique, filter swipe, that combines filtering and selection into one action for rapidly skimming individual items in a data set.