Graham Healy

How interaction methods affect image segmentation: User experience in the task

Interactive image segmentation is extensively used in photo editing when the aim is to separate a foreground object from its background so that it is available for various applications. The goal of the interaction is to get an accurate segmentation of the object with the minimal amount of human effort. To improve the usability and user experience using interactive image segmentation we present three interaction methods and study the effect of each using both objective and subjective metrics, such as, accuracy, amount of effort needed, cognitive load and preference of interaction method as voted by users. The novelty of this paper is twofold. First, the evaluation of interaction methods is carried out with objective metrics such as object and boundary accuracies in tandem with subjective metrics to cross check if they support each other. Second, we analyze Electroencephalography (EEG) data obtained from subjects performing the segmentation as an indicator of brain activity. The experimental results potentially give valuable cues for the development of easy-to-use yet efficient interaction methods for image segmentation.

Eye fixation related potentials in a target search task

Typically BCI (Brain Computer Interfaces) are found in rehabilitative or restorative applications, often allowing users a medium of communication that is otherwise unavailable through conventional means. Recently, however, there is growing interest in using BCI to assist users in searching for images. A class of neural signals often leveraged in common BCI paradigms are ERPs (Event Related Potentials), which are present in the EEG (Electroencephalograph) signals from users in response to various sensory events. One such ERP is the P300, and is typically elicited in an oddball experiment where a subjects attention is orientated towards a deviant stimulus among a stream of presented images. It has been shown that these types of neural responses can be used to drive an image search or labeling task, where we can rank images by examining the presence of such ERP signals in response to the display of images. To date, systems like these have been demonstrated when presenting sequences of images containing targets at up to 10Hz, however, the target images in these tasks do not necessitate any kind of eye movement for their detection because the targets in the images are quite salient. In this paper we analyse the presence of discriminating EEG signals when they are offset to the time of eye fixations in a visual search task where detection of target images does require eye fixations.

Object Segmentation in Images using EEG Signals

This paper explores the potential of brain-computer interfaces in segmenting objects from images. Our approach is centered around designing an effective method for displaying the image parts to the users such that they generate measurable brain reactions. When an image region, specifically a block of pixels, is displayed we estimate the probability of the block containing the object of interest using a score based on EEG activity. After several such blocks are displayed, the resulting probability map is binarized and combined with the GrabCut algorithm to segment the image into object and background regions. This study shows that BCI and simple EEG analysis are useful in locating object boundaries in images.

Neural Patterns of the Implicit Association Test.

The Implicit Association Test (IAT) is a reaction time based categorization task that measures the differential associative strength between bipolar targets and evaluative attribute concepts as an approach to indexing implicit beliefs or biases. An open question exists as to what exactly the IAT measures, and here EEG (Electroencephalography) has been used to investigate the time course of ERPs (Event-related Potential) indices and implicated brain regions in the IAT. IAT-EEG research identifies a number of early (250–450 ms) negative ERPs indexing early-(pre-response) processing stages of the IAT. ERP activity in this time range is known to index processes related to cognitive control and semantic processing. A central focus of these efforts has been to use IAT-ERPs to delineate the implicit and explicit factors contributing to measured IAT effects. Increasing evidence indicates that cognitive control (and related top-down modulation of attention/perceptual processing) may be components in the effective measurement of IAT effects, as factors such as physical setting or task instruction can change an IAT measurement. In this study we further implicate the role of proactive cognitive control and top-down modulation of attention/perceptual processing in the IAT-EEG. We find statistically significant relationships between D-score (a reaction-time based measure of the IAT-effect) and early ERP-time windows, indicating where more rapid word categorizations driving the IAT effect are present, they are at least partly explainable by neural activity not significantly correlated with the IAT measurement itself. Using LORETA, we identify a number of brain regions driving these ERP-IAT relationships notably involving left-temporal, insular, cingulate, medial frontal and parietal cortex in time regions corresponding to the N2- and P3-related activity. The identified brain regions involved with reduced reaction times on congruent blocks coincide with those of previous studies.

Improving object segmentation by using EEG signals and rapid serial visual presentation

This paper extends our previous work on the potential of EEG-based brain computer interfaces to segment salient objects in images. The proposed system analyzes the Event Related Potentials (ERP) generated by the rapid serial visual presentation of windows on the image. The detection of the P300 signal allows estimating a saliency map of the image, which is used to seed a semi-supervised object segmentation algorithm. Thanks to the new contributions presented in this work, the average Jaccard index was improved from 0.47 to 0.66 when processed in our publicly available dataset of images, object masks and captured EEG signals. This work also studies alternative architectures to the original one, the impact of object occupation in each image window, and a more robust evaluation based on statistical analysis and a weighted F-score.

Defect passivation and dark count in Geiger-mode avalanche photodiodes

An experimental study of post metal anneal conditions on dark count in Geiger-mode avalanche photodiodes (GM-APD) has been performed. The GM-APD structure will be shown to be extremely sensitive to post-metal anneals. Dark counts from measured samples decreased by a factor of two for each separate anneal in forming gas using temperatures from 425/spl deg/C to 450/spl deg/C. Conversely anneals of 250/spl deg/C in ambient increased dark count for temperature cycles up to 124 hours. Passivation and de-passivation of defect sites within the shallow junction active area are suspected as mechanisms contributing to the variations in dark count.

Exploring EEG for Object Detection and Retrieval

This paper explores the potential for using Brain Computer Interfaces (BCI) as a relevance feedback mechanism in content-based image retrieval. Several experiments are performed using a rapid serial visual presentation (RSVP) of images at different rates (5Hz and 10Hz) on 8 users with different degrees of familiarization with BCI and the dataset. We compare the feedback from the BCI and mouse-based interfaces in a subset of TRECVid images, finding that, when users have limited time to annotate the images, both interfaces are comparable in performance. Comparing our best users in a retrieval task, we found that EEG-based relevance feedback can outperform mouse-based feedback.

An outdoor spatially-aware audio playback platform exemplified by a virtual zoo

Outlined in this short paper is a framework for the construction of outdoor location-and direction-aware audio applications along with an example application to showcase the strengths of the framework and to demonstrate how it works. Although there has been previous work in this area which has concentrated on the spatial presentation of sound through wireless headphones, typically such sounds are presented as though originating from specific, defined spatial locations within a 3D environment. Allowing a user to move freely within this space and adjusting the sound dynamically as we do here, further enhances the perceived reality of the virtual environment. Techniques to realise this are implemented by the real-time adjustment of the presented 2 channels of audio to the headphones, using readings of the users head orientation and location which in turn are made possible by sensors mounted upon the headphones. Aside from proof of concept indoor applications, more user-responsive applications of spatial audio delivery have not been prototyped or explored. In this paper we present an audio-spatial presentation platform along with a primary demonstration application for an outdoor environment which we call a virtual audio zoo. This application explores our techniques to further improve the realism of the audio-spatial environments we can create, and to assess what types of future application are possible.

An investigation of triggering approaches for the rapid serial visual presentation paradigm in brain computer interfacing

The rapid serial visual presentation (RSVP) paradigm is a method that can be used to extend the P300 based brain computer interface (BCI) approach to enable high throughput target image recognition applications. The method requires high temporal resolution and hence, generating reliable and accurate stimulus triggers is critical for high performance execution. The traditional RSVP paradigm is normally deployed on two computers where software triggers generated at runtime by the image presentation software on a presentation computer are acquired along with the raw electroencephalography (EEG) signals by a dedicated data acquisition system connected to a second computer. It is often assumed that the stimulus presentation timing as acquired via events arising in the stimulus presentation code is an accurate reflection of the physical stimulus presentation. This is not necessarily the case due to various and variable latencies that may arise in the overall system. This paper describes a study to investigate in a representative RSVP implementation whether or not software-derived stimulus timing can be considered an accurate reflection of the physical stimuli timing. To investigate this, we designed a simple circuit consisting of a light diode resistor comparator circuit (LDRCC) for recording the physical presentation of stimuli and which in turn generates what we refer to as hardware triggered events. These hardware-triggered events constitute a measure of ground truth and are captured along with the corresponding stimulus presentation command timing events for comparison. Our experimental results show that using software-derived timing only may introduce uncertainty as to the true presentation times of the stimuli and this uncertainty itself is highly variable at least in the representative implementation described here. For BCI protocols such as those utilizing RSVP, the uncertainly introduced will cause impairment of performance and we recommend the use of additional circuitry to capture the physical presentation of stimuli and that these hardware-derived triggers should instead constitute the event markers to be used for subsequent analysis of the EEG.

Informed Perspectives on Human Annotation Using Neural Signals

In this work we explore how neurophysiological correlates related to attention and perception can be used to better understand the image-annotation task. We explore the nature of the highly variable labelling data often seen across annotators. Our results indicate potential issues with regard to ‘how well’ a person manually annotates images and variability across annotators. We propose such issues arise in part as a result of subjectively interpretable instructions that may fail to elicit similar labelling behaviours and decision thresholds across participants. We find instances where an individual’s annotations differ from a group consensus, even though their EEG signals indicate in fact they were likely in consensus with the group. We offer a new perspective on how EEG can be incorporated in an annotation task to reveal information not readily captured using manual annotations alone. As crowd-sourcing resources become more readily available for annotation tasks one can reconsider the quality of such annotations. Furthermore, with the availability of consumer EEG hardware, we speculate that we are approaching a point where it may be feasible to better harness an annotators time and decisions by examining neural responses as part of the process. In this regard, we examine strategies to deal with inter-annotator sources of noise and correlation that can be used to understand the relationship between annotators at a neural level.