Carina Westling

Comparing four technologies for measuring postural micromovements during monitor engagement

Objective metrics of engagement are valuable for estimating user experience or progression through interactional narratives. Postural micromovements of seated individuals during computer engagement have been previously measured with magnetic field sensors and chair-mounted force matrix detection mats. Here we compare readings from a head-mounted accelerometer, single camera sagittal motion tracking, and force distribution changes using floor-mounted force plates against a Vicon 8-camera motion capture system. Measurements were recorded on five participants who were watching or interacting with a computer monitor. Our results show that sagittal and coronal plane measurements for Vicon, the accelerometer and the single camera produced nearly identical data, were precisely synchronized in time, and in many cases proportional in amplitude. None of the systems tested were able to match the Vicons measurement of yaw.

Wearable sensor metric for fidgeting: screen engagement rather than interest causes NIMI of wrists and ankles

Measuring fidgeting is an important goal for the psychology of mind-wandering and for human computer interaction (HCI). Previous work measuring the movement of the head, torso and thigh during HCI has shown that engaging screen content leads to non-instrumental movement inhibition (NIMI). Camera-based methods for measuring wrist movements are limited by occlusions. Here we used a high pass filtered magnitude of wearable tri-axial accelerometer recordings during 2-minute passive HCI stimuli as a surrogate for movement of the wrists and ankles. With 24 seated, healthy volunteers experiencing HCI, this metric showed that wrists moved significantly more than ankles. We found that NIMI could be detected in the wrists and ankles; it distinguished extremes of interest and boredom via restlessness. We conclude that both free-willed and forced screen engagement can elicit NIMI of the wrists and ankles.

The Complex Relationship Between Empathy, Engagement and Boredom

In human computer interactions --- especially gaming --- the role of empathy has been mooted as a necessary prerequisite for higher levels of engagement and immersion. More recently other forms of engagement, including intellectual/cognitive engagement, have been proposed. In this study we present a carefully controlled dataset of human-computer interactions with a wide range of stimuli that ranged from highly engaging to boring to test these two theories. Analyzing 844 response sets to visual analogue scales (VAS) for empathy, interest, boredom, and engagement, we found that high empathy was sufficient for high engagement but is not necessary, whilst the converse was not true. We also found that empathy and boredom were incompatible with each other, but low levels of either were permissive rather than causal to the other. We conclude that there is no monotonic relationship between increasing empathy and engagement; either empathy is a sufficient (but not necessary) cause of engagement, or engagement is a necessary precursor to high empathy.

Using Wearable Inertial Sensors to Compare Different Versions of the Dual Task Paradigm during Walking

The dual task paradigm (DTP), where performance of a walking task co-occurs with a cognitive task to assess performance decrement, has been controversially mooted as a more suitable task to test safety from falls in outdoor and urban environments than simple walking in a hospital corridor. There are a variety of different cognitive tasks that have been used in the DTP, and we wanted to assess the use of a secondary task that requires mental tracking (the alternate letter alphabet task) against a more automatic working memory task (counting backward by ones). In this study we validated the x-io x-IMU wearable inertial sensors, used them to record healthy walking, and then used dynamic time warping to assess the elements of the gait cycle. In the timed 25 foot walk (T25FW) the alternate letter alphabet task lengthened the stride time significantly compared to ordinary walking, while counting backward did not. We conclude that adding a mental tracking task in a DTP will elicit performance decrement in healthy volunteers.

Automated bobbing and phase analysis to measure walking entrainment to music

In this paper, we investigate the influence of music on human walking behaviors in a public setting monitored by surveillance cameras. To this end, we propose a novel algorithm to characterize the frequency and phase of the walk. It relies on a human-by-detection tracking framework, along with a robust fitting of the human head bobbing motion. Preliminary experiments conducted on more than 100 tracks show that an accuracy greater than 85% for foot strike estimation can be achieved, suggesting that large scale analysis is at reach for finer music/walking behavior relationship studies.

A time series feature of variability to detect two types of boredom from motion capture of the head and shoulders

Boredom and disengagement metrics are key to accurately timed adaptive interventions in interactive systems. Psychological research suggests that boredom is a composite state incorporating cycles of lethargy and restlessness. Here we present innovative metrics of the components of boredom, based on motion capture and video analysis of head and shoulder movement. Healthy seated volunteers interacted with discrete, screen-presented stimuli ranging from engaging to boring, using a handheld trackball rather than a mouse, to allow for uninhibited noninstrumental shoulder movements. Our results include a feature (standard deviation of windowed ranges) potentially suitable for implementation in computer vision algorithms for early detection of disengagement.

A trigger-substrate model for smiling during an automated formative quiz: engagement is the substrate, not frustration

Introduction: Automated tutoring systems aim to respond to the learners cognitive state in order to maintain engagement. The end-users state might be inferred by interactive timings, bodily movements or facial expressions. Problematic computerized stimuli are known to cause smiling during periods of frustration. Methods: Forty-four seated, healthy participants (age range 18-35, 18 male) used a handheld trackball to answer a computer-presented, formative, 3-way multiple choice geography quiz, with 9 questions, lasting a total of 175 seconds. Frontal facial videos (10 Hz) were collected with a webcam and processed for facial expressions by CrowdEmotion using a pattern recognition algorithm. Interactivity was recorded by a keystroke logger (Inputlog 5.2). Subjective responses were collected immediately after each quiz using a panel of visual analogue scales (VAS). Results: Smiling was five-fold enriched during the instantaneous feedback segments of the quiz, and this was correlated with VAS ratings for engagement but not with happiness or frustration. Nevertheless, smiling rate was significantly higher after wrong answers compared to correct ones, and frustration was correlated with the number of questions answered incorrectly. Conclusion: The apparent disconnect between the increased smiling during incorrect answers but the lack of correlation between VAS frustration and smiles suggests a trigger-substrate model where engagement is the permissive substrate, while the noises made by the quiz after wrong answers may be the trigger.

Thigh-Derived Inertial Sensor Metrics to Assess the Sit-to-Stand and Stand-to-Sit Transitions in the Timed Up and Go (TUG) Task for Quantifying Mobility Impairment in Multiple Sclerosis

Introduction: Inertial sensors generate objective and sensitive metrics of movement disability that may indicate fall risk in many clinical conditions including multiple sclerosis (MS). The Timed-Up-And-Go (TUG) task is used to assess patient mobility because it incorporates clinically-relevant submovements during standing. Most sensor-based TUG research has focused on the placement of sensors at the spine, hip or ankles; an examination of thigh activity in TUG in multiple sclerosis is wanting. Methods: We used validated sensors (x-IMU by x-io) to derive transparent metrics for the sit-to-stand (SI-ST) transition and the stand-to-sit (ST-SI) transition of TUG, and compared effect sizes for metrics from inertial sensors on the thighs to effect sizes for metrics from a sensor placed at the L3 level of the lumbar spine. Twenty-three healthy volunteers were compared to 17 ambulatory persons with MS (PwMS, HAI ≤ 2). Results: During the SI-ST transition, the metric with the largest effect size comparing healthy volunteers to PwMS was the Area Under the Curve of the thigh angular velocity in the pitch direction–representing both thigh and knee extension; the peak of the spine pitch angular velocity during SI-ST also had a large effect size, as did some temporal measures of duration of SI-ST, although less so. During the ST-SI transition the metric with the largest effect size in PwMS was the peak of the spine angular velocity curve in the roll direction. A regression was performed. Discussion: We propose for PwMS that the diminished peak angular velocity during SI-ST directly represents extensor weakness, while the increased roll during ST-SI represents diminished postural control. Conclusions: During the SI-ST transition of TUG, angular velocities can discriminate between healthy volunteers and ambulatory PwMS better than temporal features. Sensor placement on the thighs provides additional discrimination compared to sensor placement at the lumbar spine.

Experience Design: Video Without Faces Increases Engagement But Not Empathy

Counter to prior claims that empathy is required for higher levels of engagement in human-computer interaction, our team has previously found that, in an analysis of 844 stimulus presentations, empathy is sufficient for high engagement, but is not necessary. Here, we ran a carefully controlled study of human-computer interactions with musical stimuli --- with and without visuals, and with and without recognizable people -- to directly test whether we could design an engaging stimulus that did not elicit empathy, by avoiding human faces or personal interaction. We measured subjective responses by visual analogue scale and found that the faceless stimulus was as engaging as the face-containing stimulus, but much less empathy-provoking. Therefore, we propose that empathy and engagement be considered independently during interaction design, because they are not monotonically related.

Immanent story worlds: the making of punchdrunk’s the drowned man – a Hollywood fable

Coyne’s critique of the rationalist modernist perspective in information technology proposes the relevance of a postmodernist perspective, drawing on Derrida’s deconstruction of the metaphysical and logocentrism (Coyne 1995: 102–104). Derrida’s contemporaries Deleuze and Lyotard also sought to deconstruct the metaphysical as the undeclared totalising principle of liberal humanism; Deleuze via establishing the dichotomy of the transcendental and immanence in his critique of the Cartesian subject-event relationship (Deleuze 2014: 203), and Lyotard in his discussion of narrative and scientific knowledge generation (Lyotard 1984). Twenty years after Coyne’s critique, the postmodernist perspective on information technologies is still not fully established in the design and analysis of human-computer interaction, although media theorists in the interim period, e.g. Hayles (2002) and Galloway (2004), have addressed the subject in their critique of the totalising influence present within both cyber-Romanticism and distributed network protocols, respectively. In the interest of furthering a postmodern (or even post-digital) approach to the design of interactive storyworlds, this study aims to identify principles that counter or disable the facilitation of transcendental (totalising) structures and support an immanent subject-event relationship through analysis of the work of theatre company Punchdrunk, pioneers of immersive theatre as interactive systems in physical space. The enquiry was formed around an analysis of the conditions of making and experiencing storyworlds during and after the build and run of The Drowned Man (beginning in spring 2012 and finishing in the autumn of 2014), with part of the research occurring as a participant study within the design team, complemented with semi-structured interviews with members of the company (‘Com’) and audience (‘Aud’) participants. The objective of the research was to gather observational data from the making of and participation in live Punchdrunk productions. Further data was gathered on social media, followed by analyses of lay reviews on TripAdvisor and the Punchdrunk fandom discourses on Facebook and Tumblr. The concluding section of this chapter proposes three features of Punchdrunk’s work that are key to immanence and can be implemented in digital interaction design, and applies the conclusion to the reported outcomes of Punchdrunk’s digital R&D project with MIT (Digital R&D Fund For the Arts 2012).