Steven Rick

LAB-IN-A-BOX: semi-automatic tracking of activity in the medical office

Patient-centered healthcare and increased efficiency are major goals of modern medicine, and physician–patient interaction and communication are a cornerstone of clinical encounters. The introduction of the electronic health record (EHR) has been a key component in shaping not only organization, clinical workflow and ultimately physicians’ clinical decision making, but also patient–physician communication in the medical office. In order to inform the design of future EHR interfaces and assess their impact on patient-centered healthcare, designers and researchers must understand the multimodal nature of the complex physician–patient–EHR system interaction. However, characterizing multimodal activity is difficult and expensive, often requiring manual coding of hours of video data. We present our Lab-in-a-Box solution that enables the capture of multimodal activity in real-world settings. We focus here on the medical office where our Lab-in-a-Box system exploits a range of sensors to track computer-based activity, speech interaction, visual attention and body movements, and automatically synchronize and segment this data. The fusion of multiple sensors allows us to derive initial activity segmentation and to visualize it for further interactive analysis. By empowering researchers with cutting-edge data collection tools and accelerating analysis of multimodal activity in the medical office, our Lab-in-a-Box has the potential to uncover important insights and inform the next generation of Health IT systems.

Exploring Mixed Reality in Specialized Surgical Environments

Recent technology advances in both Virtual Reality and Augmented Reality are creating an opportunity for a paradigm shift in the design of human-computer interaction systems. Delving into the Reality-Virtuality Continuum, we find Mixed Reality - systems designed to augment the physical world with virtual entities that embody characteristics of real world objects. In the medical field, Mixed Reality systems can overlay real-time and spatially accurate results onto a patients body without the need for external screens. The complexity of these systems previously required specialized prototypes, but newly available commercial products like the Microsoft HoloLens make the technology more available. Through a combination of literature review, expert analysis, and prototyping we explore the use of Mixed Reality in healthcare. From the experience of prototyping Patiently and HoloSim, two applications for augmenting medical training, we outline considerations for the future design and development of virtual interfaces grounded in reality.

Hands That Speak: An Integrated Approach to Studying Complex Human Communicative Body Movements

Gestures, the visible body movements that are ubiquitous in human behavior, are key elements of natural communication. Understanding them is fundamental to designing computing applications with more natural forms of interaction. Both sign languages and everyday gestures reveal the rich signal capacity of this modality. However, although research is developing at fast pace, we still lack in-depth understanding of the elements that create the underlying symbolic signals. This is partly due to lack of tools for studying communicative movements in context. We introduce a novel approach to address this problem based on unobtrusive depth cameras and developed an infrastructure supporting naturalistic data collection. While we focus on sign language and gestures, the tools we developed are applicable for other types of body based research applications. We report on the quality of data collection, and we show how our approach can lead to novel insights and understanding of communicative movements.