Matthew L. Lee

Lifelogging memory appliance for people with episodic memory impairment

Lifelogging technologies have the potential to provide memory cues for people who struggle with episodic memory impairment (EMI). These memory cues enable the recollection of significant experiences, which is important for people with EMI to regain a sense of normalcy in their lives. However, lifelogging technologies often collect an overwhelmingly large amount of data to review. The best memory cues need to be extracted and presented in a way that best supports episodic recollection. We describe the design of a new lifelogging system that captures photos, ambient audio, and location information and leverages both automated content/context analysis and the expertise of family caregivers to facilitate the extraction and annotation of a salient summary consisting of good cues from the lifelog. The system presents the selected cues for review in a way that maximizes the opportunities for the person with EMI to think deeply about these cues to trigger memory recollection on his own without burdening the caregiver. We compare our system with another review system that requires the caregiver to repeatedly guide the review process. Our self-guided system resulted in better memory retention and imposed a smaller burden on the caregiver whereas the caregiver-guided approach provided more opportunities for caregiver interaction.

Providing good memory cues for people with episodic memory impairment

Alzheimers disease impairs episodic memory and subtly and progressively robs people of their ability to remember their recent experiences. In this paper, we describe two studies that lead to a better understanding of how caregivers use cues to support episodic memory impairment and what types of cues are best for supporting recollection. We also show how good memory cues differ between people with and without episodic memory impairment. We discuss how this improved understanding impacts the design of lifelogging technologies for automatically capturing and extracting the best memory cues to assist overburdened caregivers and people with episodic memory impairment in supporting recollection of episodic memory.

Reflecting on pills and phone use: supporting awareness of functional abilities for older adults

Older adults often struggle with maintaining self-aware of their ability to carry out everyday activities important for independence. Unobtrusive sensors embedded in the home can monitor how older adults interact with objects around the home and can provide objective accounts of behaviors to support self-awareness. In this paper, we describe the design and four month deployment of a prototype sensing system that tracks medication taking and phone use in the homes of two older adults. We describe two case studies on 1) how they engaged with the data by looking for and explaining their own anomalous behaviors and 2) how they used the sensor data to reflect on their actions and their own self-awareness of their abilities to remain independent. Finally, we propose recommendations for the design of home sensing systems that support awareness of functional abilities for older adults using reflection.

Real-time feedback for improving medication taking

Medication taking is a self-regulatory process that requires individuals to self-monitor their medication taking behaviors, but this can be difficult because medication taking is such a mundane, unremarkable behavior. Ubiquitous sensing systems have the potential to sense everyday behaviors and provide the objective feedback necessary for self-regulation of medication taking. We describe an unobtrusive sensing system consisting of a sensor-augmented pillbox and an ambient display that provides near real-time visual feedback about how well medications are being taken. In contrast to other systems that focus on reminding before medication taking, our approach uses feedback after medication taking to allow the individual to develop their own routines through self-regulation. We evaluated this system in the homes of older adults in a 10-month deployment. Feedback helped improve the consistency of medication-taking behaviors as well as increased ratings of self-efficacy. However, the improved performance did not persist after the feedback display was removed, because individuals had integrated the feedback display into their routines to support their self-awareness, identify mistakes, guide the timing of medication taking, and provide a sense of security that they are taking their medications well. Finally, we reflect on design considerations for feedback systems to support the process of self-regulation of everyday behaviors.

Sensor-based observations of daily living for aging in place

Older adults often find it difficult to keep track of their cognitive and functional abilities required for remaining independent in their homes. Healthcare providers need objective, timely, and ecologically valid information about their patient’s behaviors at home for assessing their patient’s condition and whether behavioral problems such as non-adherence are significant problems. Ubiquitous sensors in the home can be used to monitor patient-centered observations of daily living (ODLs) about how well individuals carry out specific tasks that indicate the individual’s abilities for living independently. This article demonstrates through a series of case studies that reviewing ODL data about medication taking, phone use, and coffee making can help individual improve their self-awareness of their abilities and also motivate them to improve their performance of their behaviors at least temporarily. This article also demonstrates how physicians when reviewing ODL data about their patient’s medication adherence, phone use, and coffee making provided new, provides relevant information for physicians to refine their care plans for their patients. The ODL data indicate to physicians which functional areas the patient was performing well and which areas the patient required closer follow-up or immediate attention. Integrating ODLs into clinical workflows remains a challenge; however, the case studies in this article highlight that the preferred setting to review ODL data was during the patient’s visit to the physician’s office, so the physician and patient can have a shared understanding of the patient’s functioning at home. More proactive reviews of ODLs can be triggered by well-tuned alerts received by other members of the clinical team or informal care network. In summary, observations of daily living about the everyday actions of individuals, if reviewed, can result in greater awareness, motivation for improving behaviors, and better-informed decision making about an individual’s health care, enabling individuals to maintain their functional abilities as they age.

Embedded assessment of aging adults: A concept validation with stakeholders

Sensing systems embedded in the homes of elders have the potential to monitor individuals for early signs of functional and cognitive decline. However, it is not clear how the data collected from these embedded assessment systems can be useful for elders to support awareness and for their doctors to make better diagnoses. We conduct a concept validation of embedded assessment sensing concepts with elders, family caregivers, and clinicians. We describe their reactions to the sensing concepts, their different information needs, how they would use the information, and what limits its usefulness and provide recommendations for designers of embedded assessment systems.

Using lifelogging to support recollection for people with episodic memory impairment and their caregivers

We designed a lifelogging system to assist people with episodic memory impairment (EMI) to reminisce about recent experiences and reduce the burden on their caregivers. Our system uses a mobile passive capture approach so that the person with EMI does not have to remember to initiate capture. Our system leverages both automated content and context analysis and the expertise of the caregiver to extract the most salient cues from the lifelog. Finally, our system structures the cue review interaction so that it allows the person with EMI to think more deeply about each cue and remember the details of their experiences without repeatedly burdening the caregiver.

Eye of the Beholder: Phone-Based Text-Recognition for the Visually-Impaired

Blind and visually-impaired people cannot access essential information in the form of written text in our environment (e.g., on restaurant menus, street signs, door labels, product names and instructions, expiration dates). In this paper, we present and evaluate a mobile text-recognition system capable of extracting written information from a wide variety of sources and communicating it on-demand to the user. The user needs no additional hardware except an ordinary, Internet- enabled mobile camera-phone - a device that many visually-impaired individuals already own. This approach fills a gap in assistive technologies for the visually- impaired because it makes users aware of textual information not available to them through any other means.

Embedded assessment of wellness with smart home sensors

Embedded home sensors hold the promise of helping older adults age in place. They can help older adults maintain awareness of their functional abilities, a critical step for early detection of decline. In this proposal, I describe my research in understanding how to design and deploy home sensors that monitor how well individuals perform everyday activities. These systems collect an overwhelming amount of data, and thus I will identify the information needs of stakeholders to inform the design of salient summaries of the data for elders, their family caregivers, and their doctors to become more aware of changes functional abilities.

Creating salient summaries of home activity lifelog data

Keeping track of the fluctuations in functional abilities that elders experience is important for early detection of cognitive decline and maintaining independence. In this proposal, I describe my research in understanding how to design ubiquitous home sensor systems that can monitor how well individuals carry out everyday activities important for independence. These systems collect an overwhelmingly large amount of data and thus only the most salient details need to be presented. I will identify the information needs of stakeholders to inform the design of salient summaries of the data for elders, their family caregivers, their doctors, and their therapists to become more aware of changes functional abilities. I also describe the technical, HCI, and clinical contributions of this work.