Andrea L. Hartzler

Practical considerations in genomic decision support: The eMERGE experience

Background: Genomic medicine has the potential to improve care by tailoring treatments to the individual. There is consensus in the literature that pharmacogenomics (PGx) may be an ideal starting point for real-world implementation, due to the presence of well-characterized drug-gene interactions. Clinical Decision Support (CDS) is an ideal avenue by which to implement PGx at the bedside. Previous literature has established theoretical models for PGx CDS implementation and discussed a number of anticipated real-world challenges. However, work detailing actual PGx CDS implementation experiences has been limited. Anticipated challenges include data storage and management, system integration, physician acceptance, and more. Methods: In this study, we analyzed the experiences of ten members of the Electronic Medical Records and Genomics (eMERGE) Network, and one affiliate, in their attempts to implement PGx CDS. We examined the resulting PGx CDS system characteristics and conducted a survey to understand the unanticipated implementation challenges sites encountered. Results: Ten sites have successfully implemented at least one PGx CDS rule in the clinical setting. The majority of sites elected to create an Omic Ancillary System (OAS) to manage genetic and genomic data. All sites were able to adapt their existing CDS tools for PGx knowledge. The most common and impactful delays were not PGx-specific issues. Instead, they were general IT implementation problems, with top challenges including team coordination/communication and staffing. The challenges encountered caused a median total delay in system go-live of approximately two months. Conclusions: These results suggest that barriers to PGx CDS implementations are generally surmountable. Moreover, PGx CDS implementation may not be any more difficult than other healthcare IT projects of similar scope, as the most significant delays encountered were not unique to genomic medicine. These are encouraging results for any institution considering implementing a PGx CDS tool, and for the advancement of genomic medicine.

Automatically Detecting Failures in Natural Language Processing Tools for Online Community Text

Background The prevalence and value of patient-generated health text are increasing, but processing such text remains problematic. Although existing biomedical natural language processing (NLP) tools are appealing, most were developed to process clinician- or researcher-generated text, such as clinical notes or journal articles. In addition to being constructed for different types of text, other challenges of using existing NLP include constantly changing technologies, source vocabularies, and characteristics of text. These continuously evolving challenges warrant the need for applying low-cost systematic assessment. However, the primarily accepted evaluation method in NLP, manual annotation, requires tremendous effort and time. Objective The primary objective of this study is to explore an alternative approach—using low-cost, automated methods to detect failures (eg, incorrect boundaries, missed terms, mismapped concepts) when processing patient-generated text with existing biomedical NLP tools. We first characterize common failures that NLP tools can make in processing online community text. We then demonstrate the feasibility of our automated approach in detecting these common failures using one of the most popular biomedical NLP tools, MetaMap. Methods Using 9657 posts from an online cancer community, we explored our automated failure detection approach in two steps: (1) to characterize the failure types, we first manually reviewed MetaMap’s commonly occurring failures, grouped the inaccurate mappings into failure types, and then identified causes of the failures through iterative rounds of manual review using open coding, and (2) to automatically detect these failure types, we then explored combinations of existing NLP techniques and dictionary-based matching for each failure cause. Finally, we manually evaluated the automatically detected failures. Results From our manual review, we characterized three types of failure: (1) boundary failures, (2) missed term failures, and (3) word ambiguity failures. Within these three failure types, we discovered 12 causes of inaccurate mappings of concepts. We used automated methods to detect almost half of 383,572 MetaMap’s mappings as problematic. Word sense ambiguity failure was the most widely occurring, comprising 82.22% of failures. Boundary failure was the second most frequent, amounting to 15.90% of failures, while missed term failures were the least common, making up 1.88% of failures. The automated failure detection achieved precision, recall, accuracy, and F1 score of 83.00%, 92.57%, 88.17%, and 87.52%, respectively. Conclusions We illustrate the challenges of processing patient-generated online health community text and characterize failures of NLP tools on this patient-generated health text, demonstrating the feasibility of our low-cost approach to automatically detect those failures. Our approach shows the potential for scalable and effective solutions to automatically assess the constantly evolving NLP tools and source vocabularies to process patient-generated text.

Relevance of graph literacy in the development of patient-centered communication tools

OBJECTIVE To determine the literacy skill sets of patients in the context of graphical interpretation of interactive dashboards. METHODS We assessed literacy characteristics of prostate cancer patients and assessed comprehension of quality of life dashboards. Health literacy, numeracy and graph literacy were assessed with validated tools. We divided patients into low vs. high numeracy and graph literacy. We report descriptive statistics on literacy, dashboard comprehension, and relationships between groups. We used correlation and multiple linear regressions to examine factors associated with dashboard comprehension. RESULTS Despite high health literacy in educated patients (78% college educated), there was variation in numeracy and graph literacy. Numeracy and graph literacy scores were correlated (r=0.37). In those with low literacy, graph literacy scores most strongly correlated with dashboard comprehension (r=0.59-0.90). On multivariate analysis, graph literacy was independently associated with dashboard comprehension, adjusting for age, education, and numeracy level. CONCLUSIONS Even among higher educated patients; variation in the ability to comprehend graphs exists. PRACTICE IMPLICATIONS Clinicians must be aware of these differential proficiencies when counseling patients. Tools for patient-centered communication that employ visual displays need to account for literacy capabilities to ensure that patients can effectively engage these resources.

A patient-centered system in a provider-centered world: challenges of incorporating post-discharge wound data into practice

OBJECTIVE The proposed Meaningful Use Stage 3 recommendations require healthcare providers to accept patient-generated health data (PGHD) by 2017. Yet, we know little about the tensions that arise in supporting the needs of both patients and providers in this context. We sought to examine these tensions when designing a novel, patient-centered technology - mobile Post-Operative Wound Evaluator (mPOWEr) - that uses PGHD for post-discharge surgical wound monitoring. MATERIALS AND METHODS As part of the iterative design process of mPOWEr, we conducted semistructured interviews and think-aloud sessions using mockups with surgical patients and providers. We asked participants how mPOWEr could enhance the current post-discharge process for surgical patients, then used grounded theory to develop themes related to conflicts and agreements between patients and providers. RESULTS We identified four areas of agreement: providing contextual metadata, accessible and actionable data presentation, building on existing sociotechnical systems, and process transparency. We identified six areas of conflict, with patients preferring: more flexibility in data input, frequent data transfer, text-based communication, patient input in provider response prioritization, timely and reliable provider responses, and definitive diagnoses. DISCUSSION We present design implications and potential solutions to the identified conflicts for each theme, illustrated using our work on mPOWEr. Our experience highlights the importance of bringing a variety of stakeholders, including patients, into the design process for PGHD applications. CONCLUSION We have identified critical barriers to integrating PGHD into clinical care and describe design implications to help address these barriers. Our work informs future efforts to ensure the smooth integration of essential PGHD into clinical practice.

Integrating Patient-Reported Outcomes into Spine Surgical Care through Visual Dashboards: Lessons Learned from Human-Centered Design.

Introduction: The collection of patient-reported outcomes (PROs) draws attention to issues of importance to patients—physical function and quality of life. The integration of PRO data into clinical decisions and discussions with patients requires thoughtful design of user-friendly interfaces that consider user experience and present data in personalized ways to enhance patient care. Whereas most prior work on PROs focuses on capturing data from patients, little research details how to design effective user interfaces that facilitate use of this data in clinical practice. We share lessons learned from engaging health care professionals to inform design of visual dashboards, an emerging type of health information technology (HIT). Methods: We employed human-centered design (HCD) methods to create visual displays of PROs to support patient care and quality improvement. HCD aims to optimize the design of interactive systems through iterative input from representative users who are likely to use the system in the future. Through three major steps, we engaged health care professionals in targeted, iterative design activities to inform the development of a PRO Dashboard that visually displays patient-reported pain and disability outcomes following spine surgery. Findings: Design activities to engage health care administrators, providers, and staff guided our work from design concept to specifications for dashboard implementation. Stakeholder feedback from these health care professionals shaped user interface design features, including predefined overviews that illustrate at-a-glance trends and quarterly snapshots, granular data filters that enable users to dive into detailed PRO analytics, and user-defined views to share and reuse. Feedback also revealed important considerations for quality indicators and privacy-preserving sharing and use of PROs. Conclusion: Our work illustrates a range of engagement methods guided by human-centered principles and design recommendations for optimizing PRO Dashboards for patient care and quality improvement. Engaging health care professionals as stakeholders is a critical step toward the design of user-friendly HIT that is accepted, usable, and has the potential to enhance quality of care and patient outcomes.

Healthcare provider education to support integration of pharmacogenomics in practice: the eMERGE Network experience

Ten organizations within the Electronic Medical Records and Genomics Network developed programs to implement pharmacogenomic sequencing and clinical decision support into clinical settings. Recognizing the importance of informed prescribers, a variety of strategies were used to incorporate provider education to support implementation. Education experiences with pharmacogenomics are described within the context of each organizations prior involvement, including the scope and scale of implementation specific to their Electronic Medical Records and Genomics projects. We describe common and distinct education strategies, provide exemplars and share challenges. Lessons learned inform future perspectives. Future pharmacogenomics clinical implementation initiatives need to include funding toward implementing provider education and evaluating outcomes.

Validity and reliability of a smartphone application for the assessment of penile deformity in Peyronie's disease.

INTRODUCTION Available methods to evaluate men with Peyronies disease (PD) are limited by the inability to accurately and reproducibly measure penile deformity. AIM.: The study aims to evaluate the performance of a smartphone application for the measurement of penile curvature and narrowing. METHODS A smartphone application, the University of Washington Peyronies Examination Network (UWPEN), was developed for this purpose. To assess penile curvature, 15 single cylinders of malleable penile prostheses were molded to varying curvature angles. Three blinded observers nonsequentially measured the angle of curvature for each prosthetic cylinder using a protractor, goniometer, and UWPEN. To assess girth narrowing, six clay models of the penile shaft were constructed to represent conditions of normal, partial hourglass, circumferential hourglass, and pencil narrowing. Girth was measured using a ruler and UWPEN by the same blinded observers. MAIN OUTCOME MEASURES Statistical analyses compared intertest, interobserver, and intraobserver reliability using the interclass correlation coefficient (ICC). An ICC above 0.75 indicates excellent reproducibility among measurements. RESULTS Intertest reliability for angle measurements yielded an ICC for the three methods of 1.000. Separately, the ICC for UWPEN vs. the goniometer and protractor was 0.999 and 0.999, respectively. The interobserver ICC for UWPEN, goniometer, and protractor was 0.998, 0.999, and 1.000, respectively. Intertest reliability for girth narrowing measurements yielded an ICC of 0.991. The interobserver ICC for girth narrowing for UWPEN and the ruler was 0.978 and 0.986, respectively. Intraobserver ICC for angle measurements and girth narrowing showed high reliability for all observers and methods. CONCLUSIONS The performance of UWPEN is comparable with and highly correlated with angle measurements obtained from the goniometer and protractor as well as with girth narrowing measurements obtained from a ruler. Measurements are reproducible among different observers. UWPEN may provide a noninvasive, accurate, reliable, and widely accessible method to characterize and track PD over time.

A Prognostic Model of Surgical Site Infection Using Daily Clinical Wound Assessment

BACKGROUND Surgical site infection (SSI) remains a common, costly, and morbid health care-associated infection. Early detection can improve outcomes, yet previous risk models consider only baseline risk factors (BF) not incorporating a proximate and timely data source-the wound itself. We hypothesize that incorporation of daily wound assessment improves the accuracy of SSI identification compared with traditional BF alone. STUDY DESIGN A prospective cohort of 1,000 post open abdominal surgery patients at an academic teaching hospital were examined daily for serial features (SF), for example, wound characteristics and vital signs, in addition to standard BF, for example, wound class. Using supervised machine learning, we trained 3 Naïve Bayes classifiers (BF, SF, and BF+SF) using patient data from 1 to 5 days before diagnosis to classify SSI on the following day. For comparison, we also created a simplified SF model that used logistic regression. Control patients without SSI were matched on 5 similar consecutive postoperative days to avoid confounding by length of stay. Accuracy, sensitivity/specificity, and area under the receiver operating characteristic curve were calculated on a training and hold-out testing set. RESULTS Of 851 patients, 19.4% had inpatient SSIs. Univariate analysis showed differences in C-reactive protein, surgery duration, and contamination, but no differences in American Society of Anesthesiologists scores, diabetes, or emergency surgery. The BF, SF, and BF+SF classifiers had area under the receiver operating characteristic curves of 0.67, 0.76, and 0.76, respectively. The best-performing classifier (SF) had optimal sensitivity of 0.80, specificity of 0.64, positive predictive value of 0.35, and negative predictive value of 0.93. Features most associated with subsequent SSI diagnosis were granulation degree, exudate amount, nasogastric tube presence, and heart rate. CONCLUSIONS Serial features provided moderate positive predictive value and high negative predictive value for early identification of SSI. Addition of baseline risk factors did not improve identification. Features of evolving wound infection are discernable before the day of diagnosis, based primarily on visual inspection.

Design and Usability of Interactive User Profiles for Online Health Communities

Online health communities provide a rich source of expertise from experienced patients, but uncovering “peer mentors” with shared circumstances is like finding a needle in a haystack—a problem that will escalate as these communities grow and diversify. We investigated interactive health interest profiles (HIPs) that summarize health-related terms extracted from users’ community posts. Through iterative design, we explored practical designs that accommodate differences in users’ community participation in three HIP prototypes: Text, Word Cloud, and Timeline. By comparing prototype usability with patients and design experts, we found that patients accurately used each prototype but completed some tasks faster with the Timeline HIP. Despite this advantage, patients preferred the Text HIP. Design experts and patients agreed that simple data overviews and granular details with salient cues that invite interactivity are key design considerations for HIPs. Findings offer key design considerations for HIPs that patients find most useful when forging critical connections.

Using Heuristic Evaluation to Enhance the Visual Display of a Provider Dashboard for Patient-Reported Outcomes

Introduction: With the rising use of patient-reported outcomes (PRO) in clinical practice, there is an increasing need to understand the data visualization needs of clinical teams to support their effective use of PRO data for both individual patient decision making and broader population health applications. A human-centered design (HCD) approach can optimize the visual design of an interactive PRO system. Including Heuristic Evaluation in the HCD Toolbox: Recent literature regarding the use of HCD to design and develop PRO visualizations demonstrates the benefits of iterative methods that engage representative users who are likely to use the system in the future. However, the literature has not explored the additive value of other HCD methods such as heuristic evaluation, which involves expert examination of the interface with respect to recognized usability principles, the heuristics. Insights from Using Heuristic Evaluation: Our experience in using heuristic evaluation to enhance the design of a PRO dashboard led to several recommendations to improve the display, accessibility, and interpretability of the dashboard’s data. Heuristic evaluation can serve as a complement to HCD methods that directly engage users and thereby enhance usability.