Rachel Ramoni

Detection and characterization of usability problems in structured data entry interfaces in dentistry

BACKGROUND Poor usability is one of the major barriers for optimally using electronic health records (EHRs). Dentists are increasingly adopting EHRs, and are using structured data entry interfaces to enter data such that the data can be easily retrieved and exchanged. Until recently, dentists have lacked a standardized terminology to consistently represent oral health diagnoses. OBJECTIVES In this study we evaluated the usability of a widely used EHR interface that allow the entry of diagnostic terms, using multi-faceted methods to identify problems and work with the vendor to correct them using an iterative design method. METHODS Fieldwork was undertaken at two clinical sites, and dental providers as subjects participated in user testing (n=32), interviews (n=36) and observations (n=24). RESULTS User testing revealed that only 22-41% of users were able to successfully complete a simple task of entering one diagnosis, while no user was able to complete a more complex task. We identified and characterized 24 high-level usability problems reducing efficiency and causing user errors. Interface-related problems included unexpected approaches for displaying diagnosis, lack of visibility, and inconsistent use of UI widgets. Terminology related issues included missing and mis-categorized concepts. Work domain issues involved both absent and superfluous functions. In collaboration with the vendor, each usability problem was prioritized and a timeline set to resolve the concerns. DISCUSSION Mixed methods evaluations identified a number of critical usability issues relating to the user interface, underlying terminology of the work domain. The usability challenges were found to prevent most users from successfully completing the tasks. Our further work we will determine if changes to the interface, terminology and work domain do result in improved usability.

Are three methods better than one? A comparative assessment of usability evaluation methods in an EHR

OBJECTIVE To comparatively evaluate the effectiveness of three different methods involving end-users for detecting usability problems in an EHR: user testing, semi-structured interviews and surveys. MATERIALS AND METHODS Data were collected at two major urban dental schools from faculty, residents and dental students to assess the usability of a dental EHR for developing a treatment plan. These included user testing (N=32), semi-structured interviews (N=36), and surveys (N=35). RESULTS The three methods together identified a total of 187 usability violations: 54% via user testing, 28% via the semi-structured interview and 18% from the survey method, with modest overlap. These usability problems were classified into 24 problem themes in 3 broad categories. User testing covered the broadest range of themes (83%), followed by the interview (63%) and survey (29%) methods. DISCUSSION Multiple evaluation methods provide a comprehensive approach to identifying EHR usability challenges and specific problems. The three methods were found to be complementary, and thus each can provide unique insights for software enhancement. Interview and survey methods were found not to be sufficient by themselves, but when used in conjunction with the user testing method, they provided a comprehensive evaluation of the EHR. CONCLUSION We recommend using a multi-method approach when testing the usability of health information technology because it provides a more comprehensive picture of usability challenges.

Treatment planning in dentistry using an electronic health record: implications for undergraduate education

OBJECTIVE Treatment planning, an essential component of clinical practice, has received little attention in the dental literature and there appears to be no consistent format being followed in the teaching and development of treatment plans within dental school curricula. No investigation, to our knowledge, has been carried out to explore the subject of treatment planning since the advent of electronic health record (EHR) use in dentistry. It is therefore important to examine the topic of treatment planning in the context of EHRs. METHODS This paper reports on how 25 predoctoral dental students from two U.S. schools performed when asked to complete diagnosis and treatment planning exercises for two clinical scenarios in an EHR. Three calibrated clinical teaching faculty scored diagnosis entry, diagnosis-treatment (procedure) pairing, and sequencing of treatment according to criteria taught in their curriculum. Scores were then converted to percent correct and reported as means (with standard deviations). RESULTS Overall, the participants earned 48.2% of the possible points. Participants at School 2 earned a mean of 54.3% compared with participants at School 1, who earned 41.9%. Students fared better selecting the appropriate treatment (59.8%) compared with choosing the correct diagnoses (41.9%) but performed least favorably when organizing the sequence of their treatment plans (41.7%). CONCLUSION Our results highlight the need to improve the current process by which treatment planning is taught and also to consider the impact of technology on the fundamental skills of diagnosis and treatment planning within the modern educational setting.

The dangers of dental devices as reported in the Food and Drug Administration Manufacturer and User Facility Device Experience Database.

BACKGROUND The authors conducted a study to determine the frequency and type of adverse events (AEs) associated with dental devices reported to the Food and Drug Administration Manufacturer and User Facility Device Experience (MAUDE) database. METHODS The authors downloaded and reviewed the dental device-related AEs reported to MAUDE from January 1, 1996, through December 31,2011. RESULTS MAUDE received a total of 1,978,056 reports between January 1, 1996, and December 31, 2011. Among these reports, 28,046 (1.4%) AE reports were associated with dental devices. Within the dental AE reports that had event type information, 17,261 reported injuries, 7,777 reported device malfunctions, and 66 reported deaths. Among the 66 entries classified as death reports, 52 reported a death in the description; the remaining were either misclassified or lacked sufficient information in the report to determine whether a death had occurred. Of the dental device-associated AEs, 53.5% pertained to endosseous implants. CONCLUSIONS A plethora of devices are used in dental care. To achieve Element 1 of Agency for Healthcare Research and Qualitys Patient Safety Initiative, clinicians and researchers must be able to monitor the safety of dental devices. Although MAUDE was identified by the authors as essentially the sole source of this valuable information on adverse events, their investigations led them to conclude that MAUDE had substantial limitations that prevent it from being the broad-based patient safety sentinel the profession requires. PRACTICAL IMPLICATIONS As potential contributors to MAUDE, dental care teams play a key role in improving the professions access to information about the safety of dental devices.

Data Sharing in the Undiagnosed Diseases Network

The Undiagnosed Diseases Network (UDN) builds on the successes of the Undiagnosed Diseases Program at the National Institutes of Health (NIH UDP). Through support from the NIH Common Fund, a coordinating center, six additional clinical sites, and two sequencing cores comprise the UDN. The objectives of the UDN are to: (1) improve the level of diagnosis and care for patients with undiagnosed diseases through the development of common protocols designed by an enlarged community of investigators across the network; (2) facilitate research into the etiology of undiagnosed diseases, by collecting and sharing standardized, high‐quality clinical and laboratory data including genotyping, phenotyping, and environmental exposure data; and (3) create an integrated and collaborative research community across multiple clinical sites, and among laboratory and clinical investigators, to investigate the pathophysiology of these rare diseases and to identify options for patient management. Broad‐based data sharing is at the core of achieving these objectives, and the UDN is establishing the policies and governance structure to support broad data sharing.

Dental clinical research: an illustration of the value of standardized diagnostic terms

Abstract Objective Secondary data are a significant resource for in‐depth epidemiologic and public health research. It also allows for effective quality control and clinical outcomes measurement. To illustrate the value of structured diagnostic entry, a use case was developed to quantify adherence to current practice guidelines for managing chronic moderate periodontitis (CMP). Methods Six dental schools using the same electronic health record (EHR) contribute data to a dental data repository (BigMouth) based on the i2b2 data‐warehousing platform. Participating institutions are able to query across the full repository without being able to back trace specific data to its originating institution. At each of the three sites whose data are included in this analysis, the Dental Diagnostic System (DDS) terminology was used to document diagnoses in the clinics. We ran multiple queries against this multi‐institutional database, and the output was validated by manually reviewing a subset of patient charts. Results Over the period under study, 1,866 patients were diagnosed with CMP. Of these, 15 percent received only periodontal prophylaxis treatment, 20 percent received only periodontal maintenance treatment, and only 41 percent received periodontal maintenance treatment in combination with other AAP guideline treatments. Conclusions Our results showed that most patients with CMP were not treated according to the AAP guidelines. On the basis of this use case, we conclude that the availability and habitual use of a structured diagnosis in an EHR allow for the aggregation and secondary analyses of clinical data to support downstream analyses for quality improvement and epidemiological assessments.

Original ContributionsPatient SafetyThe dangers of dental devices as reported in the Food and Drug Administration Manufacturer and User Facility Device Experience Database

BACKGROUND The authors conducted a study to determine the frequency and type of adverse events (AEs) associated with dental devices reported to the Food and Drug Administration Manufacturer and User Facility Device Experience (MAUDE) database. METHODS The authors downloaded and reviewed the dental device-related AEs reported to MAUDE from January 1, 1996, through December 31,2011. RESULTS MAUDE received a total of 1,978,056 reports between January 1, 1996, and December 31, 2011. Among these reports, 28,046 (1.4%) AE reports were associated with dental devices. Within the dental AE reports that had event type information, 17,261 reported injuries, 7,777 reported device malfunctions, and 66 reported deaths. Among the 66 entries classified as death reports, 52 reported a death in the description; the remaining were either misclassified or lacked sufficient information in the report to determine whether a death had occurred. Of the dental device-associated AEs, 53.5% pertained to endosseous implants. CONCLUSIONS A plethora of devices are used in dental care. To achieve Element 1 of Agency for Healthcare Research and Qualitys Patient Safety Initiative, clinicians and researchers must be able to monitor the safety of dental devices. Although MAUDE was identified by the authors as essentially the sole source of this valuable information on adverse events, their investigations led them to conclude that MAUDE had substantial limitations that prevent it from being the broad-based patient safety sentinel the profession requires. PRACTICAL IMPLICATIONS As potential contributors to MAUDE, dental care teams play a key role in improving the professions access to information about the safety of dental devices.

Evaluating quality of dental care among patients with diabetes: Adaptation and testing of a dental quality measure in electronic health records

BACKGROUND Patients with diabetes are at increased risk of developing oral complications, and annual dental examinations are an endorsed preventive strategy. The authors evaluated the feasibility and validity of implementing an automated electronic health record (EHR)-based dental quality measure to determine whether patients with diabetes received such evaluations. METHODS The authors selected a Dental Quality Alliance measure developed for claims data and adapted the specifications for EHRs. Automated queries identified patients with diabetes across 4 dental institutions, and the authors manually reviewed a subsample of charts to evaluate query performance. After assessing the initial EHR measure, the authors defined and tested a revised EHR measure to capture better the oral care received by patients with diabetes. RESULTS In the initial and revised measures, the authors used EHR automated queries to identify 12,960 and 13,221 patients with diabetes, respectively, in the reporting year. Variations in the measure scores across sites were greater with the initial measure (range, 36.4-71.3%) than with the revised measure (range, 78.8-88.1%). The automated query performed well (93% or higher) for sensitivity, specificity, and positive and negative predictive values for both measures. CONCLUSIONS The results suggest that an automated EHR-based query can be used successfully to measure the quality of oral health care delivered to patients with diabetes. The authors also found that using the rich data available in EHRs may help estimate the quality of care better than can relying on claims data. PRACTICAL IMPLICATIONS Detailed clinical patient-level data in dental EHRs may be useful to dentists in evaluating the quality of dental care provided to patients with diabetes.

Applying HCI Principles in Designing Usable Systems for Dentistry

Dentistry and medicine both demand that their practice teams perform technical procedures in dynamic environments, and both have been thrust in to the electronic world after centuries of relying upon paper documentation and communication. Unlike medicine, though, dentistry’s ability to document and communicate has been hampered by the lack of an adequate and available diagnostic terminology. While the International Classification of Disease (ICD) has long been part and parcel of medical practice, dentists have only had access to standardized terms to describe procedures. This documentation gap has been thrown into stark relief in the era of electronic health records, in which structured data is most directly useful for secondary analysis. In this chapter, we describe an application of HCI principles to our effort to bring a newly developed dental diagnostic terminology into dental clinics through implementation in a dental EHR. Because we had an adequate budget, time, and access to a broad spectrum of committed stakeholders, including the EHR vendor, practicing dentists, epidemiologist, informaticians, we were able to follow an iterative participatory design approach to refine both the terminology and the interface to the terminology. A leaner approach, such as heuristic evaluation alone, would have been more appropriate had we not had access to such a wealth of resources, though such scaling-back would, of course, have impacted our ability to identify and address usability challenges. This highly collaborative work was undertaken in an effort to enhance the profession of dentistry by narrowing the gap between what is useful to secondary users of data and what is usable from the perspective of primary users.