Susana B. Martins

Evaluation of the Acceptability and Usability of a Decision Support System to Encourage Safe and Effective Use of Opioid Therapy for Chronic, Noncancer Pain by Primary Care Providers

OBJECTIVE To develop and evaluate a clinical decision support system (CDSS) named Assessment and Treatment in Healthcare: Evidenced-Based Automation (ATHENA)-Opioid Therapy, which encourages safe and effective use of opioid therapy for chronic, noncancer pain. DESIGN CDSS development and iterative evaluation using the analysis, design, development, implementation, and evaluation process including simulation-based and in-clinic assessments of usability for providers followed by targeted system revisions. RESULTS Volunteers provided detailed feedback to guide improvements in the graphical user interface, and content and design changes to increase clinical usefulness, understandability, clinical workflow fit, and ease of completing guideline recommended practices. Revisions based on feedback increased CDSS usability ratings over time. Practice concerns outside the scope of the CDSS were also identified. CONCLUSIONS Usability testing optimized the CDSS to better address barriers such as lack of provider education, confusion in dosing calculations and titration schedules, access to relevant patient information, provider discontinuity, documentation, and access to validated assessment tools. It also highlighted barriers to good clinical practice that are difficult to address with CDSS technology in its current conceptualization. For example, clinicians indicated that constraints on time and competing priorities in primary care, discomfort in patient-provider communications, and lack of evidence to guide opioid prescribing decisions impeded their ability to provide effective, guideline-adherent pain management. Iterative testing was essential for designing a highly usable and acceptable CDSS; however, identified barriers may limit the impact of the ATHENA-Opioid Therapy system and other CDSS on clinical practices and outcomes unless CDSS are paired with parallel initiatives to address these issues.

Quality of care for patients with multiple chronic conditions: the role of comorbidity interrelatedness.

ABSTRACTMultimorbidity—the presence of multiple chronic conditions in a patient—has a profound impact on health, health care utilization, and associated costs. Definitions of multimorbidity in clinical care and research have evolved over time, initially focusing on a patient’s number of comorbidities and the associated magnitude of required care processes, and later recognizing the potential influence of comorbidity characteristics on patient care and outcomes. In this article, we review the relationship between multimorbidity and quality of care, and discuss how this relationship may be mediated by the degree to which conditions interact with one another to generate clinical complexity (comorbidity interrelatedness). Drawing on established theoretical frameworks from cognitive engineering and biomedical informatics, we describe how interactions among conditions result in clinical complexity and may affect quality of care. We discuss how this comorbidity interrelatedness influences the value of existing quality guidelines and performance metrics, and describe opportunities to quantify this construct using data widely available through electronic health records. Incorporating comorbidity interrelatedness into conceptualizations of multimorbidity has the potential to enhance clinical and research efforts that aim to improve care for patients with multiple chronic conditions.

A quantitative assessment of a methodology for collaborative specification and evaluation of clinical guidelines

We introduce a three-phase, nine-step methodology for specification of clinical guidelines (GLs) by expert physicians, clinical editors, and knowledge engineers and for quantitative evaluation of the specifications quality. We applied this methodology to a particular framework for incremental GL structuring (mark-up) and to GLs in three clinical domains. A gold-standard mark-up was created, including 196 plans and subplans, and 326 instances of ontological knowledge roles (KRs). A completeness measure of the acquired knowledge revealed that 97% of the plans and 91% of the KR instances of the GLs were recreated by the clinical editors. A correctness measure often revealed high variability within clinical editor pairs structuring each GL, but for all GLs and clinical editors the specification quality was significantly higher than random (p<0.01). Procedural KRs were more difficult to mark-up than declarative KRs. We conclude that given an ontology-specific consensus, clinical editors with mark-up training can structure GL knowledge with high completeness, whereas the main demand for correct structuring is training in the ontologys semantics.

A Comparative Evaluation of Full-text, Concept-based, and Context-sensitive Search

OBJECTIVES Study comparatively (1) concept-based search, using documents pre-indexed by a conceptual hierarchy; (2) context-sensitive search, using structured, labeled documents; and (3) traditional full-text search. Hypotheses were: (1) more contexts lead to better retrieval accuracy; and (2) adding concept-based search to the other searches would improve upon their baseline performances. DESIGN Use our Vaidurya architecture, for search and retrieval evaluation, of structured documents classified by a conceptual hierarchy, on a clinical guidelines test collection. MEASUREMENTS Precision computed at different levels of recall to assess the contribution of the retrieval methods. Comparisons of precisions done with recall set at 0.5, using t-tests. RESULTS Performance increased monotonically with the number of query context elements. Adding context-sensitive elements, mean improvement was 11.1% at recall 0.5. With three contexts, mean query precision was 42% +/- 17% (95% confidence interval [CI], 31% to 53%); with two contexts, 32% +/- 13% (95% CI, 27% to 38%); and one context, 20% +/- 9% (95% CI, 15% to 24%). Adding context-based queries to full-text queries monotonically improved precision beyond the 0.4 level of recall. Mean improvement was 4.5% at recall 0.5. Adding concept-based search to full-text search improved precision to 19.4% at recall 0.5. CONCLUSIONS The study demonstrated usefulness of concept-based and context-sensitive queries for enhancing the precision of retrieval from a digital library of semi-structured clinical guideline documents. Concept-based searches outperformed free-text queries, especially when baseline precision was low. In general, the more ontological elements used in the query, the greater the resulting precision.

Designing an automated clinical decision support system to match clinical practice guidelines for opioid therapy for chronic pain

BackgroundOpioid prescribing for chronic pain is common and controversial, but recommended clinical practices are followed inconsistently in many clinical settings. Strategies for increasing adherence to clinical practice guideline recommendations are needed to increase effectiveness and reduce negative consequences of opioid prescribing in chronic pain patients.MethodsHere we describe the process and outcomes of a project to operationalize the 2003 VA/DOD Clinical Practice Guideline for Opioid Therapy for Chronic Non-Cancer Pain into a computerized decision support system (DSS) to encourage good opioid prescribing practices during primary care visits. We based the DSS on the existing ATHENA-DSS. We used an iterative process of design, testing, and revision of the DSS by a diverse team including guideline authors, medical informatics experts, clinical content experts, and end-users to convert the written clinical practice guideline into a computable algorithm to generate patient-specific recommendations for care based upon existing information in the electronic medical record (EMR), and a set of clinical tools.ResultsThe iterative revision process identified numerous and varied problems with the initially designed system despite diverse expert participation in the design process. The process of operationalizing the guideline identified areas in which the guideline was vague, left decisions to clinical judgment, or required clarification of detail to insure safe clinical implementation. The revisions led to workable solutions to problems, defined the limits of the DSS and its utility in clinical practice, improved integration into clinical workflow, and improved the clarity and accuracy of system recommendations and tools.ConclusionsUse of this iterative process led to development of a multifunctional DSS that met the approval of the clinical practice guideline authors, content experts, and clinicians involved in testing. The process and experiences described provide a model for development of other DSSs that translate written guidelines into actionable, real-time clinical recommendations.

Association of Antihypertensive Therapy and Diastolic Hypotension in Chronic Kidney Disease

The extent to which chronic kidney disease (CKD) affects achievement of blood pressure targets is not comprehensively understood. We evaluated the effects of CKD (estimated glomerular filtration rate: <60 mL/min per 1.73 m2) on achievement of blood pressure control (nondiabetic: <140/90 mm Hg; diabetic: <130/85 mm Hg) using data from the Guidelines for Drug Therapy of Hypertension Trial. This 15-month study obtained outpatient blood pressures from 3 Veteran’s Affairs institutions. Among 9985 subjects with hypertension, we evaluated the association of CKD with achieved control and antihypertensive medication use. We also explored the association between the number of antihypertensives and systolic, diastolic, and pulse pressure. After 15 months, 41% of participants met blood pressure targets. CKD was not associated with control (adjusted odds ratio: 1.04; 95% CI: 0.93 to 1.15). However, CKD was associated with higher odds of use of ≥3 medications among nondiabetic subjects (odds ratio: 1.46; 95% CI: 1.25 to 1.71) and diabetic subjects (odds ratio: 1.40; 95% CI: 1.17 to 1.66). A significant interaction was observed between CKD and the number of antihypertensives as determinants of diastolic and pulse pressures. Among non-CKD participants, a greater number of antihypertensives (0 compared with 4) was associated with wider pulse pressure (&Dgr;5.2 mm Hg; P<0.001), mainly because of higher systolic pressures (&Dgr;3.6 mm Hg; P=0.001). Among participants with CKD, although greater numbers of antihypertensives were associated with even wider pulse pressures (&Dgr;8.3 mm Hg; P<0.001), this was primarily because of lower diastolic pressures (&Dgr;4.8 mm Hg; P<0.01). Among participants with CKD, greater use of antihypertensives was associated with lower diastolic pressures. Given recent evidence suggesting adverse effects of diastolic hypotension, these results suggest potential risks in patients with CKD from aggressive attempts to control systolic blood pressure.

Epoch: an ontological framework to support clinical trials management

The increasing complexity of clinical trials has generated an enormous requirement for knowledge and information specification at all stages of the trials, including planning, documentation, implementation, and analysis. We are building a knowledge-based framework (Epoch) to support the management of clinical trials. We are tailoring this approach to the Immune Tolerance Network (ITN), an international research consortium developing new therapeutics in immune-mediated disorders. In the broad spectrum of trial management activities, we currently target two areas that are vital to the successful implementation of a trial: (1) tracking study participants as they advance through the trials, and (2) tracking biological specimens as they are processed at the trial laboratories. The core of our software architecture is a suite of ontologies that conceptualizes relevant clinical trial domain. Our approach can provide ITN and other research organizations a stable and consistent knowledge source for clinical-trial software applications.

Development and applications of the Veterans Health Administration’s Stratification Tool for Opioid Risk Mitigation (STORM) to improve opioid safety and prevent overdose and suicide.

Concerns about opioid-related adverse events, including overdose, prompted the Veterans Health Administration (VHA) to launch an Opioid Safety Initiative and Overdose Education and Naloxone Distribution program. To mitigate risks associated with opioid prescribing, a holistic approach that takes into consideration both risk factors (e.g., dose, substance use disorders) and risk mitigation interventions (e.g., urine drug screening, psychosocial treatment) is needed. This article describes the Stratification Tool for Opioid Risk Mitigation (STORM), a tool developed in VHA that reflects this holistic approach and facilitates patient identification and monitoring. STORM prioritizes patients for review and intervention according to their modeled risk for overdose/suicide-related events and displays risk factors and risk mitigation interventions obtained from VHA electronic medical record (EMR)-data extracts. Patients’ estimated risk is based on a predictive risk model developed using fiscal year 2010 (FY2010: 10/1/2009–9/30/2010) EMR-data extracts and mortality data among 1,135,601 VHA patients prescribed opioid analgesics to predict risk for an overdose/suicide-related event in FY2011 (2.1% experienced an event). Cross-validation was used to validate the model, with receiver operating characteristic curves for the training and test data sets performing well (>.80 area under the curve). The predictive risk model distinguished patients based on risk for overdose/suicide-related adverse events, allowing for identification of high-risk patients and enrichment of target populations of patients with greater safety concerns for proactive monitoring and application of risk mitigation interventions. Results suggest that clinical informatics can leverage EMR-extracted data to identify patients at-risk for overdose/suicide-related events and provide clinicians with actionable information to mitigate risk.

Ability of expert physicians to structure clinical guidelines: reality versus perception

RATIONALE, AIMS AND OBJECTIVES Structuring Textual Clinical Guidelines (GLs) into a formal representation is a necessary prerequisite for supporting their automated application. We had developed a collaborative guideline-structuring methodology that involves expert physicians, clinical editors and knowledge engineers, to produce a machine-comprehensible representation for automated support of evidence-based, guideline-based care. Our goals in the current study were: (1) to investigate the perceptions of the expert physicians and clinical editors as to the relative importance, for the structuring process, of different aspects of the methodology; (2) to assess, for the clinical editors, the inter-correlations among (i) the reported level of understanding of the guideline structuring ontologys (knowledge schemes) features, (ii) the reported ease of structuring each feature and (iii) the actual objective quality of structuring. METHODS A clinical consensus regarding the contents of three guidelines was prepared by an expert in the domain of each guideline. For each guideline, two clinical editors independently structured the guideline into a semi-formal representation, using the Asbru guideline ontologys features. The quality of the resulting structuring was assessed quantitatively. Each expert physician was asked which aspects were most useful for formation of the consensus. Each clinical editor filled questionnaires relating to: (1) the level of understanding of the ontologys features (before the structuring process); (2) the usefulness of various aspects in the structuring process (after the structuring process); (3) the ease of structuring each ontological feature (after the structuring process). Subjective reports were compared with objective quantitative measures of structuring correctness. RESULTS Expert physicians considered having medical expertise and understanding the ontological features as the aspects most useful for creation of a consensus. Clinical editors considered understanding the ontological features and the use of the structuring tools as the aspects most useful for structuring guidelines. There was a positive correlation (R = 0.87, P < 0.001) between the reported ease of understanding ontological features and the reported ease of structuring those features. However, there was no significant correlation between the reported level of understanding the features - or the reported ease of structuring by using those features - and the objective quality of the structuring of these features in actual guidelines. CONCLUSIONS Aspects considered important for formation of a clinical consensus differ from those for structuring of guidelines. Understanding the features of a structuring ontology is positively correlated with the reported ease of using these features, but neither of these subjective reports correlated with the actual objective quality of the structuring using these features.

Can Physicians Structure Clinical Guidelines? Experiments with a Mark-Up-Process Methodology

We have previously developed an architecture and a set of tools called the Digital electronic Guideline Library (DeGeL), which includes a web-based tool for structuring (marking-up) free-text clinical guidelines (GLs), namely, the URUZ Mark-up tool. In this study, we developed and evaluated a methodology and a tool for a mark-up-based specification and assessment of the quality of that specification, of procedural and declarative knowledge in clinical GLs. The methodology includes all necessary activities before, during and after the mark-up process, and supports specification and conversion of the GLs free-text representation through semi-structured and semi-formal representations into a machine comprehensible representation. For the evaluation of this methodology, three GLs from different medical disciplines were selected. For each GL, as an indispensable step, an ontology-specific consensus was created, determined by a group of expert physicians and knowledge engineers, based on GL source. For each GL, two mark-ups in a chosen GL ontology (Asbru) were created by a distinct clinical editor; each of the clinical editors created a semi-formal mark-up of the GL using the URUZ tool. To evaluate each mark-up, a gold standard mark-up was created by collaboration of physician and knowledge engineer, and a specialized mark-up-evaluation tool was developed, which enables assessment of completeness, as well as of syntactic and semantic correctness of the mark-up. Subjective and objective measures were defined for qualitative and quantitative evaluation of the correctness (soundness) and completeness of the marked-up knowledge, with encouraging results.