K Ann McKibbon

Online Access to MEDLINE in Clinical Settings: A Study of Use and Usefulness

STUDY OBJECTIVE We introduced self-service access to the medical literature database, MEDLINE, into clinical settings to assess the frequency, patterns, purposes, and success of use. DESIGN Longitudinal descriptive study. SETTING Inpatient and outpatient services of a university medical center. PARTICIPANTS All trainees and attending staff working at the service sites were invited to participate; 158 (84%) did so. INTERVENTIONS Free online access was provided to MEDLINE through GRATEFUL MED software. Participants were offered a 2-hour introduction to online searching and 2 hours of free search time. MEASUREMENTS AND MAIN RESULTS For each search, a computer program requested identification of the user and the question to be addressed. Search transactions were recorded automatically. Interviews were conducted after a random sample of searches, and search questions were given to more expert searchers to run for comparison with the original. Eighty-one percent of participants did searches on study computers, at a mean rate of 2.7 searches per month. On comparison searches, participants retrieved 55% of the number of relevant articles retrieved by reference librarians (P = 0.024) and 50% more irrelevant articles (P less than 0.001). Forty-seven percent of searches on patient problems affected clinical decisions, but often on scanty information. CONCLUSIONS MEDLINE searching from clinical settings is feasible with brief training and affects clinical decisions. However, inexperienced searchers miss many relevant citations and search inefficiently. Further studies are needed to assess the impact of searching on physician performance and patient care.

A cross-sectional study of the number and frequency of terms used to refer to knowledge translation in a body of health literature in 2006: a Tower of Babel?

BackgroundThe study of implementing research findings into practice is rapidly growing and has acquired many competing names (e.g., dissemination, uptake, utilization, translation) and contributing disciplines. The use of multiple terms across disciplines pose barriers to communication and progress for applying research findings. We sought to establish an inventory of terms describing this field and how often authors use them in a collection of health literature published in 2006.MethodsWe refer to this field as knowledge translation (KT). Terms describing aspects of KT and their definitions were collected from literature, the internet, reports, textbooks, and contact with experts. We compiled a database of KT and other articles by reading 12 healthcare journals representing multiple disciplines. All articles published in these journals in 2006 were categorized as being KT or not. The KT articles (all KT) were further categorized, if possible, for whether they described KT projects or implementations (KT application articles), or presented the theoretical basis, models, tools, methods, or techniques of KT (KT theory articles). Accuracy was checked using duplicate reading. Custom designed software determined how often KT terms were used in the titles and abstracts of articles categorized as being KT.ResultsA total of 2,603 articles were assessed, and 581 were identified as KT articles. Of these, 201 described KT applications, and 153 included KT theory. Of the 100 KT terms collected, 46 were used by the authors in the titles or abstracts of articles categorized as being KT. For all 581 KT articles, eight terms or term variations used by authors were highly discriminating for separating KT and non-KT articles (p < 0.001): implementation, adoption, quality improvement, dissemination, complex intervention (with multiple endings), implementation (within three words of) research, and complex intervention. More KT terms were associated with KT application articles (n = 13) and KT theory articles (n = 18).ConclusionsWe collected 100 terms describing KT research. Authors used 46 of them in titles and abstracts of KT articles. Of these, approximately half discriminated between KT and non-KT articles. Thus, the need for consolidation and consistent use of fewer terms related to KT research is evident.

Prediction of citation counts for clinical articles at two years using data available within three weeks of publication: retrospective cohort study

Objective To determine if citation counts at two years could be predicted for clinical articles that pass basic criteria for critical appraisal using data within three weeks of publication from external sources and an online article rating service. Design Retrospective cohort study. Setting Online rating service, Canada. Participants 1274 articles from 105 journals published from January to June 2005, randomly divided into a 60:40 split to provide derivation and validation datasets. Main outcome measures 20 article and journal features, including ratings of clinical relevance and newsworthiness, routinely collected by the McMaster online rating of evidence system, compared with citation counts at two years. Results The derivation analysis showed that the regression equation accounted for 60% of the variation (R2=0.60, 95% confidence interval 0.538 to 0.629). This model applied to the validation dataset gave a similar prediction (R2=0.56, 0.476 to 0.596, shrinkage 0.04; shrinkage measures how well the derived equation matches data from the validation dataset). Cited articles in the top half and top third were predicted with 83% and 61% sensitivity and 72% and 82% specificity. Higher citations were predicted by indexing in numerous databases; number of authors; abstraction in synoptic journals; clinical relevance scores; number of cited references; and original, multicentred, and therapy articles from journals with a greater proportion of articles abstracted. Conclusion Citation counts can be reliably predicted at two years using data within three weeks of publication.

Health Information Technology to Facilitate Communication Involving Health Care Providers, Caregivers, and Pediatric Patients: A Scoping Review

Background Pediatric patients with health conditions requiring follow-up typically depend on a caregiver to mediate at least part of the necessary two-way communication with health care providers on their behalf. Health information technology (HIT) and its subset, information communication technology (ICT), are increasingly being applied to facilitate communication between health care provider and caregiver in these situations. Awareness of the extent and nature of published research involving HIT interventions used in this way is currently lacking. Objective This scoping review was designed to map the health literature about HIT used to facilitate communication involving health care providers and caregivers (who are usually family members) of pediatric patients with health conditions requiring follow-up. Methods Terms relating to care delivery, information technology, and pediatrics were combined to search MEDLINE, EMBASE, and CINAHL for the years 1996 to 2008. Eligible studies were selected after three rounds of duplicate screening in which all authors participated. Data regarding patient, caregiver, health care provider, HIT intervention, outcomes studied, and study design were extracted and maintained in a Microsoft Access database. Stage of research was categorized using the UK’s Medical Research Council (MRC) framework for developing and evaluating complex interventions. Quantitative and qualitative descriptive summaries are presented. Results We included 104 eligible studies (112 articles) conducted in 17 different countries and representing 30 different health conditions. The most common conditions were asthma, type 1 diabetes, special needs, and psychiatric disorder. Most studies (88, 85%) included children 2 to 12 years of age, and 73 (71%) involved home care settings. Health care providers operated in hospital settings in 96 (92%) of the studies. Interventions featured 12 modes of communication (eg, Internet, intranets, telephone, video conferencing, email, short message service [SMS], and manual downloading of information) used to facilitate 15 categories of functions (eg, support, medication management, education, and monitoring). Numerous patient, caregiver, and health care relevant outcomes have been measured. Most outcomes concerned satisfaction, use, usability, feasibility, and resource use, although behavior changes and quality of life were also reported. Most studies (57 studies, 55%) were pilot phase, with a lesser proportion of development phase (24 studies, 23%) and evaluation phase (11 studies, 11%) studies. HIT interventions addressed several recurring themes in this review: establishing continuity of care, addressing time constraints, and bridging geographical barriers. Conclusions HIT used in pediatric care involving caregivers has been implemented differently in a range of disease settings, with varying needs influencing the function, form and synchronicity of information transfer. Although some authors have followed a phased approach to development, evaluation and implementation, a greater emphasis on methodological standards such as the MRC guidance for complex interventions would produce more fruitful programs of development and more useful evaluations in the future. This review will be especially helpful to those deciding on areas where further development or research into HIT for this purpose may be warranted.

A Performance and Failure Analysis of SAPHIRE with a MEDLINE Test Collection

OBJECTIVE Assess the performance of the SAPHIRE automated information retrieval system. DESIGN Comparative study of automated and human searching of a MEDLINE test collection. MEASUREMENTS Recall and precision of SAPHIRE were compared with those attributes of novice physicians, expert physicians, and librarians for a test collection of 75 queries and 2,334 citations. Failure analysis assessed the efficacy of the Metathesaurus as a concept vocabulary; the reasons for retrieval of nonrelevant articles and nonretrieval of relevant articles; and the effect of changing the weighting formula for relevance ranking of retrieved articles. RESULTS Recall and precision of SAPHIRE were comparable to those of both physician groups, but less than those of librarians. CONCLUSION The current version of the Metathesaurus, as utilized by SAPHIRE, was unable to represent the conceptual content of one-fourth of physician-generated MEDLINE queries. The most likely cause for retrieval of nonrelevant articles was the presence of some or all of the search terms in the article, with frequencies high enough to lead to retrieval. The most likely cause for nonretrieval of relevant articles was the absence of the actual terms from the query, with synonyms or hierarchically related terms present instead. There were significant variations in performance when SAPHIREs concept-weighing formulas were modified.

Developing Optimal Search Strategies for Retrieving Qualitative Studies in PsycINFO

Researchers and practitioners have problems retrieving qualitative studies. Search strategies that can easily and effectively retrieve these studies from large databases such as PsycINFO are therefore important. To determine if search strategies can identify qualitative studies, 64 journals published in 2000 were hand searched using explicit methodological criteria to identify qualitative studies. The authors tested multiple search strategies using 4,985 potential search terms in PsycINFO (Ovid Technologies) and compared the results with the hand search data to calculate retrieval effectiveness. A total of 125 qualitative studies were identified. Single-term and multiple-term strategies had sensitivities (maximizing retrieval of qualitative studies) up to 94.4% and specificities (minimizing retrieval of nonqualitative studies and reports) up to 98.6% with ranges of precision and accuracy. Search strategies included terms that were variations of interview, qualitative, themes, and experience. Formal indexing terms performed poorly. Empirically derived search strategies combining textwords can effectively, but not perfectly, retrieve qualitative studies from PsycINFO.

Core competencies in the science and practice of knowledge translation: description of a Canadian strategic training initiative

BackgroundGlobally, healthcare systems are attempting to optimize quality of care. This challenge has resulted in the development of implementation science or knowledge translation (KT) and the resulting need to build capacity in both the science and practice of KT.FindingsWe are attempting to meet these challenges through the creation of a national training initiative in KT. We have identified core competencies in this field and have developed a series of educational courses and materials for three training streams. We report the outline for this approach and the progress to date.ConclusionsWe have prepared a strategy to develop, implement, and evaluate a national training initiative to build capacity in the science and practice of KT. Ultimately through this initiative, we hope to meet the capacity demand for KT researchers and practitioners in Canada that will lead to improved care and a strengthened healthcare system.

The effectiveness of integrated health information technologies across the phases of medication management: a systematic review of randomized controlled trials.

OBJECTIVE The US Agency for Healthcare Research and Quality funded an evidence report to address seven questions on multiple aspects of the effectiveness of medication management information technology (MMIT) and its components (prescribing, order communication, dispensing, administering, and monitoring). MATERIALS AND METHODS Medline and 11 other databases without language or date limitations to mid-2010. Randomized controlled trials (RCTs) assessing integrated MMIT were selected by two independent reviewers. Reviewers assessed study quality and extracted data. Senior staff checked accuracy. RESULTS Most of the 87 RCTs focused on clinical decision support and computerized provider order entry systems, were performed in hospitals and clinics, included primarily physicians and sometimes nurses but not other health professionals, and studied process changes related to prescribing and monitoring medication. Processes of care improved for prescribing and monitoring mostly in hospital settings, but the few studies measuring clinical outcomes showed small or no improvements. Studies were performed most frequently in the USA (n=63), Europe (n=16), and Canada (n=6). DISCUSSION Many studies had limited description of systems, installations, institutions, and targets of the intervention. Problems with methods and analyses were also found. Few studies addressed order communication, dispensing, or administering, non-physician prescribers or pharmacists and their MMIT tools, or patients and caregivers. Other study methods are also needed to completely understand the effects of MMIT. CONCLUSIONS Almost half of MMIT interventions improved the process of care, but few studies measured clinical outcomes. This large body of literature, although instructive, is not uniformly distributed across settings, people, medication phases, or outcomes.

McMaster PLUS: A Cluster Randomized Clinical Trial of an Intervention to Accelerate Clinical Use of Evidence-based Information from Digital Libraries

BACKGROUND Physicians have difficulty keeping up with new evidence from medical research. METHODS We developed the McMaster Premium LiteratUre Service (PLUS), an internet-based addition to an existing digital library, which delivered quality- and relevance-rated medical literature to physicians, matched to their clinical disciplines. We evaluated PLUS in a cluster-randomized trial of 203 participating physicians in Northern Ontario, comparing a Full-Serve version (that included alerts to new articles and a cumulative database of alerts) with a Self-Serve version (that included a passive guide to evidence-based literature). Utilization of the service was the primary trial end-point. RESULTS Mean logins to the library rose by 0.77 logins/month/user (95% CI 0.43, 1.11) in the Full-Serve group compared with the Self-Serve group. The proportion of Full-Serve participants who utilized the service during each month of the study period showed a sustained increase during the intervention period, with a relative increase of 57% (95% CI 12, 123) compared with the Self-Serve group. There were no differences in these proportions during the baseline period, and following the crossover of the Self-Serve group to Full-Serve, the Self-Serve groups usage became indistinguishable from that of the Full-Serve group (relative difference 4.4 (95% CI -23.7, 43.0). Also during the intervention and crossover periods, measures of self-reported usefulness did not show a difference between the 2 groups. CONCLUSION A quality- and relevance-rated online literature service increased the utilization of evidence-based information from a digital library by practicing physicians.

The economics of health information technology in medication management: a systematic review of economic evaluations

OBJECTIVE To conduct a systematic review and synthesis of the evidence surrounding the cost-effectiveness of health information technology (HIT) in the medication process. MATERIALS AND METHODS Peer-reviewed electronic databases and gray literature were searched to identify studies on HIT used to assist in the medication management process. Articles including an economic component were reviewed for further screening. For this review, full cost-effectiveness analyses, cost-utility analyses and cost-benefit analyses, as well as cost analyses, were eligible for inclusion and synthesis. RESULTS The 31 studies included were heterogeneous with respect to the HIT evaluated, setting, and economic methods used. Thus the data could not be synthesized, and a narrative review was conducted. Most studies evaluated computer decision support systems in hospital settings in the USA, and only five of the studied performed full economic evaluations. DISCUSSION Most studies merely provided cost data; however, useful economic data involves far more input. A full economic evaluation includes a full enumeration of the costs, synthesized with the outcomes of the intervention. CONCLUSION The quality of the economic literature in this area is poor. A few studies found that HIT may offer cost advantages despite their increased acquisition costs. However, given the uncertainty that surrounds the costs and outcomes data, and limited study designs, it is difficult to reach any definitive conclusion as to whether the additional costs and benefits represent value for money. Sophisticated concurrent prospective economic evaluations need to be conducted to address whether HIT interventions in the medication management process are cost-effective.