David H. Hickam

OHSUMED: an interactive retrieval evaluation and new large test collection for research

A series of information retrieval experiments was carried out with a computer installed in a medical practice setting for relatively inexperienced physician end-users. Using a commercial MEDLINE product based on the vector space model, these physicians searched just as effectively as more experienced searchers using Boolean searching. The results of this experiment were subsequently used to create a new large medical test collection, which was used in experiments with the SMART retrieval system to obtain baseline performance data as well as compare SMART with the other searchers.

Factors associated with successful answering of clinical questions using an information retrieval system

OBJECTIVES This study sought to assess the ability of medical and nurse practitioner students to use MEDLINE to obtain evidence for answering clinical questions and to identify factors associated with the successful answering of questions. METHODS A convenience sample of medical and nurse practitioner students was recruited. After completing instruments measuring demographic variables, computer and searching attitudes and experience, and cognitive traits, the subjects were given a brief orientation to MEDLINE searching and the techniques of evidence-based medicine. The subjects were then given 5 questions (from a pool of 20) to answer in two sessions using the Ovid MEDLINE system and the Oregon Health & Science University library collection. Each question was answered using three possible responses that reflected the quality of the evidence. All actions capable of being logged by the Ovid system were captured. Statistical analysis was performed using a model based on generalized estimating equations. The relevance-based measures of recall and precision were measured by defining end queries and having relevance judgments made by physicians who were not associated with the study. RESULTS Forty-five medical and 21 nurse practitioner students provided usable answers to 324 questions. The rate of correctness increased from 32.3 to 51.6 percent for medical students and from 31.7 to 34.7 percent for nurse practitioner students. Ability to answer questions correctly was most strongly associated with correctness of the answer before searching, user experience with MEDLINE features, the evidence-based medicine question type, and the spatial visualization score. The spatial visualization score showed multi-colinearity with student type (medical vs. nurse practitioner). Medical and nurse practitioner students obtained comparable recall and precision, neither of which was associated with correctness of the answer. CONCLUSIONS Medical and nurse practitioner students in this study were at best moderately successful at answering clinical questions correctly with the assistance of literature searching. The results confirm the importance of evaluating both search ability and the ability to use the resulting information to accomplish a clinical task.

Effects of a physician communication intervention on patient care outcomes

OBJECTIVE: To determine whether an intervention designed to improve patient-physician communication increases the frequency with which physicians elicit patients’ concerns, changes other communication behaviors, and improves health care outcomes.DESIGN: Pretest-posttest design with random assignment of physicians to intervention or control groups.SETTING: General medicine clinics of a university-affiliated Veterans Affairs Hospital.PATZEWTS/PARTICIPANTS: Forty-two physicians and 348 continuity care patients taking prescription medications for chronic medical conditions.INTERVENTIONS: Intervention group physicians received 4.5 hours of training on eliciting and responding to patients’ concerns and requests, and their patients filled out the Patient Requests for Services Questionnaire prior to a subsequent clinic visit. Control group physicians received 4.5 hours of training in medical decision-making.MEASUREMENTS AND MAIN RESULTS: The frequency with which physicians elicited all of a patient’s concerns increased in the intervention group as compared with the control group (p=.032). Patients perceptions of the amount of information received from the physician did increase significantly (p<.05), but the actual magnitude of change was small. A measure of patient satisfaction with the physicians was high at baseline and also showed no significant change after the intervention. Likewise, the intervention was not associated with changes in patient compliance with medications or appointments, nor were there any effects on outpatient utilization.CONCLUSIONS: A low-intensity intervention changed physician behavior but had no effect on patient outcomes such as satisfaction, compliance, or utilization. Interventions may need to focus on physicians and patients to have the greatest effect.

Diagnosis, access and outcomes: Update of a systematic review of telemedicine services

Telemedicine services are being increasingly used. Although insurers and other payers are covering some services in the USA, the rationale for these coverage decisions is not always evidence-based. We reviewed the literature for telemedicine services that substitute for face-to-face medical diagnosis and treatment. We focused on three types of telemedicine services: store-and-forward, home-based and office/hospital-based services. Studies were included if they were relevant to at least one of the three study areas, addressed at least one key question and contained reported results. We excluded articles that did not study a service requiring face-to-face encounters (i.e. teleradiology was excluded). Our search initially identified 4083 citations. After review, 597 were judged to be potentially relevant at the title/abstract level. Following a full-text review, 106 studies were included. Store-and-forward services have been studied in many specialties, the most common being dermatology, wound care and ophthalmology. The evidence for their efficacy is mixed. Several limited studies showed the benefits of home-based telemedicine interventions in chronic diseases. Studies of office/hospital-based telemedicine suggest that telemedicine is most effective for verbal interactions, e.g. videoconferencing for diagnosis and treatment in specialties like neurology and psychiatry. There are still significant gaps in the evidence base between where telemedicine is used and where its use is supported by high-quality evidence. Further well-designed research is necessary to understand how best to deploy telemedicine services in health care.

The Treatment Advice of a Computer-Based Cancer Chemotherapy Protocol Advisor

ONCOCIN is a chemotherapy protocol advisor used experimentally in a university oncology clinic. The program combines formal protocol guidelines with judgments of oncologists who have experience adjusting therapy in complex clinical situations. We compared the chemotherapy administered by clinic physicians with the treatment that would have been recommended by ONCOCIN in 415 visits for 39 patients with lymphoma seen before the programs introduction. In 189 visits the program agreed with the therapy actually administered. In a blinded evaluation, four experts on lymphoma failed to find a significant difference between the treatments selected by physicians and those proposed by ONCOCIN. Further analyses showed that ONCOCIN tended to attenuate drug doses or delay treatment more than the experts recommended, whereas the physicians were less likely to attenuate doses to the extent the experts suggested. Our results show that ONCOCIN provides advice on lymphoma treatment similar to the treatment provided in a university oncology clinic.

Assessing predictive accuracy: How to compare brier scores

Several investigators have used the Brier index to measure the predictive accuracy of a set of medical judgments; the Brier scores of different raters who have evaluated the same patients provides a measure of relative accuracy. However, such comparisons may be difficult to interpret because of the lack of a statistical test for differentiating between two Brier scores. To demonstrate a method for addressing this issue we analyzed the judgments of five medical students, each of whom independently evaluated the same 25 patients with recurrent chest pain. Using the method we determined that two of the students gave judgments that were incompatible with the actual observed outcomes (p less than 0.05); of the three remaining students we detected a significant difference between two (p less than 0.05). These results differed from receiver operating characteristic curve area analysis, another technique used to evaluate predictive accuracy. We suggest that the proposed method can provide a useful tool for investigators using the Brier index to compare how well clinicians express uncertainty using probability judgments.

Patients’ interpretations of probability terms

Study objectives:To assess the meanings ascribed by patients to qualitative expressions of probability commonly used in medical care and to determine patient preferences for obtaining information when communicating with their physicians about medical risk numerically and/or qualitatively.Design:Cross-sectional survey of consecutive patients.Setting:A university-based Department of Veterans Affairs medical center.Participants:133 patients sequentially seen in a general medicine clinic.Measurements and results:Subjects were given a randomly ordered list of 12 common terms, each a qualitative expression of probability. They were asked to indicate what they understood to be the numerical meaning of each word. The patients’ probability estimates were found to comprise two groups of five terms each, with high intercorrelations among the probabilities assigned to the terms in each group. Mean probabilities assigned to terms in the first group all were greater than 60%, and mean probabilities assigned to terms in the second group all were below 50%. When asked whether they wanted chance information to be provided in numerical or qualitative terms, 32% reported that they wanted it only numerically; 35.3% wanted it only qualitatively; 21.8% wanted the information either way; and 8.3% wanted the information both ways.Conclusions:The numerical meanings that patients ascribe to probability terms fall into identifiable patterns. While patients vary in the actual values they assign to terms, the relative meanings of terms show consistent trends.

A task-oriented approach to information retrieval evaluation

As retrieval systems become more oriented towards end‐users, there is an increasing need for improved methods to evaluate their effectiveness. We performed a task‐oriented assessment of two MEDLINE searching systems, one which promotes traditional Boolean searching on human‐indexed thesaurus terms and the other natural language searching on words in the title, abstract, and indexing terms. Medical students were randomized to one of the two systems and given clinical questions to answer. The students were able to use each system successfully, with no significant differences in questions correctly answered, time taken, relevant articles retrieved, or user satisfaction between the systems. This approach to evaluation was successful in measuring effectiveness of system use and demonstrates that both types of systems can be used equally well with minimal training.

A Performance and Failure Analysis of SAPHIRE with a MEDLINE Test Collection

OBJECTIVE Assess the performance of the SAPHIRE automated information retrieval system. DESIGN Comparative study of automated and human searching of a MEDLINE test collection. MEASUREMENTS Recall and precision of SAPHIRE were compared with those attributes of novice physicians, expert physicians, and librarians for a test collection of 75 queries and 2,334 citations. Failure analysis assessed the efficacy of the Metathesaurus as a concept vocabulary; the reasons for retrieval of nonrelevant articles and nonretrieval of relevant articles; and the effect of changing the weighting formula for relevance ranking of retrieved articles. RESULTS Recall and precision of SAPHIRE were comparable to those of both physician groups, but less than those of librarians. CONCLUSION The current version of the Metathesaurus, as utilized by SAPHIRE, was unable to represent the conceptual content of one-fourth of physician-generated MEDLINE queries. The most likely cause for retrieval of nonrelevant articles was the presence of some or all of the search terms in the article, with frequencies high enough to lead to retrieval. The most likely cause for nonretrieval of relevant articles was the absence of the actual terms from the query, with synonyms or hierarchically related terms present instead. There were significant variations in performance when SAPHIREs concept-weighing formulas were modified.

The role of doctor's opinion in shared decision making: what does shared decision making really mean when considering invasive medical procedures?1

Objective  The goal of this study was to gain understanding about patients’ perspectives on decision making in the context of invasive medical interventions and whether patients’ decision‐making preferences influenced the type of information they desired to be provided by physicians.