Steven Bedrick

Evaluating performance of biomedical image retrieval systems--an overview of the medical image retrieval task at ImageCLEF 2004-2013.

Medical image retrieval and classification have been extremely active research topics over the past 15 years. Within the ImageCLEF benchmark in medical image retrieval and classification, a standard test bed was created that allows researchers to compare their approaches and ideas on increasingly large and varied data sets including generated ground truth. This article describes the lessons learned in ten evaluation campaigns. A detailed analysis of the data also highlights the value of the resources created.

Concussion guidelines step 1: systematic review of prevalent indicators.

BACKGROUND Currently, there is no evidence-based definition for concussion that is being uniformly applied in clinical and research settings. OBJECTIVE To conduct a systematic review of the highest-quality literature about concussion and to assemble evidence about the prevalence and associations of key indicators of concussion. The goal was to establish an evidence-based foundation from which to derive, in future work, a definition, diagnostic criteria, and prognostic indicators for concussion. METHODS Key questions were developed, and an electronic literature search from 1980 to 2012 was conducted to acquire evidence about the prevalence of and associations among signs, symptoms, and neurologic and cognitive deficits in samples of individuals exposed to potential concussive events. Included studies were assessed for potential for bias and confound and rated as high, medium, or low potential for bias and confound. Those rated as high were excluded from the analysis. Studies were further triaged on the basis of whether the definition of a case of concussion was exclusive or inclusive; only those with wide, inclusive case definitions were used in the analysis. Finally, only studies reporting data collected at fixed time points were used. For a study to be included in the conclusions, it was required that the presence of any particular sign, symptom, or deficit be reported in at least 2 independent samples. RESULTS From 5437 abstracts, 1362 full-text publications were reviewed, of which 231 studies were included in the final library. Twenty-six met all criteria required to be used in the analysis, and of those, 11 independent samples from 8 publications directly contributed data to conclusions. Prevalent and consistent indicators of concussion are (1) observed and documented disorientation or confusion immediately after the event, (2) impaired balance within 1 day after injury, (3) slower reaction time within 2 days after injury, and/or (4) impaired verbal learning and memory within 2 days after injury. CONCLUSION The results of this systematic review identify the consistent and prevalent indicators of concussion and their associations, derived from the strongest evidence in the published literature. The product is an evidence-based foundation from which to develop diagnostic criteria and prognostic indicators.

Development of a Software for Quantitative Evaluation Radiotherapy Target and Organ-at-Risk Segmentation Comparison

Modern radiotherapy requires accurate region of interest (ROI) inputs for plan optimization and delivery. Target delineation, however, remains operator-dependent and potentially serves as a major source of treatment delivery error. In order to optimize this critical, yet observer-driven process, a flexible web-based platform for individual and cooperative target delineation analysis and instruction was developed in order to meet the following unmet needs: (1) an open-source/open-access platform for automated/semiautomated quantitative interobserver and intraobserver ROI analysis and comparison, (2) a real-time interface for radiation oncology trainee online self-education in ROI definition, and (3) a source for pilot data to develop and validate quality metrics for institutional and cooperative group quality assurance efforts. The resultant software, Target Contour Testing/Instructional Computer Software (TaCTICS), developed using Ruby on Rails, has since been implemented and proven flexible, feasible, and useful in several distinct analytical and research applications.

Multimodal medical image retrieval OHSU at ImageCLEF 2008

We present results from the Oregon Health & Science Universitys participation in the medical retrieval task of ImageCLEF 2008. Our web-based retrieval system was built using a Ruby on Rails framework. Ferret, a Ruby port of Lucene was used to create the full-text based index and search engine. In addition to the textual index of annotations, supervised machine learning techniques using visual features were used to classify the images based on image acquisition modality. Our system provides the user with a number of search options including the ability to limit their search by modality, UMLS-based query expansion, and Natural Language Processing-based techniques. Purely textual runs as well as mixed runs using the purported modality were submitted. We also submitted interactive runs using user specified search options. Although the use of the UMLS metathesaurus increased our recall, our system is geared towards early precision. Consequently, many of our multimodal automatic runs using the custom parser as well as interactive runs had high early precision including the highest P10 and P30 among the official runs. Our runs also performed well using the bpref metric, a measure that is more robust in the case of incomplete judgments.

Executive summary of Concussion guidelines step 1: systematic review of prevalent indicators.

BACKGROUND: Currently, there is no evidence-based definition for concussion that is being uniformly applied in clinical and research settings. OBJECTIVE: To conduct a systematic review of the highest-quality literature about concussion and to assemble evidence about the prevalence and associations of key indicators of concussion. The goal was to establish an evidence-based foundation from which to derive, in future work, a definition, diagnostic criteria, and prognostic indicators for concussion. METHODS: Key questions were developed, and an electronic literature search from 1980 to 2012 was conducted to acquire evidence about the prevalence of and associations among signs, symptoms, and neurologic and cognitive deficits in samples of individuals exposed to potential concussive events. Included studies were assessed for potential for bias and confound and rated as high, medium, or low potential for bias and confound. Those rated as high were excluded from the analysis. Studies were further triaged on the basis of whether the definition of a case of concussion was exclusive or inclusive; only those with wide, inclusive case definitions were used in the analysis. Finally, only studies reporting data collected at fixed time points were used. For a study to be included in the conclusions, it was required that the presence of any particular sign, symptom, or deficit be reported in at least 2 independent samples. RESULTS: From 5437 abstracts, 1362 full-text publications were reviewed, of which 231 studies were included in the final library. Twenty-six met all criteria required to be used in the analysis, and of those, 11 independent samples from 8 publications directly contributed data to conclusions. Prevalent and consistent indicators of concussion are (1) observed and documented disorientation or confusion immediately after the event, (2) impaired balance within 1 day after injury, (3) slower reaction time within 2 days after injury, and/or (4) impaired verbal learning and memory within 2 days after injury. CONCLUSION: The results of this systematic review identify the consistent and prevalent indicators of concussion and their associations, derived from the strongest evidence in the published literature. The product is an evidence-based foundation from which to develop diagnostic criteria and prognostic indicators.

Report on the SIGIR 2014 Workshop on Medical Information Retrieval (MedIR)

The workshop on Medical Information Retrieval took place at SIGIR 2014 in Gold Coast, Australia on July 11. The workshop included eight oral presentations of referred papers and an invited keynote presentation. This allowed time for lively discussions among the participants. These showed the significant interest in the medical information retrieval domain and the many research challenges arising in this space which need to be addressed to give added value to the wide variety of users that can profit from medical information search, such as patients, general health professionals and specialist groups such as radiologists who mainly search for images and image related information.

Intrainstitutional EHR collections for patient-level information retrieval

Research in clinical information retrieval has long been stymied by the lack of open resources. However, both clinical information retrieval research innovation and legitimate privacy concerns can be served by the creation of intrainstitutional, fully protected resources. In this article, we provide some principles and tools for information retrieval resource‐building in the unique problem setting of patient‐level information retrieval, following the tradition of the Cranfield paradigm. We further include an analysis of parallel information retrieval resources at Oregon Health & Science University and Mayo Clinic that were built on these principles.

Improving Early Precision in the ImageCLEF Medical Retrieval Task

Oregon Health and Science University has participated in the ImageCLEFmed medical image retrieval task since 2005. Over the years of our participation, our focus has been on exploring the needs of medical end users, and developing retrieval strategies that address those needs. Given that many users of search systems never look beyond the first few results, we have attempted to emphasize early precision in the performance of our system. This chapter describes several of the approaches we have used to achieve this goal, along with the results we have seen in doing so.

Algorithmic Classification of Five Characteristic Types of Paraphasias

Purpose This study was intended to evaluate a series of algorithms developed to perform automatic classification of paraphasic errors (formal, semantic, mixed, neologistic, and unrelated errors). Method We analyzed 7,111 paraphasias from the Moss Aphasia Psycholinguistics Project Database (Mirman et al., 2010) and evaluated the classification accuracy of 3 automated tools. First, we used frequency norms from the SUBTLEXus database (Brysbaert & New, 2009) to differentiate nonword errors and real-word productions. Then we implemented a phonological-similarity algorithm to identify phonologically related real-word errors. Last, we assessed the performance of a semantic-similarity criterion that was based on word2vec (Mikolov, Yih, & Zweig, 2013). Results Overall, the algorithmic classification replicated human scoring for the major categories of paraphasias studied with high accuracy. The tool that was based on the SUBTLEXus frequency norms was more than 97% accurate in making lexicality judgments. The phonological-similarity criterion was approximately 91% accurate, and the overall classification accuracy of the semantic classifier ranged from 86% to 90%. Conclusion Overall, the results highlight the potential of tools from the field of natural language processing for the development of highly reliable, cost-effective diagnostic tools suitable for collecting high-quality measurement data for research and clinical purposes.

Relevance Judgments for Image Retrieval Evaluation

In this chapter, we review our experiences with the relevance judging process at ImageCLEF, using the medical retrieval task as a primary example. We begin with a historic perspective of the precursor to most modern retrieval evaluation campaigns, the Cranfield paradigm, as most modern system–based evaluation campaigns including ImageCLEF are modeled after it. We then briefly describe the stages in an evaluation campaign and provide details of the different aspects of the relevance judgment process. We summarize the recruitment process and describe the various systems used for judgment at ImageCLEF. We discuss the advantages and limitations of creating pools that are then judged by human experts. Finally, we discuss our experiences with the subjectivity of the relevance process and the relative robustness of the performance measures to variability in relevance judging.