Jytte Brender

STARE-HI - Statement on reporting of evaluation studies in Health Informatics

OBJECTIVE Development of guidelines for publication of evaluation studies of Health Informatics applications. METHODS An initial list of issues to be addressed in reports on evaluation studies was drafted based on experiences as editors and reviewers of journals in Health Informatics and as authors of systematic reviews of Health Informatics studies, taking into account guidelines for reporting of medical research. This list has been discussed in several rounds by an increasing number of experts in Health Informatics evaluation during conferences and by using e-mail and has been put up for comments on the web. RESULTS A set of STARE-HI principles to be addressed in papers describing evaluations of Health Informatics interventions is presented. These principles include formulation of title and abstract, of introduction (e.g. scientific background, study objectives), study context (e.g. organizational setting, system details), methods (e.g. study design, outcome measures), results (e.g. study findings, unexpected observations) and discussion and conclusion of an IT evaluation paper. CONCLUSION A comprehensive list of principles relevant for properly describing Health Informatics evaluations has been developed. When manuscripts submitted to Health Informatics journals and general medical journals adhere to these aspects, readers will be better positioned to place the studies in a proper context and judge their validity and generalisability. It will also be possible to judge better whether papers will fit in the scope of meta-analyses of Health Informatics interventions. STARE-HI may also be used for study planning and hence positively influence the quality of evaluation studies in Health Informatics. We believe that better publication of both quantitative and qualitative evaluation studies is an important step toward the vision of evidence-based Health Informatics. LIMITATIONS This study is based on experiences from editors, reviewers, authors of systematic reviews and readers of the scientific literature. The applicability of the principles has not been evaluated in real practice. Only when authors start to use these principles for reporting, shortcomings in the principles will emerge.

Paper: On the quality of neural net classifiers

This paper describes several concepts and metrics that may be used to assess various aspects of the quality of neural net classifiers. Each concept describes a property that may be taken into account by both designers and users of neural net classifiers when assessing their utility. Besides metrics for assessment of the correctness of classifiers we also introduce metrics that address certain aspects of the misclassifications. We show the applicability of the introduced quality concepts for selection among several neural net classifiers in the domain of thyroid disorders.

Factors influencing the transferability of medical decision support systems

Transferability is a key issue in the development and implementation of medical decision support systems. Such systems have up to now tended to be confined to the development site. Transferability represents the integrated effect of several more basic attributes of Decision Support Systems. These attributes can be considered to belong to two main groups: those concerned with the Medical Domain (of the system) and those concerned with the Information Technology by which the System functions. Among the Domain issues the most important are: Epidemiology, Terminology and Methodology. Concerning the Information Technology issues the most important are: Knowledge Acquisition and Representation methods, Database design and integration with inference mechanism. The effect of each of the individual factors is considered by illustrations from the literature and by studying the results of recent experiments where databases and decision support systems from different countries are interchanged. This exercise allows some planning in the design of future systems with the aim of improving overall transferability and therefore applicability.

Technology-induced errors. The current use of frameworks and models from the biomedical and life sciences literatures.

OBJECTIVE The objective of this paper is to examine the extent, range and scope to which frameworks, models and theories dealing with technology-induced error have arisen in the biomedical and life sciences literature as indexed by Medline®. METHODS To better understand the state of work in the area of technology-induced error involving frameworks, models and theories, the authors conducted a search of Medline® using selected key words identified from seminal articles in this research area. Articles were reviewed and those pertaining to frameworks, models or theories dealing with technology-induced error were further reviewed by two researchers. RESULTS All articles from Medline® from its inception to April of 2011 were searched using the above outlined strategy. 239 citations were returned. Each of the abstracts for the 239 citations were reviewed by two researchers. Eleven articles met the criteria based on abstract review. These 11 articles were downloaded for further in-depth review. The majority of the articles obtained describe frameworks and models with reference to theories developed in other literatures outside of healthcare. The papers were grouped into several areas. It was found that articles drew mainly from three literatures: 1) the human factors literature (including human-computer interaction and cognition), 2) the organizational behavior/sociotechnical literature, and 3) the software engineering literature. CONCLUSIONS A variety of frameworks and models were found in the biomedical and life sciences literatures. These frameworks and models drew upon and extended frameworks, models and theoretical perspectives that have emerged in other literatures. These frameworks and models are informing an emerging line of research in health and biomedical informatics involving technology-induced errors in healthcare.

STARE-HI – Statement on Reporting of Evaluation Studies in Health Informatics: Explanation and Elaboration

BACKGROUND Improving the quality of reporting of evaluation studies in health informatics is an important requirement towards the vision of evidence-based health informatics. The STARE-HI - Statement on Reporting of Evaluation Studies in health informatics, published in 2009, provides guidelines on the elements to be contained in an evaluation study report. OBJECTIVES To elaborate on and provide a rationale for the principles of STARE-HI and to guide authors and readers of evaluation studies in health informatics by providing explanatory examples of reporting. METHODS A group of methodologists, researchers and editors prepared the present elaboration of the STARE-HI statement and selected examples from the literature. RESULTS The 35 STARE-HI items to be addressed in evaluation papers describing health informatics interventions are discussed one by one and each is extended with examples and elaborations. CONCLUSION The STARE-HI statement and this elaboration document should be helpful resources to improve reporting of both quantitative and qualitative evaluation studies. Evaluation manuscripts adhering to the principles will enable readers of such papers to better place the studies in a proper context and judge their validity and generalizability, and thus in turn optimize the exploitation of the evidence contained therein. LIMITATIONS This paper is based on experiences of a group of editors, reviewers, authors of systematic reviews and readers of the scientific literature. The applicability of the details of these principles has to evolve as a function of their use in practice.

Methodology for constructive assessment of IT-based systems in an organisational context

Even if painted in black and white, there is no doubt that the assumptions for the application of traditional approaches for user requirements specification are more or less unfulfilled. This indicates a need for evolutionary system development combined with constructive assessment throughout the life-cycle of an IT-based solution. A methodology for constructive technology assessment is presented, which 1) covers the entire system life-cycle; 2) has users from the application domain of the future system, or their representatives, as the target users of the assessment methodology; 3) enables constructive assessment during the development of an IT-based solution; 4) is applicable independently of the system development approach; and 5) provides users of the IT-based solution with information enabling them to decide whether or not to take the system into real-life clinical usage.

A Methodology for Evaluation of Knowledge-Based Systems

Knowledge based systems (KBS) are of great potential use in medicine. However, the progress in development and actual day to day use of KBSs in medicine has been disappointing. Shortliffe (1989) believes that this is because medical decision aids are not user friendly, they do not explain their reasoning adequately, they are too time consuming to use in the actual work environment, and particularly because of physician fears of e.g. legal liability and intrusion of the systems into their practice.

User requirements specifications: a hierarchical structure covering strategical, tactical and operational requirements

The present study is a long-term review and follow-up on conclusions and recommendations on the role and contents of a User Requirements Specification Document (URD) from a previous in-depth case study. The follow-up comprises a succession of investigations and two case studies to explore the role and contents of a URD within systems development and assessment, thereby gradually extending and refining the original recommendations. Finally, the recommended three-layered structure and approach for a URD were applied in full scale within a real-life project, in which the URD serves as the basis for a Call for Tender. The preparation was entirely user-driven with the aid of a consultant for four man months. The present paper outlines the essence of the approach and the outcome of applying the recommended structure, with numerous examples of the implications.

Trends in assessment of IT-based solutions in healthcare and recommendations for the future

State-of-the-art assessments of IT-based solutions in Healthcare is discussed in this paper. Special emphasis is placed on the human and organisation-centred perspective at the development of IT-based solutions. Based on this and basic conditions for system analysis and design reported in the literature, requirements for a methodology for user-driven, constructive assessment during the entire life-cycle of the IT-based system are synthesised. Finally, the outcome is discussed in the light of the presentations on technology assessment during the MIE 97 Congress and other related literature.

Problem-oriented management of laboratory work through dynamic test scheduling

This paper gives an overview of problems inherent in dynamic test scheduling together with some model solutions. Dynamic test schedules are decision tree-like protocols for cost-efficient management of analytical production in a clinical laboratory. The present analysis is based on previous practical experience and concludes that it is not feasible to introduce dynamic test scheduling on a large scale without computer-based support, because of the increase in complexity of the laboratory work processes. Further, our experience is that it is extremely complex to incorporate the dynamic test scheduling functionality into an existing Laboratory Information System (LIS). The approach pursued in the OpenLabs (A2028) AIM Project for implementing dynamic test scheduling is to provide the necessary functionality as a stand-alone module interconnected with a LIS in an open systems solution.