Jan L. Talmon

STARE-HI - Statement on reporting of evaluation studies in Health Informatics

OBJECTIVE Development of guidelines for publication of evaluation studies of Health Informatics applications. METHODS An initial list of issues to be addressed in reports on evaluation studies was drafted based on experiences as editors and reviewers of journals in Health Informatics and as authors of systematic reviews of Health Informatics studies, taking into account guidelines for reporting of medical research. This list has been discussed in several rounds by an increasing number of experts in Health Informatics evaluation during conferences and by using e-mail and has been put up for comments on the web. RESULTS A set of STARE-HI principles to be addressed in papers describing evaluations of Health Informatics interventions is presented. These principles include formulation of title and abstract, of introduction (e.g. scientific background, study objectives), study context (e.g. organizational setting, system details), methods (e.g. study design, outcome measures), results (e.g. study findings, unexpected observations) and discussion and conclusion of an IT evaluation paper. CONCLUSION A comprehensive list of principles relevant for properly describing Health Informatics evaluations has been developed. When manuscripts submitted to Health Informatics journals and general medical journals adhere to these aspects, readers will be better positioned to place the studies in a proper context and judge their validity and generalisability. It will also be possible to judge better whether papers will fit in the scope of meta-analyses of Health Informatics interventions. STARE-HI may also be used for study planning and hence positively influence the quality of evaluation studies in Health Informatics. We believe that better publication of both quantitative and qualitative evaluation studies is an important step toward the vision of evidence-based Health Informatics. LIMITATIONS This study is based on experiences from editors, reviewers, authors of systematic reviews and readers of the scientific literature. The applicability of the principles has not been evaluated in real practice. Only when authors start to use these principles for reporting, shortcomings in the principles will emerge.

Inter-organizational future proof EHR systems A review of the security and privacy related issues

OBJECTIVES Identification and analysis of privacy and security related issues that occur when health information is exchanged between health care organizations. METHODS Based on a generic scenario questions were formulated to reveal the occurring issues. Possible answers were verified in literature. RESULTS Ensuring secure health information exchange across organizations requires a standardization of security measures that goes beyond organizational boundaries, such as global definitions of professional roles, global standards for patient consent and semantic interoperable audit logs. CONCLUSION As to be able to fully address the privacy and security issues in interoperable EHRs and the long-life virtual EHR it is necessary to realize a paradigm shift from storing all incoming information in a local system to retrieving information from external systems whenever that information is deemed necessary for the care of the patient.

Assessing the importance of features for multi-layer perceptrons

In this paper we establish a mathematical framework in which we develop measures for determining the contribution of individual features to the performance of a classifier. Corresponding to these measures, we design metrics that allow estimation of the importance of features for a specific multi-layer perceptron neural network. It is shown that all measures constitute lower bounds for the correctness that can be obtained when the feature under study is excluded and the classifier rebuilt. We also present a method for pruning input nodes from the network such that most of the knowledge encoded in its weights is retained. The proposed metrics and the pruning method are validated with a number of experiments with artificial classification tasks. The experiments indicate that the metric called replaceability results in the tightest error bounds. Both this metric and the metric called expected influence result in good rankings of the features.

Mapping the Domain of Medical Informatics

OBJECTIVES The domain of medical informatics (MI) is not well defined. It covers a wide range of research topics. Our objective is to characterize the field of MI by means of the scientific literature in this domain. METHODS We used titles and abstracts from MEDLINE records of papers published between July 1993 and July 2008, and extracted uni-, bi- and trigrams as features. Starting with the ISI category of medical informatics, we applied a semi-automated procedure to identify the set of journals and proceedings pertaining to MI. A clustering algorithm was subsequently applied to the articles from this set of publications. RESULTS MI literature can be divided into three subdomains: 1) the organization, application, and evaluation of health information systems, 2) medical knowledge representation, and 3) signal and data analysis. Over the last fifteen years, the field has remained relatively stable, although most journals have shifted their focus somewhat. CONCLUSIONS We identified the scientific literature pertaining to the field of MI, and the main areas of research. We were able to show trends in the field, and the positioning of different journals within this field.

A comparative review of patient safety initiatives for national health information technology

OBJECTIVE To collect and critically review patient safety initiatives for health information technology (HIT). METHOD Publicly promulgated set of advisories, recommendations, guidelines, or standards potentially addressing safe system design, build, implementation or use were identified by searching the websites of regional and national agencies and programmes in a non-exhaustive set of exemplar countries including England, Denmark, the Netherlands, the USA, Canada and Australia. Initiatives were categorised by type and software systems covered. RESULTS We found 27 patient safety initiatives for HIT predominantly dealing with software systems for health professionals. Three initiatives addressed consumer systems. Seven of the initiatives specifically dealt with software for diagnosis and treatment, which are regulated as medical devices in England, Denmark and Canada. Four initiatives dealt with blood bank and image management software which is regulated in the USA. Of the 16 initiatives directed at unregulated software, 11 were aimed at increasing standardisation using guidelines and standards for safe system design, build, implementation and use. Three initiatives for unregulated software were aimed at certification in the USA, Canada and Australia. Safety is addressed alongside interoperability in the Australian certification programme but it is not explicitly addressed in the US and Canadian programmes, though conformance with specific functionality, interoperability, security and privacy requirements may lead to safer systems. England appears to have the most comprehensive safety management programme for unregulated software, incorporating safety assurance at a local healthcare organisation level based on standards for risk management and user interface design, with national incident monitoring and a response function. CONCLUSIONS There are significant gaps in the safety initiatives for HIT systems. Current initiatives are largely focussed on software. With the exception of diagnostic, prognostic, monitoring and treatment software, which are subject to medical device regulations in some countries, the safety of the most common types of HIT systems such as EHRs and CPOE without decision support is not being explicitly addressed in most nations. Appropriate mechanisms for safety assurance are required for the full range of HIT systems for health professionals and consumers including all software and hardware throughout the system lifecycle. In addition to greater standardisation and oversight to ensure safe system design and build, appropriate implementation and use of HIT is critical to ensure patient safety.

Paper: On the quality of neural net classifiers

This paper describes several concepts and metrics that may be used to assess various aspects of the quality of neural net classifiers. Each concept describes a property that may be taken into account by both designers and users of neural net classifiers when assessing their utility. Besides metrics for assessment of the correctness of classifiers we also introduce metrics that address certain aspects of the misclassifications. We show the applicability of the introduced quality concepts for selection among several neural net classifiers in the domain of thyroid disorders.

Template waveform recognition applied to ECGVCG analysis

Abstract Waveform recognition in biological signals is hampered by the variability of the waveforms and the superimposed noise. A method is presented which enables the recognition of onsets or offsets or other points of interest in waveforms, using templates with amplitude-time properties. The method is illustrated with the help of the automatic recognition of begin and end points of QRS complexes. The template is found by the analysis of a learning population of waveforms, recognized beforehand by human observers. The mathematical relationship with other wave recognition methods is indicated: threshold detection, least-squares fitting. The template can be explained in terms of a set of weighted multilevel threshold detectors.

Multilead ECG analysis

This paper describes the results of our recent research in computer-assisted ECG/VCG interpretation. It comprises new developments which were initiated by the advent of relatively inexpensive microcomputers. Our previous systems performed an off-line analysis of ECGs. Currently, there is a trend to move computer power near to the patient and to provide on-line analysis of ECGs. Besides the advantage of the direct availability of the ECG interpretation, quality control will reduce the number of uninterpretable ECGs and hence the number of repeated recordings. This paper describes the requirements that were established for a system for on-line ECG analysis. The system is based on our modular approach, just like our off-line system, Modular ECG ANalysis System (MEANS). Changes in the methods and software had to be made mainly because of the simultaneity of all ECG leads and the concurrency of the processing tasks. Other modifications and extensions of the algorithms necessary to meet the requirements of on-line ECG interpretation especially those related to processing speed, are discussed, and evaluation results are presented.

The VATAM guidelines

Evaluation and assessment of the impact of information and communication technology in medicine is gaining interest. Unfortunately, till now there were no agreed upon approaches. The objective of the VATAM project is to develop guidelines that will assist assessors to set-up and execute studies. This paper describes the background of the VATAM project and provides an account of the current state of the guidelines. It concludes with an indication of the developments that will take place in the short term to further elaborate the guidelines and some considerations for consolidation of VATAMs results.

STARE-HI – Statement on Reporting of Evaluation Studies in Health Informatics: Explanation and Elaboration

BACKGROUND Improving the quality of reporting of evaluation studies in health informatics is an important requirement towards the vision of evidence-based health informatics. The STARE-HI - Statement on Reporting of Evaluation Studies in health informatics, published in 2009, provides guidelines on the elements to be contained in an evaluation study report. OBJECTIVES To elaborate on and provide a rationale for the principles of STARE-HI and to guide authors and readers of evaluation studies in health informatics by providing explanatory examples of reporting. METHODS A group of methodologists, researchers and editors prepared the present elaboration of the STARE-HI statement and selected examples from the literature. RESULTS The 35 STARE-HI items to be addressed in evaluation papers describing health informatics interventions are discussed one by one and each is extended with examples and elaborations. CONCLUSION The STARE-HI statement and this elaboration document should be helpful resources to improve reporting of both quantitative and qualitative evaluation studies. Evaluation manuscripts adhering to the principles will enable readers of such papers to better place the studies in a proper context and judge their validity and generalizability, and thus in turn optimize the exploitation of the evidence contained therein. LIMITATIONS This paper is based on experiences of a group of editors, reviewers, authors of systematic reviews and readers of the scientific literature. The applicability of the details of these principles has to evolve as a function of their use in practice.