Eugenio Alberdi

Expertise and the interpretation of computerized physiological data: implications for the design of computerized monitoring in neonatal intensive care

This paper presents the outcomes from a cognitive engineering project addressing the design problems of computerized monitoring in neonatal intensive care. Cognitive engineering is viewed, in this project, as a symbiosis between cognitive science and design practice. A range of methodologies has been used: interviews with neonatal staff, ward observations and experimental techniques. The results of these investigations are reported, focusing specifically on the differences between junior and senior physicians in their interpretation of monitored physiological data. It was found that the senior doctors made better use of the different knowledge sources available than the junior doctors. The senior doctors were able to identify more relevant physiological patterns and generated more and better inferences than did their junior colleagues. Expertise differences are discussed in the context of previous psychological research in medical expertise. Finally, the paper discusses the potential utility of these outcomes to inform the design of computerized decision support in neonatal intensive care.

ReTAX: a step in the automation of taxonomic revision

Abstract Classification, and particularly taxonomic revision, have not been generally addressed by computational models of scientific discovery. In this paper we present a framework for the automation of taxonomic revision in biological domains. This framework views taxonomy formation as an interaction of: (a) observation; (b) creation and structuring of a taxonomic hierarchy; (c) identification of relevant taxonomic descriptors; and (d) use of background knowledge. We describe a prototype system for taxonomic revision, ReTAX, which implements relevant aspects of such a framework. ReTAX receives as input a pre-established taxonomy, and is presented with new items which contradict in some way the original classification. Using a set of consistency criteria, ReTAX identifies the inconsistencies between the new information and the taxonomy. The system then applies a set of refinement operators to modify the taxonomy and resolve the inconsistencies. ReTAX has been tested on a botanical domain, replicating taxonomic revisions which had been suggested by professional botanists for the family Ericaceae. Finally, we propose extensions to ReTAX, which we hope will enable us to further develop the framework, and subsequently create an aid which taxonomists can use to revise existing taxonomies.

How to Discriminate between Computer-Aided and Computer-Hindered Decisions: A Case Study in Mammography

Background. Computer aids can affect decisions in complex ways, potentially even making them worse; common assessment methods may miss these effects. We developed a method for estimating the quality of decisions, as well as how computer aids affect it, and applied it to computer-aided detection (CAD) of cancer, reanalyzing data from a published study where 50 professionals (“readers”) interpreted 180 mammograms, both with and without computer support. Method. We used stepwise regression to estimate how CAD affected the probability of a reader making a correct screening decision on a patient with cancer (sensitivity), thereby taking into account the effects of the difficulty of the cancer (proportion of readers who missed it) and the reader’s discriminating ability (Youden’s determinant). Using regression estimates, we obtained thresholds for classifying a posteriori the cases (by difficulty) and the readers (by discriminating ability). Results. Use of CAD was associated with a 0.016 increase in sensitivity (95% confidence interval [CI], 0.003–0.028) for the 44 least discriminating radiologists for 45 relatively easy, mostly CAD-detected cancers. However, for the 6 most discriminating radiologists, with CAD, sensitivity decreased by 0.145 (95% CI, 0.034–0.257) for the 15 relatively difficult cancers. Conclusions. Our exploratory analysis method reveals unexpected effects. It indicates that, despite the original study detecting no significant average effect, CAD helped the less discriminating readers but hindered the more discriminating readers. Such differential effects, although subtle, may be clinically significant and important for improving both computer algorithms and protocols for their use. They should be assessed when evaluating CAD and similar warning systems.

Human-machine diversity in the use of computerised advisory systems: a case study

Computer-based advisory systems form with their users composite, human-machine systems. Redundancy and diversity between the human and the machine are often important for the dependability of such systems. We discuss the modelling approach we applied in a case study. The goal is to assess failure probabilities for the analysis of X-ray films for detecting cancer, performed by a person assisted by a computer-based tool. Differently from most approaches to human reliability assessment, we focus on the effects of failure diversity — or correlation — between humans and machines. We illustrate some of the modelling and prediction problems, especially those caused by the presence of the human component. We show two alternative models, with their pros and cons, and illustrate, via numerical examples and analytically, some interesting and non-intuitive answers to questions about reliability assessment and design choices for human-computer systems.

CAD in mammography: lesion-level versus case-level analysis of the effects of prompts on human decisions

ObjectTo understand decision processes in CAD-supported breast screening by analysing how prompts affect readers’ judgements of individual mammographic features (lesions). To this end we analysed hitherto unexamined details of reports completed by mammogram readers in an earlier evaluation of a CAD tool.Material and methodsAssessments of lesions were extracted from 5,839 reports for 59 cancer cases. Statistical analyses of these data focused on what features readers considered when recalling a cancer case and how readers reacted to CAD prompts.ResultsAbout 13.5% of recall decisions were found to be caused by responses to features other than those indicating actual cancer. Effects of CAD: lesions were more likely to be examined if prompted; the presence of a prompt on a cancer increased the probability of both detection and recall especially for less accurate readers in subtler cases; lack of prompts made cancer features less likely to be detected; false prompts made non-cancer features more likely to be classified as cancer.ConclusionThe apparent lack of impact reported for CAD in some studies is plausibly due to CAD systematically affecting readers’ identification of individual features, in a beneficial way for certain combinations of readers and features and a damaging way for others. Mammogram readers do not ignore prompts. Methodologically, assessing CAD by numbers of recalled cancer cases may be misleading.

Gaining Assurance in a Voter-Verifiable Voting System

The literature on e-voting systems has many examples of discussion of the correctness of the computer and communication algorithms of such systems, as well as discussions of their vulnerabilities. However, a gap in the literature concerns the practical need (before adoption of a specific e-voting system) for a complete case demonstrating that the system as a whole has sufficiently high probability of exhibiting the desired properties when in use in an actual election. This paper discusses the problem of producing such a case, with reference to a specific system: a version of the Prêt à Voter scheme for voter-verifiable e-voting. We show a possible organisation of a case in terms of four main requirements – accuracy, privacy, termination and ‘trustedness’– and show some of the detailed organisation that such a case should have, the diverse kinds of evidence that needs to be gathered and some of the interesting difficulties that arise.