Sally V. Rudmann

Coping with the complexities of multiple-solution problems: a case study

A model is proposed to account for the expertise of a skilled immunohematologist in solving multiple-solution problems. These problems, which he must deal with daily, are concerned with ensuring the safe transfusion of blood into patients. This model suggests that he copes with this difficult class of problems by: (1) Using patterns in the data to simplify the problem, hypothesizing the number of primitive solutions necessary to account for the test results and, when possibles, decomposing the problem into a set of less complex, single-solution problems. Such decompositions then enable more powerful reasoning processes; (2) making use of a mixture of data-driven and hypothesis-driven processes in order to reduce the chances that heuristic (and therefore error-prone) methods and cognitive biases will lead away from critical data; (3) Relying on a mixture of confirmatory and rule-out processes to provide converging evidence, thus reducing the changes of error; (4) uncovering his own errors through the use of “error models” that identify the conditions where one of his processes is likely to make an error (similar to the use of student models by expert tutors to diagnose mistakes made by students).

An empirical evaluation of the performance of antibody identification tasks.

Four empirical studies were conducted for better understanding of the nature of problem‐solving activities by medical technologists and medical technology students when performing antibody identification tasks. The results indicated the importance of strategies that ensure the collection of converging evidence, as these strategies protect against the fallibility of commonly used heuristics and against errors due to simple slips. The results also indicate that not only do students make significant numbers of errors, but so do practicing technologists. In one of the studies covering a 1‐year period, for instance, a group of 16 technologists made a total of 41 errors in 1057 cases. On the basis of these findings, several alternatives are proposed to reduce errors.

Successful Use of an Expert System to Teach Diagnostic Reasoning for Antibody Identification

A previously reported study indicated that, when used by an instructor as a tool to assist with tutoring in a class laboratory setting, use of the Transfusion Medicine Tutor (TMT) resulted in improvements in antibody identification performance of 87-93% (p<.001). Based on input from teachers requesting that TMT be designed for use without the presence of an instructor, a new study on the use of TMT without instructor assistance found that performance improved by 64-66% (p<.001). Finally, based on the results of these two studies, TMT was mailed to 7 sites for beta-testing. In exchange for a free copy of the kit, the instructors (and their students) were asked to fill out questionnaires. Results of these questionnaires are summarized.

The antibody identification assistant (AIDA), an example of a cooperative computer support system

Performance when using a critiquing expert system was compared to performance with no decision support for two groups of medical technologists solving antibody identification cases. The treatment group had a significantly lower misdiagnosis rate than the control group across the four cases tested (p<0.000005). This is one of the few studies conducted to evaluate critiquing as a form of decision support for practitioners solving real-world tasks. In particular, there was a trend for improved performance even on a case for which the computers knowledge was not fully competent. This is in contrast to the usual problems with people not being able to recover from faulty reasoning exhibited by a brittle, partially automated decision support system. Users of critiquing systems are doing the task themselves and given feedback in the context of what they are doing. Thus, the computer can monitor for errors in the humans reasoning, and the human has a basis for judging the computers reasoning, resulting in cooperative problem-solving between the two decision makers.

Errors in abductive reasoning

Results of efforts to extract knowledge for an expert whose job is to detect the errors made by practicing technologies are examined. These errors are discussed in terms of possible cognitive biases. The example examined involves students, technologies, and experts performing antibody identification tasks in order to construct a critiquing and intelligent tutoring system.<<ETX>>

The Transfusion Medicine Tutor: Methods and Results from the Development of an Interactive Learning Environment for Teaching Problem-Solving Skills

Investigations of students and practicing medical technologists indicate that both groups make significant numbers of errors on tasks such as antibody identification. One potential solution to help with this problem is to provide access to a computerized learning environment in which users can get exposure to a larger and much broader set of problems than would otherwise be possible. This paper describes such a learning environment, the Transfusion Medicine Tutor, and discusses the ways in which it supports guided discovery learning.

The Transfusion Medicine Tutor: Using Expert Systems Technology to Teach Domain-Specific Problem-Solving Skills

This study provides data regarding the effectiveness of the expert system-based Transfusion Medicine Tutor (TMT) when used by medical technology students to learn an important problem-solving task, the identification of alloantibodies in a patients blood for the purpose of finding compatible blood for transfusion. The results show that the students who were taught by an instructor using TMT to provide the instructional environment went from 0% correct on a pre-test case to 87%–93% correct on post-tests (N=15). This compares with an improvement rate of 20% by a control group (N=15) who used a passive version of the system with the tutoring functions turned off. The results also demonstrate the importance of relying on objective performance data rather than questionnaire data to evaluate systems, as there was no difference in the subjective responses of the students to these two different versions of the system.

The transfusion medicine tutor: a case study in the design of an intelligent tutoring system

Empirical studies of medical technology students indicated that there is a considerable need for additional skill development in performing tasks such as antibody identification. The transfusion medicine tutor (TMT) explores the use of a model of expert problem-solving to direct the application of a variety of teaching strategies in such an environment. This tutoring system focuses on red cell antibody identification, which is a problem-solving task where multiple primitive solutions can be true simultaneously. The approach to tutoring is the design of a problem-solving environment in which the computer plays an active role in tutoring the student. TMT allows students to gain experience by exploring a wide range of patient cases. The design of TMT makes it easy for the computer to detect certain types of errors. TMT allows more advanced students to compare the conclusions they have drawn from a particular test result with the computers interpretation of the same data.<<ETX>>

Design concepts underlying the use of an expert system to teach diagnostic reasoning for antibody identification

Antibody identification is a laboratory task where medical technologists must select tests to run and interpret the results in order to determine the antibodies in a patients blood. It has the classical characteristics of an abduction task, including masking and problems with noisy data. Based on a cognitive analysis of both successful and error-producing performances, a tutoring system (the transfusion medicine tutor, TMT), was developed that uses expert systems technology to provide immediate, context-sensitive feedback as students solve actual patient cases. Development of this system required consideration of aspects of artificial intelligence, education, psychology, and human factors engineering, as well as the domain of study (i.e., allo-antibody identification). In a formal field evaluation, when used by an instructor as a tool to assist with tutoring in a class laboratory setting, use of TMT resulted in improvements in antibody identification performance of 87-93% (p<.001) as compared to a passive control version which improved performance by 20%.

A Testbed for Teaching Problem Solving Skills in an Interactive Learning Environment

An interactive learning environment was developed with the goal of empirically testing the effectiveness of various teaching strategies in improving problem solving performance. The domain chosen was transfusion medicine since it involves solving complex, multiple solution problems which are typically found to be difficult (Elstein, Shulman, and Sprafka, 1978) and because normal performance of this task calls for marking data sheets with intermediate conclusions, thereby improving the chances of the computer correctly inferring the students reasoning. The testbed, called TMT (for Transfusion Medicine Tutor), monitors for errors, builds a model of what a student knows and can select teaching strategies based on human tutoring models that were developed from earlier studies. The testbed will be used to collect data of a students performance in conditions where the degree of teaching and type of feedback are manipulated. A number of broadly applicable issues can be explored in this framework such as the difference between expert and student problem solving strategies, the effectiveness of different teaching strategies, and the importance of modeling student knowledge and providing visual feedback when developing an interactive learning environment. Preliminary results of our experiments, a demonstration of the testbed, and a discussion of how it was implemented will be presented in the demonstration session.