Robert M. Rippey

Outcome of an osteoarthritis education program for low-literacy patients taught by indigenous instructors

A 10-h osteoarthritis education course was developed and evaluated for older low-income patients with osteoarthritis. Indigenous community leaders were trained to teach the course within inner-city neighborhoods of Hartford, Connecticut. Significant differences were obtained using a quasi-experimental group, pre/post-test design. There was a significant increase in knowledge (P less than 0.001) both on a verbal knowledge test and a picture story test (P less than 0.001). There was a significant increase in scores on an exercise scale (P less than 0.001). Attitude toward ones illness improved. There was a slight improvement in function which was not significant. Use of adaptive equipment increased as a direct result of the program. The course was well accepted and enjoyed by the participants.

Premature Conclusions in the Diagnosis of Iron-deficiency Anemia: Cause and Effect

Case records of patients with iron-deficiency anemia were audited using a computer-based algorithm. In 24 of 35 cases, the diagnostic conclusion seemed unjustified and characteristic of premature closure, one of four recurring errors in diagnostic reasoning described previ ously. Premature closure appeared to originate from subjects at all levels of training, to be easily and unquestioningly accepted by other physicians, and to inappropriately condition diagnostic and therapeutic decisions. Heuristics and biases described by Tversky and Kahne man are considered as contributory factors and patient care and teaching implications are discussed.

Knowledge, realism, and diagnostic reasoning in a physical diagnosis course

Two 20-item tests and a case problem were administered to 83 students in a physical diagnosis course. One test contained items related to the content of the case problem, and the other items related to the balance of the course content. Confidence scoring procedures yielded scores of both knowledge and realism on the two tests. The case problem determined the subjects ability to integrate clinical data into an accurate list of diagnoses and yielded scores reflecting incomplete assembly of clues into diagnoses (incomplete synthesis) and unjustified conclusions (premature closure). Reliability of the confidence-scored tests was significantly greater than the reliability on the same items scored as a single correct answer. Knowledge improvement scores on both tests (relevant and nonrelevant knowledge) were significantly correlated with errors of incomplete synthesis. The realism score correlated significantly with premature closures but only on the test where the item content was relevant to the case problem.

A computerized cancer information system

Cancer information was made available via a computer to patients visiting the University of Connecticut Dental Clinics. The computer program was menu driven. An option allowed users to leave messages for the system owners. The computer program generated a log of usage, user comments and items chosen. The user could choose from the following topics: (1) general information on cancer; (2) diet, nutrition and cancer; (3) smoking; (4) the environment, occupations and cancer; and (5) physical checkups. An option to leave a message for the system operators was the option chosen least. Analysis of the usage log shows that the system was used substantially while people were waiting in the clinic and that at least three or four individuals used the system every clinic day. Such information systems can provide important information to the public. The novelty of the system and the lack of supervision did not deter public use.

A computer program for administering and scoring confidence tests

In confidence-test procedures, subjects are asked to respond to all the options of multiple-choice items with weights. They may assign the entire weight to a single option or they may distribute their confidence. With Shufords truncated logarithmic scoring function (Shuford, Albert, & Massengill, 1966), incentives are provided for honest reporting of confidence, and guessing is discouraged. By utilizing adjustments based on least squares fitting of the subject response (r) to the performance-based likelihood vector (jj), a score of realism can be computed. The realism score indicates whether or not the subject was appropriately certain or cautious (Brown & Shuford, 1973). Adjustment of the knowledge score for inappropriate realism improves reliability and validity (Rippey & Voytovich, 1983). In addition, feedback from the realism score can lead to improvements in realism or suggest deficits in basic forecasting skills and knowledge (Yates, 1982). The logarithmic function has identified occasional bizarre behavior of respondents, suggestive of a pathology of reasoning (Rippey & Voytovich, 1985). The logarithmic method of scoring is used rather widely in England in a course in risk analysis offered by the Open University (1980). Because of its amenability to the scoring of traditional multiple-choice test items, the logarithmic function has been applied to studies of cognitive achievement. Other scoring functions have also been used in other studies, especially in connection with forecasting (Blattenberger & Lad, 1985; Yates, 1982). The Brier score, for example, has been widely studied because of its partitionability using components of variance methods. Confidence testing initially suffered from problems of administration and scoring (Ebel, 1968). Many of the problems were solved by the use of mainframe computer scoring (Rippey & Donato, 1978). A PLATO version of confidence testing was developed by the Rand Corporation (Landa, 1976), expanded by Rippey and Smith (1979), and improved by Anderson (1982). However, access to the system is much more immediate with microcomputers; the microcomputer program described here is a complete system for preparing tests and scoring keys, and for administering and scoring the tests, either individually or in batch mode. The individual testing procedures are most appropriate for educational use,

Adjusting Confidence Tests for Realism The Favorable Consequences

Confidence tests were administered three times during a course in physical diagnosis. Conventional measures of knowledge and written case studies were examined in order to determine acceptability, reliability, and strength of association between confidence measures and other reflections of performance. Confidence testing wasfound to be acceptable, in fact, even enjoyable, to the students. Reliability was significantly improved and realism was stable across test halves and independent of knowledge, suggesting the presence of separate, measurable attributes in clinical thinking. Although statistical significance was not reached, confidence testing seemed qualitatively better atpredictingperformance on written case histories. Students, in general, improved their performance over the year in realism assessment. Educational implications and directionsforfuture research are discussed.

Interactive Confidence Test Scoring and Interpretation

Confidence scored tests allow subjects to respond to the options on a multiple choice test by assigning probabilities to each of the item options. Reproducing scoring functions applied to the set of response probabilities produces scores which are maximized in the long run, if and only if, probabilities assigned to correct answers are equal to the proportion of times those probabilities are assigned to correct answers. This maximizing property makes it possible to separate evidence of knowledge from confidence in knowledge. Such separation has the potential for providing more reliable and valid measures of learning as well as information about the tendency to overvalue or undervalue ones information. This paper describes an interactive computer program for administering, scoring, and interpreting the results of confidence tests.

Diagnostic Reasoning in the Multiproblem Patient An Interactive, Microcomputer-Based Audit

This article describes an interactive computerprogram that allows students to enter a completely free-form problem list at the keyboard and receive immediate analysis of the accuracy and thoroughness of their diagnostic impressions. This type of exercise is a departure from traditional instruments because it focuses on patients with multiple interacting issues offering greater realism in approximating the elderly or primary care interaction. The program is briefly described and correlations with traditional audit techniques are shown. More important, development of the program had a profound effect in evolving our understanding of the teaching of diagnostic reasoning in multiproblem patients. This process is described.

Improving the Reliability and Validity of Confidence-Scored Tests by Adjusting for Realism.

Recent theoretical developments in the area of confidence testing suggest that an improvement in the reliability and validity of confidence tests should result from correcting scores for realism. Thirty-six medical and dental students were admin istered two short confidence-scored tests dealing with cellular and molecular biology. Both the uncorrected scores and the scores corrected for realism were correlated with course final test scores and with the biochemistry section of the National Medical Board Examination, Part I. The adjustment of scores for realism led to increases in reliability (split half) and predictive validity, although the differenceswere not statistically significant.

Letters: Excellence and Evaluation

portance for them of these types of activities. If we accept this argument, these areas need improvement. As a final comment, a major advantage of this type of study, using the individual as the unit, is that independent judges can estimate from these life stories the extent to which education or some specific aspect of it has had a major impact on the quality of life of specific individuals. The project report, An Empirical Study to Aid in Formulating Educational Goals, has now been published by the American Institutes for Research. This report makes several specific recommendations, but does not at all exhaust the potential insights to be gained from these developmental histories. It is planned to involve others in their interpretation and the formulation of improved policies and programs in education. John C. Flanagan American Institutes for Research Palo Alto, CA 94302