William R. Ferrell

A model of calibration for subjective probabilities

Abstract A mathematical model is developed to describe the calibration of discrete subjective probabilities and is compared with published group calibration results and with new data. The model is appropriate to probability assessment tasks, with a variety of formats, that can be considered from a signal detection point of view, such as giving the probability that a particular two-category classification is correct. The model assumes that the respondent partitions the range of a decision variable and maps the set of response probabilities onto it. Such a model can account for the systematic effect of proportion correct on the degree of under- or overconfidence; it indicates the ways in which training can affect calibration; it makes specific predictions about base rate effects; it provides a measure of “knowing what one knows”; and it gives a unifying viewpoint for a large body of experimental work on calibration.

Combining Individual Judgments

It is often assumed that n heads are better than one, that a judgment obtained from a group will be of higher quality than could be expected from an individual. This chapter considers the effectiveness of methods that have been proposed for combining individual quantitative judgments into a group judgment. For the most part, it will be found that n heads are, indeed, better than one, and at least one investigator has concluded that it does not much matter how they are combined. But the potential for improving performance is so great and the problems of achieving it so subtle that a clear understanding of the issues is essential.

A model for realism of confidence judgments: implications for underconfidence in sensory discrimination.

In a recent issue of this journal, Björkman, Juslin, and Winman (1993) presented a model of the calibration of subjective confidence judgments for sensory discrimination which they called “subjective distance theory.” They proposed that there was a robust underconfidence bias in such judgments, that the model predicted such a bias, and that two different models were needed for the calibration of subjective confidence for cognitive judgments and for sensory ones. This paper addresses issues they raised. It points out that they have not presented a new model, but rather a portion of a more general one, the “decision-variable partition model” originally proposed in Ferrell and McGoey (1980). This paper explores properties of the model and shows, contrary to Björkman, Juslin, and Winman’s hypotheses, that the model does not predict under-confidence, that the “hard-easy effect” can be observed with sensory discriminations, and that the model fits not only sensory, but also cognitive judgments.

Error Monitoring of Schoolwork by Learning Disabled Adolescents

Considerable current research and theory regarding learning disabilities concern the role of possible attentional deficits. Critical among the many sources of information in the learning situation is the correctness of the learners own response — monitoring ones own errors provides the basis for their elimination. Using a sophisticated research paradigm, described in some detail, the authors examine how learning disabled children notice their own errors compared to the process used by their normal-learning peers. — G.M.S. Monitoring on some school-related tasks (editing, writing, spelling, and synonyms) was studied in learning disabled and normal adolescents. Signal detection was used to conceptualize the monitoring process. While learning disabled and normal students appeared to use similar and appropriate criteria for detecting errors in externally generated material, the learning disabled students were less willing to call an element an error in material they produced themselves. It is suggested that external monitoring done by the teacher while the student is doing his work teaches monitoring better than correcting errors after the work is completed.

Learning Disability Classification by Bayesian Aggregation of Test Results

The feasibility of the Bayesian approach to screening for learning disability proposed by Wissink, Kass, and Ferrell (1975) is further explored. Two matched groups of children, one with and one without learning disability, were given tests related to component disabilities previously identified as being diagnostic. The probability that each child has learning disability was calculated from the test results according to the Bayesian procedure and on that basis the children were reclassified. The Bayesian method compared favorably with discriminant analysis in accuracy and is more easily applied to screening. Moreover, the lack of independence in the data did not seriously affect the results. It is concluded that the method warrants further study.

An Example of the Use of Fuzzy Set Concepts in Modeling Learning Disability

The way a particular clinician judges, from data, the degree to which a child is in the category “learning disabled” was modeled on the basis of the clinician’s statement of the traits that comprise the handicap. The model illustrates the use of fuzzy set theory to make a formal model from a vague and imprecise verbal model. The method can facilitate research in areas where understanding of phenomena is not yet well developed.

The Effect of Base Rate on Calibration of Subjective Probability for True-False Questions: Model and Experiment

Abstract The degree of calibration of subjective probabilities of events is the extent to which the observed proportion of events that occur agrees with the assigned probability values. The decision variable partition model of calibration is reviewed tutorially. It shows how numerical subjective probabilities for discrete events can be related to the perceived truth of propositions. It has been able to explain a number of experimental findings about calibration of subjective probabilities of correct response to two-alternative multiple-choice questions and to questions to which the respondent supplies the answer. In this paper, the models predictions for true-false items axe derived, and an experiment testing them is reported. The model predicts that when the subjective probability that items are true is assessed, there will be a specific effect of the base rate, the proportion of true items, but that there will be no effect when the respondent decides true or false and then reports a subjective probability of being correct. A systematic effect of task difficulty is predicted in both cases. The experimental results are in close agreement with the models predictions.

Influence Allocation Methods in Group Decision Support Systems

Influence allocation processes are voting and opinion aggregating methods that allow members to distribute some or all of their decision making influence to others in the group in order to exploit not only the groups knowledge of the alternatives, but its knowledge of itself. Only with the common use of group decision support systems (GDSS) has their use become practical. In this paper we reconsider SPAN, an influence allocation process introduced by MacKinnon (1966a). Experimental comparison shows SPAN to be significantly better at selecting a correct option from a set of options than two common voting methods. An alternative influence allocation process that we call RCON (Rational Consensus), is based on a weighting method proposed by DeGroot (1974) and has been explicated as a normative standard for combining opinion by Lehrer and Wagner (1981). The judgmental inputs to SPAN would appear to be logically related to those for RCON. Submitting the SPAN inputs from the experiment, transformed in this logical way, to the RCON process results in somewhat better performance than with SPAN. However, evidence indicated that the two methods are conceptually and psychologically sufficiently different that an experimental comparison is needed between them.

Detectability of correctness: A measure of knowing that one knows

It is important to know that one knows, to be able to discriminate what one knows from what one does not, since that is a basis for making decisions about using and augmenting ones knowledge. A measure of knowing that one knows has been proposed, the detectability of correctness, the ability to discriminate those answers that one gives that are correct from those that are incorrect. This paper presents the background of the problem of knowing that one knows, points out how previous efforts at measurement have been faulty and presents empirical results that indicate that detectability of correctness is a valid and stable measure.

Subjective detection of differences in variance from small samples

Abstract Experiments are reported testing peoples ability to detect differences in population variance from small samples. Performance was well described by a signal detection model and measures of sensitivity were obtained and compared with optimal performance. Contrary to expectation, performance with unidimensional normal distributions was close to optimal for different sample sizes and under a variety of modes of stimulus presentation ranging from points on a line to verbal categories. Performance when the source distributions were unknown was uniformly better than that of a commonly used non-parametric test. Performance was significantly reduced, however, when univariate normal samples of size 2 n were presented; as bivariate samples of size n .