Clifford R. Mynatt

Confirmation bias in a simulated research environment: An experimental study of scientific inference

Numerous authors (e.g., Popper, 1959) argue that scientists should try to falsify rather than confirm theories. However, recent empirical work (Wason and Johnson-Laird, 1972) suggests the existence of a confirmation bias, at least on abstract problems. Using a more realistic, computer controlled environment modeled after a real research setting, subjects in this study first formulated hypotheses about the laws governing events occurring in the environment. They then chose between pairs of environments in which they could: (I) make observations which would probably confirm these hypotheses, or (2) test alternative hypotheses. Strong evidence for a confirmation bias involving failure to choose environments allowing tests of alternative hypotheses was found. However, when subjects did obtain explicit falsifying information, they used this information to reject incorrect hypotheses.

Consequences of confirmation and disconfirmation in a simulated research environment

Advanced undergraduate science majors attempted for approximately 10h each to discover the laws governing a dynamic system. The system included 27 fixed objects, some of which influenced the direction of a moving particle. At a given time, any one screen of a nine-screen matrix could be observed on a plasma display screen. Confirmatory strategies were the rule, even though half the subjects had been carefully instructed in strong inference. Falsification was counterproductive for some subjects. It seems that a firm base of inductive generalizations, supported by confirmatory research, is a prerequisite to useful implementation of a falsification strategy.

Strategies of rule discovery in an inference task

It has long been known that subjects in certain inference tasks will seek evidence which can confirm their present hypotheses, even in situations where disconfirmatory evidence could be more informative. We sought to alter this tendency in a series of experiments which employed a rule discovery task, the 2-4-6 problem first described by Wason. The first experiment instructionally modified subjects confirmatory tendencies. While a disconfirmatory strategy was easily induced, it did not lead to greater efficiency in discovering the rule. The second experiment introduced subjects to the possibility of disconfirmation only after they had developed a strongly held hypothesis through the use of confirmatory evidence. This manipulation also failed to alter the efficiency of rule discovery. In the third experiment, subjects were taught to use multiple hypotheses at each step, in the manner of Platts “Strong Inference”. This operation actually worsened performance. Finally, in the fourth experiment, the structure of the problem was altered slightly by asking subjects to seek two interrelated rules. A dramatic increase in performance resulted, perhaps because information which in previous tasks was seen as merely erroneous could now be related to an alternative rule. The four studies have broad implications for the psychological study of inference processes in general, and for the study of scientific inference in particular.

Information relevance, working memory, and the consideration of alternatives

People routinely focus on one hypothesis and avoid consideration of alternative hypotheses on problems requiring decisions between possible states of the world–-for example, on the “pseudodiagnosticity” task (Doherty, Mynatt, Tweney, & Schiavo, 1979). In order to account for behaviour on such “inference” problems, it is proposed that people can hold in working memory, and operate upon, but one alternative at a time, and that they have a bias to test the hypothesis they think true. In addition to being an ex post facto explanation of data selection in inference tasks, this conceptualization predicts that there are situations in which people will consider alternatives. These are: 1. “action” problems, where the alternatives are possible courses of action; 2. “inference” problems, in which evidence favours an alternative hypothesis. Experiment 1 tested the first prediction. Subjects were given action or inference problems, each with two alternatives and two items of data relevant to each alternative. They received probabilistic information about the relation between one datum and one alternative and picked one value from among the other three possible pairs of such relations. Two findings emerged; (1) a strong tendency to select information about only one alternative with inferences; and (2) a strong tendency, compared to inferences, to select information about both alternatives with actions. Experiment 2 tested the second prediction. It was predicted that data suggesting that one alternative was incorrect would lead many subjects to consider, and select information about, the other alternative. For actions, it was predicted that this manipulation would have no effect. Again the data were as predicted.

Responsibility attribution in groups and individuals

Tested the major assumption of the diffusion of responsibility hypothesis (i.e. that group members involved in acts with negative consequences should attribute less responsibility to themselves than either individuals who experience negative consequences or groups or individuals who experience positive consequences). Using 80 male undergraduates, this assumption was tested in a 2 × 2 design in which individuals or groups gave advice that led to success or failure. Group failure Ss assumed less responsibility than Ss in the other 3 conditions. Furthermore, individual failure Ss rated the outcomes as less bad and saw themselves as having less influence over the advisee. Other areas of social psychological research in which the combination of responsibility and negative consequences seems to be a critical factor are discussed. (26 ref) (PsycINFO Database Record (c) 2006 APA, all rights reserved)

On people's understanding of the diagnostic implications of probabilistic data

Two lines of prior research into the conditions under which people seek information are examined in light of two statistical definitions of diagnosticity. Five experiments are reported. In two, subjects selected information in order to test a hypothesis. In the remaining three, they selected information in order to convince someone else of the truth of a known hypothesis. A total of 567 university students served as subjects. The two primary conclusions were as follows: (1) When the task is highly structured by the environment, subjects select information diagnostically, and (2) when the task is less structured, so that subjects must seek relevant information not manifest, they select information pseudodiagnostically. Possible relations to other laboratory inference tasks and to clinical judgment are discussed.

Rationality and Disconfirmation: Further Evidence

Recent laboratory research on the psychology of problem solving has important implications for an understanding of the psychology of science. Such work is reviewed, and supports the claim made by Hardin in this journal that pursuit of confirmatory data is highly adaptive in scientific problem solving. Hardins reservations about the generalizability of such research to real science are considered, and rejected on empirical grounds. Further research bearing on the same issue is discussed, revealing that confirmatory strategies alone are less useful than mixed strategies that employ confirmatory and disconfirmatory approaches. The validity of the claim is supported by brief consideration of Michael Faradays research strategies.

Why software testing is sometimes ineffective: two applied studies of positive test strategy

The term positive test strategy describes the tendency to test a hypothesis with test cases that confirm (i.e., aim to support) rather than disconfirm the hypothesis. Most demonstrations of this phenomenon have involved relatively abstract problems. The authors suggest that people use a positive test strategy in a more applied setting as well, that is, in computer software testing. In 2 experiments, they examined how Ss with varying expertise performed functional testing of software. There was substantial evidence of the use of a positive test strategy: Ss tended to test only those functions and aspects of the software that were specifically described in the specifications as what the software was supposed to do. This effect was only partially mitigated by increasing expertise among testers and by more complete program specifications

The influence of feedback and diagnostic data on pseudodiagnosticity

Pseudodiagnosticity is the tendency of people to select diagnostically worthless data in an opinion revision task and then to make a judgment based on those data. University students were presented four problems in which they selected data and then made a judgment based on those data. Knowledge of results was provided on each trial. Half the subjects were “force-fed” truly diagnostic data. Getting the correct answer virtually guaranteed that subjects repeated the data selection strategy that preceded it. Data selection following an incorrect answer was contingent upon whether the subjects had seen diagnostically relevant data.

When self-interest and altruism conflict.

Two experiments studied (a) individual and group decision making under conditions of conflict between self-interest and altruism and (b) retribution of harm-doing by victims of individuals and groups. In Exp I, 60 undergraduate males were assigned to conditions of individual or group decision making or were victims of individuals or groups. Groups were far more punitive than individuals but reported only slightly diminished responsibility for the fate of the victim. Exp II, with 54 male undergraduates, showed that victim retribution is a function of punishment received and not a function of the source of punishment. Results are discussed within the framework of processes of group decision making. (43 ref) (PsycINFO Database Record (c) 2006 APA, all rights reserved)