Tim Rakow

Uncertain Judgements: Eliciting Experts' Probabilities: O'Hagan/Uncertain Judgements: Eliciting Experts' Probabilities

Elicitation is the process of extracting expert knowledge about some unknown quantity or quantities, and formulating that information as a probability distribution. Elicitation is important in situations, such as modelling the safety of nuclear installations or assessing the risk of terrorist attacks, where expert knowledge is essentially the only source of good information. It also plays a major role in other contexts by augmenting scarce observational data, through the use of Bayesian statistical methods. However, elicitation is not a simple task, and practitioners need to be aware of a wide range of research findings in order to elicit expert judgements accurately and reliably. Uncertain Judgements introduces the area, before guiding the reader through the study of appropriate elicitation methods, illustrated by a variety of multi-disciplinary examples.

Individual differences in sensitivity to health communications: consideration of future consequences.

There are reliable individual differences in the extent to which people consider the long- and short-term consequences of their behaviors. Such differences, assessed by the Consideration of Future Consequences (CFC) Scale (A. Strathman, F. Gleicher, D. S. Boninger, & C. S. Edwards, 1994), are hypothesized to influence the impact of a persuasive communication. In an experimental study, the time frame of occurrence of positive and negative consequences of engaging in a new colorectal cancer-screening program was manipulated in a sample of two hundred twenty 50-69-year-old men and women. CFC moderated (a) the processing of short- versus long-term consequences and (b) the persuasive impact of the different communications on behavioral intentions. Low CFC individuals produced more positive thoughts and were more persuaded when positive consequences were short term and negative consequences were long term. The opposite was true for high CFC individuals.

Think, blink or sleep on it? The impact of modes of thought on complex decision making

This paper examines controversial claims about the merit of “unconscious thought” for making complex decisions. In four experiments, participants were presented with complex decisions and were asked to choose the best option immediately, after a period of conscious deliberation, or after a period of distraction (said to encourage “unconscious thought processes”). In all experiments the majority of participants chose the option predicted by their own subjective attribute weighting scores, regardless of the mode of thought employed. There was little evidence for the superiority of choices made “unconsciously”, but some evidence that conscious deliberation can lead to better choices. The final experiment suggested that the task is best conceptualized as one involving “online judgement” rather than one in which decisions are made after periods of deliberation or distraction. The results suggest that we should be cautious in accepting the advice to “stop thinking” about complex decisions.

The role of experience in decisions from description

We extend research on the distinction between decisions from experienceor description to situations in which people are given perfect information about outcome probabilitiesand have experience in an environment which matches the described information. Participants read a description of a die with more sides of one color than another (e.g., 4 black and 2 white sides) and were then asked either to predict the outcomes of rolls of the die or to select the best strategy for betting on the most likely outcome for each roll in a hypothetical game. Experience in the environment (trials), contingency (probability of the more likely alternative), and outcome feedback all had significant effects on the adoption of the optimal strategy (always predicting the most likely outcome), despite their normative irrelevance. Comparisons of experience with description-only conditions suggested that experience exerted an influence on performance if it was active—making predictions—but not if it was passive—observing outcomes. Experience had anegative initial impact on optimal responding: participants in description-only conditions selected the optimal strategy more often than those with 60 trials of prediction experience, a finding that reflects the seduction of “representative” responding.

The determinants of parents' beliefs about the intelligence of their children: A study from Hong Kong

One hundred and ninety-three Hong Kong parents (mean age 42.2 years) were given a structured interview / questionnaire concerning their own and their childrens self-estimated overall and multiple intelligence. Previous research suggested that males tend to give higher overall “g” estimates to their children and themselves than do females, as well as higher scores on mathematical and spatial intelligence (Furnham, 2001). Further, studies in the West suggest that parents think their children are significantly brighter than they are and that their sons are brighter than their daughters. Estimates were lower than those found in Western populations but, even so, males rated their own mathematical and spatial intelligence higher than did females. Hong Kong Chinese parents did not think their sons were brighter than their daughters. The seven multiple intelligences factored into three clear factors for self and children, and regressions indicated that it was “academic” intelligence (verbal, mathematical, spatia...

Evaluating Three Criteria for Establishing Cue-Search Hierarchies in Inferential Judgment

The authors identify and provide an integration of 3 criteria for establishing cue-search hierarchies in inferential judgment. Cues can be ranked by information value according to expected information gain (Bayesian criterion), cue-outcome correlation (correlational criterion), or ecological validity (accuracy criterion). All criteria significantly predicted information acquisition behavior; however, in 3 experiments, the most successful predictor was the correlational criterion (followed by the Bayesian). Although participants showed sensitivity to task constraints, searching for less information when it was more expensive (Experiment 1) and when under time constraints (Experiment 2), concomitant changes in the relative frequency of acquisition of cues with different information values were not observed. A rational analysis illustrates why such changes in the frequency of acquisition would be beneficial, and reasons for the failure to observe such behavior are discussed.

Prospective Research in Health Service Settings: Health Psychology, Science and the ‘Hawthorne’ Effect

Health service providers sometimes express concern about the impact of prospective survey research upon patient behaviour. To date, there is little available evidence from which to estimate the likelihood of any ‘Hawthorne’ effect on patient behaviour in health service settings. We analysed data from one of our own surveys to investigate whether inviting people to participate in research had any impact on their subsequent uptake of a screening service. Findings showed that people sent a questionnaire were slightly faster to take up screening than those not sent a questionnaire. We obtained no significant difference in absolute service uptake rate at six months.

Doomed to repeat the successes of the past: history is best forgotten for repeated choices with nonstationary payoffs.

Many everyday tasks involve repeated choices in which past outcomes are used to estimate payoffs but in which present payoffs may differ from past ones. Two experiments with 10 decision problems employing the decisions-from-feedback paradigm examined the choice between two risky options, wherein the payoff probabilities for one option could change over a sequence of trials. Participants either saw the outcomes associated with each option, or additionally were given a “history” summarizing the outcomes of previous trials. Participants adapted quickly to new problems, but adapted slowly to payoff changes. Providing a history improved initial choices, but had a null or negative impact on later ones—although, appropriately, the summary received less weight in later trials. An associative choice model captured changes in preference, but not initial patterns of choice. The findings emphasize the adaptive value of forgetting in unstable decision environments, but illustrate how providing additional relevant information may hinder this.

The role of working memory in information acquisition and decision making: Lessons from the binary prediction task

The effects of memory constraints upon information acquisition and decision making were examined in two experiments using binary prediction tasks, where participants observe outcomes for two options before deciding which one to bet upon. Our studies extend previous investigations to the case where participants learn the structure of the task through observation, but where information acquisition is separated from the task of prediction. Participants with higher cognitive capacity (larger memory span or higher intelligence) were more likely to adopt the “maximizing” strategy (always selecting the more frequent alternative). This finding conflicts with some recent investigations of similar tasks, a contrast that implies that the presence of feedback on choices may be important in determining the strategic actions of high-capacity individuals. Participants selecting the optimal strategy were in turn more efficient in their data acquisition. The behaviour of participants adopting suboptimal choice strategies was consistent with prediction based upon a “narrow window of experience”—that is, seeking to match the characteristics of small samples of observations.

Forgetting the past: Individual differences in recency in subjective valuations from experience.

Recent research investigating decisions from experience suggests that not all information is treated equally in the decision process, with more recently encountered information having a greater impact. We report 2 studies investigating how this differential treatment of sequentially encountered information affects subjective valuations of risky prospects when observations of past outcomes must be used to estimate the prospects payoff distribution, and examine how individual differences in cognitive capacities influence information usage. In Study 1 we found that a sliding window of information model that averages a subset of (only) the most recently encountered outcomes (samples) fit the subjective valuation data for a portion of individuals better than models that integrate all observed outcomes. This pattern of results is replicated in Study 2, in which we also found that the amount of information used to form valuations varies greatly between individuals, and that individual difference in memory span explains a portion of this variation. Combined, these results suggest a process in which information usage is in part reliant on cognitive capacity, and where information aggregation appears to be memory based rather than online, providing new insight into the processes involved in the construction of valuation in experiential decisions.