Lawrence D. Phillips

Calibration of probabilities: the state of the art to 1980

From the subjectivist point of view (de Finetti, 1937/1964), a probability is a degree of belief in a proposition. It expresses a purely internal state; there is no “right,” “correct,” or “objective” probability residing somewhere “in reality” against which ones degree of belief can be compared. In many circumstances, however, it may become possible to verify the truth or falsity of the proposition to which a probability was attached. Today, one assesses the probability of the proposition “it will rain tomorrow.” Tomorrow, one looks at the rain gauge to see whether or not it has rained. When possible, such verification can be used to determine the adequacy of probability assessments. Winkler and Murphy (1968b) have identified two kinds of “goodness” in probability assessments: normative goodness, which reflects the degree to which assessments express the assessors true beliefs and conform to the axioms of probability theory, and substantive goodness, which reflects the amount of knowledge of the topic area contained in the assessments. This chapter reviews the literature concerning yet another aspect of goodness, called calibration. If a person assesses the probability of a proposition being true as .7 and later finds that the proposition is false, that in itself does not invalidate the assessment. However, if a judge assigns .7 to 10,000 independent propositions, only 25 of which subsequently are found to be true, there is something wrong with these assessments.

Drug harms in the UK: a multicriteria decision analysis

BACKGROUND Proper assessment of the harms caused by the misuse of drugs can inform policy makers in health, policing, and social care. We aimed to apply multicriteria decision analysis (MCDA) modelling to a range of drug harms in the UK. METHODS Members of the Independent Scientific Committee on Drugs, including two invited specialists, met in a 1-day interactive workshop to score 20 drugs on 16 criteria: nine related to the harms that a drug produces in the individual and seven to the harms to others. Drugs were scored out of 100 points, and the criteria were weighted to indicate their relative importance. FINDINGS MCDA modelling showed that heroin, crack cocaine, and metamfetamine were the most harmful drugs to individuals (part scores 34, 37, and 32, respectively), whereas alcohol, heroin, and crack cocaine were the most harmful to others (46, 21, and 17, respectively). Overall, alcohol was the most harmful drug (overall harm score 72), with heroin (55) and crack cocaine (54) in second and third places. INTERPRETATION These findings lend support to previous work assessing drug harms, and show how the improved scoring and weighting approach of MCDA increases the differentiation between the most and least harmful drugs. However, the findings correlate poorly with present UK drug classification, which is not based simply on considerations of harm. FUNDING Centre for Crime and Justice Studies (UK).

CALIBRATION OF PROBABILITIES: THE STATE OF THE ART+

From the subjectivist point of view (de Finetti, 1937) a probability is a degree of belief in a proposition whose truth has not been ascertained. A probability expresses a purely internal state; there is no “right” or “correct” probability that resides somewhere “in reality” against which it can be compared. However, in many circumstances, it may become possible to verify the truth o£ falsity of the proposition to which a probability was attached. Today, we assess the probability of the proposition“it will rain tomorrow”. Tomorrow, we go outside and look at the rain gauge to see whether or not it has rained. When verification is possible, we can use it to gauge the adequacy of our probability assessments.

A theory of requisite decision models

A requisite decision model is defined as a model whose form and content are sufficient to solve a particular problem. The model is constructed through an interactive and consultative process between problem owners and specialists (decision analysts). The process of generating the model uses participants’ sense of unease about current model results to further development of the model. Sensitivity analyses facilitate the emergence of new intuitions about the problem; when no new intuitions arise, the model is considered requisite. At all stages of development, the model represents the social reality of the shared understanding of the problem by the problem owners. The goal of creating a requisite model is to help construct a new reality, to create a future. Validating a requisite decision model is done with reference to a requisite validation model whose form will be recognizably multi-attributed and whose content may draw on a variety of disciplines both scientific and clinical. A requisite model is more likely to be adequate if problem owners contributing to its development represent a variety of views, if the adversarial process is used to advantage, and if the specialist can provide a neutral perspective and setting. The role of decision analysis is to provide a framework for the development of a coherent model and to provide structure to thinking. While requisite models may be applicable in other areas of social science, they certainly highlight the need for a psychology of what people can do.

Estimating the Harms of Nicotine-Containing Products Using the MCDA Approach

Background: An international expert panel convened by the Independent Scientific Committee on Drugs developed a multi-criteria decision analysis model of the relative importance of different types of harm related to the use of nicotine-containing products. Method: The group defined 12 products and 14 harm criteria. Seven criteria represented harms to the user, and the other seven indicated harms to others. The group scored all the products on each criterion for their average harm worldwide using a scale with 100 defined as the most harmful product on a given criterion, and a score of zero defined as no harm. The group also assessed relative weights for all the criteria to indicate their relative importance. Findings: Weighted averages of the scores provided a single, overall score for each product. Cigarettes (overall weighted score of 100) emerged as the most harmful product, with small cigars in second place (overall weighted score of 64). After a substantial gap to the third-place product, pipes (scoring 21), all remaining products scored 15 points or less. Interpretation: Cigarettes are the nicotine product causing by far the most harm to users and others in the world today. Attempts to switch to non-combusted sources of nicotine should be encouraged as the harms from these products are much lower.

Cultural Variation In Probabilistic Thinking: Alternative Ways Of Dealing With Uncertainty

Abstract This study reviews research on cultural differences in “probabilistic thinking” and presents some intra- and inter-cultural findings. Strong differences are shown to exist between people raised under Asian and British cultures on measures of this ability. These differences were found to out-weigh any influence of subculture, religion, occupation, arts/science orientation and sex. Generally, Asians were found to adopt a less finely differentiated view of uncertainty both numerically and verbally than did the British sample. Numerical probabilities assessed by the Asians were more extreme and less realistic than those assessed by the British sample. Possible antecedents of these differences are outlined, and cultural differences in probabilistic thinking are shown to be compatible with decryptions of cultural differences in business decision making. It is argued that there are qualitative cultural differences in ways of dealing with uncertainty.

Cultural Differences in Viewing Uncertainty and Assessing Probabilities

This paper examines possible cultural influences on probabilistic thinking. By probabilistic thinking we mean the tendency to view the world in terms of uncertainty, the ascribing of different degrees of uncertainty to events, and the ability meaningfully to express that uncertainty either verbally or as a numerical probability. Although one of us (Phillips, 1970) has previously suggested that culture may influence the cognitive processes involved in probability assessment, we have not found a single study in the literature wholly devoted to this topic. However, in observing subjects in probability assessment experiments, we and other investigators have informally noted differences, particularly in overall comprehension of the tasks, that seem to be associated with subjects’ cultural backgrounds. The research reported here suggests that there is some substance to these informal observations.

Balancing benefit and risk of medicines: a systematic review and classification of available methodologies.

The need for formal and structured approaches for benefit–risk assessment of medicines is increasing, as is the complexity of the scientific questions addressed before making decisions on the benefit–risk balance of medicines. We systematically collected, appraised and classified available benefit–risk methodologies to facilitate and inform their future use.

Nuclear Risk Management on Stage: A Decision Analysis Perspective on the UK's Committee on Radioactive Waste Management

In 2003, the UK government set up a broad-based Committee on radioactive waste management (CoRWM) to look at the UKs policy on radioactive waste management with a view to jumpstarting a stalled policy process. The committees brief was to come up with a set of recommendations that would protect the public and the environment, and be capable of inspiring public confidence. After consulting widely with the public and stakeholders, and drawing on advice from scientists and other experts, CoRWM arrived at a remarkably well-received set of recommendations. On the basis of our experiences of working on CoRWMs multi-criteria decision analysis of different management options, study of CoRWM documentation, and interviews with committee members, we describe the explicit and implicit principles that guided CoRWM. We also give an account of the process by which CoRWM arrived at its conclusions, covering four phases: framing, shortlisting, option assessment, and integration; and four cross-cutting activities: public and stakeholder engagement (PSE), science and engineering input, ethics and social science input, and learning from overseas practice. We finish by outlining some of the key developments in the UKs radioactive waste management process, which followed on from the publication of CoRWMs report, and present our reflections for the benefit of the risk and decision analysts of future committees that, like CoRWM, are charged with recommending to government on the management of technically complex and risky technologies, drawing on extensive public and stakeholder consultation.

European rating of drug harms

Background: The present paper describes the results of a rating study performed by a group of European Union (EU) drug experts using the multi-criteria decision analysis model for evaluating drug harms. Methods: Forty drug experts from throughout the EU scored 20 drugs on 16 harm criteria. The expert group also assessed criteria weights that would apply, on average, across the EU. Weighted averages of the scores provided a single, overall weighted harm score (range: 0–100) for each drug. Results: Alcohol, heroin and crack emerged as the most harmful drugs (overall weighted harm score 72, 55 and 50, respectively). The remaining drugs had an overall weighted harm score of 38 or less, making them much less harmful than alcohol. The overall weighted harm scores of the EU experts correlated well with those previously given by the UK panel. Conclusion: The outcome of this study shows that the previous national rankings based on the relative harms of different drugs are endorsed throughout the EU. The results indicates that EU and national drug policy measures should focus on drugs with the highest overall harm, including alcohol and tobacco, whereas drugs such as cannabis and ecstasy should be given lower priority including a lower legal classification.