Gene Rowe

Public Participation Methods: A Framework for Evaluation

There is a growing call for greater public involvement in establishing science and technology policy, in line with democratic ideals. A variety of public participation procedures exist that aim to consult and involve the public, ranging from the public hearing to the consensus conference. Unfortunately, a general lack of empirical consideration of the quality of these methods arises from confusion as to the appropriate benchmarks for evaluation. Given that the quality of the output of any participation exercise is difficult to determine, the authors suggest the need to consider which aspects of the process are desirable and then to measure the presence or quality of these process aspects. To this end, a number of theoretical evaluation criteria that are essential for effective public participation are specified. These comprise two types: acceptance criteria, which concern features of a method that make it acceptable to the wider public, and process criteria, which concern features of the process that are liable to ensure that it takes place in an effective manner. Future research needs to develop instruments to measure these criteria more precisely and identify the contextual and environmental factors that will mediate the effectiveness of the different participation methods.

The Delphi technique as a forecasting tool: issues and analysis

Abstract This paper systematically reviews empirical studies looking at the effectiveness of the Delphi technique, and provides a critique of this research. Findings suggest that Delphi groups outperform statistical groups (by 12 studies to two with two ‘ties’) and standard interacting groups (by five studies to one with two ‘ties’), although there is no consistent evidence that the technique outperforms other structured group procedures. However, important differences exist between the typical laboratory version of the technique and the original concept of Delphi, which make generalisations about ‘Delphi’ per se difficult. These differences derive from a lack of control of important group, task, and technique characteristics (such as the relative level of panellist expertise and the nature of feedback used). Indeed, there are theoretical and empirical reasons to believe that a Delphi conducted according to ‘ideal’ specifications might perform better than the standard laboratory interpretations. It is concluded that a different focus of research is required to answer questions on Delphi effectiveness, focusing on an analysis of the process of judgment change within nominal groups.

A Typology of Public Engagement Mechanisms

Imprecise definition of key terms in the “public participation” domain have hindered the conduct of good research and militated against the development and implementation of effective participation practices. In this article, we define key concepts in the domain: public communication, public consultation, and public participation. These concepts are differentiated according to the nature and flow of information between exercise sponsors and participants. According to such an information flow perspective, an exercise’s effectiveness may be ascertained by the efficiency with which full, relevant information is elicited from all appropriate sources, transferred to (and processed by) all appropriate recipients, and combined(when required) to give an aggregate/consensual response. Key variables that may theoretically affect effectiveness—and on which engagement mechanisms differ—are identified and used to develop a typology of mechanisms. The resultant typology reveals four communication, six consultation, and four participation mechanism classes. Limitations to the typology are discussed, and future research needs identified.

Evaluating Public-Participation Exercises: A Research Agenda

The concept of public participation is one of growing interest in the UK and elsewhere, with a commensurate growth in mechanisms to enable this. The merits of participation, however, are difficult to ascertain, as there are relatively few cases in which the effectiveness of participation exercises have been studied in a structured (as opposed to highly subjective) manner. This seems to stem largely from uncertainty in the research community as to how to conduct evaluations. In this article, one agenda for conducting evaluation research that might lead to the systematic acquisition of knowledge is presented. This agenda identifies the importance of defining effectiveness and of operationalizing one’s definition (i.e., developing appropriate measurement instruments and processes). The article includes analysis of the nature of past evaluations, discussion of potential difficulties in the enactment of the proposed agenda, and discussion of some potential solutions.

Delphi: A reevaluation of research and theory

Abstract This paper examines critically the Delphi technique to determine whether it succeeds in alleviating the “process loss” typical of interacting groups. After briefly reviewing the technique, we consider problems with Delphi from two perspectives. First, we examine methodological and technical difficulties and the problems these have brought about in experimental applications. We suggest that important differences exist between the typical laboratory Delphi and the original concept of Delphi. These differences, reflecting a lack of control of important group characteristics/factors (such as the relative level of panelist expertise), make comparisons between Delphi studies unrealistic, as are generalizations from laboratory studies to the ideal of Delphi. This conclusion diminishes the power of those former Delphi critiques that have largely dismissed the procedure because of the variability of laboratory study results. Second, having noted the limited usefulness of the majority of studies for answering questions on the effectiveness of Delphi, we look at the technique from a theoritical/ mechanical perspective. That is, by drawing upon ideas/findings from other areas of research, we attempt to discern whether the structure of the Delphi procedure itself might reasonably be expected to function as intended. We conclude that inadequacies in the nature of feedback typically supplied in applications of Delphi tend to ensure that any small gains in the resolution of “process loss” are offset by the removal of any opportunity for group “process gain”. Some solutions to this dilemma are advocated; they are based on an analysis of the process of judgment change within groups and a consideration of factors that increase the validity of statistical/ nominal groups over their constituent individual components.

Expert Opinions in Forecasting: The Role of the Delphi Technique

Expert opinion is often necessary in forecasting tasks because of a lack of appropriate or available information for using statistical procedures. But how does one get the best forecast from experts? One solution is to use a structured group technique, such as Delphi, for eliciting and combining expert judgments. In using the Delphi technique, one controls the exchange of information between anonymous panelists over a number of rounds (iterations), taking the average of the estimates on the final round as the group judgment. A number of principles are developed here to indicate how to conduct structured groups to obtain good expert judgments. These principles, applied to the conduct of Delphi groups, indicate how many and what type of experts to use (five to 20 experts with disparate domain knowledge); how many rounds to use (generally two or three); what type of feedback to employ (average estimates plus justifications from each expert); how to summarize the final forecast (weight all experts’ estimates equally); how to word questions (in a balanced way with succinct definitions free of emotive terms and irrelevant information); and what response modes to use (frequencies rather than probabilities or odds, with coherence checks when feasible). Delphi groups are substantially more accurate than individual experts and traditional groups and somewhat more accurate than statistical groups (which are made up of noninteracting individuals whose judgments are aggregated). Studies support the advantage of Delphi groups over traditional groups by five to one with one tie, and their advantage over statistical groups by 12 to two with two ties. We anticipate that by following these principles, forecasters may be able to use structured groups to harness effectively expert opinion.

Evaluation of a Deliberative Conference

The concept of “public participation” is currently one of great interest to researchers and policy makers. In response to a perceived need for greater public involvement in decision making and policy formation processes on the part of both policymakers and the general public, a variety of novel mechanisms have been developed, such as the consensus conference and citizens jury, to complement traditional mechanisms, such as the public meeting. However, the relative effectiveness of the various mechanisms is unclear, as efforts at evaluation have been sparse. In this article, the authors describe an evaluation of a two-day “deliberative conference” on the topic of radiation dose assessment. The authors detail the evaluation framework that they adopt and describe the instruments that they have developed to determine the attainment (or otherwise) of the evaluationcriteria stipulated in that framework. They then describe the participation exercise that they have evaluated. Finally, they apply the instruments to assess the effectiveness of the exercise, and discuss the results and their implications for the conduct of evaluations and the use of this particular participation mechanism.

Differences in Expert and Lay Judgments of Risk: Myth or Reality?

This article evaluates the nine empirical studies that have been conducted on expert versus lay judgments of risk. Contrary to received wisdom, this study finds that there is little empirical evidence for the propositions (1) that experts judge risk differently from members of the public or (2) that experts are more veridical in their risk assessments. Methodological weaknesses in the early research are documented, and it is shown that the results of more recent studies are confounded by social and demographic factors that have been found to correlate with judgments of risk. Using a task-analysis taxonomy, a template is provided for the documentation of future studies of expert-lay differences/similarities that will facilitate analytic comparison.

Difficulties in evaluating public engagement initiatives: reflections on an evaluation of the UK GM Nation? public debate about transgenic crops

In the realm of risk management, and policy-making more generally, “public engagement” is often advocated as an antidote to pathologies associated with traditional methods of policy-making, and associated deficit-model-driven communication strategies. The actual benefits of public engagement are, however, difficult to establish without thorough evaluation of specific engagement processes. Unfortunately, rigorous evaluation is difficult, and, perhaps for this reason, it has rarely been undertaken. In this paper we highlight a number of these difficulties in the light of our experiences in evaluating a major engagement initiative, namely the GM Nation? publice debate on the possible commercialization of transgenic crops, which took place in Britain in 2003. The difficulties we identify seem likely to be relevant to many, if not most, engagement evaluations. They are concerned with both theoretical/normative (how one should evaluate) and practical (how one does evaluate) issues. We suggest a number of possible solutions to these evaluation difficulties.

Perceptions of food risk management among key stakeholders : Results from a cross-European study

In designing and implementing appropriate food risk management strategies, it is important to examine how key stakeholders perceive both the practice and effectiveness of food risk management. The objective of this study is to identify similarities and differences in perceptions of, and attitudes to, food risk management practices held by consumers and experts with an interest in food safety. Focus groups were conducted in five European countries chosen for their (hypothesised) cultural differences in attitudes towards risk: Denmark, Germany, Greece, Slovenia and the UK. Content analysis was carried out on the resulting texts and (sub) categories were identified within the analysis framework to facilitate the capture of emerging themes. Five key themes were identified as common to the perceptions of both consumers and experts, although these are not represented in the same way by both groups. These key themes are: (1) efforts made by the responsible authorities to manage food risks; (2) responsibility for prevention and management of food risks; (3) how priorities are established within regulatory systems; (4) scientific progress and its implications for food risk management; and (5) media attention and food safety incidents. Although some similarities emerged between the groups, differences were also identified. For example, experts appeared to be highly negative about media influences, whereas consumers appeared more indifferent about media influences and motives. These different perspectives need to be addressed in order to reduce the perceptual distance between key stakeholders, and in particular, to enhance consumer confidence in the food risk management system. Based on the study findings, recommendations for food risk management policies are outlined.