Christian Kohl

Selecting appropriate methods of knowledge synthesis to inform biodiversity policy

Responding to different questions generated by biodiversity and ecosystem services policy or management requires different forms of knowledge (e.g. scientific, experiential) and knowledge synthesis. Additionally, synthesis methods need to be appropriate to policy context (e.g. question types, budget, timeframe, output type, required scientific rigour). In this paper we present a range of different methods that could potentially be used to conduct a knowledge synthesis in response to questions arising from knowledge needs of decision makers on biodiversity and ecosystem services policy and management. Through a series of workshops attended by natural and social scientists and decision makers we compiled a range of question types, different policy contexts and potential methodological approaches to knowledge synthesis. Methods are derived from both natural and social sciences fields and reflect the range of question and study types that may be relevant for syntheses. Knowledge can be available either in qualitative or quantitative form and in some cases also mixed. All methods have their strengths and weaknesses and we discuss a sample of these to illustrate the need for diversity and importance of appropriate selection. To summarize this collection, we present a table that identifies potential methods matched to different combinations of question types and policy contexts, aimed at assisting teams undertaking knowledge syntheses to select appropriate methods.

A framework for stakeholder engagement during systematic reviews and maps in environmental management

People have a stake in conservation and environmental management both for their own interests and the sake of the environment itself. Environmental decision-making has changed somewhat in recent decades to account for unintentional impacts on human wellbeing. The involvement of stakeholders in environmental projects has been recognised as critical for ensuring their success and equally for the syntheses of evidence of what works, where, and for whom, providing key benefits and challenges. As a result of increased interest in systematic reviews of complex management issues, there is a need for guidance in best practices for stakeholder engagement. Here, we propose a framework for stakeholder engagement in systematic reviews/systematic maps, highlighting recommendations and advice that are critical for effective, efficient and meaningful engagement of stakeholders. The discussion herein aims to provide a toolbox of stakeholder engagement activities, whilst also recommending approaches from stakeholder engagement research that may prove to be particularly useful for systematic reviews and systematic maps.

Enhancing the interpretation of statistical P values in toxicology studies: implementation of linear mixed models (LMMs) and standardized effect sizes (SESs)

Abstract In this paper, we compare the traditional ANOVA approach to analysing data from 90-day toxicity studies with a more modern LMM approach, and we investigate the use of standardized effect sizes. The LMM approach is used to analyse weight or feed consumption data. When compared to the week-by-week ANOVA with multiple test results per week, this approach results in only one statement on differences in weight development between groups. Standardized effect sizes are calculated for the endpoints: weight, relative organ weights, haematology and clinical biochemistry. The endpoints are standardized, allowing different endpoints of the same study to be compared and providing an overall picture of group differences at a glance. Furthermore, in terms of standardized effect sizes, statistical significance and biological relevance are displayed simultaneously in a graph.

Can systematic reviews inform GMo risk assessment and risk management

Systematic reviews represent powerful tools to identify, collect, synthesize, and evaluate primary research data on specific research questions in a highly standardized and reproducible manner. They enable the defensible synthesis of outcomes by increasing precision and minimizing bias whilst ensuring transparency of the methods used. This makes them especially valuable to inform evidence-based risk analysis and decision making in various topics and research disciplines. Although seen as a “gold standard” for synthesizing primary research data, systematic reviews are not without limitations as they are often cost, labor and time intensive and the utility of synthesis outcomes depends upon the availability of sufficient and robust primary research data. In this paper, we (1) consider the added value systematic reviews could provide when synthesizing primary research data on genetically modified organisms (GMO) and (2) critically assess the adequacy and feasibility of systematic review for collating and analyzing data on potential impacts of GMOs in order to better inform specific steps within GMO risk assessment and risk management. The regulatory framework of the EU is used as an example, although the issues we discuss are likely to be more widely applicable.

Variability of control data and relevance of observed group differences in five oral toxicity studies with genetically modified maize MON810 in rats.

The data of four 90-day feeding trials and a 1-year feeding trial with the genetically modified (GM) maize MON810 in Wistar Han RCC rats performed in the frame of EU–funded project GRACE were analysed. Firstly, the data obtained from the groups having been fed the non–GM maize diets were combined to establish a historical control data set for Wistar Han RCC rats at the animal housing facility (Slovak Medical University, Bratislava, Slovakia). The variability of all parameters is described, and the reference values and ranges have been derived. Secondly, the consistency of statistically significant differences found in the five studies was analysed. In order to do so, the body weight development, organ weight, haematology and clinical biochemistry data were compared between the studies. Based on the historical control data, equivalence ranges for these parameters were defined, and the values measured in the GM maize–fed groups were compared with these equivalence ranges. Thirdly, the (statistical) power of these feeding studies with whole food/feed was assessed and detectable toxicologically relevant group differences were derived. Linear mixed models (LMM) were applied, and standardized effect sizes (SES) were calculated in order to compare different parameters as well as to provide an overall picture of group and study differences at a glance. The comparison of the five feeding trials showed a clear study effect in the control data. It also showed inconsistency both in the frequency of statistically significant differences and in the difference values between control and test groups.

Inclusive development and prioritization of review questions in a highly controversial field of regulatory science

How to best assess potential health, environmental and other impacts of genetically modified organisms (GMOs) and how to interpret the resulting evidence base have been long-standing controversial issues in the EU. As a response, transparency and inclusiveness became a major focus of regulatory science activities in the GMO impact area. Nevertheless, nearly three decades of controversies resulted in a heavily polarized policy environment, calling for further efforts. Against this backdrop the EU funded project GRACE explored the value of evidence synthesis approaches for GMO impact assessment and developed an evidence synthesis framework with a strong emphasis on openness, stakeholder engagement, transparency, and responsiveness to tackle regulatory science challenges. This framework was tested and implemented in the course of 14 systematic reviews or maps conducted on selected review questions spanning potential health, environmental, and socioeconomic impacts of GMOs. An inclusive development and prioritisation of review questions is of key importance in evidence synthesis as it helps to provide a better link between stakeholder demands and concerns and policy relevant outcomes. This paper, therefore, places a particular focus on the stakeholder involvement strategy developed and experiences gathered during this particular step in the course of the GRACE project. Based on this experience, possible lessons for future engagement exercises in highly controversial fields of regulatory science are discussed.

Including non-public data and studies in systematic reviews and systematic maps.

Systematic reviews and maps should be based on the best available evidence, and reviewers should make all reasonable efforts to source and include potentially relevant studies. However, reviewers may not be able to consider all existing evidence, since some data and studies may not be publicly available. Including non-public studies in reviews provides a valuable opportunity to increase systematic review/map comprehensiveness, potentially mitigating negative impacts of publication bias. Studies may be non-public for many reasons: some may still be in the process of being published (publication can take a long time); some may not be published due to author/publisher restrictions; publication bias may make it difficult to publish non-significant or negative results. Here, we consider what forms these non-public studies may take and the implications of including them in systematic reviews and maps. Reviewers should carefully consider the advantages and disadvantages of including non-public studies, weighing risks of bias against benefits of increased comprehensiveness. As with all systematic reviews and maps, reviewers must be transparent about methods used to obtain data and avoid risks of bias in their synthesis. We make tentative suggestions for reviewers in situations where non-public data may be present in an evidence base.

Does the growing of Bt maize change abundance or ecological function of non-target animals compared to the growing of non-GM maize? A systematic review protocol

BackgroundSince 1996, genetically modified (GM) crops have been grown on an ever increasing area worldwide. Maize producing a Cry protein from the bacterium Bacillus thuringiensis (Bt) was among the first GM crops released for commercial production and it is the only GM crop currently cultivated in Europe. A major part of the regulatory process that precedes the commercial release of GM crops is the environmental risk assessment. Because Bt maize is modified to produce insecticidal proteins, potential interactions with non-target organisms are a major area of concern to be addressed in the risk assessment. In particular, beneficial arthropods that provide important agro-ecological services, such as pollination, decomposition, and biological control are the focus. This systematic review will evaluate if the growing of Bt maize changes abundance or ecological function of non-target animals compared to the growing of conventional, non-GM maize. The review will be limited to plot or field level data including field margins. Potential cropping system effects and off-field effects will not be addressed. Bt maize will be compared to conventional maize either untreated or treated with chemical insecticides.MethodsStakeholders from academia, competent authorities, industry, and civil society organizations were given the opportunity to comment on the review question and an earlier draft of this review protocol.Keyword searches will be conducted in a range of abstracting and full text literature databases. Retrieved records will be screened against a set of inclusion criteria, first on title and abstract level, then on full text level. Selected studies will be evaluated for risks of bias (quality assessment). Data on field/plot characteristics, maize cultivars, insecticide treatments, non-target animal taxa, sampling methods, and response variables of populations and ecological functions will be extracted. Meta-analysis will be conducted using the effect size estimator Hedge’s d on a range of comparisons and including sensitivity analysis. The review process will be fully documented in CADIMA, an open access online data portal for evidence synthesis.

Proposed criteria for the evaluation of the scientific quality of mandatory rat and mouse feeding trials with whole food/feed derived from genetically modified plants

In recent years, animal feeding trials conducted with whole food/feed have been a focal issue in the controversy on the safety assessment of genetically modified (GM) plants and derived food/feed. Within the scientific community and among stakeholders, quite different views have been expressed on how these studies should be conducted, analysed and interpreted, what they might add in terms of information relevant to safety and whether 90-day rodent feeding trials should be mandatory. Despite the fact that the Commission Implementing Regulation (EU) No. 503/2013 (specifying the requirements for the risk assessment of GM food/feed) requests mandatory 90-day feeding trials for GM plants with single transformation events, the controversy continues. This is due to the fact that in 2016 the European Commission will have to review this particular provision in the legislation (ibid, Article 12), and because of questions raised by long-term feeding studies with GM maize.

What is the available evidence for the application of genome editing as a new tool for plant trait modification and the potential occurrence of associated off-target effects: a systematic map protocol

BackgroundPlant breeding is a developing process and breeding methods have continuously evolved over time. In recent years, genome editing techniques such as clustered regularly interspaced short palindromic repeats/CRISPR associated proteins (CRISPR/Cas), transcription activator-like effector nucleases (TALENs), zinc-finger nucleases (ZFN), meganucleases (MN) and oligonucleotide-directed mutagenesis (ODM) enabled a precise modification of DNA sequences in plants. Genome editing has already been applied in a wide range of plant species due to its simplicity, time saving and cost-effective application compared to earlier breeding techniques including classical mutagenesis. Although genome editing techniques induce much less unintended modifications in the genome (off-target effects) compared to classical mutagenesis techniques, off-target effects are a prominent point of criticism as they might cause genomic instability, cytotoxicity and cell death.MethodsThe aim of this systematic map is to address the following primary question: “What is the available evidence for the application of genome editing as a new tool for plant trait modification and the potential occurrence of associated off-target effects”? The primary question will be considered by two secondary questions: One is aimed at the traits being modified by genome editing in plants and the other explores the occurrence of off-target effects. The systematic map will focus on model plants as well as on plants produced for agricultural production that were subjected to genome editing techniques. Academic and grey literature will be searched in English and German language. Inclusion/exclusion criteria were developed for the two secondary questions and will be applied on title/abstract and full text stage. Included studies will be catalogued in a searchable and open access database and study results will be summarized using descriptive statistics. Furthermore, the extracted data will serve as a preparatory step for further in-depth analysis, e.g. by a systematic review.