Alison O'Mara-Eves

Pinpointing needles in giant haystacks: use of text mining to reduce impractical screening workload in extremely large scoping reviews

In scoping reviews, boundaries of relevant evidence may be initially fuzzy, with refined conceptual understanding of interventions and their proposed mechanisms of action an intended output of the scoping process rather than its starting point. Electronic searches are therefore sensitive, often retrieving very large record sets that are impractical to screen in their entirety. This paper describes methods for applying and evaluating the use of text mining (TM) technologies to reduce impractical screening workload in reviews, using examples of two extremely large-scale scoping reviews of public health evidence (choice architecture (CA) and economic environment (EE)). Electronic searches retrieved >800,000 (CA) and >1 million (EE) records. TM technologies were used to prioritise records for manual screening. TM performance was measured prospectively. TM reduced manual screening workload by 90% (CA) and 88% (EE) compared with conventional screening (absolute reductions of ≈430 000 (CA) and ≈378 000 (EE) records). This study expands an emerging corpus of empirical evidence for the use of TM to expedite study selection in reviews. By reducing screening workload to manageable levels, TM made it possible to assemble and configure large, complex evidence bases that crossed research discipline boundaries. These methods are transferable to other scoping and systematic reviews incorporating conceptual development or explanatory dimensions.

Reducing systematic review workload through certainty-based screening

Graphical abstract

Is Consumer Response to Plain/Standardised Tobacco Packaging Consistent with Framework Convention on Tobacco Control Guidelines? A Systematic Review of Quantitative Studies

Background and Objectives Standardised or ‘plain’ tobacco packaging was introduced in Australia in December 2012 and is currently being considered in other countries. The primary objective of this systematic review was to locate, assess and synthesise published and grey literature relating to the potential impacts of standardised tobacco packaging as proposed by the guidelines for the international Framework Convention on Tobacco Control: reduced appeal, increased salience and effectiveness of health warnings, and more accurate perceptions of product strength and harm. Methods Electronic databases were searched and researchers in the field were contacted to identify studies. Eligible studies were published or unpublished primary research of any design, issued since 1980 and concerning tobacco packaging. Twenty-five quantitative studies reported relevant outcomes and met the inclusion criteria. A narrative synthesis was conducted. Results Studies that explored the impact of package design on appeal consistently found that standardised packaging reduced the appeal of cigarettes and smoking, and was associated with perceived lower quality, poorer taste and less desirable smoker identities. Although findings were mixed, standardised packs tended to increase the salience and effectiveness of health warnings in terms of recall, attention, believability and seriousness, with effects being mediated by the warning size, type and position on pack. Pack colour was found to influence perceptions of product harm and strength, with darker coloured standardised packs generally perceived as containing stronger tasting and more harmful cigarettes than fully branded packs; lighter coloured standardised packs suggested weaker and less harmful cigarettes. Findings were largely consistent, irrespective of location and sample. Conclusions The evidence strongly suggests that standardised packaging will reduce the appeal of packaging and of smoking in general; that it will go some way to reduce consumer misperceptions regarding product harm based upon package design; and will help make the legally required on-pack health warnings more salient.

Alcohol advertising and public health: systems perspectives versus narrow perspectives

Background Alcohol consumption is influenced by a complex causal system of interconnected psychological, behavioural, social, economic, legal and environmental factors. These factors are shaped by governments (eg, licensing laws and taxation), by consumers (eg, patterns of alcohol consumption drive demand) and by alcohol industry practices, such as advertising. The marketing and advertising of alcoholic products contributes to an ‘alcogenic environment’ and is a modifiable influence on alcohol consumption and harm. The public health perspective is that there is sufficient evidence that alcohol advertising influences consumption. The alcohol industry disputes this, asserting that advertising only aims to help consumers choose between brands. Methods We review the evidence from recent systematic reviews, including their theoretical and methodological assumptions, to help understand what conclusions can be drawn about the relationships between alcohol advertising, advertising restrictions and alcohol consumption. Conclusions A wide evidence base needs to be drawn on to provide a system-level overview of the relationship between alcohol advertising, advertising restrictions and consumption. Advertising aims to influence not just consumption, but also to influence awareness, attitudes and social norms; this is because advertising is a system-level intervention with multiple objectives. Given this, assessments of the effects of advertising restrictions which focus only on sales or consumption are insufficient and may be misleading. For this reason, previous systematic reviews, such as the 2014 Cochrane review on advertising restrictions (Siegfried et al) contribute important, but incomplete representations of ‘the evidence’ needed to inform the public health case for policy decisions on alcohol advertising. We conclude that an unintended consequence of narrow, linear framings of complex system-level issues is that they can produce misleading answers. Systems problems require systems perspectives.

Does Being Overweight Impede Academic Attainment? A Systematic Review.

Objectives: To examine evidence from studies exploring the relationship between childhood obesity and educational attainment. Design: A systematic review of secondary analyses and observational studies published in English after 1997 examining attainment as measured by grade point average or other validated measure, in children aged 6 to16 years, in high-income countries. Methods: Eleven databases from the fields of public health, education and social science were searched, along with 19 specialist registers and catalogues. Hand searching of relevant journals, contacting of experts and citation searching were undertaken. Two reviewers used standardized tools to independently carry out data extraction and assess the quality of included studies. Evidence was synthesized in a narrative summary. Results: Twenty-nine studies were identified for inclusion. Overall, the evidence suggested that higher weight is weakly associated with lower educational attainment among children and young people. Differences between average attainment of overweight and non-overweight children were marginal, with potentially negligible real-world implications for test scores. Limited evidence suggested that little variation in achievement was explained by weight status alone. Almost half the studies found that other factors, such as socioeconomic status, may better explain much of the negative association between obesity and attainment. Conclusion: Theoretical and methodological inconsistencies were evident both within and between many of the studies. As such, the results of the included studies must be interpreted with caution. If the negative association between obesity and attainment is accepted, it still remains doubtful whether obesity is exerting a socially important effect upon educational attainment.

How can additional secondary data analysis of observational data enhance the generalisability of meta-analytic evidence for local public health decision making?

This paper critically explores how survey and routinely collected data could aid in assessing the generalisability of public health evidence. We propose developing approaches that could be employed in understanding the relevance of public health evidence, and investigate ways of producing meta-analytic estimates tailored to reflect local circumstances, based on analyses of secondary data. Currently, public health decision makers face challenges in interpreting global review evidence to assess its meaning in local contexts. A lack of clarity on the definition and scope of generalisability, and the absence of consensus on its measurement, has stunted methodological progress. The consequence of failing to tackle generalisability means that systematic review evidence often fails to fulfil its potential contribution in public health decision making. Three approaches to address these problems are considered and emerging challenges discussed: (1) purposeful exploration after a review has been conducted, and we present a framework of potential avenues of enquiry and a worked example; (2) recalibration of the results to weight studies differentially based on their similarity to conditions in an inference population, and we provide a worked example using UK Census data to understand potential differences in the effectiveness of community engagement interventions among sites in England and Wales; (3) purposeful exploration before starting a review to ensure that the findings are relevant to an inference population. The paper aims to demonstrate how a more nuanced treatment of context in reviews of public health interventions could be achieved through greater engagement with existing large sources of secondary data.

Interpretive analysis of 85 systematic reviews suggests that narrative syntheses and meta-analyses are incommensurate in argumentation.

Using Toulmins argumentation theory, we analysed the texts of systematic reviews in the area of workplace health promotion to explore differences in the modes of reasoning embedded in reports of narrative synthesis as compared with reports of meta‐analysis. We used framework synthesis, grounded theory and cross‐case analysis methods to analyse 85 systematic reviews addressing intervention effectiveness in workplace health promotion. Two core categories, or ‘modes of reasoning’, emerged to frame the contrast between narrative synthesis and meta‐analysis: practical–configurational reasoning in narrative synthesis (‘what is going on here? What picture emerges?’) and inferential–predictive reasoning in meta‐analysis (‘does it work, and how well? Will it work again?’). Modes of reasoning examined quality and consistency of the included evidence differently. Meta‐analyses clearly distinguished between warrant and claim, whereas narrative syntheses often presented joint warrant–claims. Narrative syntheses and meta‐analyses represent different modes of reasoning. Systematic reviewers are likely to be addressing research questions in different ways with each method. It is important to consider narrative synthesis in its own right as a method and to develop specific quality criteria and understandings of how it is carried out, not merely as a complement to, or second‐best option for, meta‐analysis.

Using the realist perspective to link theory from qualitative evidence synthesis to quantitative studies: broadening the matrix approach.

BACKGROUND This study describes an approach for the use of a specific type of qualitative evidence synthesis in the matrix approach, a mixed studies reviewing method. The matrix approach compares quantitative and qualitative data on the review level by juxtaposing concrete recommendations from the qualitative evidence synthesis against interventions in primary quantitative studies. However, types of qualitative evidence syntheses that are associated with theory building generate theoretical models instead of recommendations. Therefore, the output from these types of qualitative evidence syntheses cannot directly be used for the matrix approach but requires transformation. This approach allows for the transformation of these types of output. METHOD The approach enables the inference of moderation effects instead of direct effects from the theoretical model developed in a qualitative evidence synthesis. Recommendations for practice are formulated on the basis of interactional relations inferred from the qualitative evidence synthesis. In doing so, we apply the realist perspective to model variables from the qualitative evidence synthesis according to the context-mechanism-outcome configuration. FINDINGS A worked example shows that it is possible to identify recommendations from a theory-building qualitative evidence synthesis using the realist perspective. We created subsets of the interventions from primary quantitative studies based on whether they matched the recommendations or not and compared the weighted mean effect sizes of the subsets. The comparison shows a slight difference in effect sizes between the groups of studies. The study concludes that the approach enhances the applicability of the matrix approach.

Text mining for search term development in systematic reviewing: A discussion of some methods and challenges

Using text mining to aid the development of database search strings for topics described by diverse terminology has potential benefits for systematic reviews; however, methods and tools for accomplishing this are poorly covered in the research methods literature. We briefly review the literature on applications of text mining for search term development for systematic reviewing. We found that the tools can be used in 5 overarching ways: improving the precision of searches; identifying search terms to improve search sensitivity; aiding the translation of search strategies across databases; searching and screening within an integrated system; and developing objectively derived search strategies. Using a case study and selected examples, we then reflect on the utility of certain technologies (term frequency-inverse document frequency and Termine, term frequency, and clustering) in improving the precision and sensitivity of searches. Challenges in using these tools are discussed. The utility of these tools is influenced by the different capabilities of the tools, the way the tools are used, and the text that is analysed. Increased awareness of how the tools perform facilitates the further development of methods for their use in systematic reviews.

Community engagement in public health interventions to reduce health inequalities: mapping the evidence against policy objectives

Abstract Background The presence of health inequalities is one of the largest issues currently facing public health policy making in developed countries. Interventions that engage the community in their design or delivery are thought more likely to address the specific needs of disadvantaged populations, and community engagement has consequently become central to guidance and strategies for promoting public health. We aimed to map the evidence on community engagement approaches to address health inequalities, with a focus on the mechanisms of engagement used, and to consider the coverage of current evidence to inform policy making on health inequalities. Methods We did systematic searches for systematic reviews to identify relevant controlled trials assessing public health interventions with members of the community searching these databases using web interfaces: DoPHER, Campbell, Cochrane Reviews library, DARE, HTA database, and Web of Knowledge HTA search. Systematic reviews were used as a source because of the systematic way in which they describe included studies. The search was limited to studies that address health inequalities, either through targeting a disadvantaged population or through subgroup analyses, and that were done in developed (OECD) countries. We extracted data from the studies (using independent quality assurance procedures) and judged their quality using the Cochrane Risk of Bias method. Data were obtained on the mechanisms through which the community were involved, the health issue targeted by the intervention, and characteristics of the sample or population and intervention. Data were held in EPPI-Reviewer 4 and all reports of relevant studies were included, making no distinction between published or unpublished sources. We mapped the aims of the interventions in our review with the six policy objectives proposed in the UK Marmot review (2010) to establish the extent to which there was evidence to inform policy in this area. For example, we looked at how many studies addressed the objective of getting the best start in life through a focus on issues such as breastfeeding and antenatal care. Findings We identified 361 reports of 319 studies meeting our criteria. Most contained some form of health education about topics, including sexual health (60) or healthy eating (31). Almost half the studies included the community in the design or planning of the intervention through consultation or collaboration. Almost three-quarters were led by, or delivered in collaboration with, community members; only a quarter encompassed the community in the assessment of the intervention. In terms of overlap with UK health inequalities policy objectives, 219 studies addressed a social determinant of health or modifiable risk factor prioritised in the Marmot review (eg, 64 studies assessed ill health prevention); some modifiable risk factors were in fewer studies (eg, 14 studies assessed sustainable and safe places and communities). Interpretation Despite some gaps, public health science provides a wealth of evidence about community engagement to address health inequalities, which could be used to inform policy making. Outputs from this project include a new conceptual framework and analysis of the effectiveness and cost-effectiveness of such approaches. Funding This Abstract refers to independent research commissioned by the National Institute for Health Research.