Alexandra Pollitt

A global call for action to include gender in research impact assessment

Global investment in biomedical research has grown significantly over the last decades, reaching approximately a quarter of a trillion US dollars in 2010. However, not all of this investment is distributed evenly by gender. It follows, arguably, that scarce research resources may not be optimally invested (by either not supporting the best science or by failing to investigate topics that benefit women and men equitably). Women across the world tend to be significantly underrepresented in research both as researchers and research participants, receive less research funding, and appear less frequently than men as authors on research publications. There is also some evidence that women are relatively disadvantaged as the beneficiaries of research, in terms of its health, societal and economic impacts. Historical gender biases may have created a path dependency that means that the research system and the impacts of research are biased towards male researchers and male beneficiaries, making it inherently difficult (though not impossible) to eliminate gender bias. In this commentary, we – a group of scholars and practitioners from Africa, America, Asia and Europe – argue that gender-sensitive research impact assessment could become a force for good in moving science policy and practice towards gender equity. Research impact assessment is the multidisciplinary field of scientific inquiry that examines the research process to maximise scientific, societal and economic returns on investment in research. It encompasses many theoretical and methodological approaches that can be used to investigate gender bias and recommend actions for change to maximise research impact. We offer a set of recommendations to research funders, research institutions and research evaluators who conduct impact assessment on how to include and strengthen analysis of gender equity in research impact assessment and issue a global call for action.

The impact of the National Institute for Health Research Health Technology Assessment programme, 2003–13: a multimethod evaluation

BACKGROUNDnThe National Institute for Health Research (NIHR) Health Technology Assessment (HTA) programme supports research tailored to the needs of NHS decision-makers, patients and clinicians. This study reviewed the impact of the programme, from 2003 to 2013, on health, clinical practice, health policy, the economy and academia. It also considered how HTA could maintain and increase its impact.nnnMETHODSnInterviews (nu2009=u200920): senior stakeholders from academia, policy-making organisations and the HTA programme. Bibliometric analysis: citation analysis of publications arising from HTA programme-funded research. Researchfish survey: electronic survey of all HTA grant holders. Payback case studies (nu2009=u200912): in-depth case studies of HTA programme-funded research.nnnRESULTSnWe make the following observations about the impact, and routes to impact, of the HTA programme: it has had an impact on patients, primarily through changes in guidelines, but also directly (e.g. changing clinical practice); it has had an impact on UK health policy, through providing high-quality scientific evidence - its close relationships with the National Institute for Health and Care Excellence (NICE) and the National Screening Committee (NSC) contributed to the observed impact on health policy, although in some instances other organisations may better facilitate impact; HTA research is used outside the UK by other HTA organisations and systematic reviewers - the programme has an impact on HTA practice internationally as a leader in HTA research methods and the funding of HTA research; the work of the programme is of high academic quality - the Health Technology Assessment journal ensures that the vast majority of HTA programme-funded research is published in full, while the HTA programme still encourages publication in other peer-reviewed journals; academics agree that the programme has played an important role in building and retaining HTA research capacity in the UK; the HTA programme has played a role in increasing the focus on effectiveness and cost-effectiveness in medicine - it has also contributed to increasingly positive attitudes towards HTA research both within the research community and the NHS; and the HTA focuses resources on research that is of value to patients and the UK NHS, which would not otherwise be funded (e.g. where there is no commercial incentive to undertake research). The programme should consider the following to maintain and increase its impact: providing targeted support for dissemination, focusing resources when important results are unlikely to be implemented by other stakeholders, particularly when findings challenge vested interests; maintaining close relationships with NICE and the NSC, but also considering other potential users of HTA research; maintaining flexibility and good relationships with researchers, giving particular consideration to the Technology Assessment Report (TAR) programme and the potential for learning between TAR centres; maintaining the academic quality of the work and the focus on NHS need; considering funding research on the short-term costs of the implementation of new health technologies; improving the monitoring and evaluation of whether or not patient and public involvement influences research; improve the transparency of the priority-setting process; and continuing to monitor the impact and value of the programme to inform its future scientific and administrative development.

Understanding the relative valuation of research impact: a best–worst scaling experiment of the general public and biomedical and health researchers

Objectives (1) To test the use of best–worst scaling (BWS) experiments in valuing different types of biomedical and health research impact, and (2) to explore how different types of research impact are valued by different stakeholder groups. Design Survey-based BWS experiment and discrete choice modelling. Setting The UK. Participants Current and recent UK Medical Research Council grant holders and a representative sample of the general public recruited from an online panel. Results In relation to the studys 2 objectives: (1) we demonstrate the application of BWS methodology in the quantitative assessment and valuation of research impact. (2) The general public and researchers provided similar valuations for research impacts such as improved life expectancy, job creation and reduced health costs, but there was less agreement between the groups on other impacts, including commercial capacity development, training and dissemination. Conclusions This is the second time that a discrete choice experiment has been used to assess how the general public and researchers value different types of research impact, and the first time that BWS has been used to elicit these choices. While the 2 groups value different research impacts in different ways, we note that where they agree, this is generally about matters that are seemingly more important and associated with wider social benefit, rather than impacts occurring within the research system. These findings are a first step in exploring how the beneficiaries and producers of research value different kinds of impact, an important consideration given the growing emphasis on funding and assessing research on the basis of (potential) impact. Future research should refine and replicate both the current study and that of Miller et al in other countries and disciplines.

Estimating the Returns to United Kingdom Publicly Funded Musculoskeletal Disease Research in Terms of Net Value of Improved Health Outcomes

BackgroundBuilding on an approach applied to cardiovascular and cancer research, we estimated the economic returns from United Kingdom public- and charitable-funded musculoskeletal disease (MSD) research that arise from the net value of the improved health outcomes in the United Kingdom.MethodsTo calculate the economic returns from MSD-related research in the United Kingdom, we estimated (1) the public and charitable expenditure on MSD-related research in the United Kingdom between 1970 and 2013; (2) the net monetary benefit (NMB), derived from the health benefit in quality adjusted life years (QALYs) valued in monetary terms (using a base-case value of a QALY of £25,000) minus the cost of delivering that benefit, for a prioritised list of interventions from 1994 to 2013; (3) the proportion of NMB attributable to United Kingdom research; and (4) the elapsed time between research funding and health gain. The data collected from these four key elements were used to estimate the internal rate of return (IRR) from MSD-related research investments on health benefits. We analysed the uncertainties in the IRR estimate using a one-way sensitivity analysis.ResultsExpressed in 2013 prices, total expenditure on MSD-related research from 1970 to 2013 was £3.5 billion, and for the period used to estimate the rate of return, 1978-1997, was £1.4 billion. Over the period 1994–2013 the key interventions analysed produced 871,000 QALYs with a NMB of £16 billion, allowing for the net NHS costs resulting from them and valuing a QALY at £25,000. The proportion of benefit attributable to United Kingdom research was 30% and the elapsed time between funding and impact of MSD treatments was 16xa0years. Our best estimate of the IRR from MSD-related research was 7%, which is similar to the 9% for CVD and 10% for cancer research.ConclusionsOur estimate of the IRR from the net health gain to public and charitable funding of MSD-related research in the United Kingdom is substantial, and justifies the research investments made between 1978 and 1997. We also demonstrated the applicability of the approach previously used in assessing the returns from cardiovascular and cancer research. Inevitably, with a study of this kind, there are a number of important assumptions and caveats that we highlight, and these can inform future research.

ISRIA statement: ten-point guidelines for an effective process of research impact assessment

As governments, funding agencies and research organisations worldwide seek to maximise both the financial and non-financial returns on investment in research, the way the research process is organised and funded is becoming increasingly under scrutiny. There are growing demands and aspirations to measure research impact (beyond academic publications), to understand how science works, and to optimise its societal and economic impact. In response, a multidisciplinary practice called research impact assessment is rapidly developing. Given that the practice is still in its formative stage, systematised recommendations or accepted standards for practitioners (such as funders and those responsible for managing research projects) across countries or disciplines to guide research impact assessment are not yet available.In this statement, we propose initial guidelines for a rigorous and effective process of research impact assessment applicable to all research disciplines and oriented towards practice. This statement systematises expert knowledge and practitioner experience from designing and delivering the International School on Research Impact Assessment (ISRIA). It brings together insights from over 450 experts and practitioners from 34 countries, who participated in the school during its 5-year run (from 2013 to 2017) and shares a set of core values from the school’s learning programme. These insights are distilled into ten-point guidelines, which relate to (1) context, (2) purpose, (3) stakeholders’ needs, (4) stakeholder engagement, (5) conceptual frameworks, (6) methods and data sources, (7) indicators and metrics, (8) ethics and conflicts of interest, (9) communication, and (10) community of practice.The guidelines can help practitioners improve and standardise the process of research impact assessment, but they are by no means exhaustive and require evaluation and continuous improvement. The prima facie effectiveness of the guidelines is based on the systematised expert and practitioner knowledge of the school’s faculty and participants derived from their practical experience and research evidence. The current knowledge base has gaps in terms of the geographical and scientific discipline as well as stakeholder coverage and representation. The guidelines can be further strengthened through evaluation and continuous improvement by the global research impact assessment community.

Does a biomedical research centre affect patient care in local hospitals

BackgroundBiomedical research can have impacts on patient care at research-active hospitals. We qualitatively evaluated the impact of the Oxford Biomedical Research Centre (Oxford BRC), a university-hospital partnership, on the effectiveness and efficiency of healthcare in local hospitals. Effectiveness and efficiency are conceptualised in terms of impacts perceived by clinicians on the quality, quantity and costs of patient care they deliver.MethodsFirst, we reviewed documentation from Oxford BRC and literature on the impact of research activity on patient care. Second, we interviewed leaders of the Oxford BRC’s research to identify the direct and indirect impacts they expected their activity would have on local hospitals. Third, this information was used to inform interviews with senior clinicians responsible for patient care at Oxford’s acute hospitals to discover what impacts they observed from research generally and from Oxford BRC’s research work specifically. We compared and contrasted the results from the two sets of interviews using a qualitative approach. Finally, we identified themes emerging from the senior clinicians’ responses, and compared them with an existing taxonomy of mechanisms through which quality of healthcare may be affected in research-active settings.ResultsWe were able to interview 17 research leaders at the Oxford BRC and 19 senior clinicians at Oxford’s acute hospitals. The research leaders identified a wide range of beneficial impacts that they expected might be felt at local hospitals as a result of their research activity. They expected the impact of their research activity on patient care to be generally positive. The senior clinicians responsible for patient care at those hospitals presented a more mixed picture, identifying many positive impacts, but also a smaller number of negative impacts, from research activity, including that of the Oxford BRC. We found the existing taxonomy of benefit types to be helpful in organising the findings, and propose modifications to further improve its usefulness.ConclusionsImpacts from research activity on the effectiveness and efficiency of patient care at the local acute hospitals, as perceived by senior clinicians, were more often beneficial than harmful. The Oxford BRC contributed to those impacts.