Steve Hanney

How Can Payback from Health Services Research Be Assessed

Throughout the world there is a growing recognition that health care should be research-led. This strengthens the requirement for expenditure on health services research to be justified by demonstrating the benefits it produces. However, payback from health research and development is a complex concept and little used term. Five main categories of payback can be identified: Knowledge; research benefits; political and administrative benefits; health sector benefits; and broader economic benefits. Various models of research utilization together with previous assessments of payback from research helped in the development of a new conceptual model of how and where payback may occur. The model combines an input-output perspective with an examination of the permeable interfaces between research and its environment. The model characterizes research projects in terms of Inputs, Processes, and Primary Outputs. The last consist of knowledge and research benefits. There are two interfaces between the project and its environment. The first (Project Specification, Selection and Commissioning) is the link with Research Needs Assessment. The second (Dissemination) should lead to Secondary Outputs (which are policy or administrative decisions), and usually Applications (which take the form of behavioural changes), from which Impacts or Final Outcomes result. It is at this final stage that health and wider economic benefits can be measured. A series of case studies were used to assess the feasibility both of applying the model and the payback categorization. The paper draws various conclusions from the case studies and identifies a range of issues for further work.

Knowledge for better health: a conceptual framework and foundation for health research systems

Health research generates knowledge that can be utilized to improve health system performance and, ultimately, health and health equity. We propose a conceptual framework for health research systems (HRSs) that defines their boundaries, components, goals, and functions. The framework adopts a systems perspective towards HRSs and serves as a foundation for constructing a practical approach to describe and analyse HRSs. The analysis of HRSs should, in turn, provide a better understanding of how research contributes to gains in health and health equity. In this framework, the intrinsic goals of the HRS are the advancement of scientific knowledge and the utilization of knowledge to improve health and health equity. Its four principal functions are stewardship, financing, creating and sustaining resources, and producing and using research. The framework, as it is applied in consultation with countries, will provide countries and donor agencies with relevant inputs to policies and strategies for strengthening HRSs and using knowledge for better health.

The information sources and journals consulted or read by UK paediatricians to inform their clinical practice and those which they consider important: a questionnaire survey

BackgroundImplementation of health research findings is important for medicine to be evidence-based. Previous studies have found variation in the information sources thought to be of greatest importance to clinicians but publication in peer-reviewed journals is the traditional route for dissemination of research findings. There is debate about whether the impact made on clinicians should be considered as part of the evaluation of research outputs. We aimed to determine first which information sources are generally most consulted by paediatricians to inform their clinical practice, and which sources they considered most important, and second, how many and which peer-reviewed journals they read.MethodsWe enquired, by questionnaire survey, about the information sources and academic journals that UK medical paediatric specialists generally consulted, attended or read and considered important to their clinical practice.ResultsThe same three information sources – professional meetings & conferences, peer-reviewed journals and medical colleagues – were, overall, the most consulted or attended and ranked the most important. No one information source was found to be of greatest importance to all groups of paediatricians. Journals were widely read by all groups, but the proportion ranking them first in importance as an information source ranged from 10% to 46%. The number of journals read varied between the groups, but Archives of Disease in Childhood and BMJ were the most read journals in all groups. Six out of the seven journals previously identified as containing best paediatric evidence are the most widely read overall by UK paediatricians, however, only the two most prominent are widely read by those based in the community.ConclusionNo one information source is dominant, therefore a variety of approaches to Continuing Professional Development and the dissemination of research findings to paediatricians should be used. Journals are an important information source. A small number of key ones can be identified and such analysis could provide valuable additional input into the evaluation of clinical research outputs.

Evaluating the Benefits from Health Research and Development Centres A Categorization, a Model and Examples of Application

The most appropriate criteria and the best approach for evaluating centres concerned with research into health and health services, are matters of academic and policy debate. There is interest in assessing the impact made by research on policy and practice as well as using more traditional peer review of the knowledge produced. In this context the article describes the development of a multidimensional categorization of benefits, or payback, from research and development (R&D) and a model for conducting evaluations of impact. This categorization and model were used during an assessment of two R&D centres funded by a regional office of the National Health Service in the UK. The acceptability of such an approach is discussed in the light of relevant proposals from other authors, changes in the nature of knowledge production and the consequent role of multiple stakeholders.

Research impact: A narrative review

Impact occurs when research generates benefits (health, economic, cultural) in addition to building the academic knowledge base. Its mechanisms are complex and reflect the multiple ways in which knowledge is generated and utilised. Much progress has been made in measuring both the outcomes of research and the processes and activities through which these are achieved, though the measurement of impact is not without its critics. We review the strengths and limitations of six established approaches (Payback, Research Impact Framework, Canadian Academy of Health Sciences, monetisation, societal impact assessment, UK Research Excellence Framework) plus recently developed and largely untested ones (including metrics and electronic databases). We conclude that (1) different approaches to impact assessment are appropriate in different circumstances; (2) the most robust and sophisticated approaches are labour-intensive and not always feasible or affordable; (3) whilst most metrics tend to capture direct and proximate impacts, more indirect and diffuse elements of the research-impact link can and should be measured; and (4) research on research impact is a rapidly developing field with new methodologies on the horizon.

Assessing Benefits from Department of Health and National Health Service Research & Development

Public services in the UK are increasingly expected to account for their outputs and performance. This article describes a retrospective evaluation of the benefits from the R&D funded by a regional office of the NHS Executive. The methods adopted enabled the various elements of the regions R&D portfolio to be examined and provided a basis for the development of a plan for regular monitoring. The proposals for implementation are now also feeding into a debate as to how health services R&D might best be monitored nationally in a way that is compatible with the norms and practices of research in other contexts.

Using Categorisations of Citations When Assessing the Outcomes from Health Research

SummaryThis paper describes an attempt to explore how far a categorisation of citations could be used as part of an assessment of the outcomes from health research. A large-scale project to assess the outcomes from basic, or early clinical, research is being planned, but before proceeding with such a project it was thought important to test and refine the developing methods in a preliminary study. Here we describe the development, and initial application, of one element of the planned methods: an approach to categorising citations with the aim of tracing the impact made by a body of research through several generations of papers. The results from this study contribute to methodological development for the large-scale project by indicating that: only for a small minority of citing papers is the cited paper of considerable importance; the number of times a paper is cited can not be used to indicate the importance of that paper to the articles that cite it; and self-citations could play an important role in facilitating the eventual outcomes achieved from a body of research.

Assessing benefits from health research projects: can we use questionnaires instead of case studies?

An evaluative framework was developed by the Health Economics Research Group (HERG) for the UKs Department of Health (DH) to assess the benefits from DH-funded RD they could be strengthened in various ways in response to weaknesses exposed in the case studies. Copyright , Beech Tree Publishing.

Desarrollo y aplicación del Modelo Payback para la evaluación del impacto socioeconómico de la investigación en salud

Existe una presion creciente para que los impactos socioeconomicos de la investigacion en salud sean evaluados. La demanda por parte de los gobiernos para justificar el gasto del dinero publico es cada vez mayor. Ademas existe la creencia de que evaluando la forma en que se manifiestan los impactos o beneficios de la investigacion sanitaria se ayuda a mantener informados a los responsables de dicha investigacion de forma que aumenten los impactos futuros. Desde mediados de la decada de los noventa, Buxton y Hanney, en el Health Economics Research Group, de la Universidad de Brunel, han estado desarrollando y aplicando el Modelo Payback para valorar los impactos de la investigacion sanitaria. En colaboracion con otros colegas, lo han aplicado en una serie de estudios para evaluar la rentabilidad de los programas de investigacion en diversos campos (diabetes mellitus, artritis y enfermedad cardiovascular) y en varios paises (como Reino Unido, Paises Bajos, Irlanda, Australia y Canada). Otros equipos de investigadores han aplicado el Modelo Payback en, por ejemplo, Espana y Hong Kong. El Modelo Payback esta formado por dos elementos; el primero consiste en la categorizacion multidimensional de los beneficios de la investigacion sanitaria. Esta cubre cinco categorias principales, que van desde la tradicional produccion de conocimiento y la formacion y la focalizacion de la investigacion hasta los impactos en la politica y el desarrollo de productos, pasando por los beneficios sanitarios y economicos. El segundo elemento es un modelo logico que permita identificar el mejor modo de evaluar estos impactos. Si bien el Modelo Payback puede requerir muchos recursos para ser aplicado, su uso ha proporcionado casos muy ilustrativos acerca de la rentabilidad que se genera por la investigacion y ha ayudado a mantener informados a los responsables de la investigacion.

Evaluating the NHS research and development programme: Will the programme give value for money?

Whilst a strong argument can be made that, from an academic perspective, scientific research is of intrinsic value, from the perspective of the National Health Service (NHS), research and development (R&D) is a means to an end and not an end in itself. This means that the current amount spent on R&D (0428 million, or 1.3% of the total NHS budget) must be viewed as a decision to forgo that level of spending on the direct provision of health care in the belief that, if invested in R&D, this expenditure will offer a greater future return. In this context, it is clearly important that a systematic attempt is made to assess whether the NHS R&D programme provides value for money, and yields a return to the NHS. This paper first explains the importance and difficulty of assessing value for money, before outlining some of the alternative techniques that have been proposed. It then describes the non-additive multidimensional approach that we have developed for the Department of Health (DoH)24. It provides an account of some of the lessons learnt from this work, and indicates what still needs to be done to establish whether the R&D programme represents value for money.