Vathsala I. Stone

Modeling technology innovation: How science, engineering, and industry methods can combine to generate beneficial socioeconomic impacts

BackgroundGovernment-sponsored science, technology, and innovation (STI) programs support the socioeconomic aspects of public policies, in addition to expanding the knowledge base. For example, beneficial healthcare services and devices are expected to result from investments in research and development (R&D) programs, which assume a causal link to commercial innovation. Such programs are increasingly held accountable for evidence of impact—that is, innovative goods and services resulting from R&D activity. However, the absence of comprehensive models and metrics skews evidence gathering toward bibliometrics about research outputs (published discoveries), with less focus on transfer metrics about development outputs (patented prototypes) and almost none on econometrics related to production outputs (commercial innovations). This disparity is particularly problematic for the expressed intent of such programs, as most measurable socioeconomic benefits result from the last category of outputs.MethodsThis paper proposes a conceptual framework integrating all three knowledge-generating methods into a logic model, useful for planning, obtaining, and measuring the intended beneficial impacts through the implementation of knowledge in practice. Additionally, the integration of the Context-Input-Process-Product (CIPP) model of evaluation proactively builds relevance into STI policies and programs while sustaining rigor.ResultsThe resulting logic model framework explicitly traces the progress of knowledge from inputs, following it through the three knowledge-generating processes and their respective knowledge outputs (discovery, invention, innovation), as it generates the intended socio-beneficial impacts. It is a hybrid model for generating technology-based innovations, where best practices in new product development merge with a widely accepted knowledge-translation approach. Given the emphasis on evidence-based practice in the medical and health fields and “bench to bedside” expectations for knowledge transfer, sponsors and grantees alike should find the model useful for planning, implementing, and evaluating innovation processes.ConclusionsHigh-cost/high-risk industries like healthcare require the market deployment of technology-based innovations to improve domestic society in a global economy. An appropriate balance of relevance and rigor in research, development, and production is crucial to optimize the return on public investment in such programs. The technology-innovation process needs a comprehensive operational model to effectively allocate public funds and thereby deliberately and systematically accomplish socioeconomic benefits.

Development of a measure of knowledge use by stakeholders in rehabilitation technology

Objectives: Uptake of new knowledge by diverse and diffuse stakeholders of health-care technology innovations has been a persistent challenge, as has been measurement of this uptake. This article describes the development of the Level of Knowledge Use Survey instrument, a web-based measure of self-reported knowledge use. Methods: The Level of Knowledge Use Survey instrument was developed in the context of assessing effectiveness of knowledge communication strategies in rehabilitation technology. It was validated on samples representing five stakeholder types: researchers, manufacturers, clinician–practitioners, knowledge brokers, and consumers. Its structure is broadly based on Rogers’ stages of innovation adoption. Its item generation was initially guided by Hall et al’s Levels of Use framework. Item selection was based on content validity indices computed from expert ratings (n1 = 4; n2 = 3). Five representative stakeholders established usability of the web version. The version included 47 items (content validity index for individual items >0.78; content validity index for a scale or set of items >0.90) in self-reporting format. Psychometrics were then established for the version. Results: Analyses of data from small (n = 69) and large (n = 215) samples using the Level of Knowledge Use Survey instrument suggested a conceptual model of four levels of knowledge use—Non-awareness, Awareness, Interest, and Use. The levels covered eight dimensions and six user action categories. The sequential nature of levels was inconclusive due to low cell frequencies. The Level of Knowledge Use Survey instrument showed adequate content validity (≈ 0.88; n = 3) and excellent test–retest reliability (1.0; n = 69). It also demonstrated good construct validity (n = 215) for differentiating among new knowledge outputs (p < 0.001) and among stakeholder types (0.001 < p ≤ 0.013). It showed strong responsiveness to change between baseline and follow-up testing (0.001 < p ≤ 0.002; n = 215). Conclusion: The Level of Knowledge Use Survey instrument is valid and reliable for measuring uptake of innovations across diffuse stakeholders of rehabilitation technologies and therefore also for tracking changes in knowledge use.

The Voice of the Customer—Part 2: Benchmarking Battery Chargers Against the Consumer's Ideal Product

The Rehabilitation Engineering Research Center on Technology Evaluation and Transfer is exploring how the end users of assistive technology devices define the ideal device. This work is called the Consumer Ideal Product program. In this work, end users identify and establish the importance of a broad range of product design features, along with the related product support and service provided by manufacturers and vendors. This paper describes a method for systematically transforming end-user defined requirements into a form that is useful and accessible to product designers, manufacturers, and vendors. In particular, product requirements, importance weightings, and metrics are developed from the Consumer Ideal Product battery charger outcomes. Six battery charges are benchmarked against these product requirements using the metrics developed. The results suggest improvements for each products design, service, and support. Overall, the six chargers meet roughly 45-75% of the ideal products requirements. Many of the suggested improvements are low-cost changes that, if adopted, could provide companies a competitive advantage in the marketplace.

Level Of Knowledge Use Survey (LOKUS) instrument: Documenting knowledge use by stakeholders

Sponsors and scholars engaged in applied technological R&D are increasingly challenged to demonstrate evidence that their project outputs are adopted by stakeholders outside of academia. Potential knowledge users in the Assistive Technology field include clinicians, consumers, manufacturers, policy makers and information brokers. This paper first summarizes prior work to create the LOKUS instrument, a web-based tool designed to survey the level of knowledge use by these stakeholders. It then explores the value of the instrument for measuring change in knowledge use over time.

Cultura de avaliação e política de avaliação como guias para a sua prática: reflexões a respeito da experiência brasileira

Nao ha duvida de que as politicas de avaliacao tornam o trabalho do avaliador uma tarefa mais facil por darem mais transparencia ao processo avaliativo e mais seguranca as pessoas envolvidas. No entanto, isso so se da quando essas politicas sao apresentadas a stakeholders (interessados) em potencial, em uma linguagem clara, ja que elas sao divulgadas e utilizadas para guiar a pratica. Sugere-se, assim, que a construcao de uma cultura de avaliacao e necessaria para que uma politica de avaliacao seja implementada efetivamente de modo a ser plenamente utilizada. Esse artigo apresenta, de maneira concisa, o que os autores entendem por “cultura de avaliacao” e “politica de avaliacao”; discute a importância de politicas de avaliacao na pratica da avaliacao e da meta-avaliacao; e destaca as possiveis consequencias da ausencia dessas politicas. A partir de tais consideracoes, surgem recomendacoes que tem como intuito inspirar o desenvolvimento e a implementacao dessas politicas. Finalmente, sao apresentadas reflexoes baseadas na experiencia brasileira com relacao as inter-relacoes “cultura-politica-pratica” na avaliacao.

Evaluation culture and evaluation policy as guides to practice: reflections on the Brazilian experience

Nao ha duvida de que as politicas de avaliacao tornam o trabalho do avaliador uma tarefa mais facil por darem mais transparencia ao processo avaliativo e mais seguranca as pessoas envolvidas. No entanto, isso so se da quando essas politicas sao apresentadas a stakeholders (interessados) em potencial, em uma linguagem clara, ja que elas sao divulgadas e utilizadas para guiar a pratica. Sugere-se, assim, que a construcao de uma cultura de avaliacao e necessaria para que uma politica de avaliacao seja implementada efetivamente de modo a ser plenamente utilizada. Esse artigo apresenta, de maneira concisa, o que os autores entendem por “cultura de avaliacao” e “politica de avaliacao”; discute a importância de politicas de avaliacao na pratica da avaliacao e da meta-avaliacao; e destaca as possiveis consequencias da ausencia dessas politicas. A partir de tais consideracoes, surgem recomendacoes que tem como intuito inspirar o desenvolvimento e a implementacao dessas politicas. Finalmente, sao apresentadas reflexoes baseadas na experiencia brasileira com relacao as inter-relacoes “cultura-politica-pratica” na avaliacao.

Cultura de evaluación y política de evaluación como guías para su práctica: reflexiones sobre la experiencia brasileña

Nao ha duvida de que as politicas de avaliacao tornam o trabalho do avaliador uma tarefa mais facil por darem mais transparencia ao processo avaliativo e mais seguranca as pessoas envolvidas. No entanto, isso so se da quando essas politicas sao apresentadas a stakeholders (interessados) em potencial, em uma linguagem clara, ja que elas sao divulgadas e utilizadas para guiar a pratica. Sugere-se, assim, que a construcao de uma cultura de avaliacao e necessaria para que uma politica de avaliacao seja implementada efetivamente de modo a ser plenamente utilizada. Esse artigo apresenta, de maneira concisa, o que os autores entendem por “cultura de avaliacao” e “politica de avaliacao”; discute a importância de politicas de avaliacao na pratica da avaliacao e da meta-avaliacao; e destaca as possiveis consequencias da ausencia dessas politicas. A partir de tais consideracoes, surgem recomendacoes que tem como intuito inspirar o desenvolvimento e a implementacao dessas politicas. Finalmente, sao apresentadas reflexoes baseadas na experiencia brasileira com relacao as inter-relacoes “cultura-politica-pratica” na avaliacao.