Chris L. S. Coryn

A Systematic Review of Theory-Driven Evaluation Practice From 1990 to 2009

Although the general conceptual basis appeared far earlier, theory-driven evaluation came to prominence only a few decades ago with the appearance of Chen’s 1990 book Theory-Driven Evaluations. Since that time, the approach has attracted many supporters as well as detractors. In this paper, 45 cases of theory-driven evaluations, published over a twenty-year period, are systematically examined to ascertain how closely theory-driven evaluation practices comport with the key tenants of theory-driven evaluation as described and prescribed by prominent theoretical writers. Evidence derived from this review to repudiate or substantiate many of the claims put forth both by critics of and advocates for theory-driven forms of evaluation are presented and an agenda for future research on the approach is recommended.

Development and evaluation of the short version of the Patient Assessment of Chronic Illness Care instrument

Objectives: The Patient Assessment of Chronic Illness Care (PACIC) was developed for measuring the extent to which patients receive care congruent with the chronic care model (CCM). The purpose of this study was to develop a short version of the PACIC with better psychometric properties than the original instrument. Methods: Two samples of 529 and 361 type 2 diabetic patients completed a modified PACIC. A short-version PACIC instrument was developed and validated using parallel analysis to determine the number of factors, confirmatory factor analysis (CFA) within an exploratory factor analysis framework (E/CFA) was conducted to explore the measurement structure of the full instrument, and a CFA was performed to confirm the hypothesized structure. Results: The results demonstrated that the PACIC is unidimensional and that it can be reduced to 11 items with no loss in psychometric properties. No demographic variables or clinical assays were found to be related to the PACIC. Discussion: A short-version PACIC is now available and ready for use in research with diabetic patients. Its use is encouraged in future research, particularly in the exploration of its validity against actual CCM services delivered and long-term clinical outcomes.

Contemporary Thinking About Causation in Evaluation: A Dialogue With Tom Cook and Michael Scriven:

Legitimate knowledge claims about causation have been a central concern among evaluators and applied researchers for several decades and often have been the subject of heated debates. In recent years these debates have resurfaced with a renewed intensity, due in part to the priority currently being given to randomized experiments by many funders of evaluation studies, such as the Institute for Educational Sciences. In this dialogue, which took place at Western Michigan University in October 2008, two of the field’s leading theorists and methodologists, Thomas D. Cook and Michael Scriven, described their current thinking and views about causation and causal inference in evaluation. They also discussed recent methodological developments for cause-probing investigations that sometimes produce results comparable to those produced by randomized experiments. Both Cook and Scriven prepared clarifying postscripts after reading the edited transcript.

The Politics and Consequences of Including Stakeholders in International Development Evaluation

Participatory evaluation approaches have a relatively long history of advocacy and application in the international development evaluation community. Despite widespread use and apparent resonance with practitioners and donors alike, very little empirical research exists on why and how participatory evaluation approaches are used in international development settings. In this article, we present results derived from a mixed method investigation of a sample of practicing international development evaluators regarding their perceptions of how and why stakeholders are included in international development evaluations. Findings suggest that participatory evaluation approaches are interpreted and practiced in widely differing ways. Implications for international development evaluation practice and future research are discussed.

Adding a Time-Series Design Element to the Success Case Method to Improve Methodological Rigor An Application for Nonprofit Program Evaluation

Brinkerhoffs Success Case Method (SCM) was developed with the specific purpose of assessing the impact of organizational interventions (e.g., training and coaching) on business goals by analyzing extreme groups using case study techniques and storytelling. As an efficient and cost-effective method of evaluative inquiry, SCM is attractive in other contexts as well, although few examples of such uses are to be found in the published literature. However, modifications of the SCM concept and design are sometimes necessary for implementing the approach in nonprofit environments where business goals are not necessarily an explicit objective. This method note demonstrates how SCM was modified and extended to a social service context, in which the program evaluated was aimed at reducing chronic homelessness and unemployment. Modifications included defining success in a nonprofit setting and adding a time-series element to the design features of traditional SCM to increase methodological rigor.

An application of a new evidence grading system to research on the chronic care model.

Methodological quality undergirds all evidence-based medicine because without strong evidence supporting or refuting the efficacy of an intervention, the movement toward basing medical decisions and practice on scientific evidence is not sustainable. Recently, the consensus that had existed regarding the hierarchy of evidence produced by a study design was challenged on the basis that existing guidelines failed to properly define key terms, weight the merits of certain non-randomized controlled trials, and employ a comprehensive list of study design limitations to render evaluative conclusions, to name a few of the challenges. The present study introduces a new grading system that overcomes, or at the very least greatly diminishes, these challenges. This new method is applied to the literature on the Chronic Care Model and the results are then compared to several of the most popular grading guidelines currently in use. These results revealed substantial differences between the guidelines in accordance with previous research that challenged existing methods. Furthermore, the present study lends support to the proposed grading guideline although further research into its validity and reliability is needed.

Measles, Mumps, and Rubella Vaccines and Diagnoses of Autism Spectrum Disorders among Children: A Meta‐Analysis

Purpose: The authors of the study sought, through a meta-analysis of primary studies, to same for those who receive a measles, mumps, and rubella (MMR) vaccination and those who do not?” Background: Despite evidence from numerous sources failing to demonstrate consistent evidence of ASD diagnoses as a consequence of receiving MMR vaccinations, parents/guardians sometimes forego vaccinating their children. Methods: Three researchers searched for and obtained relevant studies, and two researchers independently applied a standardized data extraction form to the included studies. Results: Seven independent effect sizes were calculated, yielding a pooled random-effects OR* = 0.25 (95% CI, LL = 0.09, UL = 0.76) (*denotes random-effects estimate). Under the fixedeffect model, the OR = 0.33 (95% CI, LL = 0.25, UL = 0.45). Conclusions: The odds of ASD diagnoses attributable to receiving or not receiving an MMR vaccination are not indistinguishable within clinical limits of indifference. Even so, the odds of ASD diagnoses are substantially smaller for those receiving an MMR vaccination than for those who do not.

Social capital, substance use disorder and depression among youths

ABSTRACT Background: Social capital – the network of social connections that exists among people – is known to be related to depression and substance use among adults. However, little is known about these relationships among adolescents, even though this age group is vulnerable due to factors of peer pressure, family, neighborhood, and maturational changes. Objectives: To evaluate the associations among social capital, substance use disorder and depression on a sample of 17 705 respondents between the ages of 12 and 17 in the 2009 National Survey of Drug Use and Health. Methods: Structural equation modeling was used to examine social capital; responses to 48 items differentiated into two factors that measured structural social and cognitive social capital. Adolescent depression and substance use disorder were measured as past-year major depressive episodes and substance use disorder according to DSM-IV criteria. Results: Structural social capital was associated with substance use disorder (β = −0.12; p = 0.001) and depression (β = −0.19; p = 0.001). Cognitive social capital was associated with substance use disorder (β = −0.17; p = 0.001), but not with depression (β = −0.002; p > 0.005). Substance use disorder mediated the association between structural and cognitive social capital and depression (β = 0.06; p = 0.001). Conclusion: There was support for associations among youth structural and cognitive social capital, substance use disorder and depression. These findings suggest that additional research of a longitudinal nature is needed to determine causal connections among social capital, depression and substance use disorder for adolescents.

The Influence of Parental and Communication Style On Consumer Socialization: A Meta-Analysis Informs Marketing Strategy Considerations Involving Parent–Child Interventions

ABSTRACT This meta-analysis merged family communication pattern (FCP) and parental socialization literature streams to offer renewed perspectives on how parents intervene in media-related consumer-socialization interactions with children. With only one exception, FCP-type effects were not different from theorized corresponding parental socialization-style effect sizes. This supports prior literature suggesting equivalency between specific FCP and parental socialization pairs. Furthermore, when certain FCP–parental socialization pairs were compared with other pairs on socialization interactions, such as control, coviewing, and discussing media strategy, differences were found that prior theorizing would have supported. Implications of these results are discussed, including what they suggest for managers.

The effects of mobile phone use on academic performance: A meta-analysis

Abstract Purpose Although the mobile phone has been conspicuously proliferated in the past decades, little is known about its influence; especially its effect on student learning and academic performance. Although there is a growing interest in mobile devices and their correlates and consequences for children, effects vary across related studies and the magnitude of the overall effect remains unclear. The purpose of this study is to further examine any relationships that may exist between mobile phone use and educational achievement. Research design A meta-analysis of research conducted on the relationship between mobile phone use and student educational outcomes over a 10-year period (2008–2017) was conducted. The operational definition of cell phone use to guide the implementation of this study is: any measure of mobile phone use, whether considered normative or problematic, that quantifies the extent to which a person uses a phone, feels an emotional or other dependence on a phone, or categorizes the types of uses and situations in which use occurs. Studies examining use for the express purpose of educational improvement are not included, as the aim of this study is to ascertain the effects of normal smartphone use. The operational definition of academic achievement to guide the implementation of this study is: any measure that quantifies the extent to which a student or group of students is performing or feels he or she is performing to a satisfactory level, including but not limited to letter grades and test scores, knowledge and skill acquisition, and self-reported measures of academic ability or difficulty. Findings The overall meta-analysis indicated that the average effect of mobile phone usage on student outcomes was r =  −0.162 with a 95% confident interval of −0.196 to −0.128. The effect sizes of moderator variables (education level, region, study type, and whether the effect size was derived from a Beta coefficient, and mobile phone use construct) were analyzed. The results of this study and their implications for both research and practice are discussed.