Daniel M. Maggin

An Application of the What Works Clearinghouse Standards for Evaluating Single-Subject Research Synthesis of the Self-Management Literature Base

The use of single-subject research in the development and evaluation of academic, psychological, and behavioral interventions has led to the experimental validation of an array of treatment options for a diverse set of educational challenges. However, the synthesis of these bodies of research has been the subject of considerable debate. In an effort to utilize the findings from studies using single-subject methodologies to identify effective educational practices, the What Works Clearinghouse developed a set of criteria to evaluate the strength of evidence for various strategies. In this article, an application of these standards is demonstrated using a body of self-management intervention studies drawn from a recently published systematic review. The utility of the standards for identifying evidence-based practices validated with single-subject research methods is discussed. In addition, a comparison of the What Works Clearinghouse procedures to previously developed methods for identifying evidence-based practices with single-subject research is provided, and implications for research and practice are described.

A systematic evidence review of school-based group contingency interventions for students with challenging behavior ☆

The purpose of this review was to synthesize the research underlying group contingency interventions to determine whether there is sufficient evidence to support their use for managing the classroom behavior of students with behavioral difficulties. An application of the What Works Clearinghouse (WWC) procedures for evaluating single-subject research revealed that the research investigating group contingencies demonstrated sufficient rigor, evidence, and replication to label the intervention as evidence-based. These findings were further supported across five quantitative indices of treatment effect. The results associated with the application of the WWC procedures and quantitative evaluations were supplemented with additional systematic coding of methodological features and study characteristics to evaluate the populations and conditions under which the effects of the group contingency best generalize. Findings associated with this coding revealed that the lack of detailed reporting across studies limited our ability to determine for whom and under what conditions group contingencies are best suited.

Is Performance Feedback for Educators an Evidence-Based Practice? A Systematic Review and Evaluation Based on Single-Case Research:

Optimal levels of treatment fidelity, a critical moderator of intervention effectiveness, are often difficult to sustain in applied settings. It is unknown whether performance feedback, a widely researched method for increasing educators’ treatment fidelity, is an evidence-based practice. The purpose of this review was to evaluate the current research on performance feedback as a strategy to promote the implementation of school-based practices. Studies were evaluated according to What Works Clearinghouse (WWC; Kratochwill et al., 2010) technical guidelines for single-case design, utilizing both the design and evidence standards to determine whether studies provided sufficient evidence for the effectiveness of performance feedback. Results indicate that performance feedback can be termed an evidence-based intervention based on criteria set by the WWC. Implications for future research are described.

Curriculum-based measurement of oral reading (R-CBM): A diagnostic test accuracy meta-analysis of evidence supporting use in universal screening

A great deal of research over the past decade has examined the appropriateness of curriculum-based measurement of oral reading (R-CBM) in universal screening. Multiple researchers have meta-analyzed available correlational evidence, yielding support for the interpretation of R-CBM as an indicator of general reading proficiency. In contrast, researchers have yet to synthesize diagnostic accuracy evidence, which pertains to the defensibility of the use of R-CBM for screening purposes. The overall purpose of this research was to therefore conduct the first meta-analysis of R-CBM diagnostic accuracy research. A systematic search of the literature resulted in the identification of 34 studies, including 20 peer-reviewed articles, 7 dissertations, and 7 technical reports. Bivariate hierarchical linear models yielded generalized estimates of diagnostic accuracy statistics, which predominantly exceeded standards for acceptable universal screener performance. For instance, when predicting criterion outcomes within a school year (≤9 months), R-CBM sensitivity ranged between .80 and .83 and specificity ranged between .71 and .73. Multiple moderators of R-CBM diagnostic accuracy were identified, including the (a) R-CBM cut score used to define risk, (b) lag in time between R-CBM and criterion test administration, and (c) percentile rank corresponding to the criterion test cut score through which students were identified as either truly at risk or not at risk. Follow-up analyses revealed substantial variability of extracted cut scores within grade and time of year (i.e., fall, winter, and spring). This result called into question the inflexible application of a single cut score across contexts and suggested the potential necessity of local cut scores. Implications for practices, directions for future research, and limitations are discussed.

A Systematic Evidence Review of the Check-In/Check-Out Program for Reducing Student Challenging Behaviors

Three-tiered models of prevention are increasingly being adopted by schools to address the behavioral needs of students. A critical component of multitiered systems are secondary interventions used with students in need of behavioral support but who are not candidates for individualized interventions. Despite the importance of secondary interventions, questions remain regarding which approaches have sufficient empirical support to warrant their use. The purpose of this review was, therefore, to examine the research underlying the Check-In/Check-Out (CICO) program, a widely used secondary intervention, to determine the strengths, limitations, and generality of the accumulated research. The What Works Clearinghouse (WWC) procedures for evaluating single-case and group-based research were applied with results indicating mixed support for the program. Specifically, there were a sufficient number of single-case research studies to deem the CICO program as evidence-based, while the group-based research had no demonstrated effects. These findings are discussed in terms of future research on the CICO program and the broader implications for selecting and implementing secondary interventions in school settings.

Introduction to the Special Series Issues and Advances of Synthesizing Single-Case Research

This special series addresses issues and advances in the synthesis of single-case research. In this introduction, we provide an overview of the conceptual and procedural challenges associated with evaluating and aggregating findings drawn from single-case research. Key issues relating to the development of reliable and objective procedures for synthesizing single-case data are reviewed and situated within the context of the nomothetic and idiographic research paradigms. The potential for syntheses of single-case research to augment more traditional approaches for informing the development of evidence-based practices in special education is then reviewed. We conclude with a brief overview of the articles contained in this special series and provide a summary of the collective contributions for moving toward a widely accepted framework for synthesizing single-case research findings.

Evaluating single-case research data for systematic review: a commentary for the special issue

The purpose of this commentary is to provide observation on the statistical procedures described throughout this special section from the perspective of researchers with experience in conducting systematic reviews and meta-analyses of single-case research to address issues of evidence-based practice. It is our position that both visual and statistical analyses are complimentary methods for evaluating single-case research data for these purposes. Given the recent developments regarding the use of single-case research to inform evidence-based practice and policy, the developments described in the present issue will be contextualized within the need for a widely accepted process for data evaluation to assist with extending the impact of single-case research. The commentary will, therefore, begin with providing an overview of the conceptual underpinnings of a systematic review of single-case research and will be followed by a discussion of several features that are essential to the development of a conceptually sound and widely used statistical procedure for single-case research. The commentary will conclude with recommendations and guidelines for the use of both visual and statistical analyses within primary research reports and recommendations for future research.

Reliability, Validity, and Usability of Data Extraction Programs for Single-Case Research Designs:

Single-case experimental designs (SCEDs) have been increasingly used in recent years to inform the development and validation of effective interventions in the behavioral sciences. An important aspect of this work has been the extension of meta-analytic and other statistical innovations to SCED data. Standard practice within SCED methods is to display data graphically, which requires subsequent users to extract the data, either manually or using data extraction programs. Previous research has examined issues of reliability and validity of data extraction programs in the past, but typically at an aggregate level. Little is known, however, about the coding of individual data points. We focused on four different software programs that can be used for this purpose (i.e., Ungraph, DataThief, WebPlotDigitizer, and XYit), and examined the reliability of numeric coding, the validity compared with real data, and overall program usability. This study indicates that the reliability and validity of the retrieved data are independent of the specific software program, but are dependent on the individual single-case study graphs. Differences were found in program usability in terms of user friendliness, data retrieval time, and license costs. Ungraph and WebPlotDigitizer received the highest usability scores. DataThief was perceived as unacceptable and the time needed to retrieve the data was double that of the other three programs. WebPlotDigitizer was the only program free to use. As a consequence, WebPlotDigitizer turned out to be the best option in terms of usability, time to retrieve the data, and costs, although the usability scores of Ungraph were also strong.

Intensive Interventions for Students With Emotional and Behavioral Disorders: Issues, Theory, and Future Directions

Behavioral, emotional, and social problems are a ubiquitous feature of childhood. At some point in development, nearly all youth experience circumstances and contexts where they do not seem to fit, have difficulty controlling their emotions and impulses, and struggle to regulate their behavior with others. Epidemiological studies suggest that more than 30% of youth experience a Diagnostic and Statistical Manual of Mental Disorders (DSM) level difficulty during childhood or adolescence (Forness, Freeman, Paparella, Kauffman, & Walker, 2012). For some youth, the manifestation of disorder is transitory (Costello, Foley, & Angold, 2006). For example, prevalence rates of special education identification for Emotional and Behavioral Disorder (EBD) depend on whether estimates reflect single or cumulative time points (Forness et al., 2012). With appropriate supports, many youth who have momentary emotional and behavioral problems will go on to experience adequate adjustment and positive outcomes. Nonetheless, current estimates suggest that a significant proportion of youth—between 5% and 13%—experience serious and sustained or recurring adjustment difficulties that constrain their future adaptation (Casanueva et al., in press). The focus of this special issue is on youth for whom EBD is a chronic condition or a condition that seriously impairs present academic, behavioral, emotional, and social functioning at the risk of negatively affecting important adolescent and early adulthood outcomes. More specifically, this issue focuses on those students for whom long-term, intensive intervention will be required. At any given time, approximately 1% of school-aged children receive services for EBD (Forness et al., 2012). A high proportion of youth who receive services for EBD have relatively low academic grades, poor social skills, and high rates of suspension as compared with students in other disability categories and nondisabled youth (Bradley, Henderson, & Monfore, 2004). Compared with students in other disability areas and nondisabled youth, students with EBD and learning disabilities have higher rates of truancy (Chen, Culhane, Metraux, Park, & Venable, 2015) and students with EBD are also more likely to experience out of home placements and involvement with other child service sectors (Chen et al., 2016). Furthermore, youth with EBD are more likely to have lower rates of high school completion, post-secondary education, and independent living, and higher rates of unemployment, arrest, and parole and probation than youth in the general population (Wagner & Newman, 2012). However, it should be noted that youth who have similar academic, behavioral, and social risks as youth with EBD, but who are not necessarily identified for EBD special education services, also experience poor outcomes such as school dropout, substance use, and criminality (Janosz, Le Blanc, Boulerice, & Tremblay, 2000; Lipsey & Derzon, 1998; Walker & Sprague, 1999) and are in need of intensive intervention. Problematic outcomes of youth with EBD are not manifested in a vacuum. Rather, they reflect correlated constraints. Youth develop as an integrated whole with behavioral, biophysical, cognitive, psychological, and sociological variables operating together to contribute to individual functioning (Farmer, Gatzke-Kopp, Lee, Dawes, & Talbott, 2016). This means that problematic outcomes, including school failure, school dropout, involvement in substance use, criminality, and adolescent and early adulthood mental health disorders, tend to reflect a system of correlated or interconnected factors (Bergman, Andershed, & Andershed, 2009; Cairns & Cairns, 1994; Cicchetti & Toth, 2009). The importance of correlated constraints for 661498 EBXXXX10.1177/1063426616661498Journal of Emotional and Behavioral DisordersMaggin et al. research-article2016

An Analysis of Preference Relative to Teacher Implementation of Intervention

The purpose of this study was to conduct a preference trial as a preliminary test of preference effects on teacher behavior relative to implementation (adoption, adherence, quality). Teachers were randomly assigned to “preference” or “no-preference” groups and then trained to implement the intervention. Direct observation occurred immediately after initial training, after 6 weeks of coaching support, and after 4 weeks of no support. Results showed that, when compared with the no-preference group, teachers who had the opportunity to exert a preference adopted the intervention sooner and sustained higher fidelity and quality of implementation independent of coaching. Furthermore, though most teachers in the no-preference group did adopt the intervention and demonstrate high fidelity following coaching, implementation did not sustain after the withdrawal of coaching.