Melissa A. Collier-Meek

Is Performance Feedback for Educators an Evidence-Based Practice? A Systematic Review and Evaluation Based on Single-Case Research:

Optimal levels of treatment fidelity, a critical moderator of intervention effectiveness, are often difficult to sustain in applied settings. It is unknown whether performance feedback, a widely researched method for increasing educators’ treatment fidelity, is an evidence-based practice. The purpose of this review was to evaluate the current research on performance feedback as a strategy to promote the implementation of school-based practices. Studies were evaluated according to What Works Clearinghouse (WWC; Kratochwill et al., 2010) technical guidelines for single-case design, utilizing both the design and evidence standards to determine whether studies provided sufficient evidence for the effectiveness of performance feedback. Results indicate that performance feedback can be termed an evidence-based intervention based on criteria set by the WWC. Implications for future research are described.

An exploratory investigation of teachers' intervention planning and perceived implementation barriers.

Increasingly teachers are the primary implementer responsible for providing evidence-based interventions to students. However, there is little knowledge regarding the extent to which teachers plan for intervention implementation, receive implementation support, or identify and address implementation barriers. This study explores survey data from over 1200 preschool through grade 12 teachers from 46 public school districts in a Northeastern state. Results indicate that teachers spend significant time engaging in intervention-related behavior and may be a primary source responsible for selecting student interventions. However, the current extent to which they plan for implementation and present levels of implementation support are inadequate to produce high levels of sustained intervention implementation. In addition, almost 60% of implementation barriers reported related to aspects of the intervention itself. Findings from this study provide guidance for future research and preliminary recommendations for ameliorating implementation barriers and proactively supporting treatment integrity in schools.

A critical review of videoconferencing software to support school consultation

With recent advances in technology, several viable options now exist for school consultants to communicate with teachers via synchronous online videoconferencing (VC) without the need for specialized equipment. Unfortunately, few studies have evaluated the effectiveness of school consultation through VC, so practitioners are provided little guidance when determining if VC is suited to these specific needs. In this article, we identify the VC software packages most commonly used in the growing telehealth literature and discuss how features common to these programs might support effective school teleconsultation. We focus our attention on six specific capabilities: (a) onscreen document sharing; (b) group VC; (c) instant messaging; (d) recording capabilities; (e) integrated cloud storage; and (f) ethical compliance in regard to client confidentiality. Potential applications of these features are discussed, as well as practical and ethical considerations for trainers and school practitioners.

Assessment of Consultation and Intervention Implementation: A Review of Conjoint Behavioral Consultation Studies

Reviews of treatment outcome literature indicate treatment integrity is not regularly assessed. In consultation, two levels of treatment integrity (i.e., consultant procedural integrity [CPI] and intervention treatment integrity [ITI]) provide relevant implementation data. Specifically, assessment of CPI and ITI are necessary to conclude (a) consultation is functionally related to consultee implementation behavior and (b) intervention implementation is functionally related to student outcomes. In this article, study characteristics and the presence of treatment integrity at both levels are examined in 21 studies utilizing Conjoint Behavioral Consultation, a model of consultation that includes multiple consultees. Results indicate that in approximately half of studies, CPI, ITI, or both are assessed and, when reported, treatment integrity is moderately high across both levels. However, there are distinct differences in the assessment and reporting of these levels of treatment integrity. Limitations and implications for consultation research and treatment integrity reporting are discussed.

A comparison of problem identification interviews conducted face-to-face and via videoconferencing using the consultation analysis record☆

School psychologists who experience challenges delivering face-to-face consultation may utilize videoconferencing to facilitate their consultation activities. Videoconferencing has been found to be an effective method of service delivery in related fields and emerging research suggests that it may be effective for providing teacher training and support in school settings. In this exploratory investigation, we used the Consultation Analysis Record (Bergan & Tombari, 1975) and its four indices to assess the effectiveness of conducting problem identification interviews via videoconferencing versus face-to-face. Overall, findings indicated significant differences across these two conditions, with videoconference interviews coded as having higher indices of content relevance, process effectiveness, and message control, but lower content focus, compared to face-to-face interviews. As these indices have been positively associated with favorable consultation outcomes, the results provide initial support for the effectiveness of consultation delivered via videoconferencing.

When Is Teleconsultation Acceptable to School Psychologists

ABSTRACT School psychologists report a desire to increase consultation with teachers but are limited by competing job demands. Consultation via videoconference (i.e., teleconsultation) offers one possible means of increasing consultant availability to teachers, but little is known about the acceptability of this technology in actual practice. In this exploratory study, we conducted an online survey to assess the conditions under which school consultants would use teleconsultation to assist teachers remotely. Sixty-two school psychology practitioners, trainers, and trainees completed a choice-based conjoint survey designed to assess teleconsultation acceptability in the context of varying commute times, consultee comfort levels, and student problem severities. Findings suggest important differences in teleconsultation acceptability in the context of real-world tradeoffs. Implications of these results are discussed, as well as the potential for using conjoint analysis to better understand technology decisions among school consultants.

Embedding Positive Behavior Intervention and Supports in Afterschool Programs.

A fterschool programs (ASPs) provide ‘‘safe, structured learning environments for students outside the traditional school day’’ and are an ‘‘integral part’’ of education (National Governor’s Association [NGA], 2009, p. 7). These programs vary in their structure but typically operate for 2 to 3 hours following the end of the school day. They may be broad or generic in focus (e.g., include different activities related to academics [e.g., tutoring, homework support], enrichment [e.g., sports, art, music], and service [e.g., volunteering, community service]; NGA, 2009), or they may have a very specific focus (e.g., vocational preparedness for high school students). Generally speaking, ASPs offer a critical service to families by providing safe environments for students during ‘‘risky’’ after school hours (i.e., 3–6 p.m.; Palmer, Anderson, & Sabatelli, 2009). There is growing recognition that ASPs provide opportunities for positive youth development. Many ASPs focus on behavior and socioemotional challenges, provide evidence-based interventions to improve homework completion and academic skills, and offer physical activities and nutritious foods (Durlak & Weissberg, 2007). Although ASPs have operated for many years, the focus on their potential for added learning time and results-based accountability is relatively recent (James-Burdumy, Dynarski, & Deke, 2007). Nationally, many of these programs are funded to help close the education achievement gap for disadvantaged children by providing academic, enrichment, recreation, and family literacy supports while keeping youth safe (James-Burdumy et al.). There are significant challenges to attaining these positive student outcomes, many of which relate to the imbalance between the relatively intensive needs of participants, limited resources, and staff qualifications available in programs (Farrell, Collier-Meek, Randall, & Saxena, 2012). Considering the increased likelihood that children and youth who attend ASPs belong to atrisk groups (Farrell et al.), these students are more likely to experience behavior challenges at school. Furthermore, because behavior problems are more likely to occur in unstructured settings (Newcomer, Colvin, & Lewis, 2009), it is likely that discipline concerns are more prevalent in ASPs as compared with school settings. Indeed, behavior management is oft-cited by ASP professionals as a desired element of training (Grossman, Campbell, & Raley, 2007; Palmer et al., 2010), and in some locales, youth in ASPs have a higher incidence of serious discipline infractions (Connecticut Commissioner of Education, 2008). The ASP research base is not sufficiently well developed to provide guidance on the staff characteristics and training models that are most effective in supporting youth engagement in programming and ultimately in producing positive outcomes. Thus, ASPs face a combination of challenges including: (a) substantial mandates to address academics, improve behavioral-social functioning, practice in a culturally competent manner, and meet the needs of families; (b) a population of students at-risk whose needs may be more intensive than those of the average student; and (c) an explicit need for staff training in behavior management, limited resources and opportunities for professional development, and high staff turnover (Connecticut Commissioner of Education, 2008). These challenges emphasize a corresponding need for effective and efficient systems and practices to create an environment in which staff members meet the promise of ASPs. These programs have an opportunity to reap many benefits by utilizing evidence-based education initiatives, such as Positive Behavioral Interventions and Supports (PBIS), to promote a positive environment, support participant and staff behavior, and enhance outcomes. In support of extending PBIS into ASPs, this paper includes an overview of PBIS, specifies its relevance to ASPs, outlines initial research support, and reviews considerations related to preparation, implementation, and evaluation of PBIS in ASPs. Finally, we make recommendations for embedding elements of PBIS into ASPs.

How are treatment integrity data assessed? Reviewing the performance feedback literature.

Collecting treatment integrity data is critical for (a) strengthening internal validity within a research study, (b) determining the impact of an intervention on student outcomes, and (c) assessing the need for implementation supports. Although researchers have noted the increased inclusion of treatment integrity data in published articles, there has been limited attention to how treatment integrity is assessed. The purpose of this study was to review how treatment integrity is typically assessed in single-case design performance feedback articles. Doctoral students in school psychology coded 58 performance feedback studies to determine how treatment integrity data are assessed in this literature, and common intervention characteristics linked to treatment integrity assessment. Findings indicate that, although data were collected most frequently via direct observation by a researcher targeting intervention adherence, intervention characteristics varied widely across studies. Implications of results are discussed.

Examining Dimensions of Treatment Intensity and Treatment Fidelity in Mathematics Intervention Research for Students at Risk

To prevent academic failure and promote long-term success, response-to-intervention (RtI) is designed to systematically increase the intensity of delivering research-based interventions. Interventions within an RtI framework must not only be effective but also be implemented with treatment fidelity and delivered with the appropriate level of treatment intensity to improve student mathematics achievement. The purpose of this systematic review was to explore the mathematics intervention research for students at risk of mathematics failure by examining intervention characteristics related to treatment fidelity and intensity. Results from 66 math intervention studies conducted from 2004 to 2015 were summarized. The majority of studies monitored treatment fidelity and provided details of some aspects of treatment intensity (i.e., dose, group size). However, interventionists’ characteristics, implementation characteristics, and treatment design were less frequently reviewed. Implications for future research and educational practices are discussed.

Highlighting the Utility of the Consultation Analysis Record for Consultation Research and Training

ABSTRACT What we say in consultation is important, and thus studying spoken messages has been an important topic in consultation research and training for decades. Here we raise awareness of the Consultation Analysis Record (CAR; Bergan & Tombari, 1975), a well-established verbal interaction coding system that has largely disappeared from the professional literature. Following a description of the CAR and six summary findings from research that has employed it, we turn our attention to its future utility, including facilitating its use with technology and its applications to multitiered systems of support teams and procedural integrity research. The central thesis is that the CAR remains a valuable tool to study processes and outcomes of consultation as well as to train consultants, and it should be used more often for these purposes.