Elisa M. Torres

Cross-Validation of the Implementation Leadership Scale (ILS) in Child Welfare Service Organizations

The Implementation Leadership Scale (ILS) is a brief, pragmatic, and efficient measure that can be used for research or organizational development to assess leader behaviors and actions that actively support effective implementation of evidence-based practices (EBPs). The ILS was originally validated with mental health clinicians. This study validates the ILS factor structure with providers in community-based organizations (CBOs) providing child welfare services. Participants were 214 service providers working in 12 CBOs that provide child welfare services. All participants completed the ILS, reporting on their immediate supervisor. Confirmatory factor analyses were conducted to examine the factor structure of the ILS. Internal consistency reliability and measurement invariance were also examined. Confirmatory factor analyses showed acceptable fit to the hypothesized first- and second-order factor structure. Internal consistency reliability was strong and there was partial measurement invariance for the first-order factor structure when comparing child welfare and mental health samples. The results support the use of the ILS to assess leadership for implementation of EBPs in child welfare organizations.

Validation of the Implementation Leadership Scale (ILS) in Substance use Disorder Treatment Organizations

There have been recent calls for pragmatic measures to assess factors that influence evidence-based practice (EBP) implementation processes and outcomes. The Implementation Leadership Scale (ILS) is a brief and efficient measure that can be used for research or organizational development purposes to assess leader behaviors and actions that actively support effective EBP implementation. The ILS was developed and validated in mental health settings. This study validates the ILS factor structure with providers in alcohol and other drug (AOD) use treatment agencies. Participants were 323 service providers working in 72 workgroups from three AOD use treatment agencies. Confirmatory factor analyses and reliability analyses were conducted to examine the psychometric properties of the ILS. Convergent and discriminant validity were also assessed. Confirmatory factor analyses demonstrated good fit to the hypothesized first and second order factor structure. Internal consistency reliability was excellent. Convergent and discriminant validity was supported. The ILS psychometric characteristics, reliability, and validity were supported in AOD use treatment agencies. The ILS is a brief and pragmatic measure that can be used for research and practice to assess leadership for EBP implementation in AOD use treatment agencies.

The Evidence-based Practice Attitude Scale-36 (EBPAS-36): a brief and pragmatic measure of attitudes to evidence-based practice validated in US and Norwegian samples

BackgroundShort and valid instruments for measuring factors facilitating or hindering implementation efforts are called for. This article describes (1) the adaptation of a shorter version of the Evidence-based Practice Attitude Scale (EBPAS-50 items), and (2) the psychometric properties of the shortened version in both US and Norwegian data.MethodsThe US participants were mental health service providers (N = 418) recruited from clinics providing mental health services in San Diego County, California. The Norwegian participants were psychologists, psychiatric nurses, and psychology students (N = 838) recruited from the Norwegian Psychological Association and the Norwegian Nurses Organization. A confirmatory factor analysis (CFA) approach was used.ResultsThe reduction resulted in 36 items named EBPAS-36, and the original 12 factor model was maintained. The EBPAS-36 had acceptable model fit, as indicated by a low degree of misspecification errors in both the US (RMSEA = .045 (CI90% .040–.049); SRMR = .05) and the Norwegian data (RMSEA = .052 (CI90% .047–.056, SRMR = .07). Incremental model fit was fair in the US (CFI = .93, TLI = .91) and in the Norwegian samples (CFI = .91, TLI = .89). The internal consistency (Cronbach’s α) in the US and the Norwegian samples were good for the total EBPAS-36 score (.79 and .86, respectively) and were ranged from adequate to excellent for the subscales (US .60–.91 and Norway .61–.92).ConclusionsThe EBPAS-36 has adequate psychometric properties both in US and Norwegian samples, hence indicating cross-cultural validity. It is a brief, pragmatic, and more user-friendly instrument than the EBPAS-50, yet maintains a broad scope by retaining the original 12 measurement domains.

Leading for the long haul: a mixed-method evaluation of the Sustainment Leadership Scale (SLS)

BackgroundDespite our progress in understanding the organizational context for implementation and specifically the role of leadership in implementation, its role in sustainment has received little attention. This paper took a mixed-method approach to examine leadership during the sustainment phase of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework. Utilizing the Implementation Leadership Scale as a foundation, we sought to develop a short, practical measure of sustainment leadership that can be used for both applied and research purposes.MethodsData for this study were collected as a part of a larger mixed-method study of evidence-based intervention, SafeCare®, sustainment. Quantitative data were collected from 157 providers using web-based surveys. Confirmatory factor analysis was used to examine the factor structure of the Sustainment Leadership Scale (SLS). Qualitative data were collected from 95 providers who participated in one of 15 focus groups. A framework approach guided qualitative data analysis. Mixed-method integration was also utilized to examine convergence of quantitative and qualitative findings.ResultsConfirmatory factor analysis supported the a priori higher order factor structure of the SLS with subscales indicating a single higher order sustainment leadership factor. The SLS demonstrated excellent internal consistency reliability. Qualitative analyses offered support for the dimensions of sustainment leadership captured by the quantitative measure, in addition to uncovering a fifth possible factor, available leadership.ConclusionsThis study found qualitative and quantitative support for the pragmatic SLS measure. The SLS can be used for assessing leadership of first-level leaders to understand how staff perceive leadership during sustainment and to suggest areas where leaders could direct more attention in order to increase the likelihood that EBIs are institutionalized into the normal functioning of the organization.

Validation of the Implementation Leadership Scale (ILS) with Supervisors’ Self-Ratings

Although often discussed, there is a lack of empirical research on the role of leadership in the management and delivery of health services. The implementation leadership scale (ILS) assesses the degree to which leaders are knowledgeable, proactive, perseverant, and supportive during evidence-based practice (EBP) implementation. The purpose of this study was to examine the psychometric properties of the ILS for leaders’ self-ratings using a sample of mental health clinic supervisors (N = 119). Supervisors (i.e., leaders) completed surveys including self-ratings of their implementation leadership. Confirmatory factor analysis, reliability, and validity of the ILS were evaluated. The ILS factor structure was supported in the sample of supervisors. Results demonstrated internal consistency reliability and validity. Cronbach alpha’s ranged from 0.92 to 0.96 for the ILS subscales and 0.95 for the ILS overall scale. The factor structure replication and reliability of the ILS in a sample of supervisors demonstrates its applicability with employees across organizational levels.

Validating the Implementation Climate Scale (ICS) in child welfare organizations.

There is increasing emphasis on the use of evidence-based practices (EBPs) in child welfare settings and growing recognition of the importance of the organizational environment, and the organizations climate in particular, for how employees perceive and support EBP implementation. Recently, Ehrhart, Aarons, and Farahnak (2014) reported on the development and validation of a measure of EBP implementation climate, the Implementation Climate Scale (ICS), in a sample of mental health clinicians. The ICS consists of 18 items and measures six critical dimensions of implementation climate: focus on EBP, educational support for EBP, recognition for EBP, rewards for EBP, selection or EBP, and selection for openness. The goal of the current study is to extend this work by providing evidence for the factor structure, reliability, and validity of the ICS in a sample of child welfare service providers. Survey data were collected from 215 child welfare providers across three states, 12 organizations, and 43 teams. Confirmatory factor analysis demonstrated good fit to the six-factor model and the alpha reliabilities for the overall measure and its subscales was acceptable. In addition, there was general support for the invariance of the factor structure across the child welfare and mental health sectors. In conclusion, this study provides evidence for the factor structure, reliability, and validity of the ICS measure for use in child welfare service organizations.

The Development of a Brief Version of the Posttraumatic Cognitions Inventory (PTCI-9):

Negative posttraumatic cognitions lead to the development and maintenance of posttraumatic stress disorder symptoms. There is a need for a brief measure to assess these cognitions. Participants were administered the Posttraumatic Cognitions Inventory (PTCI) and measures of mental health symptomatology. These data were used to develop a brief version of the PTCI (PTCI-9) in 223 male and female veterans, which was then examined in a sample of 117 female civilians. Confirmatory factor analyses demonstrated an acceptable fit in both samples. The PTCI-9 total and subscale scores showed strong internal consistencies (Cronbach’s αs = .80-.87) and strong correlations with the PTCI in veterans (rs = .90-.96) and civilians (rs = .91-.96). Measurement invariance testing demonstrated partial invariance between the two samples. The PTCI-9 significantly correlated with measures of PTSD, depression, and quality of life. These findings demonstrate that the PTCI-9 is a reliable and valid measure of posttraumatic cognitions that can reduce patient and provider burden.