Oliver T. Massey

A proposed model for the analysis and interpretation of focus groups in evaluation research

Focus groups have an established history in applied research and evaluation. The fundamental methods of the focus group technique have been well discussed, as have their potential advantages. Less guidance tends to be provided regarding the analysis of data resulting from focus groups or how to organize and defend conclusions drawn from the analysis. This article reviews the methodology of the focus group with an emphasis on thematic analysis of latent data at three levels, articulated, attributional, and emergent. The three levels are described and illustrated with respect to their value and contribution to evaluation within the framework of the group method and qualitative standards of thematic analysis.

Three critical views of functioning: Comparisons of assessments made by individuals with mental illness, their case managers, and family members

Level of functioning measures are regularly used by mental health agencies to identify the strengths and abilities of individuals with mental illness. In spite of the important use of these measures, little research has compared the case managers perception of functioning to that of individual consumers or their family members. In the current study, a comparison of the three groups produced moderate correlations in assessments of the consumers functioning. Case managers rated the functioning of consumers significantly lower than did family members or consumers themselves with regard to Vocational Skills, Vocational Motivation, and Community Living Skills. The relevance of the three perspectives for community based service provision is discussed.

School violence interventions in the safe schools/healthy students initiative: evaluation of two early intervention programs

Abstract The Safe Schools/Healthy Students Initiative was designed to encourage collaboration among school districts and local community service providers in the provision of behavioral health prevention and early intervention efforts. These efforts would address the physical safety of students as well as provide mental health, violence prevention, and social skills services. One local SS/HS Initiative brought together community and school collaborators in an ambitious agenda that included 14 distinct programs that addressed the needs of over 110,000 students in a large school district. The purpose of the current paper is to report the results of the evaluation of two of the programs designed to reduce violent and disruptive behavior in schools. The programs include a school-based anger management program and a community-based, alternative-to-suspension program. Working in cooperation with program staff and the school district, quasi-experimental designs were used to measure change over time for students. The two studies demonstrate the application of multiple methodologies in evaluating the effectiveness of prevention and early intervention efforts with the aim of providing data to support program improvement and sustainability.

A study of the utility of the child behavior checklist with residentially placed children

An evaluation study was conducted to determine the potential usefulness of the Child Behavior Checklist (CBCL) with children placed in residential settings. The CBCL, the Global Assessment Scale, the Progress Evaluation Scales, and the Jesness Behavior Checklist were completed for 97 children in community residential treatment. Children categorized by the CBCL as Externalizers were found to be more obtrusive, hostile, alienated, nonconforming, attention seeking, and sensitive, and less responsible, insightful, and considerate than were Internalizers. The total problem score, the Internalizing scale score, and the Externalizing scale score on the CBCL were significantly correlated with problem severity, predicted length of stay, and a measure of placement appropriateness. These three total scores proved to be useful measures of residentially placed children. Results suggest that a high score on the Externalizing scale may reflect the presence of behaviors difficult to deal with in a residential environment. A high score on the Internalizing scale may be no less serious, but involves behaviors less disruptive of the group setting. The potential value of the CBCL for evaluating residentially placed children is discussed.

The development of factor-based level of functioning scales from a level of care instrument

A factor analysis of data from 1,295 administrations of a level of care instrument produced a seven-factor solution. The scales developed from these factors through a simple unit weighting procedure were internally reliable and replicable on an additional sample of clients. The seven factors are discussed in terms of level of functioning measures. Results suggest that community living skills and self-care skills are two among several robust dimensions of level of functioning in psychiatric clients. Implications are discussed.

Socioeconomic status and behavior problems: addressing the context for school safety

Abstract Violence in the schools is a fundamental concern for students, educators and communities as they collectively address issues that make schools safer. A longitudinal analysis of disciplinary referrals was conducted with a large countywide school district to measure differences across socioeconomic status and school level. These variables proved to be critical in identifying and targeting schools that may need to be the focus of prevention and intervention efforts. Other suggestions related to the effects of school violence on academic instruction and methods for the targeting of services are discussed along with implications for educators and mental health professionals in school based settings.

The utility of level of functioning to characterize community residential settings

An assessment of client functioning was utilized to characterize differences among community residential placement facilities, with the aim of understanding the relationships among settings. Clients recommended for placement in one of nine settings differed significantly on seven scales measuring community living skills, self-care skills, sociability, nuisance behaviors, violence, irritability and nursing needs. Results indicate that community residential settings, although characterized as forming a continuum, may be better conceptualized as an array of settings, where clients may differ in some but not all functional abilities. Facilities may also serve different functions for clients of different ages, either because of expectations concerning older clients or due to different patterns of functioning. Illustrations of the implications that can be drawn for purposes of policy-making and service planning are provided.

The use of consumer functioning to model placement decisions

Mental health systems are regularly confronted with the difficult problem of identifying appropriate community settings for individuals with mental illness. The author describes a study that determined if consumer functioning could effectively model placement decisions. Results suggest consistency in the way functioning is related to placement issues and indicate that the approach may be applied across systems. Implications of the relevance of functioning for community placement are also discussed.

Broadening measures of success: results of a behavioral health translational research training program

BackgroundWhile some research training programs have considered the importance of mentoring in inspiring professionals to engage in translational research, most evaluations emphasize outcomes specific to academic productivity as primary measures of training program success. The impact of such training or mentoring programs on stakeholders and local community organizations engaged in translational research efforts has received little attention. The purpose of this evaluation is to explore outcomes other than traditional academic productivity in a translational research graduate certificate program designed to pair graduate students and behavioral health professionals in collaborative service-learning projects.MethodsSemi-structured qualitative interviews with scholars, community mentors, and academic mentors were conducted regarding a translational research program to identify programmatic impacts. Interviews were transcribed and coded by the research team to identify salient themes related to programmatic outcomes.ResultsResults are framed using the Translational Research Impact Scale which is organized into three overarching domains of potential impact: (1) research-related impacts, (2) translational impacts, and (3) societal impacts. This evaluation demonstrates the program’s impact in all three domains of the TRIS evaluation framework. Graduate certificate participants (scholars) reported that gaining experience in applied behavioral health settings added useful skills and expertise to their present careers and increased their interest in pursuing translational research. Scholars also described benefits resulting from networks gained through participation in the program, including valuable ties between the university and community behavioral health organizations.ConclusionsThis evaluation of the outcomes of a graduate certificate program providing training in translational research highlights the need for more community-oriented and practice-based measures of success. Encouraging practitioner involvement in translational research is vital to translate knowledge into practice and to enable practice-based needs to inform research and policy. A more flexible approach to measuring programmatic success in research training programs can help bridge the knowledge translation gap.

Issues in the assessment of client level of functioning in mental health: An introduction

This special section of the Journal of Mental Health Administration is a collection of papers which deal with the use of level of functioning measures in mental health settings. These papers offer examples of efforts to come to grips with the difficult task of assessing the abilities of clients in the service delivery system. The importance of assessing client functioning rests with the need for adequately documenting the basis for the provision of services, the degree to which the client improves over time and the effectiveness of interventions. In addition to the significant individual contribution each paper makes to our understanding of functional assessment, together the five papers in this section serve two additional purposes. First, the papers illustrate the broad applicability of level of functioning within the mental health system. Functional assessments not only offer a way of characterizing the residual disability which may result from an individuals mental illness but also offer insight regarding the effectiveness of services and a prescription of the interventions which may be required to return the individual to an active position in the community. Functional assessments can also be used to help determine the availability and organization of services. On a less ambitious note, level of functioning information offers a description of those who seek services. One of the simplest values of level of functioning information may be to identify, for public planning and policy-making, who constitutes the chronically mentally ill population. Each of these applications is critically important for meaningful and cost-effective delivery of services. Second, the papers illustrate the broad approaches used in assessing functioning in mental health systems. The divergent needs for information result in very broad interpretations of the standard for level of functioning instruments. Mental health systems have relied on level of functional assessments which are based on single questions, rating the client from one to seven, one to nine, or zero to 90, as well as instruments composed of 80 or more questions covering a dozen or more dimensions. Differences in instrumentation are due not only to differences in the purpose of the assessment but also to more pragmatic concerns such as the systems capacity to develop instrumentation and to collect, analyze and disseminate information in a timely fashion. Among the instruments referred to in this section, readers may be most familiar with the simple oneitem summary measures of functioning. These measures have been used to describe client populations and determine change over time and are sometimes proposed for use in determining the chronic mentally ill population for purposes of P.L. 99-660. A well-kn0wn example of a one-question scale is the Global Assessment Scale.~ For readers who use or have considered using these summary measures, the paper by Pokorny 2 will be of greatest relevance. Pokornys paper describes summary measures of functioning in state mental health authorities and offers needed recommendations and warnings for their use. These global measures have been found to be very appealing to mental health providers because of