Karen E. Kirkhart

Eyes on the Prize: Multicultural Validity and Evaluation Theory:

Examining evaluation theory in cultural context is an important component of evaluating theory; however, it is not the ultimate goal. The foundational element in good evaluation is validity, and appreciating the cultural location of evaluation theory is an important building block in the argument supporting multicultural validity. Multicultural validity requires congruence between theory and context. This article first addresses validity as the foundational aspiration of all evaluation, then moves to a succinct discussion of culture. Suggestions for reflecting on the cultural location of evaluation theory and cultural dimensions of context are described, illustrated by three different approaches to establishing cultural congruence—Indigenous Evaluation Framework (IEF), Talent Development (TD) Model, and Utilization-Focused Evaluation (UFE). Closing comments return to validity as the fundamental prize and to the centrality of culture in supporting it.

1994 Conference Theme: Evaluation and Social Justice Seeking Multicultural Validity: A Postcard from the Road

The main thesis of my Address today is that we as evaluators need to have a clearer understanding of the ways in which multicultural influences shape and are shaped by our work. The construct multicultural validity is proposed as the vehicle for organizing concerns about pluralism and diversity in evaluation, and as a way to reflect upon the cultural boundness of our work. I will propose that multicultural validity be conceptualized as a central dimension of validity, treated with the same respect, routinization, and scrutiny as other dimensions; that is, it should be a visible focus of concern in evaluation theory, methodology, practice, and metaevaluation. The subtitle is intended to reflect the fact that

Evaluation in the community mental health centers program: A bold new reproach?

The Federal Community Mental Health Centers Program (CMHC)-from 1963 to 1981-was heralded as a revolution in mental health care. Championed by many, and severely criticized by others, the actual impact of the program on the nations mental health remains unclear. The authorization to evaluate the CMHC Program came originally from congressional legislation (PL 90-174), and later from the policies and regulations of NIMH under a series of Federal laws, notably PL 94-63. From 1976-1980, two dominant evaluation strategies were prevalent: funds expended by NIMH each year for studies of CMHC services or program-wide evaluations, and a much larger expenditure by CMHCs to conduct their own, independent evaluations following federal guidelines. As the Centers Program was turned over to the states in the form of block grants (PL 97-35), a group of professionals involved with setting and carrying out federal CMHC evaluation policy of both varieties met in public forum to debate the impact of these two evaluation approaches. While some participants cited gains in evaluation technology and impact upon local management of CMHCs, others found the lack of a coordinated and systematic approach to evaluating the CMHC Program to have been an opportunity missed. The impact of CMHC evaluation efforts are also discussed in terms of their major contribution to the field of evaluation research as a whole.

Defining Evaluator Competencies: New Light on an Old Issue

This issue of EN focuses on the Standards for Evaluations of Educational Programs, Projects, and Products. In addition to the contributions enumerated by the invited reviewers, the Startdards offer a framework for understanding the skills and competencies needed to conduct program evaluations. Such an understanding is necessary to design appropriate evaluation training programs, to improve the quality and relevance of existing evaluation training programs, and to strengthen the evaluation profession. As Cronbach (1980) points out:

Analyzing mental health evaluation: : Moral and ethical dimensions

Now that evaluators have been sensitized to the importance of moral and ethical issues in their work, it is time to move beyond generalities and examine the moral and ethical implications of specific evaluation models in specific settings. This paper proposes a framework that can be used to examine moral and ethical dimensions of evaluation and illustrates it by analyzing a selected model of mental health evaluation. Such a systematic identification of moral issues can improve evaluation practice both proactively, by shaping the training of evaluators, and retrospectively, by contributing to meta-evaluation.

Assessing Evaluation Performance and Use Test-Retest Reliability of Standardized Instruments

Two studies were conducted to determine the stability of self-assessment of evaluation performance and use over a six-week period. Two instruments, the Evaluation Performance Questionnaire (EPQ) and the Evaluation Utilization Questionnaire (EUQ) were developed to assess internal evaluation of community mental health centers (CMHCs). One hundred one evaluators of CMHCs were surveyed in Study 1, and 175 directors and evaluators were surveyed in Study 2; response rates were 54% and 60%, respectively. Test-retest reliability coefftcients were calculated for total and subscale scores, and scale means were compared across time. Results show that the EPQ and EUQ provide reliable information about the level and scope of evaluation activity and the use of evaluation information within community mental health centers. The value of these tools for research, administration, evaluation, and practice is discussed

The Standards: Implications for Professional Licensure and Accreditation

Program evaluation faces an issue common to all emerging professions: how to assure the quality of its professional activities. The Joint Committee’s Standard A2 puts it succinctly: &dquo;The persons conducting the evaluation should be trustworthy and competent to perform the evaluation.&dquo; Two common approaches to assuring competence and, by implication, trustworthiness among members of a profession are licensure and accreditation. _ ’

Value bases of case management evaluation

Abstract Case management is designed to promote access to integrated, quality care for consumers, yet evaluations of case management show mixed results. This paper argues that part of the confusion stems from an unclarity about the value frames used to arrive at judgments of the merit or worth of case management. This paper reviews six value frames commonly applied to human service evaluation and examines the particular relevance of each perspective for the evaluation of case management; they are: consumer needs, consumer wants, professional standards, program goals and objectives, models of practice, and judged or believed standards of merit. Illustrative studies are cited to tie the conceptual framework to current literature and provide direction for future evaluations. The paper outlines a process for selecting value frames and links their selection to six broader issues in case management evaluation: evaluation design and methodology, values underlying programs, definition of the independent variable, political and organizational contexts of evaluation, resource constraints, and metaevaluation. In conclusion, the paper argues that client need should be a predominant value frame in evaluating case management.

Evaluation in Mental Health Centers Assessing the Hierarchical Model

This study presents a data-based examination of program evaluation in community mental health centers from two perspectives. First, survey data from 111 centers are analyzed according to a hierarchical model of evaluation performance proposed in previous research. Second, these data are reanalyzed using multidimensional scaling to examine empirical relationships among types of evaluation activities. Although evaluation patterns are consistent with the hierarchical model, the empirical model suggests important new relationships that refine prior conceptualizations. Results are discussed in the context of research on evaluation and professional issues for evaluators.

Information resources for program evaluators

Abstract Mechanisms for access are given regarding key information resources on the subject of program evaluation. Included are listings of important books, journals and newsletters; brief descriptions of the major professional societies in the program evaluation field (including membership information); a listing of resources for evaluation training materials; and a brief discussion of targeted research on evaluation that is contributing to the development of resource materials. The resources indentified are mainly intended for use by individuals conducting evaluation research studies, and for those engaging in training and research on program evaluation. A concluding section highlights some potentially useful information retrieval devices and strategies, and examines briefly future developments in program evaluation that may be related to resource materials now available or being created.