R. Ayres

Getting to the Bottom Line: A Method for Synthesizing Findings Within Mixed-method Program Evaluations

Evaluators who are concerned more with pragmatics than with competing epistemologies have brought multi- and mixed-method evaluations into common practice. Program evaluators commonly use multiple methods and mixed data to capture both the breadth and depth of information pertaining to the evaluand, and to strengthen the validity of findings. However, multiple or mixed methods may yield incongruent results, and evaluators may find themselves reporting seemingly conflicting findings to program staff, policy makers, and other stakeholders. Our purpose is to offer a method for synthesizing findings within multi- or mixed-method evaluations to reach defensible evaluation (primarily summative) conclusions. The proposed method uses a set of criteria and analytic techniques to assess the worth of each data source or type and to establish what each says about program effect. Once on a common scale, simple math allows synthesis across data sources or types. The method should prove a useful tool for evaluators.

Quest for Quality: Recruitment, Retention, Professional Development, and Performance Evaluation of Teachers and Principals in Baltimore City's Public Schools.

This article rests in the context of the 2001 evaluation of Baltimore City Public School System’s (BCPSS) 5-year-old reform effort, conducted by Westat and its collaborators.2 In that context, the article describes findings on key human resource issues for the BCPSS. Briefly, the part of the evaluation on which this article is based was charged with appraising the success of efforts to improve the quality, stability, and effectiveness of the BCPSS instructional workforce (teachers and principals). The overall design of the evaluation, as well as findings in other key areas of the school system reform effort (e.g., governance and management, instruc-tional interventions, achievement outcomes) are described in the other articles that comprise this special issue. Four areas of the human resource effort are examined for both teachers and principals in the Baltimore City Public School System (BCPSS): (a) recruitment policies and practices, (b) mentoring and other retention strategies, (c) professional development, and (d) performance evaluation. Four methods of data gathering were used: one quantitative (survey), and three qualitative (document review, key informant interview, and focus group interview). These have been described elsewhere in this special issue, and will not be further detailed here.