Paul R. Brandon

Stakeholder Participation for the Purpose of Helping Ensure Evaluation Validity: Bridging the Gap Between Collaborative and Non-collaborative Evaluations:

Collaborative evaluations, in which program stakeholders participate extensively, typically are conducted for the primary purpose of enhancing the use of evaluation findings, and non-collaborative evaluations, in which stakeholders do not participate extensively, typically are conducted for the primary purpose of generating valid findings. This article shows how the gap can be bridged between these two types of evaluations. The article synthesizes, and elaborates on, a small body of recent research that showed that considerable interaction with stakeholders during the evaluation of small education programs helped achieve the raison d’etre of non-collaborative evaluations-that is, to enhance validity. It is also shown that theories and methods of stakeholder participation for the purpose of enhancing validity, which by and large have been ignored in the collaborative evaluation literature, can improve the quality of collaborative studies.

Children’s Mathematics Achievement in Hawaii: Sex Differences Favoring Girls

Most reviews of the literature have concluded that boys have surpassed girls in mathematics achievement at some point in their schooling. In Hawaii, however, some studies have suggested that sex differences in mathematics achievement have favored girls. In this paper, norm-referenced mathematics achievement test results for Hawaii public school students in four ethnic groups and four grades are examined for sex differences. Results are examined for sex differences varying by mathematical skill, among high-achieving students, and among ethnic groups. The findings show that the Hawaii public school girls have higher achievement levels than the boys. The boys achieve their highest scores in mathematical reasoning, and the girls achieve their highest scores in computation. High-achieving girls outperform high-achieving boys. Sex differences favoring girls among Caucasian students are less than they are among Japanese-American, Filipino-American, and Hawaiian students. The relationship of the sex differences to sociocultural factors is examined, and the practical consequences of the differences are discussed.

Gender Differences in Young Asian Americans' Educational Attainments.

In this paper, data from the 1986 follow-up of a national probability sample of 1980 high school seniors are examined: (a) to determine whether gender differences exist in the educational attainment of Asian-American youth and compare these differences with those found among other groups (Blacks, Hispanics, Native Americans, and Whites), (b) to confirm that immigrant status and ethnicity account for Asian-American mens and womens educational attainment (as suggested by the literature), and (c) to determine if immigrant status and ethnicity are differentially related to the mens and womens educational attainment. The findings suggest that young Asian-American females reach high levels of educational attainment more quickly than young Asian-American males, and that the differences are particularly noteworthy among immigrants (or the children of immigrants) and Chinese Americans.

The Strength of the Methodological Warrants for the Findings of Research on Program Evaluation Use.

Considerable research has been conducted on the use of the findings of program evaluation, but little, if any, attention has been paid to the soundness of the methods of this research. If the methods are not sound or not well described in the research, the strength of the conclusions of the research is unknown. The authors examine the empirical studies reported in five widely cited reviews of the literature on program evaluation use and summarize the types of methods used, the professions for which they were conducted, and the extent to which they met two basic criteria for soundness. It was concluded that, by and large, the body of studies does not adequately address the criteria. With care, the findings of the research literature on the use of evaluations might provide guidance for evaluators, but they lack sufficient scientific credibility for other uses.

The development and validation of The Inquiry Science Observation Coding Sheet

Evaluation reports increasingly document the degree of program implementation, particularly the extent to which programs adhere to prescribed steps and procedures. Many reports are cursory, however, and few, if any, fully portray the long and winding path taken when developing evaluation instruments, particularly observation instruments. In this article, we describe the development of an observational method for evaluating the degree to which K-12 inquiry science programs are implemented, including the many steps and decisions that occurred during the development, and present evidence for the reliability and validity of the data that we collected with the instrument. The article introduces a method for measuring the adherence of inquiry science implementation and gives evaluators a full picture of what they might expect when developing observation instruments for assessing the degree of program implementation.

Involving Program Beneficiaries in the Early Stages of Evaluation: Issues of Consequential Validity and Influence.

Program decision makers are typically involved in many steps of participatory evaluations, including the specification of program attributes that evaluations address. Program beneficiaries should also participate in this early step, however, because their contributions broaden the scope of evaluations, thereby enhancing the consequential validity of findings. In this article, we discuss an approach for involving beneficiaries in specifying program attributes for evaluations to address. Methods are outlined for sharing influence among program decision makers and beneficiaries, with decision makers retaining the authority to make final decisions. We describe an evaluation in which this approach was applied, present evidence on how the approach enhanced consequential validity, and discuss the constraints on its use.

Teacher Involvement in School-Conducted Needs Assessments Issues of Decision-Making Process and Validity

Little attention in the literature on stakeholder shared decision making in program evaluations has been given to involving stakeholders in developing the process of decision making. If stakeholders are not fully involved in developing this process, decisions may be biased and lead to invalid evaluation findings. In this article, the authors present a theoretical model suggesting how stakeholder involvement in establishing the process of decision making in school-conducted needs assessments affects the validity of the selected needs, with validity defined as teacher agreement with the selection of needs. Using LISREL VII, the model was tested using teacher survey data. The findings show that teachers were not heavily involved in developing decision- making processes. However, they agreed with the selected needs, and their involvement in de veloping the decision-making processes accounted for a substantial portion of their agreement.

SAS Code for Calculating Intraclass Correlation Coefficients and Effect Size Benchmarks for Site-Randomized Education Experiments.

When evaluators plan site-randomized experiments, they must conduct the appropriate statistical power analyses. These analyses are most likely to be valid when they are based on data from the jurisdictions in which the studies are to be conducted. In this method note, we provide software code, in the form of a SAS macro, for producing statistical power analyses using such data. The macro is designed to estimate statistical parameters for multiple school subjects and multiple grades in a single computer run. Among other convenient features, the macro calculates intraclass correlation coefficient confidence intervals, which are essential for planning studies conservatively. We also provide a SAS macro for calculating benchmarks–again, using locally-based–data for interpreting the feasibility of achieving the minimum detectable effect sizes that are calculated in statistical power analyses. The benchmarks are potentially useful for interpreting the meaning of the effect sizes that are achieved in completed studies, as well. We describe the macros, describe how we verified their accuracy, show how they can be useful to education evaluators, and give examples of their use with statewide educational assessment data. It is our hope that the macros will help evaluators use local data when conducting group-randomized studies.

Evaluators' Use of Peer Debriefing: Three Impressionist Tales:

Program evaluators are sometimes requested to help their clients identify ways to use evaluation findings, but little guidance for this task has been provided in the literature. How might evaluators grapple with this dilemma? This article discusses the use of peer debriefing as a method for helping evaluators during the final stage of the evaluation of a curriculum in a school of a large state university and presents the implications of these findings for program evaluators and qualitative researchers. The authors employ an impressionist tale format to discuss the impact of the peer debriefing process on the peer debriefer, the principal investigator, and the project director. The method is found to be helpful to evaluators who face issues of roles, the use of expertise, and interpersonal dynamics.

Teacher empowerment and the implementation of school‐based reform

As increasing concern has been given to reshaping teaching and learning processes in schools, empowering teachers has become an important means of implementing school reform. The focus of this research was to investigate how the purposeful reform of school decision‐making responsibilities affects teacher participation and leadership in selecting critical needs to address during the school improvement process. Results indicate that involvement in the process of setting up decision making and selecting the content of school needs affect teachers′ agreement with selected needs. Teacher expertise and leadership opportunities were found to affect their participation in school decision making. Discusses results in terms of their implications for school reform.