Charalambos Y. Charalambous

When Rater Reliability Is Not Enough Teacher Observation Systems and a Case for the Generalizability Study

In recent years, interest has grown in using classroom observation as a means to several ends, including teacher development, teacher evaluation, and impact evaluation of classroom-based interventions. Although education practitioners and researchers have developed numerous observational instruments for these purposes, many developers fail to specify important criteria regarding instrument use. In this article, the authors argue that for classroom observation to succeed in its aims, improved observational systems must be developed. These systems should include not only observational instruments but also scoring designs capable of producing reliable and cost-efficient scores and processes for rater recruitment, training, and certification. To illustrate how such a system might be developed and improved, the authors provide an empirical example that applies generalizability theory to data from a mathematics observational instrument.

A Comparative Analysis of the Addition and Subtraction of Fractions in Textbooks from Three Countries

In this paper, we report on a comparison of the treatment of addition and subtraction of fractions in primary mathematics textbooks used in Cyprus, Ireland, and Taiwan. To this end, we use a framework specifically developed to investigate the learning opportunities afforded by the textbooks, particularly with respect to the presentation of the content and the textbook expectations as manifested in the associated tasks. We found several similarities and differences among the textbooks regarding the topics included and their sequencing, the constructs of fractions, the worked examples, the cognitive demands of the tasks, and the types of responses required of students. The findings emphasized the need to examine textbooks in order to understand differences in instruction and achievement across countries. Indeed, we postulate that within a given country there may exist a recognizable “textbook signature.” We also draw on the results and the challenges inherent in our analysis to provide suggestions and directions for future textbook analysis studies.

Teacher Knowledge, Curriculum Materials, and Quality of Instruction: Unpacking a Complex Relationship.

The set of papers presented in this issue comprise a multiple-case study which attends to instructional resources—teacher knowledge and curriculum materials—to understand how they individually and jointly contribute to instructional quality. We approach this inquiry by comparing lessons taught by teachers with differing mathematical knowledge for teaching who were using either the same or different editions of a US Standards-based curriculum. This introductory paper situates the work reported in the next four case-study papers by outlining the analytic framework guiding the exploration and detailing the methods for addressing the research questions.

Validating Arguments for Observational Instruments: Attending to Multiple Sources of Variation

Measurement scholars have recently constructed validity arguments in support of a variety of educational assessments, including classroom observation instruments. In this article, we note that users must examine the robustness of validity arguments to variation in the implementation of these instruments. We illustrate how such an analysis might be used to assess a validity argument constructed for the Mathematical Quality of Instruction instrument, focusing in particular on the effects of varying the rater pool, subject matter content, observation procedure, and district context. Variation in the subject matter content of lessons did not affect rater agreement with master scores, but the evaluation of other portions of the validity argument varied according to the composition of the rater pool, observation procedure, and district context. These results demonstrate the need for conducting such analyses, especially for classroom observation instruments that are subject to multiple sources of variation.

Using educational effectiveness research to design international comparative studies: turning limitations into new perspectives

This paper argues that Educational Effectiveness Research (EER) could be a foundation upon which studies of the International Association of the Evaluation of Education (IEA) could be based, both in terms of their design and in terms of data analysis. First, it is supported that multilevel modelling techniques, some of the most frequently used statistical techniques in the domain of EER, offer a wider spectrum in analysing IEA data. An illustrative example is used to provide support for this argument. Specifically, TIMSS 1999 data concerning student self-perceptions and achievement in both mathematics and science were analysed using single- and multilevel analyses. While single-level analysis resulted in positive correlations at the micro-level and negative correlations at the macro-level, the findings of multilevel analysis did not support such contradictious relationships. Second, the paper argues that the design of international comparative studies could be based on the theoretical models of EER that attempt to explain the variation in student achievement by taking into account variables at different levels. To support this argument a large number of variables taken from the student, teacher and school questionnaires of TIMSS 1999 associated with the comprehensive model of educational effectiveness were added to a multilevel model. Implications for the design of comparative studies are drawn.

Attending to General and Mathematics-Specific Dimensions of Teaching: Exploring Factors Across Two Observation Instruments

ABSTRACT New systems that seek to evaluate teachers with regard to their classroom quality often rely on observation instruments that capture general instructional pedagogies. However, decades of research suggest that content-specific dimensions of instruction also are important to differentiate teachers and improve student outcomes. We explore the degree of overlap between a general and a content-specific instrument when capturing upper elementary teachers’ mathematics instruction. To do so, we conducted exploratory and confirmatory factor analyses on data from more than 2,000 videotaped lessons scored using both the Classroom Assessment Scoring System, a general instrument, and the Mathematical Quality of Instruction, a content-specific instrument. Findings indicate that there is some overlap between instruments but that preferred factor structures include both general and content-specific practices.

Working at the Nexus of Generic and Content-Specific Teaching Practices: An Exploratory Study Based on TIMSS Secondary Analyses

For years scholars have attended to either generic or content-specific teaching practices attempting to understand instructional quality and its effects on student learning. Drawing on the TIMSS 2007 and 2011 databases, this exploratory study empirically tests the hypothesis that attending to both types of practices can help better explain student performance. Capitalizing on one generic and one content-specific framework, we tested this hypothesis for grade 8 mathematics. The analyses for the entire sample showed that more variance was explained when considering both types of practices in tandem than that explained when considering each type in isolation. Important insights were also gleaned when testing this hypothesis for each individual country, since notable differences were observed across participating countries. In discussing the study findings, which were relatively consistent across the two TIMSS cycles, we draw implications for the development of theoretical frameworks capturing instructional quality and the design of international comparative studies.

Exploring the reliability of generic and content-specific instructional aspects in physical education lessons

ABSTRACT Heightened accountability pressures and an increased emphasis on teaching quality have directed scholarly attention to scrutinizing instruction, particularly with respect to issues of validity and reliability. However, these attempts have largely been directed toward “core” content areas and investigated generic or content-specific instructional aspects separately. Focusing on a less explored area, physical education, and concurrently attending to both instructional aspects, in this exploratory study, we examined whether the optimal lesson-rater combination needed to obtain reliable teaching-quality estimates differs depending on the type of instructional aspects considered. Data analysis of 147 lessons using generalizability theory suggested that either a 3-lesson-2-rater or a 4-lesson-1-rater combination yields sufficiently reliable estimates for nearly all dimensions examined (generic or content specific). Quality of student practice, however, required 11 lessons scored by 2 raters. These findings underline the importance of examining individual dimensions within a given observational instrument and the merit of carefully selecting and training raters.

Investigating the Knowledge Needed for Teaching Mathematics An Exploratory Validation Study Focusing on Teaching Practices

Central in the frameworks proposed to capture the knowledge needed for teaching mathematics is the assumption that teachers need more than pure subject-matter knowledge. Validation studies exploring this assumption by recruiting contrasting populations are relatively scarce. Drawing on a sample of 644 Greek-Cypriots preservice and inservice elementary school teachers and university students with strong mathematical background, this study explored participants’ knowledge with respect to four teaching practices: providing and evaluating explanations; selecting and using representations; analyzing student errors, misconceptions, and non-conventional solutions; and selecting tasks. Results emerging from an item response theory (IRT) model and non-parametric tests showed significant differences among these groups, largely in the first two practices. Clinical interviews with a voluntary sub-sample of these participants provided insights into the unexpected findings related to the other two practices. The theoretical, methodological, and practical implications of these findings are discussed, and directions for future studies are offered.

The Impact of Stress on Pressure Ulcer Wound Healing Process and on the Psychophysiological Environment of the Individual Suffering from them

Introduction: The occurrence of a pressure ulcer constitutes a major psychological and physiological burden and it has been linked with a reduced quality of life and increased stress of the individual. Objective: The main objective was to investigate the impact that stress has on pressure ulcer healing process and on the psychophysiological environment of the individual suffering from them. Method: The scientific literature was reviewed through Cinahl, Pub-med, EBSCO, Medline and Google scholar. The articles were chosen due to their direct correlation with the objective under study and their scientific relevance. Results: Increased stress has been demonstrated to increase the glucocorticoids levels affecting negatively the production of wound healing cytokines (IL1α, IL1β and TNFα). Matrix metalloproteases has been identified to be unregulated in occasions of increased stress in acute wounds. Stress has also been correlated with poor health behaviors that may not have a direct link on the wound healing process, although they can in part explain or enhance some of the effects of stress on wound healing. Conclusion: The correlation between stress and wound healing in acute wounds has been thoroughly investigated and its negative effects have been established. The presence of a pressure ulcer can have a detrimental impact on the stress level of an individual although further investigation is needed to establish the role of stress in chronic wounds such as pressure ulcers.