Rob Cavanagh

A turn toward specifying validity criteria in the measurement of Technological Pedagogical Content Knowledge (TPACK)

Abstract The impetus for this paper stems from a concern about directions and progress in the measurement of the Technological Pedagogical Content Knowledge (TPACK) framework for effective technology integration. In this paper, we develop the rationale for using a seven-criterion lens, based upon contemporary validity theory, for critiquing empirical investigations and measurements using the TPACK framework. This proposed seven-criterion lens may help researchers map out measurement principles and techniques that ensure reliable and valid measurement in TPACK research. Our critique of existing TPACK research using these criteria as a frame suggests several areas of theorizing and practice that are likely impeding the press for measurement. First are contradictions and confusion about the epistemology of TPACK. Second is the lack of clarity about the purpose of TPACK measurement. Third is the choice and use of measurement models and techniques. This article illustrates these limitations with examples from current TPACK and measurement-based research and discusses directions and guidelines for further research.

Pedagogic principal leadership

T he effectiveness of schools in educating students is highly dependent upon the nature of leadership within the individual school. While principals are formally required to lead the school, leadership is not the sole province of the principalship. Indeed, most schools are characterised by a combination of formal and informal leadership as evidenced by teachers assuming responsibility for particular tasks and programmes. Although the leadership of schools is a complex phenomenon, the outcomes of successful school leadership are readily identifiable. These outcomes centre upon the quality of pedagogy provided by teachers and the engagement of students in learning. However, pedagogic change is difficult (Planning & Evaluation Service 2000) and, as Stigler and Hiebert (1999) noted, teachers tend to replicate the culture and pedagogy of their personal experiences at school as students. So the challenge for school leaders centres upon working with teachers to transform classroom pedagogy. The following discussion is a synthesis of literature and research into school leadership and changing teachers’ pedagogic practices. In particular, it identifies the key factors in bringing about pedagogic change by concluding with proposal of a conceptual framework of pedagogic leadership.

Establishing the Validity of Rating Scale Instrumentation in Learning Environment Investigations

Rating scale instruments have been widely used in learning environment research for many decades. Arguments for their sustained use require provision of evidence commensurate with contemporary validity theory. The multiple-type conception of validity (e.g. content, criterion and construct), that persisted until the 1980s was subsumed into a unified view by Messick. He re-conceptualised types of validity as aspects of evidence for an overall judgment about construct validity. A validity argument relies on multiple forms of evidence. For example, the content, substantive, structural, generalisability aspect, external, and consequential aspects of validity evidence. The theoretical framework for the current study comprised these aspects of validity evidence with the addition of interpretability. The utility of this framework as a tool for examining validity issues in rating scale development and application was tested. An investigation into student engagement in classroom learning was examined to identify and assess aspects of validity evidence. The engagement investigation utilised a researcher-completed rating scale instrument comprising eleven items and a six-point scoring model. The Rasch Rating Scale model was used for scaling of data from 195 Western Australian secondary school students. Examples of most aspects of validity evidence were found, particularly in the statistical estimations and graphical displays generated by the Rasch model analysis. These are explained in relation to the unified theory of validity. The study is significant. It exemplifies contemporary validity theory in conjunction with modern measurement theory. It will be of interest to learning environment researchers using or considering using rating scale instruments.

The influence of students’ cognitive preferences on the selection of science and mathematics subjects

This study examined the subject‐choice process engaged in by a sample of Western Australian school students. The theoretical background was provided by career education models in which aspects of cognitive preferences are related to the choice of particular science and mathematics subjects. CareerMate, a computerised career counselling instrument, was used to identify cognitive preferences in the four dimensions of energy projection, experience preference, helping and closure. Aspects of the students’ preferences were correlated with their subject choices and gender and subject specific relationships emerged. There were significant differences in cognitive preferences between the academically capable females who chose matriculation physical science and mathematics subjects and those who did not. These differences need to be considered when encouraging females to select such subjects. If these subjects are to attract capable females currently choosing alternatives, then changes to the curriculum content an...

The possible misfit of Csikszentmihalyi’s dimensions of flow in the contemporary roles of school leaders

In many jurisdictions school leaders are being placed under increased accountability and stress, which then affects their ability to address the real issue of education − improving students’ learning. Flow Theory, developed by the Hungarian−American psychologist Mihalyi Csikszentmihalyi, has a high degree of relevance to the issue of the way modern school leaders carry out their roles. A purposive sample of school leaders (N = 8) was interviewed about their Flow experiences, in both in-school and out-of-school situations. The nine dimensions of Flow (Jackson and Csikszentmihalyi, 1999) were used as a conceptual guide in the e-interviews. Each of the school leaders gave powerful descriptions of their memorable out-of-school deep-Flow experiences, but their in-school experiences of Flow appeared to have far less impact. The data analysis showed that with this sample of school leaders only four of Jackson and Csikszentmihalyi’s nine dimensions of Flow were identifiable in their in-school experiences. The misfit of Csikszentmihalyi’s dimensions of Flow is important, and needs re-interrogation in future research. Importantly, the moral dimension of the school leaders’ job was identified by the respondents as the most important facilitator of Flow in both public and private schools, and this may provide the key to improved school leader resilience and motivation.

The Engagement in Classroom Learning of Year 10 and 11 Western Australian Students

The consideration of issues related to student engagement in classroom learning has taken on increasing importance in Western Australia since the passing of legislation to raise the school leaving age to 17 years, which came into effect in 2008. There are now more students retained at schools in Years 11 and 12 than previously. Engaging these students in learning is of the upmost importance for secondary schools. This paper presents a hypothesised model of student engagement in classroom learning that is based on the principles of Flow Theory (i.e. a person achieves a state of flow when there is a match in high skills and high challenges). The hypothesised model proposes that student engagement occurs when there is a balance between student learning capabilities (skills) and the expectations of student learning (challenges). Each of these comprised sub-constructs, of which there were 11 in total. The research sought to determine which of the 11 subconstructs that comprise the student engagement in classroom learning were the most difficult and which were easier to identify in Year 10 and 11 students. It also sought to determine whether membership of different groups of students accounted for variance in the calibrated scores (these groups being gender; school year; subject; and whether it was a favourite or least favourite subject). The sample comprised 112 Year 10 and 11 students from metropolitan and rural government schools in Western Australia. Each student was assigned a rating from zero to five by two researchers on each of the 11 sub-constructs. The Rasch Rating Scale Model was used for analysis of the quantitative data. Firstly, the raters experienced differing levels of difficulty in identifying the respective sub-constructs in the students. That is, the 11 items in the instrument presented varying levels of difficulty of affirmation. Secondly, the engagement scores differed by gender (boys displaying lower levels of engagement) and whether favourite or least favourite subject was reported (favourite subjects displaying higher levels of engagement). The year of schooling of the student and the subject area (e.g. English, Mathematics, Science, and Society and Environment) did not account for variance in engagement scores. The implications of these findings are discussed.

Measurement as a Medium for Communication and Social Action I: A Phenomenological View of Science and Society

Two sequential papers were written in response to the assertion that Measurement is a Medium for Communication and Social Action. The first provides a Phenomenological View of Science and Society, the second explores the Promise and Power of being Amodern. Paper One commences by outlining examples of how phenomenological constructs are relevant to understanding measurement, communication, the human condition, and in general, science and society. The subsequent detailed critique of classical and modern phenomenological concepts applies an analytic frame comprising four characteristics of phenomenology: Back to the Things Themselves; Authentic Method; Unity of Subject and Object; and The World of the Text. Each is examined in relation to the incumbent philosophical considerations along with the consequences for reconceptualizing science. The paper concludes with turn toward an unmodern or amodern frame of reference based on this reconceptualization.

Differences between Mothers' and Fathers' Ratings of Family Functioning with the Family Assessment Device: The Validity of Combined Parent Scores.

The psychometric properties of the General Functioning subscale of the McMaster Family Assessment Device were examined using the Rasch Model (N = 237 couples). Mothers’ and fathers’ ratings of the General Functioning subscale of the McMaster Family Assessment Device are recommended, provided these are analyzed separately. More than a quarter of couples differed significantly in their ratings.

An investigation of differential need for psychological services across learning environments

The role of school psychologists in Western Australia has been reviewed a number of times since the establishment of services to schools. Current practice whereby school psychologist allocation to a range of schools is achieved, continues to rely on school student population figures, its socioeconomic index and an appraisal of the school’s ‘difficulty’ level. Psychological services are then allocated accordingly, the decision-making mechanism based on an ad hoc conception of school need. The research reported in this paper concentrates on the issue of trying to establish what aspects or characteristics of learning environments constitute a greater or esser level of need for services and then attempts to measure this need in an objective evidence-based manner. The various elements of school need for psychological services are posited to cluster around constructs extrapolated from the domains of service reported in the international professional literature. These are characteristics of students, characteristics of schools and teacher expertise. The three constructs constitute the preliminary conceptual framework for the study upon which the empirical investigation was based. The study was conducted in three phases: First, item development, theoretical framework refinement utilising data collected from a questionnaire; second, development of a pool of appropriate items, piloting and trialling; and third, utilising the refined linear scale to measure a sample of schools need for psychological services. Data were obtained from samples of principals, teachers and school psychologists working in two Department of Education and Training (DET) school districts. Data analysis employed the Rasch Rating Scale Model and Analyses of Variance. Data fitting the model confirmed that a uni-dimensional trait was measured. Data-to-model fit was estimated by item difficulty thresholds, individual item fit statistics, the person Separation Index and Principal Components Factors loadings of residuals. The results demonstrated that the linear scale instrument developed in the research provided an authentic measure of school need and that the measures of the phase three schools differed significantly from each other. The empirical findings of the study are discussed in the context of their application in informing decisions about the level of psychological services that should be provided to schools congruent with the psychological needs of their students.

Secondary School Students’ Engagement in Learning Japanese as a Second Language

The learning of Asian languages is a significant feature of national and state education policies. For example, the multi-million dollar National Asian Languages and Studies in Schools Program which was designed to increase participation and engagement in learning Asian languages. While much of the impetus for this press is due to international trade and economic priorities, the curriculum area of languages other than English is also important for educative and cultural reasons. Of the four Asian languages typically taught in local schools (Indonesian, Japanese Korean and Mandarin), Japanese has the highest enrolment. The research reported in this paper focussed on the engagement of Western Australian secondary school students in their classroom learning of Japanese. While the study of second language instruction and teaching is situated within the field of second language acquisition, it also applies conventional educational theory. For example, cognitive, meta-cognitive and socio-affective constructs. This similarity is reflected in the model of student engagement that informed instrumentation decisions in the investigation of local Japanese classroom learning. Engagement was conceptualised as a function of student capability for learning and the expectations placed on this learning. Capability was defined in terms of self-esteem, self-concept, resilience, self-regulation and self-efficacy. Expectations were defined as facets of learning for understanding – expectations of explanation, interpretation, application and having perspective, empathy and self-knowledge. A self-report instrument was administered to 278 Year 8 to twelve students. The instrument comprised 50 statements about attributes of students and their learning. Students responded on a four-category response scale. The data were tested against the Rasch rating Scale Model. Data fitting the model shows a unidimensional trait was measured and the measure was invariant. Data-to-model fit was assessed by estimation of item difficulty thresholds, individual item fit statistics, the Person Separation Index and Principal Components Factor loadings of residuals. The difficulty students had in affirming individual statements and groups of statements were also estimated to indicate common and less common perceptions of Japanese classroom learning.