Pamela A. Moss

Can There Be Validity Without Reliability

Reliability has traditionally been taken for granted as a necessary but insufficient condition for validity in assessment use. My purpose in this article is to illuminate and challenge this presumption by exploring a dialectic between psychometric and hermeneutic approaches to drawing and warranting interpretations of human products or performances. Reliability, as it is typically defined and operationalized in the measurement literature (e.g., American Educational Research Association [AERA], American Psychological Association, & National Council on Measurement in Education, 1985; Feldt & Brennan, 1989), privileges standardized forms of assessment. By considering hermeneutic alternatives for serving the important epistemological and ethical purposes that reliability serves, we expand the range of viable high-stakes assessment practices to include those that honor the purposes that students bring to their work and the contextualized judgments of teachers.

Shifting Conceptions of Validity in Educational Measurement: Implications for Performance Assessment

Recent developments in the philosophy of validity, highlighting the importance of investigating the consequences of assessment use, provide theoretical support for the move toward performance assessment. The problem for validity researchers is finding the appropriate set of criteria and standards to simultaneously support the validity of an assessment-based interpretation and the validity of its impact on the educational system. My intent is to provide an integrative and critical review of the guidance available for conducting validity inquiry in the context of performance assessment. In the first section is a summary of the emerging consensus among measurement scholars—not reflected in the 1985 Standards for Educational and Psychological Testing (American Educational Research Association [AERA], American Psychological Association [APA], & National Council on Measurement in Education [NCME])—about the centrality of construct validity to the evaluation of any assessment-based interpretation and about the importance of expanding the concept of validity to include explicit consideration of the consequences of assessment use. The description of this emerging consensus suggests general epistemological principles for validity inquiry. In the second section is a description and synthesis of various categories of questions, evidence, or criteria that have been used to describe or guide validity inquiry, either for assessment in general or for performance assessment in particular. These analytic schemes, like the traditional construct-content-criterion categories, highlight specific issues that their authors consider important for validity researchers to address. Each balances technical concerns about such issues as reliability, generalizability, and comparability with concerns about the consequences of assessment. In the final section is an overview of concerns, expressed largely by interpretive researchers, about validity criteria that privilege standardized forms of assessment, whether performance-based or multiple-choice. These arguments suggest the importance of further expanding the conception of validity inquiry to treat as problematic the epistemological principles used to warrant validity conclusions.

Enlarging the Dialogue in Educational Measurement: Voices From Interpretive Research Traditions

In this article, I argue that the theory and practice of educational measurement would benefit by expanding the dialogue among measurement professionals to include voices from research traditions different from ours and from the communities we study and serve. To that end, I explore a critical dialectic between a naturalist conception of social science, from which the tradition of educational measurement has evolved, and an interpretive conception of social science. By moving between theory and concrete illustrations, I contrast the ways in which interpretations are typically constructed and grounded within the different traditions and consider the ethical and political consequences of these epistemological choices. This contrast among research traditions reflects an exercise in what Messick (1989) calls a “Singerian” mode of inquiry, where one method of inquiry is evaluated in terms of another to highlight the assumptions and values underlying each. The dialectic implied in such a reciprocal critique—particularly when undertaken with the respectful intent of comprehending the alternative perspective as fully as possible (Bernstein, 1992)—provides a generative model through which assessment theory and practice can evolve.

Collaboration as Dialogue: Teachers and Researchers Engaged in Conversation and Professional Development

This article, in both form and substance, seeks to reconceptualize the role of collaboration in professional development. The nature of collaboration and collaborative relationships in professional development research is reviewed, and an alternative conceptualization is offered. Because our work spans multiple sites, we are able to highlight the particularities of our experiences across sites and to contrast the individual stories of each teacher and researcher as opposed to essentializing our story into generalities about these groups. Writing about this work together required the construction of a narrative form that could incorporate and honor the voices of everyone. The story/data is presented as a Readers Theatre—a written script, based on meeting dialogues and interactions. This format highlights the problem of writing in collaborative research, as well as the differences in collaborative experiences among teachers and researchers. It also seeks to challenge traditional conceptions of the roles of teachers and researchers as theorizers about and disseminators of knowledge.

Assessment, equity, and opportunity to learn

Providing all students with a fair opportunity to learn (OTL) is perhaps the most pressing issue facing the U.S. education system. Moving beyond conventional notions of OTL – as access to content, often content tested; access to resources; or access to instructional processes – the authors reconceptualize OTL in terms of interaction among learners and elements of their learning environments. Drawing on sociocultural, sociological, psychometric, and legal perspectives, this book provides historical critique, theory and principles, and concrete examples of practice through which learning, teaching, and assessment can be re-envisioned to support fair OTL for all students. This book offers educators, researchers, and policy analysts new to sociocultural perspectives a readable and engaging introduction to fresh ideas for conceptualizing, enhancing, and assessing OTL; encourages those who already draw on sociocultural resources to focus attention on OTL and assessment; and nurtures collaboration among members of discourse communities who have rarely engaged one another’s work.

Learning From Our Differences: A Dialogue Across Perspectives on Quality in Education Research

The dialogue re-presented in this article is intended to foster mutual engagement—and opportunity for learning—across different perspectives on research within the education research community. Participants in the dialogue each addressed the following questions: (1) What are the touchstones by which you judge quality or rigor in education research (for a single study, a set of studies, or a “field” or community of researchers in dialogue)? What is your chief concern or fear that the touchstones guard against? (2) Where do you see challenges to your perspective in the perspectives of other members of the panel? How might your perspective evolve to respond to those challenges? Given all of this, what are the implications for the preparation of education researchers? Opening and closing comments set the dialogue in historical context, highlight issues raised, and suggest next steps for collaborative learning from the diversity of perspectives in our field.

Educational Standards, Assessment, and the Search for Consensus

In this article, we critically examine the nature of the “consensus” reflected in educational standards used to orient high-stakes assessment programs. We analyze two complementary cases of practice in the assessment of teaching. One focuses on the discourse of standards creation and one examines how standards like these are typically used to orient assessment development and judgments about individual performance. We offer two (partially competing) theoretical perspectives that might illuminate and guide our practices in this currently undertheorized and underexamined area of standards development. One is based in the discourse ethics of Jürgen Habermas and one is based in critical elaborations of Hans-Georg Gadamer’s philosophical hermeneutics. We argue that conventional consensus-seeking approaches to the development and public review of educational standards tend to mask diversity and relinquish authority for consequential decisions to assessment developers who work in far less public circumstances. We draw on hermeneutic philosophy to offer a more pluralist approach that allows dissensus to be represented and taken into account in the assessment process.

Opportunities to Learn in Practice and Identity

The theory of learning in many schools today is based on what I would call the “content fetish” (Gee 2004). The content fetish is the view that any academic area (whether physics, sociology, or history) is composed of a set of facts or a body of information and that the way learning should work is through teaching and testing such facts and information. However, for some current learning theorists, “know” is a verb before it is a noun, “knowledge” (Barsalou 1999a, 1999b; Bereiter and Scardamalia 1993; Clark 1997; Glenberg 1997; Glenberg and Robertson 1999; Lave and Wenger 1991; Rogoff 1990). Any actual domain of knowledge, academic or not, is first and foremost a set of activities (special ways of acting and interacting so as to produce and use knowledge) and experiences (special ways of seeing, valuing, and being in the world). Physicists do physics. They talk physics. And when they are being physicists, they see and value the world in a different way than do non-physicists. The same applies for good anthropologists, linguists, urban planners, army officers, doctors, artists, literary critics, historians, and so on (diSessa 2000; Lave 1996; Ochs, Gonzales, and Jacoby 1996; Shaffer 2004). Yet if much decontextualized, overt information and skill-and-drill on facts does not work as a theory of learning, neither does “anything goes,” “just turn learners loose in rich environments,” “no need for teachers” (Kirschner, Sweller, and Clark 2006). These are the progressive counterpart of the traditionalists’ skill-and-drill, and they, too, are problematic as a theory of learning. Learners are novices, and leaving them to float among rich experiences with no guidance only triggers human beings’ great penchant for finding creative but spurious patterns and generalizations that send them down

An Integrative Approach to Portfolio Evaluation for Teacher Licensure

The purpose of our overall research agenda is to develop and evaluate a methodology for the assessment of teachers in which experienced teachers, serving as judges, engage in dialogue to integrate multiple sources of evidence about a candidate to reach a sound conclusion. The project that provides the venue for this research agenda is the Interstate New Teacher Assessment and Support Consortium (INTASC), which is developing a portfolio assessment system to assist participating states in making a decision about teacher licensure. To develop the theoretical foundation necessary to support and evaluate such dialogic and integrative assessment practices, we turn, in part, to the tradition of philosophical hermeneutics, as a complement to psychometrics. In this article, we characterize and assess the processes in which judges, trying out an integrative approach to portfolio evaluation for the first time, engage as they collaboratively construct and document their conclusions, and we locate this work in the larger research agenda. The premise of this project, which is being carefully evaluated in the course of inquiry, is that these integrative practices cannot only lead to an epistemologically sound evaluation of teaching but also promote an ongoing professional dialogue of critical reflection on teaching practice.

Continuing the Dialogue on Collaboration

In this article, we seek to establish a dialogue with the authors of “The Challenge of Studying Collaboration” (John-Steiner, Weber, & Minnis, 1998), a piece written in response to our previous AERJ article (Clark et al., 1996). Although we agree with some parts of their critique, we resist some of their suggestions because of our differing sets of assumptions about the appropriate goals and criteria for scholarly inquiry. The dialectic between our differing approaches to studying, theorizing, and representing collaboration illuminates issues that we hope others may find useful in examining their own assumptions about the nature and scope of scholarly inquiry.