Andrew Maul

Rethinking Traditional Methods of Survey Validation

ABSTRACT It is commonly believed that self-report, survey-based instruments can be used to measure a wide range of psychological attributes, such as self-control, growth mindsets, and grit. Increasingly, such instruments are being used not only for basic research but also for supporting decisions regarding educational policy and accountability. The validity of such instruments is typically investigated using a classic set of methods, including the examination of reliability coefficients, factor or principal components analyses, and correlations between scores on the instrument and other variables. However, these techniques may fall short of providing the kinds of rigorous, potentially falsifying tests of relevant hypotheses commonly expected in scientific research. This point is illustrated via a series of studies in which respondents were presented with survey items deliberately constructed to be uninterpretable, but the application of the aforementioned validation procedures nonetheless returned favorable-appearing results. In part, this disconnect may be traceable to the way in which operationalist modes of thinking in the social sciences have reinforced the perception that attributes do not need to be defined independently of particular sets of testing operations. It is argued that affairs might be improved via greater attention to the manner in which definitions of psychological attributes are articulated and greater openness to treating beliefs about the existence and measurability of psychological attributes as hypotheses rather than assumptions—in other words, as beliefs potentially subject to revision.

As Pragmatic as Theft Over Honest Toil: Disentangling Pragmatism From Operationalism

“The uniformity of weights and measures cannot displease anyone but those lawyers who fear a diminution in the number of trials, and those merchants who fear anything that renders the operations of commerce easy and simple” (Condorcet, as cited in Hand, 2016, p. 6) The concept of pragmatism has come to garner widespread appeal over the past century, particularly in the context of methodological innovations in the social and psychological sciences. In addition to connoting an orientation toward practicality and real-world usefulness—and who among us would not want to claim such things—pragmatism is often used to suggest freedom from slavish devotion to abstract principles or theoretical concerns. The assumption of this freedom has led to a diversification of activities being referred to as measurement across different scientific disciplines and other areas of human activity. Many of these activities bear little resemblance to canonical instances of measurement in the physical sciences (e.g., the measurement of length or temperature). Pragmatism, then, may seem to offer an alternative method of justification for the dependability and usefulness of the knowledge acquired as a result of these more liberally–defined measurement activities. Of course, whether pragmatism actually succeeds in offering such a justification is another matter. As a philosophical approach to truth and knowledge, the Pragmatist school may have much to offer to measurement theorists. However, in many contexts the idea of pragmatism is invoked but not carefully defined. One obvious danger resulting from this is that different users of the term will have different intuitions about what it means, raising the possibility that the term is used in a manner that is explanatorily soothing but semantically and scientifically vacuous. This vacuum may be easily filled by anti-realist positions that ultimately serve no pragmatic purpose other than to render the operations of measurement deceptively easy and simplistic.

A meta-structural understanding of measurement

It is not always clear to what extent the logic and vocabulary of measurement as used in different scientific disciplines are mutually coherent, nor how measurement can be demarcated from, say, opinion. In recent decades there have been a number of attempts to provide necessary and/or sufficient sets of conditions for when measurement is achieved, usually in terms either of inputs (e.g., whether an evaluated property is a quantity), or outputs (e.g., whether a procedure assigns numbers according to a rule). We argue instead that the public trust attributed to measurement is best justified in terms of the structural features of the process rather than of its inputs or outputs.

Learning Progressions, Vertical Scales, and Testable Hypotheses: Promising Intuitions and Points for Clarification

Briggs and Peck (this issue) call for greater care in the conceptualization of the target attributes of students, or “what it is that is growing from grade to grade.” In particular, they argue that learning progressions can serve as valuable bases for the design of assessments and for the interpretation of assessment results, especially in terms of change over time (or “growth”). Against the backdrop of the history of science in general, it must be regarded as remarkable that Briggs and Peck’s argument—the core of which is simply that the meaningfulness of claims regarding changes in an attribute depends on clarity about the attribute itself—could be regarded as at all novel. But against the backdrop of (especially, but not exclusively, U.S.-based) educational assessment and psychometrics, in which technical concerns have arguably been foregrounded at the expense of substantive and conceptual concerns, their contribution is valuable and timely. However, I believe their points can be productively pushed even further. In particular, I argue that even greater care can and should be taken in attending to (a) the distinction between claims about the actual state of affairs in the world and claims about the ways in which we choose to model or represent such affairs via our concepts and language and, relatedly, (b) the distinction between elements of our belief systems we take to be immune to revision (i.e., as assumptions or stipulations) versus those taken to be open to revision (i.e., as hypotheses).

Moving beyond Traditional Methods of Survey Validation.

“This is more than the usual wish to transcend one’s predecessors, for it includes a rebellion against the philosophical impulse itself, which is felt as humiliating and unrealistic. It is natural to feel victimized by philosophy, but this particular defensive reaction goes too far. It is like the hatred of childhood and results in a vain effort to grow up too early, before one has gone through the essential formative confusions and exaggerated hopes that have to be experienced on the way to understanding anything. Philosophy is the childhood of the intellect, and a culture that tries to skip it will never grow up.” —Thomas Nagel (1989)

The building of knowledge, language, and decision-making about climate change science: a cross-national program for secondary students

ABSTRACT The United Nations’ declaration on climate change education in December 2014 has sparked a renewal of policies and programs initiated during the ‘Decade of Education for Sustainable Development’ (DESD, 2005–2014), aimed at promoting awareness, understanding, and civic action for environmental sustainability within learning communities all around the world. We present findings from a dialogic, multimodal, and literacies-based educational project designed to provide secondary students (N = 141) from four countries with the resources to read about and discuss evidence regarding climate change from seminal studies with peers and a core group of scientists (N = 7). Post-program interviews revealed a significant increase in language use related to evidence-based reasoning. Students also demonstrated an increased propensity to recycle. These findings support the hypothesis that providing opportunities for students to read and discuss seminal scientific sources incites positive changes in beliefs, attitudes, and behaviors related to climate change and climate science, and understandings of the nature of scientific evidence and argumentation.