Joel R. Levin

Statistical Practices of Educational Researchers: An Analysis of their ANOVA, MANOVA, and ANCOVA Analyses

Articles published in several prominent educational journals were examined to investigate the use of data analysis tools by researchers in four research paradigms: between-subjects univariate designs, between-subjects multivariate designs, repeated measures designs, and covariance designs. In addition to examining specific details pertaining to the research design (e.g., sample size, group size equality/inequality) and methods employed for data analysis, the authors also catalogued whether (a) validity assumptions were examined, (b) effect size indices were reported, (c) sample sizes were selected on the basis of power considerations, and (d) appropriate textbooks and/or articles were cited to communicate the nature of the analyses that were performed. The present analyses imply that researchers rarely verify that validity assumptions are satisfied and that, accordingly, they typically use analyses that are nonrobust to assumption violations. In addition, researchers rarely report effect size statistics, nor do they routinely perform power analyses to determine sample size requirements. Recommendations are offered to rectify these shortcomings.

Single-Case Intervention Research Design Standards

In an effort to responsibly incorporate evidence based on single-case designs (SCDs) into the What Works Clearinghouse (WWC) evidence base, the WWC assembled a panel of individuals with expertise in quantitative methods and SCD methodology to draft SCD standards. In this article, the panel provides an overview of the SCD standards recommended by the panel (henceforth referred to as the Standards) and adopted in Version 1.0 of the WWC’s official pilot standards. The Standards are sequentially applied to research studies that incorporate SCDs. The design standards focus on the methodological soundness of SCDs, whereby reviewers assign the categories of Meets Standards, Meets Standards With Reservations, and Does Not Meet Standards to each study. Evidence criteria focus on the credibility of the reported evidence, whereby the outcome measures that meet the design standards (with or without reservations) are examined by reviewers trained in visual analysis and categorized as demonstrating Strong Evidence, Moderate Evidence, or No Evidence. An illustration of an actual research application of the Standards is provided. Issues that the panel did not address are presented as priorities for future consideration. Implications for research and the evidence-based practice movement in psychology and education are discussed. The WWC’s Version 1.0 SCD standards are currently being piloted in systematic reviews conducted by the WWC. This document reflects the initial standards recommended by the authors as well as the underlying rationale for those standards. It should be noted that the WWC may revise the Version 1.0 standards based on the results of the pilot; future versions of the WWC standards can be found at http://www.whatworks.ed.gov.

The Mnemonic Keyword Method

The keyword method is a two-stage procedure for remembering materials that have an associative component. In the case of foreign vocabulary learning, for example, the learner first must acquire a stable association between the unfamiliar foreign word and a familiar English word that sounds like a salient part of the foreign word. The acoustically similar English word is the keyword. The learner then encodes a meaningful interaction between the keyword and the foreign word’s definition. The method has been investigated most extensively with respect to recall of definitions from vocabulary words. However, other aspects of vocabulary learning also are considered, as are potential classroom applications of the keyword method based on a variety of curricular content. Future research directions are emphasized throughout the discussion.

Activity and imagined activity can enhance young children's reading comprehension

The Indexical Hypothesis suggests a new method for enhancing children’s reading comprehension. Young readers may not consistently “index,” or map, words to the objects the words represent. Consequently, these readers fail to derive much meaning from the text. The instructional method involves manipulating toy objects referred to in the text (e.g., a barn, a tractor, a horse, in a text about a farm) to simulate the actions described in the text. Correctly manipulating the objects forces indexing and facilitates the derivation of meaning. Both actual manipulation and imagined manipulation resulted in markedly better (compared with rereading) memory for and comprehension of the text material, thereby lending strong support to the Indexical Hypothesis. Can young children’s reading comprehension be enhanced? Are there potent reading-comprehension strategies that can be identified and prescribed (see, e.g., Ehri, Nunes, Stahl, & Willows, 2001; Symons, McGoldrick, Snyder, & Pressley, 1990)? The Indexical Hypothesis (Glenberg & Kaschak, 2002; Glenberg & Robertson, 2000; Kaschak & Glenberg, 2000) suggests a new sort of answer to these old questions. Because children may not consistently index, or map, written words to the objects the words represent, even when the words are pronounced, these children fail to derive meaning from the text. Consequently, reading becomes an unengaging exercise in word calling. If, as we hypothesize, early young readers’ performance can be enhanced by increased indexing, then the following instructional intervention is suggested: While children read texts about events taking place in a particular scenario (e.g., on a farm), objects referred to in the text (e.g., a toy barn, tractor, and horse) are made available, and the children are asked to manipulate those objects to simulate the content of the sentences. Such manipulation should force indexing, thereby facilitating the children’s derivation of meaning. We begin by reviewing the Indexical Hypothesis and some of the research that supports it. The review includes a description of three precedents suggesting that manipulation of objects being read about should enhance children’s reading comprehension. 1 We then present three experiments conducted with first- and secondgrade readers. These experiments demonstrate large (e.g., 50% and more) positive effects of manipulation on children’s recall and application of the material just read. In addition, in Experiment 3, children are trained to imagine manipulating the toys rather than actually manipulating them. This imagined manipulation produces a modest degree of transfer (i.e., strategy maintenance in the absence of instructions). Finally, we contrast the explanation of poor reading comprehension provided by the Indexical Hypothesis with several other accounts based on fluency, inference making, and integration.

Enhancing the Scientific Credibility of Single-Case Intervention Research: Randomization to the Rescue.

In recent years, single-case designs have increasingly been used to establish an empirical basis for evidence-based interventions and techniques in a variety of disciplines, including psychology and education. Although traditional single-case designs have typically not met the criteria for a randomized controlled trial relative to conventional multiple-participant experimental designs, there are procedures that can be adopted to create a randomized experiment in this class of experimental design. Our two major purposes in writing this article were (a) to review the various types of single-case design that have been and can be used in psychological and educational intervention research and (b) to incorporate randomized experimental schemes into these designs, thereby improving them so that investigators can draw more valid conclusions from their research. For each traditional single-case design type reviewed, we provide illustrations of how various forms of randomization can be introduced into the basic design structure. We conclude by recommending that traditional single-case intervention designs be transformed into more scientifically credible randomized single-case intervention designs whenever the research conditions under consideration permit.

A controlled, powerful multiple-comparison strategy for several situations.

The multiple-comparison procedure originally proposed by R. A. Fisher (1935) for the 1-way analysis of variance context has several desirable properties when K (the number of groups) is equal to 3. In this article, the logic of the procedure is described in conjunction with those properties. A discussion follows of how the Fisher procedure can be similarly applied in a number of other K=3 (and, more generally, 2-degree-of-freedom) hypothesis-testing situations. Finally, the Fisher logic is combined with recent sequential applications of the Bonferroni inequality to illustrate the utility and versatility of that combination for the applied researcher

Memory strategy monitoring in adults and children

In five experiments, adults and 11- to 13-year-old children were exposed to two study strategies for vocabulary learning: repetition of words with meanings and associative elaboration (the keyword method). Subjects were asked to choose one of the two study methods for learning a 24-item list of new vocabulary words. Adults demonstrated awareness of the relative efficacy of the two study strategies only following practice with the two methods accompanied by a test on the practice list. They did not exhibit an awareness of the relative utility of the two methods when the practice test was omitted. Children, in addition to having to experience the practice list and test, required explicit feedback about their performance with the two strategies before they demonstrated clear awareness that the elaboration method was superior. The data are relevant to recent positions on the role of metacognitive experiences for metacognitive knowledge and cognitive actions.

On pictures in prose

For some time now, researchers and educators have questioned the value of pictures for prose-learning. However, there is abundant empirical evidence to document the positive value of pictures. Consistent learning gains are associated with the use of pictures when experiments adhere to these five ground rules: (1) prose passages are presented orally; (2) the subjects are children; (3) the passages are fictional narratives; (4) the pictures overlap the story content; and (5) learning is demonstrated by factual recall.

Four Cognitive Principles of Learning-Strategy Instruction

Four cognitive principles of learning-strategy instruction are proposed. These principles call for strategy researchers to: (a) develop learning strategies that are appropriate to ones desired cognitive outcomes, (b) conduct routine component analyses of both learning strategies and learner processes, (c) take into account the match between the particular learning strategy and the learners specific knowledge and skills, and (d) perform controlled empirical validations of learning-strategy effectiveness. A preliminary model that distinguishes among the processes of understanding, remembering, and applying is presented to amplify the first cognitive principle.

Developmental constraints associated with children's use of the keyword method of foreign language vocabulary learning

Abstract Second- and sixth-grade (ages 7 and 11 years, respectively) subjects were instructed to learn simple Spanish vocabulary nouns using the keyword method. To remember a foreign word translation the keyword method user must: (1) associate the foreign word to an English word (the keyword) that sounds like part of the foreign word; and (2) remember an image of the keyword and translation referents interacting. Second-grade keyword users who were provided with interactive pictures remembered more vocabulary items than those who generated their own imagery links when given separate pictures of the keyword and translation referents. Second graders who generated their own linking images when given only the keywords and translation words recalled fewer items than both picture groups, and were not significantly different from control subjects. Sixth graders in the three imagery-link variations performed at comparable levels and better than control subjects. The results are in complete accord with previous speculations about the development of imagery strategies in children.