Huub van den Bergh

On multi-level modeling of data from repeated measures designs: a tutorial

Data from repeated measures experiments are usually analyzed with conventional ANOVA. Three well-known problems with ANOVA are the sphericity assumption, the design effect (sampling hierarchy), and the requirement for complete designs and data sets. This tutorial explains and demonstrates multi-level modeling (MLM) as an alternative analysis tool for repeated measures data. MLM allows us to estimate variance and covariance components explicitly. MLM does not require sphericity, it takes the sampling hierarchy into account, and it is capable of analyzing incomplete data. A fictitious data set is analyzed with MLM and ANOVA, and analysis results are compared. Moreover, existing data from a repeated measures design are re-analyzed with MLM, to demonstrate its advantages. Monte Carlo simulations suggest that MLM yields higher power than ANOVA, in particular under realistic circumstances. Although technically complex, MLM is recommended as a useful tool for analyzing repeated measures data from speech research.

Educational Leadership and Pupil Achievement: The Choice of a Valid Conceptual Model to Test Effects in School Effectiveness Research.

School effectiveness research often uses multilevel models in which only direct effects of characteristics of schools on pupil achievement are modelled. Recently, more attention is given to conceptual models that assume indirect and antecedent effects. In this paper, we elaborate on these models and show that the findings from school effectiveness research are influenced by the conceptual model underlying the analyses. We do this by focusing on the effect of “integrated leadership” on 2 outcome measures: mathematics achievement and mother tongue achievement. Four different conceptual models will be tested by means of multilevel structural equation modelling. Therefore we use data from a school effectiveness research in Flanders (850 fourth graders and 847 sixth graders within 47 schools). The results of these analyses show that the conclusion whether integrated leadership has an effect or not on pupil achievement, is dependent on the choice of the conceptual model.

Changes in cognitive activities during the writing process and relationships with text quality

This study investigates effective temporal organisations of writing processes in novice writers, using protocol analysis and focusing on task representation and formulating processes. Ninth grade students (n = 36) wrote an argumentative text under thinking-aloud conditions. Writers did not only differ with respect to the number of task representation and formulating activities, but also with respect to the moment on which these activities were performed. Task representation activities are positively related to text quality, but only during the initial phases of the writing process. Formulating activities are negatively correlated with text quality in the beginning, but positively in the end of the process.

On the construct validity of multiple-choice items for reading comprehension

In this study 590 third-grade students took one of four reading comprehension tests with either multiple- choice items or open-ended items. Each also took 32 tests indicating 16 semantic Structure-of-Intellect (si) abilities. Four conditions or groups were distinguished on the basis of the reading comprehension tests. The four 33 x 33 correlation matrices were analyzed si multaneously with a four-group LISREL model. The 16 intellectual abilities explained approximately 62% of the variance in true reading comprehension scores. None of the SI abilities proved to be differentially re lated to item type. Therefore, it was concluded that item type for reading comprehension is congeneric with respect to the SI abilities measured. Index terms: construct validity, item format, free response, reading comprehension, Structure-of-Intellect model.In this study 590 third-grade students took one of four reading comprehension tests with either multiple- choice items or open-ended items. Each also took 32 tests indicating 16 semantic Structure-of-Intellect (si) abilities. Four conditions or groups were distinguished on the basis of the reading comprehension tests. The four 33 x 33 correlation matrices were analyzed si multaneously with a four-group LISREL model. The 16 intellectual abilities explained approximately 62% of the variance in true reading comprehension scores. None of the SI abilities proved to be differentially re lated to item type. Therefore, it was concluded that item type for reading comprehension is congeneric with respect to the SI abilities measured. Index terms: construct validity, item format, free response, reading comprehension, Structure-of-Intellect model.

Measuring the quality of governmental websites in a controlled versus an online setting with the 'Website Evaluation Questionnaire'

The quality of governmental websites is often measured with questionnaires that ask users for their opinions on various aspects of the website. This article presents the Website Evaluation Questionnaire (WEQ), which was specifically designed for the evaluation of governmental websites. The multidimensional structure of the WEQ was tested in a controlled laboratory setting and in an online real-life setting. In two studies we analyzed the underlying factor structure, the stability and reliability of this structure, and the sensitivity of the WEQ to quality differences between websites. The WEQ proved to be a valid and reliable instrument with seven clearly distinct dimensions. In the online setting higher correlations were found between the seven dimensions than in the laboratory setting, and the WEQ was less sensitive to differences between websites. Two possible explanations for this result are the divergent activities of online users on the website and the less attentive way in which these users filled out the questionnaire. We advise to relate online survey evaluations more strongly to the actual behavior of website users, for example, by including server log data in the analysis.

Writing as a learning tool: Testing the role of students' writing strategies

The claim that writing facilitates students’ learning, although widely accepted, has little support from empirical research. A possible explanation for the lack of empirical evidence is that writing-to-learn research has disregarded that students use different writing strategies. The purpose of the present experimental study is to test whether it is effective to adapt writing-to-learn tasks to different writing strategies when teaching literature. A course “Learning to write argumentative texts about literature” was developed in two different versions: one adapted to a planning writing strategy, the other to a revising writing strategy. Participants were 113 tenth-grade high school students in the Netherlands. Our hypothesis is an adaptation hypothesis: we expect that the more a student will use a planning writing strategy, the more the student will profit from the lessons in the planning condition, and that the more a student uses a revising writing strategy, the more beneficial the revising condition will be. However, results show that for improving literary interpretation skill, a course adapted to the planning writing strategy is more effective for almost all students.RésuméL’affirmation selon laquelle l’écriture favorise l’apprentissage chez les collégiens est largement acceptée mais peu soutenue par des résultats de recherches empiriques. Une explication possible à ce manque d’évidences empiriques tient au fait que la recherche relative à l’apprentissage de l’écrit a négligé l’étude des différentes stratégies d’écriture des collégiens. L’objectif de cette étude expérimentale est de voir s’il est efficace d’adapter une stratégie d’écriture réécrivant et une stratégie planifiant à des activités d’apprentissage de production écrite dans l’enseignement de la littérature. Un cours intitulé “Apprendre à écrire un texte argumentatif sur la littérature” a été proposé sous deux formes différentes: l’une pour la stratégie d’écriture planifiant, l’autre pour la stratégie réécrivant. Les participants étaient 113 collégiens en classe de troisième aux Pays-Bas. Notre hypothèse est une hypothèse d’adaptation: on s’attend à ce qu’un collégien ayant une stratégie d’écriture planifiant profitera davantage des leçons adaptées à cette stratégie et que les leçons de type sculptural seront bénéfiques pour les lycéens ayant une stratégie d’écriture réécrivant. Cependant, les résultats montrent que, pour améliorer la compétence d’interprétation littéraire, un cours adapté a une stratégie d’écriture planifiant est plus efficace pour quasiment tous les collégiens.

The effect of observational learning on students’ performance, processes, and motivation in two creative domains

BACKGROUND Previous research has shown that observation can be effective for learning in various domains, for example, argumentative writing and mathematics. The question in this paper is whether observational learning can also be beneficial when learning to perform creative tasks in visual and verbal arts. AIMS We hypothesized that observation has a positive effect on performance, process, and motivation. We expected similarity in competence between the model and the observer to influence the effectiveness of observation. Sample.  A total of 131 Dutch students (10(th) grade, 15 years old) participated. METHOD Two experiments were carried out (one for visual and one for verbal arts). Participants were randomly assigned to one of three conditions; two observational learning conditions and a control condition (learning by practising). The observational learning conditions differed in instructional focus (on the weaker or the more competent model of a pair to be observed). RESULTS We found positive effects of observation on creative products, creative processes, and motivation in the visual domain. In the verbal domain, observation seemed to affect the creative process, but not the other variables. The model similarity hypothesis was not confirmed. CONCLUSIONS Results suggest that observation may foster learning in creative domains, especially in the visual arts.

Effective learning activities in observation taks when learning to write and read argumentative texts

On repeated occasions, observational learning has proved itself to be an effective instruction method. Experimental studies have shown to be effective for complex tasks such as reading and writing for both teachers and students as models. The problem when interpreting the results of such research is that, in observation tasks, several mental activities play a simultaneous role. In this study we therefore set out to identify the effective elements of observation tasks. We focused on two elements of the observation tasks, both aimed at stimulating monitoring activities: evaluation of the model’s performance and elaboration on this evaluation. We have also distinguished between elaboration on the observed products (the models’ written answers), and elaboration on the observed processes (the models’ verbalisations of their mental activities).The data were subjected to a LISREL analysis. First of all, it was observed that subjects who performed “evaluation” and “productelaboration” better, and “process-elaboration” more often in one lesson, also performed these activities better or more often in the subsequent lesson. Next, we observed an effect of aptitude on the learning activities: pre skill scores influence “evaluation” and “product-elaboration”. The most important finding is that “evaluation” and “product-elaboration” contribute positively to argumentative writing skills. It is discussed that these findings confirm the importance of the monitoring, evaluative and reflective activities when learning complex tasks as writing.RésuméComparativement à l’apprentissage par l’action, la méthode d’apprentissage par l’observation c’est plus efficace. On sait queles études expérimentales s’appliquent bien à destâches complexes comme la lecture et la production écrite, à la fois chez les enseignants et chez les élèves, pris comme modèles. Portant l’interprétation des résultats expérimentaux pose un problème puis que dans les tâches d’observation, plusieurs activités interviennent simultanément. Dans la présente étude, nous voulons identifier les éléments réellement critiques dans ces tâches d’observation. Nous nous sommes intéressés à deux d’entre eux: ‘l’évaluation’ et ‘l’élaboration’ (du ‘produit’ et du ‘processus’).Pour le traitement des donées nous avons utilisée par la méthode LISREL. En premier lieu, les sujets qui, dans une première leçon, réalisent plus souvent correctement les activités d’apprentissage ‘évaluation’ et ‘élaboration du produit’, et réalisent plus souvent ‘élaboration du processus’, réussissent de la même façon dans le leçon suivante. En outre, nous avons constaté quel’existe un effet d’aptitude: les aptitudes établies préalablement influencent ‘l’évaluation’ et ‘l’élaboration du produit’. Le plus important résultat c’est des effets positifs de ‘l’évaluation’ et de ‘l’élaboration du produit’ sur les performances d’écrire argumentatives. Ce résultat confirme l’importance des activités monitor, évaluation et réflexion pendant l’apprentissage des taches complexes comme écrire.

Importance in instructional text: teachers' and students' perceptions of task demands

History teachers (N = 22) and their 11th-grade students (N = 451) rated the relative importance of sections of an instructional text on which teachers would be giving a test. Multilevel analysis revealed only a moderate correspondence between teachers in their importance ratings. This suggested that there were large differences between the task demands of individual teachers; therefore, students needed to attune to the demands their own teacher set. However, only a moderate correspondence was found between ratings of individual students and those of their teacher. Although this correspondence varied between teachers, between students, and between text sections, results suggest that in general, the participating students did not have a very clear or sound perception of task demands.

Reducing interrater variability and improving health care: a meta-analytical review

OBJECTIVES In the scientific literature about reliability, the main approach to increasing reliability seems to involve increasing the number of observers and improving the instrument used. Other aspects for improving reliability - like the training of raters - seem to receive less notice. It is worth asking whether this technical approach could be complemented by training the user of the instrument. A systematic meta-analytical review of the research literature was performed to answer this question and examine the effectiveness of planned interventions for improving interrater reliability of health care professionals. METHOD The databases of PubMed (MEDLINE), Embase, Omega and PsycINFO were searched. The inclusion criteria were met by 57 studies. Details extracted from the studies included the study design, the number of observers and the number of observed cases, the intervention, the type of instrument (whether or not it was highly technical), and statistical information about the agreement before and after the intervention. Interventions were categorized into three groups: training of professionals, improving the diagnostic instrument and a combination of training and improving the instrument. A meta-analysis was performed by means of linear regression. RESULTS The interventions were arranged according to their effectiveness in improving the diagnostic instrument (mean change: β = 0.13), training combined with improving the instrument (mean change: β = 0.10) and training (mean change: β = 0.09). CONCLUSION On average, although all types of interventions are effective, improving the diagnostic instrument seems to be the most effective. Especially when highly technical instruments were concerned, improvement proved to be very effective (β = 0.52). Because instrumental variables constitute a major source of error, improving the instrument is an important approach. However, this review offers solid arguments that can complement the literature and practice, with a focus on training the user of the instrument.