Marion Tillema

Quantifying the quality difference between L1 and L2 essays: A rating procedure with bilingual raters and L1 and L2 benchmark essays

It is the consensus that, as a result of the extra constraints placed on working memory, texts written in a second language (L2) are usually of lower quality than texts written in the first language (L1) by the same writer. However, no method is currently available for quantifying the quality difference between L1 and L2 texts. In the present study, we tested a rating procedure for enabling quality judgments of L1 and L2 texts on a single scale. Two main features define this procedure: 1) raters are bilingual or near native users of both the L1 and L2; 2) ratings are performed with L1 and L2 benchmark texts. Direct comparisons of observed L1 and L2 scores are only warranted if the ratings with L1 and L2 benchmarks are parallel tests and if the ratings are reliable. Results showed that both conditions are met. Effect sizes (Cohen’s d) indicate that, while score variances are large, there is a relatively large added L2 effect: in the investigated population, L2 text scores were much lower than L1 text scores. The tested rating procedure is a promising method for cross-national comparisons of writing proficiency.

Process execution of writing and reading: considering text quality, learner and task characteristics

We have conducted systematic reflections, data reanalyses, and incorporated results from several studies to promote discussion, enhance understanding, and build theory. Two models guide our research and analyses: The Descriptive Interactive Process (DIP) model (Fig. 20.1, left), and the Experimental Interactive Process (EIP) model (Fig. 20.1, right). In the DIP model, the main idea is to study processes: What happens during task execution, and how does the process change accordingly? The complexity can be illustrated by adding three components to the model: (a) quality of the output—what variation in processes is related to variation in output quality?; (b) task characteristics—what degree do processes vary with task features (e.g., computer versus pen-and-paper writing)?; and (c) learner characteristics—what degree doesthe way skilled versus unskilled writers adjust their process to tasks vary? In the EIP model (Fig. 20.1, right), the general aim is to detect the effect of interventions on processes: Do different instructional variables affect thetarget process differently? This model can be extended by adding the product variable—Do instructional variables affect the target process differently, and does the product quality vary accordingly?—and learner characteristics: Does the way instructional variables affect the target process vary with regard to learner characteristics? Do good writers profit as much from the experimental instruction as poor writers? Does the experimental instruction change the processes carried out while writing in the same way for good and poor writers?

Generalizability of text quality scores

This chapter explores the generalizability of ratings on text quality of ninth-grade students in pre-university education and first-year university students. For first-year university students, one does not need to rate as many texts as for ninth-graders. Especially, when writing in a foreign language, first-year students prove to be very stable writers. The stability of text quality scores appears to depend on the way texts are rated as well. So-called analytic scoring schemes seem to result in reliable but topic-dependent text quality scores. Holistic ratings, on the other hand, appear to give raters less support (resulting in a lower inter-rater agreement) but result in less topic-dependent scores. Hence, in order to generalize, writers do not need to write that many texts if these texts are holistically rated. Implications of these results for experimental studies on writing and educational effectiveness of writing pedagogies are discussed. Keywords:analytic scoring schemes; first-year students; generalizability; holistic ratings; ninth-graders; text quality scores; writing skills

List of Volumes

This section of this book titled Measuring Writing: Recent Insights into Theory, Methodology and Practices contains a list of volumes. Keywords: writing ability, formative assessments

Chapter 2.02.03: Formulating Activities in L1 and L2 and Their Relation With Text Quality

There have been many studies into the relation between cognitive processes during writing in the mother tongue (L1) and writing in a second or foreign language (L2). As task situations during writing continuously change, so does the probability of occurrence of cognitive activities. This chapter focuses on the orchestration of formulating activities during writing in L1 and L2, and their relation with text quality. It shows that the occurrence of formulating activities differs between L1 and L2. The occurrence of formulating activities in L1 follows a more complex pattern than the occurrence of formulating in L2. Furthermore, the correlation between formulating activities is more complex in L1 than in L2. Finally, this correlation appears to be highly task dependent in L1 but not in L2. Keywords: cognitive processes; foreign language; formulating activities; mother tongue; text quality