Ian Schagen

Adding value in educational research—the marriage of data and analytical power

The last 20 years have seen the development of sophisticated techniques for analysing individual data within a hierarchical context and the growing availability of good datasets to which these techniques can be applied. The modelling of pupil performance controlling for prior attainment has led to a type of analysis commonly titled ‘value-added’. Concern for school factors which affect pupil progress has given rise to ‘school effectiveness’ research. This article outlines the history of these linked movements, mainly with reference to England and Wales, but developments in other countries are also outlined. It discusses some of the objections that have been raised, and comments on possible future directions.

Fallback in attainment on transfer at age 11: evidence from the Summer Literacy Schools evaluation

Summary Summer Literacy Schools were introduced as a pilot scheme by the Government in the summer of 1997 in an attempt to improve childrens literacy skills at the age of 11, the time of transfer from primary to secondary education. The initiative was evaluated by collecting the pupils’ results from the national test taken in May and comparing these with results of a similar test administered in September. Results for a control group who had not attended summer schools were also analysed. The analysis revealed that the scores of both groups declined significantly between the pre‐test and the post‐test, and no significant difference in the extent of the decline was found between the summer school pupils and the control group. Further research is needed on the possibility that the transition from primary to secondary school is associated generally with a significant decline in attainment.

Multilevel Analysis of the Key Stage 1 National Curriculum Assessment Data in 1995

Abstract In a follow‐up to previous work (Schagen, 1994), results from Key Stage 1 National Curriculum tests in 1995 have been analysed using multilevel modelling. The analysis estimates the effects of different background variables, at both the pupil and school levels. Some investigation of the influence of reading attainment on mathematics test scores, over and above the effect of general mathematical attainment, was carried out. The results showed the significant effects of certain background variables, including sex, age, nursery education and ethnic background, which persisted even when controlling for Teacher Assessment levels. Mathematics tests scores were shown to be related to reading levels, even when teacher‐assessed mathematical attainment was taken into account.

The use of standardized residuals to derive value‐added measures of school performance

Measures of school performance based on pupil attainment are becoming more sophisticated, with DfES piloting ‘contextualized value‐added’ measures. However, most such measures are based on simple sums of residuals about ‘expected’ values, but it is clear from national data that distributions of performance are not symmetrical and of constant variance. A method is explored which takes account of the exact shape of the performance distribution to produce ‘standardized residuals’ which can be aggregated to the school level and give consistent measures on a fixed metric. Examples are given based on national data.

Insights into School Effectiveness from Analysis of OFSTED's School Inspection Database.

ABSTRACT The OFSTED database contains detailed numerical judgements derived from the inspection of the majority of secondary schools in England. Factor analysis techniques were used to derive relevant factors at the school and pupil levels, which were analysed in relation to both lesson observation grades and examination outcomes. The results are discussed in relation to the usefulness of the OFSTED database for school effectiveness research.

Different for Girls? An Exploration of the Impact of "Playing for Success.".

Playing for Success is a national initiative aimed at providing study support for underachieving young people. Based in football grounds, it uses the medium and environment of football as a motivating force. A few commentators have raised questions about the initiative in relation to equality of opportunity, suggesting that girls and pupils from ethnic minorities may be disadvantaged by the initiative’s association with football. The NFER completed the second national evaluation of the initiative in 2001. This paper presents evidence showing that Playing for Success is equally beneficial for girls and boys, and for pupils from different ethnic backgrounds.

Multilevel Analysis of the Key Stage 1 National Curriculum Assessment Data in 1991 and 1992.

abstract Data collected as part of the national evaluation of the Key Stage 1 assessments in English, mathematics and science have been used to explore the relationships between the attainment of pupils in these subjects at age 7 and background variables collected at both the pupil and school levels. The method of analysis was multilevel modelling, and this shows some interesting results, highlighting for example the effect of nursery education.

Concorde and discord: the art of multilevel modelling

A recent issue of International Journal of Research & Methods in Education (IJRME) contained a challenging article by Stephen Gorard (2007) in which he attacks aspects of current practice in statistical modelling, with particular focus on multilevel modelling (MLM). We believe there is much in his argument which is misleading or misconceived, both at a general and a detailed level, and this article explains where the difficulties arise. We start by a brief overview of some of the basic principles of statistical modelling. We then discuss MLM and some of the criticisms in general. Finally, in the last section before conclusions we detail some of the mis-statements in the Gorard article.

A methodology for judging departmental performance within schools

Summary Overall performance statistics for schools are of little value in school self‐improvement because they do not differentiate finely enough between subject departments. What is needed is to identify subject departments which are ‘adding value’ by producing better results than might have been expected, as well as those departments which are under‐performing. Direct comparisons of GCSE results between subjects are problematical, and have led to a variety of supposedly sophisticated techniques which purport to allow for subject difficulty, but which are themselves difficult to operationalize and comprehend, as well as being of uncertain validity. A method for comparing subject grades within schools with expected results based on a group of schools is presented, which is simple and robust and has been used in practice to identify departments ‘adding significant value’.

Faith schools and pupils' progress through primary education

The impact of faith schools on the performance and progress of their pupils has been studied using data from the National Pupil Database (NPD). The value‐added analysis was carried out using multilevel modelling, controlling for prior attainment as well as a range of background variables, including ethnicity, sex, eligibility for free school meals (FSM), alternative measures of deprivation based on census information, special educational needs (SEN) and English as an additional language (EAL). The analysis confirmed that all faith schools, in particular, Roman Catholic and Church of England schools, made slightly more progress with their pupils than non‐faith schools. It also showed that pupils with SEN attending faith schools performed better at key stage 2 than pupils with SEN in non‐faith schools.