Ben Levin

A framework for mandatory impact evaluation to ensure well informed public policy decisions

Trillions of dollars are invested yearly in programmes to improve health, social welfare, education, and justice (which we will refer to generally as public programmes). Yet we know little about the eff ects of most of these attempts to improve peoples’ lives, and what we do know is often not used to inform decisions. We propose that governments and non-governmental organisations (NGOs) address this failure responsibly by mandating more systematic and transparent use of research evidence to assess the likely eff ects of public programmes before they are launched, and the better use of well designed impact evaluations after they are launched. Resources for public programmes will always be scarce. In low-income and middle-income countries, where there are often particularly severe constraints on resources and many competing priorities, available resources have to be used as effi ciently as possible to address important challenges and goals, such as the Millennium Development Goals. Use of research evidence to inform decisions is crucial. As suggested by Hassan Mshinda, the Director-General of the Commission for Science and Technology in Tanzania: “If you are poor, actually you need more evidence before you invest, rather than if you are rich.” But neither the problem nor the need for solutions is limited either to health or countries of low and middle income. Expenditures and the potential for waste are greatest in high-income countries, which also have restricted resources and unmet needs, particularly during a fi nancial crisis. Having good evidence to inform diffi cult decisions can be politically attractive, as shown, for example, by the US Government’s decision to include US

Mobilising Research Knowledge in Education.

1·1 billion for comparative research (including systematic reviews and clinical trials) as part of its

Governments and education reform: some lessons from the last 50 years

787 billion economic stimulus bill. To paraphrase Billy Beane, Newt Gingrich, and John Kerry, who have argued for a health-care system that is driven by robust comparative clinical evidence by substituting policy makers for doctors: “Evidence-based health care would not strip [policymakers] of their decision-making authority nor replace their expertise. Instead, data and evidence should complement a lifetime of experience, so that [policymakers] can deliver the best quality care at the lowest possible cost.” Lancet 2010; 375: 427–31

Leadership for evidence-informed education

The field of knowledge mobilisation (KM) addresses the multiple ways in which stronger connections can be made between research, policy and practice. This paper reviews the current situation around knowledge mobilisation in education. It addresses changing understandings of KM, considers some of the main issues in conducting empirical research in the field, and looks at the state of activity to promote and increase KM, offering commentary and suggestions in each area.

Schools in challenging circumstances: A reflection on what we know and what we need to know

Over the last few decades many efforts have been made to address education issues through policy at various levels. Looking at these efforts around the world suggests that they have often been motivated more by beliefs than by evidence of impact. Not only are the wrong policies often adopted, but effective implementation of education policy is often lacking. In part this is because governments face particular constraints on what they can do. Education reform efforts would be stronger if they gave more attention to reliable research evidence and a greater focus to what is known about effective teaching.

Media–government relations in education

This article reviews the evidence on the way that schools and school systems find, share and use research in their work, with a particular focus on the role of school and district leaders in increasing the prominence and value of research in their work. It reviews the evidence on the ways in which practitioners in education find and use research and the role of organisation factors in promoting the use of research. The article concludes with a discussion of steps that school and district leaders could take to increase the profile of research in their organisations.

What Did You Do in School Today

A collection of papers looking at school improvement in schools facing challenging circumstances is very welcome. The field of school improvement has been criticized, with some merit, for sometimes giving the impression that school context is not that important, and that any school can, with appropriate action, improve significantly. Most advocates of school improvement do recognize the importance of the social context of the school, but, as several of these papers point out, we are still learning to understand how context affects the work of improvement. In that sense, these papers illustrate both the significant contributions that have been made by the school improvement literature and the challenges still facing this field. The papers in this collection cover a wide range of issues related to school improvement. Harris, Chapman, Muijs, Russ, and Stoll look at improvement efforts and results in eight schools in a de-industrialized area of England. These schools had all shown several years of increases in externally measured student outcomes and reported a strong focus on literacy and numeracy, extensive use of student data to drive change, and extensive teacher professional development. The paper by Reynolds, Harris, Clarke, Harris, and James also reports on school improvement work with eight schools in England, focusing in this case more on the process of engaging schools as external consultants and creating effective internal commitment to

The Challenge of Large-Scale Literacy Improvement.

Government and the media are intimately connected in many ways though each distrusts the other. This paper, based on the author’s experience in government, describes the ways that governments try to deal with the media and discusses three of the main complaints about media coverage—its tendency to simplify, to assign blame, and to be primarily focused on the short term. It concludes with some suggestions for possible improvements.Government and the media are intimately connected in many ways though each distrusts the other. This paper, based on the author’s experience in government, describes the ways that governments try to deal with the media and discusses three of the main complaints about media coverage—its tendency to simplify, to assign blame, and to be primarily focused on the short term. It concludes with some suggestions for possible improvements.

Results without Rancor or Ranking Ontario's Success Story

Engaging students in secondary school appears to be a concern in schools around the world. The Canadian Education Association has made adolescent learners a focus of its work for the last several years. Among its various initiatives in this area, the CEA has been working with a dozen school districts involving a hundred secondary schools on a project called “What did you do in school today” (WDYDIST)? (http://www.cea-ace.ca/res.cfm?subsection=wdy). The project “is grounded in the conviction that, in order to raise the achievement levels of all students and to narrow the gaps between students, we have to guarantee that all young people are engaged in their learning and that all receive effective and intellectually challenging instruction.” And who could argue with that? One part of this project involved surveying some 32,000 students in grades 6 through 12 about their engagement in extracurricular activities, sense of belonging in school, academic engagement (such as attendance), and intellectual engagement (such as interest and enjoyment of classes). About 70% of the students reported reasonable levels of engagement in the first three of the four areas, but only 37% reported being intellectually engaged in language arts or mathematics classes. Furthermore, engagement in all areas except sense of belonging was lower for older students. Levels of engagement were significantly related to students’ socioeconomic background, but differences between schools were even greater, with some schools reporting levels of engagement that were double those of other schools, even controlling for students’ background. The survey also asked about five areas of classroom and school climate: effective learning time, teacher/student relations, classroom discipline, expectations for success, and instructional challenge. Not surprisingly, students were more engaged when there were high expectations for their success and positive teacher/student relations. Intellectual engage-

THE COMPLETE STORY: A POPULATION‐ BASED PERSPECTIVE ON SCHOOL PERFORMANCE AND EDUCATIONAL TESTING

This paper discusses the challenge of making large-scale improvements in literacy in schools across an entire education system. Despite growing interest and rhetoric, there are very few examples of sustained, large-scale change efforts around school-age literacy. The paper reviews 2 instances of such efforts, in England and Ontario. After describing main features of these reforms, the paper presents 4 main reasons that such efforts are not more frequent: (a) the educational challenge of changing very large numbers of schools and classrooms on a sustained basis, (b) the bureaucratic challenge of improving the connections among different areas of social policy in pursuit of better outcomes for students, (c) the learning challenge of organizing complex systems to do this work while continually modifying the approach in light of new evidence and system feedback, and (d) the political challenge of galvanizing and maintaining the effort required to support these other changes.