Maaike Smit

Professional Development in the Use of Data: From Data to Knowledge in Data Teams

ABSTRACT Schools need support in the use of data. To provide this support, a data team intervention was developed. A prior study conducted in the Netherlands showed that several factors can enable or hinder the work of data teams. The current replication study focuses on the factors influencing data use in data teams and the perceived effects of the data teams’ work, but looking at data teams in Sweden. The results of this qualitative study show that the data teams’ work is influenced by the same factors as in the Netherlands: Data characteristics (e.g., relevance of the data), team characteristics (e.g., heterogeneity of the team), and school organizational characteristics (e.g., school leader support).

Case Study: English Language Results

This chapter uses a second case study to extensively describe what working with the data team™ procedure is like in practice. This chapter describes the process for a team that worked on a subject-specific problem pertaining to English language results in the 10th grade. The team defined the following problem definition: “We are unhappy about the English achievement scores in the 10th grade, because the grade has been 5.5 on average over the past four years. We want the average grade in the 10th grade to be at least 5.9 next year and at least 6.1 the year after that.” The team investigates several hypotheses, concerning middle school location, 5th grade English achievement scores, scores from previous years, and reading skills. In this chapter we describe how the data team investigated these hypotheses, what their conclusions were, and how they were able to improve English language results based on the collected data.

Case: High School Graduation Rates

This chapter extensively covers what working with the data teamTM procedure is like in practice. It describes a team working on a school-wide problem regarding low graduation rates at their high school. It is loosely based on our experiences during the “Datateams” project, for which we supervised 37 teams as they worked with this method. In this chapter, a data team works on the following problem definition: “We are unhappy about the fact that an average of 32% of the students failed to graduate over the past three years. Next year, no more than 25% of the students may fail to graduate. The year after that, we want to reduce this percentage to no more than 20%.” The team investigates several hypotheses, concerning local graduation requirements and state requirements, student motivation, middle school scores, students cutting classes, feedback, and supervision. In this chapter, we describe how the data team investigated these hypotheses, what their conclusions were, and how they were able to improve graduation rates based on the collected data.

Introducing the Data Team™ Procedure in the School

This chapter describes what needs to be done when a school would like to start working with the data teamTM procedure. First of all, it is advisable to bring in external supervision from the start. Research into the data teamTM procedure shows that long-term external supervision is crucial to a team’s effectiveness. Secondly, there are some preliminary conditions that must be taken care of before adopting this method concerning characteristics of the data (e.g., what are the available data in the school), characteristics of the school organization (e.g., scheduled time for data team meetings), and individual and team characteristics (e.g., who joins the team). Thirdly, there are some factors that play a part during the process of data use. Several of these factors are important both as preliminary conditions and during the process itself. During the process, characteristics of the data (e.g., access to high quality data), characteristics of the school organization (e.g., data-informed leadership), and individual and team characteristics (e.g., data literacy) can enable or hinder the work of the data team.

Step 3: Data Collection

The team has selected one or several hypotheses and now needs to collect data for their research. The team should strive to make use of data that are already available within the school whenever possible. Teams make use of various kinds of data, including input data (data pertaining to how students enter the school, such as data about student characteristics or the elementary schools they came from), process data (data pertaining to the learning processes that occur within the school), and output data (all data about results for the school). Sometimes, it may not be possible to investigate a hypothesis using available data. The team will then have to collect new data. In these cases, we recommend making use of existing high-quality research instruments whenever possible. Before the team starts collecting the data, it is important first to discuss what data are needed, exactly (about what target group? on what aspect? covering how many years?), and how the team wants the data to be presented. Once it is clear what data need to be collected and how these data should be presented, the team will draw up a plan of action for data collection. This plan of action states who will do what, and when. In this chapter, the process of data collection is described, and various examples are provided.

Step 8: Evaluation

The team has investigated various hypotheses. Based on the hypotheses that ended up being accepted, measures were taken to resolve the problem that was identified during step 1. Now it is time to evaluate whether the measures were actually effective. First of all, it is important to conduct an evaluation of the process and monitor the measures that were implemented. In order to determine whether these measures are having the desired effect, it is important to verify whether the measures are being enacted as the team intended. If a measure is not being executed (properly) or if it is leading to unwanted reactions from the target group, it is unlikely that the desired goal is being achieved. Next, the effects need to be evaluated. The evaluation of the effects is about whether the measures that were implemented have been effective for resolving the problem. The team has to ask two important questions: (1) Have the causes of the problem been eliminated? (2) Has the problem been resolved and the goal achieved, as formulated during step 1? Data need to be collected here as well. If the team can demonstrate that the problem has been resolved and that the goal (the desired situation), as outlined during step 1, has been achieved, the circle will be complete. In this chapter, we will describe how to evaluate the process and effects using several real life examples.

Step 1: Problem Definition

The first step of the data teamTM procedure is defining the problem. What problem or subject does the team want to work on? The subject a team wants to work on may be school-wide or subject-specific in nature, or both. Examples of school-wide topics are problems regarding the percentage of students progressing to the next grade level, problems regarding dropout rates, or graduation rates. Examples of subject-specific issues include unsatisfactory English achievement results in grade 6, disappointing scores for math in grade 9 and low assessment scores on physics in grade 10. The team collects data concerning the current situation. The existence of the problem and its magnitude need to be demonstrated. It is not uncommon for the problem to turn out to be different from what was expected. Next, the team will discuss the situation it wants to achieve. The current situation has been assessed, but now the desired situation—a goal—must be linked to that current situation. Once the current situation and the desired situation are clear, these can be brought together in a problem definition. The target group is clear, and the scope of the problem and the goal have been identified. In this chapter, we will describe in depth how to develop a problem definition, and we will also provide several examples.

Step 6: Interpretation and Conclusions

The data analysis makes it possible to interpret the data and draw conclusions about the hypothesis or research question. If a team’s hypothesis is correct, and they have identified an important cause of the problem, they can proceed to step 7: taking measures. However, if the team has not been able to accept the hypothesis, they have to go back to step 2 and formulate a new hypothesis. It is also possible for a team to think they have identified part of the cause of a problem. The team can then proceed in two ways: they will take measures based on the accepted hypothesis, and they will conduct further research investigating a new hypothesis. There are various options when it comes to qualitative data as well. The team may have found the answer to their research question, which allows them to move on to step 7 and start taking improvement measures. It is also possible that the team has acquired more insight into the research question at hand, although they have not yet found a definitive answer. They can then decide to draw up a new hypothesis based on the qualitative results. When working on a research question, it is also possible for a team to identify a cause of the problem, while there appear to be other causes as well. In that case, the team takes measures to resolve the known cause, while also reverting back to step 2 to draw up a new hypothesis or research question.

The Road to Sustainability

Schools often start using the data team™ procedure with the ambition of making data use part of the school’s regular operations. Working with the data team™ procedure therefore has to transition from a one-time project into a sustainable change: a new “routine” implemented at all layers of the school to use data in discussions and when making decisions. In this chapter, we will describe three important factors for sustainability: (1) Keeping a record of the innovation through development of new procedures and materials, (2) professional development of personnel, and (3) organizational development regarding both structural and cultural aspects of the innovation. We will also describe three types of sustainability: (1) a sustainable solution to the problem the data team has investigated, (2) the continued use and integration of the data team™ procedure itself, and (3) data use within the school. In this chapter, we will describe how to work on these three types of sustainability.

Step 7: Implementing Improvement Measures

Once the team has identified a cause of the problem at hand, they can move on to step 7: taking improvement measures. However, just because the cause of a problem has been identified, this does not mean that it is also immediately clear what measure is best suited to address this cause. In step 7, teams therefore start by gathering ideas about what measures might resolve the problem at hand. In order to draw up a clear overview of measures, a team can make use of a variety of sources, such as the knowledge and experience of team members and colleagues in the school, networks such as the teacher’s union, practitioner journals, scientific literature, the internet and the experiences of other schools. Selecting the “right” measure from a list of possible measures often requires additional effort. The first step of this process is to verify whether the measure actually ties into the conclusions that have been formulated. Next, the team decides what criteria to use to compare the various measures and make the right choice. This can include such criteria as feasibility, proven effects, the expected speed of the effects and the costs of implementing the measure. Moving from measures to implementation requires a plan of action. A plan of action outlines the actions that need to be taken to implement each selected measure, who is to take these actions, what means are needed to do so and what the deadline is.