F.J.G. Janssens

Effects and side effects of inspections and accountability in education: an overview of empirical studies

This paper presents an overview of studies into effects and side effects of control mechanisms in education. We focus on effects and side effects of inspection visits and public performance indicators. A first conclusion is that the studies do not provide us with a clear answer to the question of whether inspections have positive causal effects on the quality of schools. Results of studies of publications on public performance indicators are more unambiguous. They lead us to the conclusion that although principals and teachers believe performance indicators are important, parents and pupils take very little notice of these indicators when choosing schools. A third conclusion concerns the occurrence of side effects of school inspections and other control mechanisms in education. Several of the studies discussed clearly refer to the existence of these side effects, such as ‘window dressing’ and other types of ‘gaming’.

Reducing interrater variability and improving health care: a meta-analytical review

OBJECTIVES In the scientific literature about reliability, the main approach to increasing reliability seems to involve increasing the number of observers and improving the instrument used. Other aspects for improving reliability - like the training of raters - seem to receive less notice. It is worth asking whether this technical approach could be complemented by training the user of the instrument. A systematic meta-analytical review of the research literature was performed to answer this question and examine the effectiveness of planned interventions for improving interrater reliability of health care professionals. METHOD The databases of PubMed (MEDLINE), Embase, Omega and PsycINFO were searched. The inclusion criteria were met by 57 studies. Details extracted from the studies included the study design, the number of observers and the number of observed cases, the intervention, the type of instrument (whether or not it was highly technical), and statistical information about the agreement before and after the intervention. Interventions were categorized into three groups: training of professionals, improving the diagnostic instrument and a combination of training and improving the instrument. A meta-analysis was performed by means of linear regression. RESULTS The interventions were arranged according to their effectiveness in improving the diagnostic instrument (mean change: β = 0.13), training combined with improving the instrument (mean change: β = 0.10) and training (mean change: β = 0.09). CONCLUSION On average, although all types of interventions are effective, improving the diagnostic instrument seems to be the most effective. Especially when highly technical instruments were concerned, improvement proved to be very effective (β = 0.52). Because instrumental variables constitute a major source of error, improving the instrument is an important approach. However, this review offers solid arguments that can complement the literature and practice, with a focus on training the user of the instrument.

Evaluating instruments for regulation of health care in the Netherlands

OBJECTIVES Reliable and valid judgments are necessary for regulatory authorities to merit confidence from care institutions and society and preserve authority. Moreover, limited reliability and validity of regulatory judgments increase the risk of limited improvement of the quality of health care. The goal of the study is to obtain insight in (dis) advantages of different regulatory instruments for regulation of health care. METHOD In this study, the reliability and validity of judgments generated by a lightly structured and highly structured regulatory instrument used by the Dutch Health Care Inspectorate are compared. RESULTS Results indicate that the lightly structured instrument causes a large variety in discussed topics in regulatory visits: indicators pointing out potential risks in care are not always part of these discussions, by which incentives to improve care remain unjustly undone. Both types of instruments show variations in the meaning of judgments, indicating validity problems. CONCLUSION The results of our study suggest that regulation of health care requires thorough appraisal of instruments. Several requirements are identified: first, an instrument that justifies the complexity of care with an accompanying explicit set of standards is necessary. Second, commitment of inspectors to the instrument is essential. And third, training of inspectors is indispensable.

Analyzing the Assumptions of a Policy Program: An Ex-Ante Evaluation of "Educational Governance" in the Netherlands.

This article demonstrates the value of theoretical ex-ante evaluation of policy programs. We show how a specific policy might not achieve its objectives and illustrate the elements of the policy, which need improvement. These conclusions are based on a reconstruction and evaluation of the theory behind a policy program that aims to increase productivity of schools in the Netherlands, the policy of ‘‘educational governance.’’ We chose this program for its interesting combination of measures on internal quality assurances as well as accountability and supervision. The evaluation clearly shows a flaw in the theory underlying the policy program, which threatens its potential effectiveness. Furthermore, we show that there is a risk of contrary and incompatible interests among actors, as well as some practical reasons why the program might not work. Adjustment of the program is recommended before implementation.

Toward a model of school inspections in a polycentric system.

Many education systems are developing towards more lateral structures where schools collaborate in networks to improve and provide (inclusive) education. These structures call for bottom-up models of network evaluation and accountability instead of the current hierarchical arrangements where single schools are evaluated by a central agency. This paper builds on available research about network effectiveness to present evolving models of network evaluation. Network effectiveness can be defined as the achievement of positive network level outcomes that cannot be attained by individual organizational participants acting alone. Models of network evaluation need to take into account the relations between network members, the structure of the network, its processes and its internal mechanism to enforce norms in order to understand the achievement and outcomes of the network and how these may evolve over time. A range of suitable evaluation models are presented in this paper, as well as a tentative school inspection framework which is inspired by these models. The final section will present examples from Inspectorates of Education in Northern Ireland and Scotland who have developed newer inspection models to evaluate the effectiveness of a range of different networks.

Experimental studies to improve the reliability and validity of regulatory judgments on health care in the Netherlands: a randomized controlled trial and before and after case study

Rationale, aims and objectives We examined the effect of two interventions on both the reliability and validity of regulatory judgments: adjusting the regulatory instrument and attending a consensus meeting. Method We adjusted the regulatory instrument. With a randomized controlled trial (RCT) we examined the effect of the adjustments we made to the instrument. In the consensus meeting inspectors discussed cases and had to reach consensus about the order of the cases. We used a before and after case study to assess the effect of the consensus meeting. We compared the judgments assigned in the RCT with the unadjusted instrument with the judgments assigned with the unadjusted instrument after the consensus meeting. Moreover we explored the effect of increasing the number of inspectors per regulatory visit based on the estimates of the two interventions. Results The consensus meeting improved the agreement between inspectors; the variance between inspectors was smallest (0.03) and the reliability coefficient was highest (0.59). Validity is assessed by examining the relation between the assigned judgments and the corporate standard and expressed by a correlation coefficient. This coefficient was highest after the consensus meeting (0.48). Adjustment of the instrument did not increase reliability and validity coefficients. Conclusions Participating in a consensus meeting improved reliability and validity. Increasing the number of inspectors resulted in both higher reliability and validity values. Organizing consensus meetings and increasing the number of inspectors per regulatory visit seem to be valuable interventions for improving regulatory judgments.

De relatie tussen normen en oordelen in het toezicht op de gezondheidszorg

SamenvattingIn dit onderzoek hebben we de nauwkeurigheid van oordelen van inspecteurs van de Inspectie voor de Gezondheidszorg (IGZ) onderzocht. Het gaat om oordelen over vier criteria van verantwoorde zorg die gegeven zijn in toezichtsbezoeken in verpleeghuizen in 2005/2006. Deze bezoeken vonden plaats in het kader van het Gefaseerd Toezicht (GT). We hebben de onderbouwingen van 615 oordelen over de criteria “decubitus”, “voldoende hulp bij eten en drinken”, “permanent toezicht huiskamers” en “zorgzwaartemeting” geanalyseerd. Onderzocht is in hoeverre de argumenten uit de onderbouwingen van de oordelen overeenkomen met de normen uit het toezichtskader van de IGZ. Hierdoor kon worden nagegaan of de door de inspecteurs feitelijk gegeven oordelen overeenkomen met de oordelen die op basis van de aanwezige argumenten bij een strikte toepassing van de IGZ-normen gegeven hadden moeten worden. De nauwkeurigheid van de feitelijke oordelen zijn door twee onafhankelijke beoordelaars geanalyseerd. Het onderzoek laat zien dat er sprake is van een validiteitsprobleem: de betekenis van de feitelijke oordelen verschilt regelmatig. Bij de vier onderzochte criteria leidt dit tot 52% vals-positieve oordelen en 1% vals-negatieve oordelen. Er is geen verband tussen het percentage vals-positieve oordelen en het gemiddelde oordeel over de vier criteria. Vals-positieve oordelen komen voor bij zowel inspecteurs die lager dan gemiddeld oordelen, als bij inspecteurs die hoger dan gemiddeld oordelen. Het geven van een te positief oordeel vergeleken met de IGZ-norm, komt bij alle in het onderzoek betrokken inspecteurs voor en is daarmee een situatiegebonden kenmerk van deze inspecteurs. Het percentage vals-positieve oordelen is afhankelijk van het criterium dat wordt beoordeeld. Daarmee zijn vals-positieve oordelen ook afhankelijk van de kwaliteit van het gehanteerde instrument,AbstractThe relation between the employment of standards and judgment in supervision on healthcareObjective: To evaluate the validity of judgments given by inspectors working with risk based supervision for the Dutch Health Care Inspectorate. Design: Descriptive, qualitative, retrospective study. Method: A total of 615 foundations of arguments from 26 inspectors of the Dutch Healthcare inspectorate were collected from 182 supervision reports. These judgments were given to 4 criteria which were specified for nursing homes in 2005/2006: “pressure ulcers”, “adequate help during meals”, “permanent supervision in living rooms” and “registration of severity of care”. The precision of the foundations of arguments was analyzed by two independent observers. A comparison between those arguments and the conventions designed by the Dutch Health Care Inspectorate was made to determine whether the arguments correspond to those conventions. Results. Within the source of 615 foundations of arguments, 53% of the foundations were inaccurate. In 52% of the cases, the judgments inspectors gave were identified as false-positive and 1% as false-negative. No dependency has been found between the percentage of false-positive judgments and the average judgment of the inspector. The percentages of false-positive judgments differ between criteria. Conclusion: A problem in the validity of judgments has been demonstrated. Giving false-positive judgments is not a personal characteristic of an inspector: all inspectors show this type of behavior, even inspectors who were receiving training. The occurrence of false-positive judgments seems to depend on the criteria that are being evaluated. Therefore, giving false-positive judgments depends on the quality of the instrument.