Mauricio Romero

The Performance of Risk Adjustment Models in Colombian Competitive Health Insurance Market

We introduce new risk groups to a standard capitation formula and evaluate risk selection incentives of insurers. The study uses a unique data set of almost 24 million affiliates to Government’s mandatory health insurance system. This data set is very rich in the sense of reporting all claims during year 2010, basic demographic variables, initial diagnostic, health services, pharmaceuticals used, etc. It compromises more than 300 million claims. We construct two diagnostic related groups: an adaptation of the 3M algorithm, and a ad hoc diagnostic related group constructed by the authors. Using standard linear capitations formulas we evaluate incentives for cream skimming using several measures. In general, results show a notable improvement in the explanatory power of health expenditures by introducing the ad hoc diagnostic related groups to the standard Colombian risk adjustment formula. With the new risk groups the R2 of the model is 13.53% as opposed to 1.45% of the current formula. Furthermore, for users in the highest expenditure quintile, expected expenditure is 71% of actual expenditure, as opposed to 27% under the current formula. This suggest there is much space for improving the current Colombian capitation formula using information that is currently available.

On the Optimality of Answer-Copying Indices Theory and Practice

Multiple-choice exams are frequently used as an efficient and objective method to assess learning, but they are more vulnerable to answer copying than tests based on open questions. Several statistical tests (known as indices in the literature) have been proposed to detect cheating; however, to the best of our knowledge, they all lack mathematical support that guarantees optimality in any sense. We partially fill this void by deriving the uniformly most powerful (UMP) test under the assumption that the response distribution is known. In practice, however, we must estimate a behavioral model that yields a response distribution for each question. As an application, we calculate the empirical type I and type II error rates for several indices that assume different behavioral models using simulations based on real data from 12 nationwide multiple-choice exams taken by fifth and ninth graders in Colombia. We find that the most powerful index among those studied, subject to the restriction of preserving the type I error, is one based on the work of Wollack and is superior to the index developed by Wesolowsky.

How to make replication the norm

The publishing system builds in resistance to replication. Paul Gertler, Sebastian Galiani and Mauricio Romero surveyed economics journals to find out how to fix it. The publishing system builds in resistance to replication. Paul Gertler, Sebastian Galiani and Mauricio Romero surveyed economics journals to find out how to fix it.

Risk Adjustment Revisited Using Machine Learning Techniques

Risk adjustment is vital in health policy design. Risk adjustment defines the annual capitation payments to health insurers and is a key determinant of insolvency risk for health insurers. In this study we compare the current risk adjustment formula used by Colombia’s Ministry of Health and Social Protection against alternative specifications that adjust for additional factors. We show that the current risk adjustment formula, which conditions on demographic factors and their interactions, can only predict 30% of total health expenditures in the upper quintile of the expenditure distribution. We also show the government’s formula can improve significantly by conditioning ex ante on measures indicators of 29 long-term diseases. We contribute to the risk adjustment literature by estimating machine learning based models and showing non-parametric methodologies (e.g., boosted trees models) outperform linear regressions even when fitted in a smaller set of regressors.

Can Outsourcing Improve Liberia's Schools? Preliminary Results from Year One of a Three-Year Randomized Evaluation of Partnership Schools for Liberia

After one year, public schools managed by private contractors in Liberia raised student learning by 60 percent, compared to standard public schools. But costs were high, performance varied across contractors, and contracts authorized the largest contractor to push excess pupils and underperforming teachers onto other government schools.

Incentives for Replication in Economics

Replication is a critical component of scientific credibility as it increases our confidence in the reliability of the knowledge generated by original research. Yet, replication is the exception rather than the rule in economics. In this paper, we examine why replication is so rare and propose changes to the incentives to replicate. Our study focuses on software code replication, which seeks to replicate the results in the original paper using the same data as the original study and verifying that the analysis code is correct. We analyse the effectiveness of the current model for code replication in the context of three desirable characteristics: unbiasedness, fairness and efficiency. We find substantial evidence of “overturn bias” that likely leads to many false positives in terms of “finding” or claiming mistakes in the original analysis. Overturn bias comes from the fact that replications that overturn original results are much easier to publish than those that confirm original results. In a survey of editors, almost all responded they would in principle publish a replication study that overturned the results of the original study, but only 29% responded that they would consider publishing a replication study that confirmed the original study results. We also find that most replication effort is devoted to so called important papers and that the cost of replication is high in that posited data and software are very hard to use. We outline a new model for the journals to take over replication post acceptance and prepublication that would solve the incentive problems raised in this paper.

A Derivation of the Optimal Answer-Copying Index and Some Applications

Multiple choice exams are frequently used as an efficient and objective instrument to evaluate knowledge. Nevertheless, they are more vulnerable to answer-copying than tests based on open questions. Several statistical tests (known as indices) have been proposed to detect cheating but to the best of our knowledge they all lack a mathematical support that guarantees optimality in any sense. This work aims at filling this void by deriving the uniform most powerful (UMP) test assuming the response distribution is known. In practice we must estimate a behavioral model that yields a response distribution for each question. We calculate the empirical type-I and type-II error rates for several indices, that assume different behavioral models, using simulations based on real data from twelve nation wide multiple choice exams taken by 5th and 9th graders in Colombia. We find that the index with the highest power among those studied, subject to the restriction of preserving the type-I error, is the one that uses a nominal response model for item answering, conditions on the answers of the individual suspected of being the source of copy and calculates critical values via a normal approximation. This index was first studied by Wollack (1997) and later by W. Van der Linden and Sotaridona (2006) and is superior to the indices studied and developed by Wesolowsky (2000) and Frary, Tideman, and Watts (1977). Furthermore, we compare the performance of the indices on examination rooms with different levels of proctoring and find that increasing the level of proctoring can reduce copying by as much as 50% and that simple strategies such as having different students answer different portions of the test at different times canal so reduce cheating by over 50%. Finally, a Bonferroni type false discovery rate procedure is used to detect massive cheating. The application is straightforward and we believe it could be use to make entire examination rooms retake an exam under stricter surveillance conditions.