Learning Decision Thresholds for Risk-Stratification Models from Aggregate Clinician Behavior

Abstract

Deploying a risk-stratification model to guide clinical practice often requires the choice of a cutoff on the model s output - called the decision threshold - to trigger a subsequent action such as an electronic alert. Choosing this cutoff is not always straightforward. Leveraging the collective information in treatment decisions made in real life, we propose an approach that learns reference decision thresholds from physician practice. Using the example of prescribing a statin for primary prevention of cardiovascular disease based on 10-year risk calculated by the 2013 Pooled Cohort Equations, we demonstrate the feasibility of using real world data to learn the implicit decision threshold that reflects existing physician behavior. Learning a decision threshold in this manner allows for evaluation of a proposed operating point against the threshold reflective of the standard of care. Furthermore, this approach can be used to monitor and audit model-guided clinical decision-making following model deployment.