Identifying Significant Predictive Bias in Classifiers

We present a novel subset scan method to detect if a probabilistic binary classifier has statistically significant bias -- over or under predicting the risk -- for some subgroup, and identify the characteristics of this subgroup. This form of model checking and goodness-of-fit test provides a way to interpretably detect the presence of classifier bias and poor classifier fit, not just in one or two dimensions of features of a priori interest, but in the space of all possible feature subgroups. We use subset scan and parametric bootstrap methods to efficiently address the difficulty of assessing the exponentially many possible subgroups. We also suggest several useful extensions of this method for increasing interpretability of predictive models and prediction performance.