Counterfactual Cross-Validation: Stable Model Selection Procedure for Causal Inference Models

What is the most effective way to select the best causal model among potential candidates? In this paper, we propose a method to effectively select the best conditional average treatment effect (CATE) predictors from a set of candidates using only an observational validation set. When conducting a model selection or tuning hyperparameters, we are interested in choosing the best model or hyperparameter value. Thus, we focus on accurately preserving the rank order of the CATE prediction performance of causal model candidates. For this purpose, we propose a new model selection procedure that preserves the true ranking of the model performance and minimizes the upper bound of the finite sample uncertainty in model selection. Consistent with the theoretical properties, empirical evaluations demonstrate that our proposed method is more likely to select the best model and set of hyperparameters for both model selection and hyperparameter tuning tasks.