A Learning-Theoretic Framework for Certified Auditing with Explanations

Responsible use of machine learning requires models be audited for undesirable properties. While a number of auditing algorithms have been proposed in prior work, how to do principled auditing in a general setting has remained ill-understood. This work proposes a formal learning-theoretic framework for auditing, and uses it to investigate if and how model explanations can help audits. Specifically, we propose algorithms for auditing linear classifiers for feature sensitivity using label queries as well as two kinds of explanations, and provide performance guarantees. Our results illustrate that while counterfactual explanations can be extremely helpful for auditing, anchor explanations may not be as beneficial in the worst case.