Automated Experiments on Ad Privacy Settings: A Tale of Opacity, Choice, and Discrimination

To partly address people's concerns over web tracking, Google has created the Ad Settings webpage to provide information about and some choice over the profiles Google creates on users. We present AdFisher, an automated tool that explores how user behaviors, Google's ads, and Ad Settings interact. Our tool uses a rigorous experimental design and analysis to ensure the statistical significance of our results. It uses machine learning to automate the selection of a statistical test. We use AdFisher to find that Ad Settings is opaque about some features of a user's profile, that it does provide some choice on ads, and that these choices can lead to seemingly discriminatory ads. In particular, we found that visiting webpages associated with substance abuse will change the ads shown but not the settings page. We also found that setting the gender to female results in getting fewer instances of an ad related to high paying jobs than setting it to male.