Researchers often use rankings of popular websites when measuring security practices, evaluating defenses or analyzing ecosystems. However, little is known about the data collection and processing methodologies of these rankings. In this paper, we uncover how both inherent properties and vulnerabilities to adversarial manipulation of these rankings may affect the conclusions of security studies. To that end, we compare four main rankings used in recent studies in terms of their agreement with each other, stability, representation bias and potential impact on research cases. Additionally, we introduce and illustrate novel ways for the adversary to manipulate the rankings and bend research results to their will. Overall, our study reveals significant shortcomings of current rankings and calls for a more cautious approach from the research community.
View on arXiv