Are Sampling Heuristics Necessary in Object Detectors?

To alleviate the imbalance between foregrounds and backgrounds, prevalent object detectors to date are always equipped with sampling heuristics, which have been regarded as a necessary component thus far. In this paper, we challenge this paradigm. Our investigation reveals that, with careful training and inference schemes, the well-known RetinaNet could still achieve similar accuracy even without Focal Loss. Inspired by this observation, we propose Sampling-Free mechanism as an alternative to sampling heuristics, which addresses the imbalance from aspects of initialization, loss and inference, thus avoiding laborious hyper-parameters tuning in sampling heuristics. As extensive experimental results will demonstrate, sampling-free mechanism works well for one-stage, two-stage and anchor-free object detectors, with the better performance achieved than sampling-based models. Moreover, it is also effective for instance segmentation. Given the public available implementation \url{https://github.com/ChenJoya/sampling-free}, we sincerely expect our discovery could simplify object detectors training.