Optimal control of false discovery criteria in the two-group model

The highly influential two group model in testing a large number of statistical hypotheses assumes that the test statistics come from a mixture of a high probability null distribution and a low probability alternative. Optimal control of the marginal false discovery rate (mFDR), in the sense that it provides maximal power (expected true discoveries) subject to mFDR control, is known to be achieved by thresholding the local false discovery rate (locFDR) with a fixed threshold. In this paper we address the challenge of controlling optimally the popular false discovery rate (FDR) or positive FDR (pFDR) rather than mFDR in the two group model. These criteria are less conservative than the mFDR criterion, so they make more rejections in expectation. We derive the optimal multiple testing (OMT) policies for these two error criteria. These policies turn out to be thresholding the locFDR with a threshold that is a function of the entire set of statistics. By carefully studying the structure of the optimal policies, we develop an efficient algorithm for finding these policies. With this algorithm, we can easily derive and apply these OMT procedures for problems with thousands of hypotheses. We show that for $K=5000$ hypotheses there can be significant power gain in OMT with FDR or pFDR versus mFDR control. The OMT policies with FDR and pFDR control coincide when the probability of zero rejections is zero for the OMT policy with FDR control. When the signal is weak, the OMT policy with pFDR control has a significantly lower probability of zero rejections than the OMT policy with FDR control. Our results lead us to suggest that the pFDR is (arguably) the preferred error measure to control optimally for the two group model.