Optimization Theory in Statistics

This paper addresses the issues of optimization theory and related numerical issues within the context of Statistics. Focusing on the problem of concave regression, several estimation techniques for nonparametric shape-constrained regression are classified, analyzed and compared qualitatively and quantitatively through numerical simulations. In particular, their main features, strengths and limitations for solving large instances of the problem are examined through this paper. Several improvements to enhance numerical stability and bound the computational cost are proposed. For each analyzed algorithm, the pseudo-code and its corresponding code in Scilab are provided. The results from this study demonstrate that the choice of the optimization approach strongly impacts algorithmic performances. Interestingly, it is also shown that, currently, there are not available methods able to solve efficiently large instance of the concave regression problems (more than many thousands of points). We suggest that further research should focus on finding a way to exploit and adapt classical multi-scale strategy to compute an approximate solution.