Feature Selection with Annealing for Regression, Classification and Ranking

Many computer vision and medical imaging problems are faced with learning from large datasets, with millions of observations and features. In this paper we propose a novel efficient algorithm for variable selection and learning on such datasets, optimizing a likelihood with sparsity constraints. The iterative algorithm alternates parameter updates with tightening the constraints by gradually removing variables based on a criterion and a schedule. We present a generic approach for optimizing any differentiable loss function and present applications to regression, classification and ranking. We use one dimensional piecewise linear response functions to introduce nonlinear dependence on the selected variables and a second order prior on the response functions to avoid overfitting. We obtain theoretical guarantees of convergence and variable selection consistency for regression and logistic regression. Experiments on real and synthetic data show that the proposed method compares very well with other state of the art methods in regression, classification and ranking while being computationally very efficient.