$\ell_2$-$\ell_0$ regularization path tracking algorithms

Sparse signal approximation can be formulated as the mixed $\ell_2$-$\ell_0$ minimization problem $\min_x J(x;\lambda)=\|y-Ax\|_2^2+\lambda\|x\|_0$. We propose two heuristic search algorithms to minimize J for a continuum of $\lambda$-values, yielding a sequence of coarse to fine approximations. Continuation Single Best Replacement is a bidirectional greedy algorithm adapted from the Single Best Replacement algorithm previously proposed for minimizing J for fixed $\lambda$. $\ell_0$ regularization path track is a more complex algorithm exploiting that the $\ell_2$-$\ell_0$ regularization path is piecewise constant with respect to $\lambda$. Tracking the $\ell_0$ regularization path is done in a sub-optimal manner by maintaining (i) a list of subsets that are candidates to be solution supports for decreasing $\lambda$'s and (ii) the list of critical $\lambda$-values around which the solution changes. Both algorithms gradually construct the $\ell_0$ regularization path by performing single replacements, i.e., adding or removing a dictionary atom from a subset. A straightforward adaptation of these algorithms yields sub-optimal solutions to $\min_x \|y-Ax\|_2^2$ subject to $\|x\|_0\leq k$ for contiguous values of $k\geq 0$ and to $\min_x \|x\|_0$ subject to $\|y-Ax\|_2^2\leq\varepsilon$ for continuous values of $\varepsilon$. Numerical simulations show the effectiveness of the algorithms on a difficult sparse deconvolution problem inducing a highly correlated dictionary A.