Information Theoretic Validity of Penalized Likelihood

Building upon past work, which developed information theoretic notions of when a penalized likelihood procedure can be interpreted as codelengths arising from a two stage code and when the statistical risk of the procedure has a redundancy risk bound, we present new results and risk bounds showing that the l1 penalty in Gaussian Graphical Models fits the above story. We also show how twice the traditional l0 penalty times plus lower order terms which stay bounded on the whole parameter space has a conditional two stage description length interpretation and present risk bounds for this penalized likelihood procedure.