Asynchronous COMID: the theoretic basis for sparse gradient tricks on Parameter Server

Asynchronous FTRL and $L2$ norm done at server are two widely used tricks to improve training efficiency, but their convergences are not well-proved. In this paper, we propose asynchronous COMID algorithm and prove its convergence. Then, we establish the equivalence between asynchronous COMID and the above two tricks. Thus, the convergences of above two tricks are also proved. Experimental results show asynchronous COMID reduces the burden of the network without any harm on the convergence speed and final output.