Analysis of gradient descent methods with non-diminishing, bounded errors

In this paper, we present easily verifiable, sufficient conditions for both stability and convergence (to the minimum set) of gradient descent ($GD$) algorithms with bounded, non-diminishing errors. These errors often arise from using gradient estimators or because the objective function is noisy to begin with. Our work extends the contributions of Mangasarian \& Solodov and Bertsekas \& Tsitsiklis. Our framework improves over the aforementioned ones in that both stability (almost sure boundedness) and convergence are guaranteed even in the case of $GD$ with non-diminishing errors. We present a simplified, yet effective implementation of $GD$ using $SPSA$ with constant sensitivity parameters. Further, unlike other papers no `additional' restrictions are imposed on the step-size and so also on the learning rate when used to implement machine learning algorithms. Finally, we present the results of some experiments to validate the theory.