Probabilistic Contraction Analysis of Iterated Random Operators

Consider a contraction operator $T$ over a Polish space $\mathcal X$ with a fixed point $x^\star$. Assume that one can approximate the operator $T$ by a random operator $\hat T^n$ using $n\in\mathbb{N}$ independent and identically distributed samples of a random variable. Consider the sequence $(\hat X^n_k)_{k\in\mathbb{N}}$, which is generated by $\hat X^n_{k+1} = \hat T^n(\hat X^n_k)$ and is a random sequence. In this paper, we prove that under certain conditions on the random operator, (i) the distribution of $\hat X^n_k$ converges to a unit mass over $x^\star$ as $k$ and $n$ goes to infinity, and (ii) the probability that $\hat X^n_k$ is far from $x^\star$ as $k$ goes to infinity can be made arbitrarily small by an appropriate choice of $n$. We also find a lower bound on the probability that $\hat X^n_k$ is far from $x^\star$ as $k\rightarrow \infty$. We apply the result to study probabilistic convergence of certain randomized value and Q-value iteration algorithms for discounted and average cost Markov decision processes.