Convergence of random variables

In probability theory, there exist several different notions of convergence of random variables. The convergence (in one of the senses presented below) of sequences of random variables to some limiting random variable is an important concept in probability theory, and its applications to statistics and stochastic processes. For example, if the average of n independent, identically distributed random variables Y_i, i = 1, ..., n, is given by

<math>X_n = \frac{1}{n}\sum_{i=1}^n Y_i\,,<math>

then as n goes to infinity, X_n converges in probability (see below) to the common mean, μ, of the random variables Y_i. This result is known as the weak law of large numbers. Other forms of convergence are important in other useful theorems, including the central limit theorem.

Throughout the following, we assume that (X_n) is a sequence of random variables, and X is a random variable, and all of them are defined on the same probability space (Ω, F, P).

Contents

1 Convergence in distribution 2 Convergence in probability 2.1 Lemma 2.2 Proof 3 Almost sure convergence 4 Convergence in rth mean 5 Converse implications 6 References

Convergence in distribution

Suppose that F₁, F₂, ... is a sequence of cumulative distribution functions corresponding to random variables X₁, X₂, ..., and that F is a distribution function corresponding to a random variable X. We say that the sequence X_n converges towards X in distribution, if

<math>\lim_{n\rightarrow\infty} F_n(a) = F(a),<math>

for every real number a at which F is continuous. Since F(a) = Pr(X ≤ a), this means that the probability that the value of X is in a given range is very similar to the probability that the value of X_n is in that range, provided n is large enough. Convergence in distribution is often denoted by adding the letter 'D' over an arrow indicating convergence:

<math>X_n \, \begin{matrix} {\,}_D \\ {\,}^{\rightarrow} \end{matrix} \, X.<math>

Convergence in distribution is the weakest form of convergence, and is sometimes called weak convergence. It does not, in general, imply any other mode of convergence. However, convergence in distribution is implied by all other modes of convergence mentioned in this article, and hence, it is the most common and often the most useful form of convergence of random variables. It is the notion of convergence used in the central limit theorem and the (weak) law of large numbers.

A useful result, which may be employed in conjunction with law of large numbers and the central limit theorem, is that if a function  g: R → R  is continuous, then if  X_n  converges in distribution to  X, then so too does  g(X_n)  converge in distribution to  g(X). (This may be proved using Skorokhod's representation theorem.)

Convergence in distribution is also called convergence in law, since the word "law" is sometimes used as a synonym of "probability distribution."

Convergence in probability

We say that the sequence X_n converges towards X in probability if

<math>\lim_{n\rightarrow\infty}P\left(\left|X_n-X\right|\geq\varepsilon\right)=0<math>

for every ε > 0. Convergence in probability is, indeed, the (pointwise) convergence of probabilities. Pick any ε > 0 and any δ > 0. Let P_n be the probability that X_n is outside a tolerance ε of X. Then, if X_n converges in probability to X then there exists a value N such that, for all n ≥ N, P_n is itself less than δ.

Convergence in probability is the notion of convergence used in the weak law of large numbers. Convergence in probability implies convergence in distribution. To prove it, it's convenient to prove the following, simple lemma:

Lemma

Be X, Y random variables, c a real number and ε > 0; then

<math>P(Y\leq c)\leq P(X\leq c+\varepsilon)+P(\left|Y - X\right|>\varepsilon)<math>

In fact,

<math>P(Y\leq c)=P(Y\leq c,X\leq c+\varepsilon)+P(Y\leq c,X>c+\varepsilon)<math>

<math>=P(Y\leq c \vert X\leq c+\varepsilon)P(X\leq c+\varepsilon)+P(Y\leq c,c

<math>\leq P(X\leq c+\varepsilon)+P(Y - X<- \varepsilon)\leq P(X\leq c+\varepsilon)+P(\left|Y - X\right|>\varepsilon)<math>

since

<math>P(\left|Y - X\right|>\varepsilon)=P(Y - X>\varepsilon)+P(Y - X<-\varepsilon)\geq P(Y - X<-\varepsilon).<math>

Proof

For every ε > 0, due to the preceding lemma, we have:

<math>P(X_n\leq a)\leq P(X\leq a+\varepsilon)+P(\left|X_n - X\right|>\varepsilon)<math>

<math>P(X\leq a-\varepsilon)\leq P(X_n \leq a)+P(\left|X_n - X\right|>\varepsilon)<math>

So, we have:

<math>P(X\leq a-\varepsilon)-P(\left|X_n - X\right|>\varepsilon)\leq P(X_n \leq a)\leq P(X\leq a+\varepsilon)+P(\left|X_n - X\right|>\varepsilon)<math>

Taking the limit for <math>n\rightarrow\infty<math>, we obtain:

<math>P(X\leq a-\varepsilon)\leq \lim_{n\rightarrow\infty} P(X_n \leq a)\leq P(X\leq a+\varepsilon)<math>

But <math>P(X\leq a)<math> is the cumulative distribution function <math>F_X(a)<math>, which is continuous by hypothesis, that is:

<math>\lim_{\varepsilon \rightarrow 0^+} F_X(a-\varepsilon)=\lim_{\varepsilon \rightarrow 0^+} F_X(a+\varepsilon)=F_X(a)<math>

and so, taking the limit for <math>\varepsilon \rightarrow 0^+<math>, we obtain:

<math>\lim_{n\rightarrow\infty} P(X_n \leq a)=P(X \leq a)<math>

Almost sure convergence

We say that the sequence X_n converges almost surely or almost everywhere or with probability 1 or strongly towards X if

<math>P\left(\lim_{n\rightarrow\infty}X_n=X\right)=1.<math>

This means that you are virtually guaranteed that the values of X_n approach the value of X, in the sense (see almost surely) that events for which X_n does not converge to X have probability 0. Using the probability space (Ω, F, P) and the concept of the random variable as a function from Ω to R, this is equivalent to the statement

<math>P\left(\{\omega \in \Omega: X_n(\omega) \rightarrow X \,\, \mathrm{as} \,\, n \rightarrow \infty\}\right) = 1.<math>

Almost sure convergence implies convergence in probability, and hence implies convergence in distribution. It is the notion of convergence used in the strong law of large numbers.

Convergence in rth mean

We say that the sequence X_n converges in rth mean or in the L^r norm towards X, if r ≥ 1, E|X_n| < ∞ for all n, and

<math>\lim_{n\rightarrow\infty}\mathrm{E}\left(\left|X_n-X\right|^r\right)=0<math>

where the operator E denotes the expected value. Convergence in rth mean tells us that the expectation of the rth power of the difference between X_n and X converges to zero.

The most important cases of convergence in rth mean are:

• When X_n converges in rth mean to X for r = 1, we say that X_n converges in mean to X.
• When X_n converges in rth mean to X for r = 2, we say that X_n converges in mean square to X.

Convergence in rth mean, for r ≥ 1, implies convergence in probability (by Chebyshev's inequality), while if r > s ≥ 1, convergence in rth mean implies convergence in sth mean. Hence, convergence in mean square implies convergence in mean.

Converse implications

The chain of implications between the various notions of convergence, above, are noted in their respective sections, but it is sometimes important to establish converses to these implications. No other implications other than those noted above hold in general, but a number of special cases do permit converses:

• If X_n converges in distribution to a constant c, then X_n converges in probability to c.

• If X_n converges in probability X, and if Pr(|X_n| ≤ b) = 1 for all n and some b, then X_n converges in rth mean to X for all r ≥ 1. In other words, if X_n converges in probability to X and all random variables X_n are almost surely bounded above and below, then X_n converges to X also in any rth mean.

• If for all ε > 0,

<math>\sum_n P\left(|X_n - X| > \varepsilon\right) < \infty,<math>

then X_n converges almost surely to X. In other words, if X_n converges in probability to X sufficiently quickly (i.e. the above sum converges for all ε > 0), then X_n also converges almost surely to X.

References

1. G.R. Grimmett and D.R. Stirzaker (1992). Probability and Random Processes, 2nd Edition. Clarendon Press, Oxford, pp 271--285. ISBN 0198536658.pl:Zbieżność według rozkładu