Let X1,X2,…X_1,X_2,\ldots and XX be random variables defined on a common probability space (Ω,F,P).\left(\Omega,\mathcal{F},\mathbb{P}\right). The sequence converges surely to XX if and only if Xn(ω)→X(ω)X_n(\omega)\rightarrow X(\omega) for every outcome ω∈Ω.\omega\in\Omega. Sure convergence is pointwise convergence of the random variables as functions on the sample space. It is too strong to be useful, since it is sensitive to outcomes of probability zero, and the modes of convergence below weaken it in different ways.

Let X1,X2,…X_1,X_2,\ldots and XX be random variables defined on a common probability space. The sequence converges almost surely to X,X, written Xn→a.s.X,X_n\xrightarrow{a.s.}X, if and only if the set of outcomes on which Xn(ω)→X(ω)X_n(\omega)\rightarrow X(\omega) has probability one.

P({ω∈Ω:lim⁡n→∞Xn(ω)=X(ω)})=1\mathbb{P}\left(\left\{\omega\in\Omega:\lim_{n\rightarrow\infty}X_n(\omega)=X(\omega)\right\}\right)=1

Let X1,X2,…X_1,X_2,\ldots and XX be random variables defined on a common probability space. The sequence converges in probability to X,X, written Xn→pX,X_n\xrightarrow{p}X, if and only if the probability of a deviation of any fixed size tends to zero.

lim⁡n→∞P(∣Xn−X∣>ε)=0  ∀ ε>0\lim_{n\rightarrow\infty}\mathbb{P}\left(\lvert X_n-X\rvert>\varepsilon\right)=0\,\,\forall\,\varepsilon>0

Let X1,X2,…X_1,X_2,\ldots and XX be random variables defined on a common probability space and r≥1.r\geq 1. The sequence converges in rr-th mean to X,X, written Xn→LrX,X_n\xrightarrow{L^r}X, if and only if E[∣Xn∣r]<∞\mathbb{E}\left[\lvert X_n\rvert^r\right]<\infty for all nn and the rr-th moment of the difference tends to zero.

lim⁡n→∞E[∣Xn−X∣r]=0\lim_{n\rightarrow\infty}\mathbb{E}\left[\lvert X_n-X\rvert^r\right]=0

Convergence in mean square is convergence in 22-nd mean. Convergence in mean is convergence in 11-st mean.

Let X1,X2,…X_1,X_2,\ldots and XX be random variables with cdfs FXnF_{X_n} and FX,F_X, not necessarily defined on a common probability space. The sequence converges in distribution or converges weakly to X,X, written Xn→dX,X_n\xrightarrow{d}X, if and only if the cdfs converge at every point where the limiting cdf is continuous.

lim⁡n→∞FXn(x)=FX(x)  ∀ x  at  which  FX  is  continuous\lim_{n\rightarrow\infty}F_{X_n}\left(x\right)=F_X\left(x\right)\,\,\forall\,x\,\,\mathrm{at}\,\,\mathrm{which}\,\,F_X\,\,\mathrm{is}\,\,\mathrm{continuous}

Convergence in distribution has the following properties:

  • it is a statement about the distributions of XnX_n alone, so Xn→dXX_n\xrightarrow{d}X is also written Xn→dPX;X_n\xrightarrow{d}\mathbb{P}_X;
  • the exclusion of discontinuity points is needed so that, for example, the constants 1n\frac{1}{n} converge in distribution to 00 even though FXn(0)=0F_{X_n}\left(0\right)=0 does not converge to FX(0)=1.F_X\left(0\right)=1.

Let X1,X2,…X_1,X_2,\ldots and XX be random variables. The portmanteau theorem states that convergence in distribution is equivalent to convergence of the expectations of all bounded continuous functions.

Xn→dX⇔lim⁡n→∞E[g(Xn)]=E[g(X)]  ∀ bounded  continuous  g:R→RX_n\xrightarrow{d}X\Leftrightarrow\lim_{n\rightarrow\infty}\mathbb{E}\left[g\left(X_n\right)\right]=\mathbb{E}\left[g\left(X\right)\right]\,\,\forall\,\mathrm{bounded}\,\,\mathrm{continuous}\,\,g:\mathbb{R}\rightarrow\mathbb{R}

The condition of the portmanteau theorem is the definition of convergence in distribution for random vectors.

Let X1,X2,…X_1,X_2,\ldots and XX be random variables with characteristic functions φXn\varphi_{X_n} and φX.\varphi_X. Lévy’s continuity theorem states that Xn→dXX_n\xrightarrow{d}X if and only if the characteristic functions converge pointwise, φXn(t)→φX(t)\varphi_{X_n}\left(t\right)\rightarrow\varphi_X\left(t\right) for all t∈R,t\in\mathbb{R}, and that if φXn\varphi_{X_n} converges pointwise to a function that is continuous at 00 then that function is the characteristic function of some random variable XX and Xn→dX.X_n\xrightarrow{d}X. The same holds for mgfs when they exist on a common open interval containing 0.0.

The modes of convergence are related as follows:

  • almost sure convergence implies convergence in probability;
  • convergence in rr-th mean implies convergence in probability, by Markov’s inequality applied to ∣Xn−X∣r;\lvert X_n-X\rvert^r;
  • convergence in ss-th mean implies convergence in rr-th mean for 1≤r<s,1\leq r<s, by Lyapunov’s inequality;
  • convergence in probability implies convergence in distribution;
  • convergence in distribution to a constant cc implies convergence in probability to c;c;
  • convergence in probability implies almost sure convergence along some subsequence.

Neither almost sure convergence nor convergence in rr-th mean implies the other, and no other implications hold in general. For example, if Xn=n 1AnX_n=n\,\mathbf{1}_{A_n} with P(An)=1n\mathbb{P}\left(A_n\right)=\frac{1}{n} then Xn→p0X_n\xrightarrow{p}0 but E[Xn]=1\mathbb{E}\left[X_n\right]=1 for every n,n, so XnX_n does not converge in mean.

Let A1,A2,…A_1,A_2,\ldots be events with ∑n=1∞P(An)<∞.\sum_{n=1}^{\infty}\mathbb{P}\left(A_n\right)<\infty. The Borel–Cantelli lemma states that with probability one only finitely many of the events occur. Applied to An={∣Xn−X∣>ε},A_n=\left\{\lvert X_n-X\rvert>\varepsilon\right\}, the lemma gives the standard route to almost sure convergence: if ∑n=1∞P(∣Xn−X∣>ε)<∞\sum_{n=1}^{\infty}\mathbb{P}\left(\lvert X_n-X\rvert>\varepsilon\right)<\infty for every ε>0\varepsilon>0 then Xn→a.s.X.X_n\xrightarrow{a.s.}X. Concentration inequalities that give summable tail bounds therefore yield almost sure convergence.

Let X1,X2,…X_1,X_2,\ldots and XX be random variables and g:R→Rg:\mathbb{R}\rightarrow\mathbb{R} be continuous at every point of a set that XX lies in with probability one. The continuous mapping theorem states that convergence of XnX_n to XX is preserved by gg in each mode.

Xn→a.s.X⇒g(Xn)→a.s.g(X)X_n\xrightarrow{a.s.}X\Rightarrow g\left(X_n\right)\xrightarrow{a.s.}g\left(X\right) Xn→pX⇒g(Xn)→pg(X)X_n\xrightarrow{p}X\Rightarrow g\left(X_n\right)\xrightarrow{p}g\left(X\right) Xn→dX⇒g(Xn)→dg(X)X_n\xrightarrow{d}X\Rightarrow g\left(X_n\right)\xrightarrow{d}g\left(X\right)

The continuous mapping theorem also holds for random vectors and continuous g:Rk→Rm.g:\mathbb{R}^k\rightarrow\mathbb{R}^m. In particular, sums, products, powers and ratios with non-zero denominators of sequences that converge jointly also converge.

Let X1,X2,…,X_1,X_2,\ldots, X,X, Y1,Y2,…Y_1,Y_2,\ldots be random variables and cc be a constant with Xn→dXX_n\xrightarrow{d}X and Yn→pc.Y_n\xrightarrow{p}c. Slutsky’s theorem states that sums, products and quotients of the two sequences converge in distribution to the corresponding combinations of the limits.

Xn+Yn→dX+c,XnYn→dcX,XnYn→dXc  (c≠0)X_n+Y_n\xrightarrow{d}X+c,\qquad X_nY_n\xrightarrow{d}cX,\qquad\frac{X_n}{Y_n}\xrightarrow{d}\frac{X}{c}\,\,\left(c\neq 0\right)

Convergence in distribution of XnX_n to XX and of YnY_n to YY separately does not imply convergence in distribution of (Xn,Yn)\left(X_n,Y_n\right) or of Xn+Yn,X_n+Y_n, since the joint distribution is not determined by the marginals, so Slutsky’s theorem covers the case that is needed most often. Slutsky’s theorem allows unknown constants in a limit, such as a standard deviation, to be replaced by estimates that converge in probability to them without changing the limiting distribution.

Let θ∈R,\theta\in\mathbb{R}, X1,X2,…X_1,X_2,\ldots be random variables with n(Xn−θ)→dN(0,σ2)\sqrt{n}\left(X_n-\theta\right)\xrightarrow{d}\mathrm{N}\left(0,\sigma^2\right) and g:R→Rg:\mathbb{R}\rightarrow\mathbb{R} be differentiable at θ\theta with g′(θ)≠0.g'\left(\theta\right)\neq 0. The delta method states that the transformed sequence is asymptotically normal with variance scaled by the squared derivative.

n(g(Xn)−g(θ))→dN(0,g′(θ)2σ2)\sqrt{n}\left(g\left(X_n\right)-g\left(\theta\right)\right)\xrightarrow{d}\mathrm{N}\left(0,g'\left(\theta\right)^2\sigma^2\right)

The delta method follows from the first-order Taylor expansion g(Xn)≈g(θ)+g′(θ)(Xn−θ),g\left(X_n\right)\approx g\left(\theta\right)+g'\left(\theta\right)\left(X_n-\theta\right), since Xn→pθX_n\xrightarrow{p}\theta makes the remainder negligible, together with Slutsky’s theorem.

Let θ∈R,\theta\in\mathbb{R}, X1,X2,…X_1,X_2,\ldots be random variables with n(Xn−θ)→dN(0,σ2)\sqrt{n}\left(X_n-\theta\right)\xrightarrow{d}\mathrm{N}\left(0,\sigma^2\right) and g:R→Rg:\mathbb{R}\rightarrow\mathbb{R} be twice differentiable at θ\theta with g′(θ)=0g'\left(\theta\right)=0 and g′′(θ)≠0.g''\left(\theta\right)\neq 0. The second-order delta method states that the transformed sequence converges at rate nn to a scaled chi-squared distribution, using the next term of the expansion.

n(g(Xn)−g(θ))→dσ2g′′(θ)2χ12n\left(g\left(X_n\right)-g\left(\theta\right)\right)\xrightarrow{d}\frac{\sigma^2g''\left(\theta\right)}{2}\chi^2_1

Let θ‾∈Rk,\underline{\theta}\in\mathbb{R}^k, X‾1,X‾2,…\underline{X}_1,\underline{X}_2,\ldots be random vectors with n(X‾n−θ‾)→dNk(0‾,Σ)\sqrt{n}\left(\underline{X}_n-\underline{\theta}\right)\xrightarrow{d}\mathrm{N}_k\left(\underline{0},\Sigma\right) and g:Rk→Rg:\mathbb{R}^k\rightarrow\mathbb{R} be differentiable at θ‾\underline{\theta} with ∇g(θ‾)≠0‾.\nabla g\left(\underline{\theta}\right)\neq\underline{0}. The multivariate delta method states that

n(g(X‾n)−g(θ‾))→dN(0,∇g(θ‾)⊺Σ∇g(θ‾)).\sqrt{n}\left(g\left(\underline{X}_n\right)-g\left(\underline{\theta}\right)\right)\xrightarrow{d}\mathrm{N}\left(0,\nabla g\left(\underline{\theta}\right)^\intercal\Sigma\nabla g\left(\underline{\theta}\right)\right).

For vector-valued g‾\underline{g} the limiting covariance matrix is Jg‾(θ‾)ΣJg‾(θ‾)⊺.J_{\underline{g}}\left(\underline{\theta}\right)\Sigma J_{\underline{g}}\left(\underline{\theta}\right)^\intercal. The delta method is the standard tool for finding the limiting distribution of a smooth function of an average, such as a ratio, a logarithm or a variance.