Let (Ω,F,P)\left(\Omega,\mathcal{F},\mathbb{P}\right) be a probability space and (S,S)\left(S,\mathcal{S}\right) be a measurable space. A random variable is a function X:Ω→SX:\Omega\rightarrow S such that the inverse image of every measurable subset of SS is an event.

{ω∈Ω:X(ω)∈A}∈F  ∀ A∈S\left\{\omega\in\Omega:X\left(\omega\right)\in A\right\}\in\mathcal{F}\,\,\forall\,A\in\mathcal{S}

Let X:Ω→SX:\Omega\rightarrow S be a random variable. The set SS is the state space of XX and S\mathcal{S} is its σ\sigma-algebra. The random variable XX is real-valued if and only if S⊆R.S\subseteq\mathbb{R}. The notation X∈AX\in A denotes the event {ω∈Ω:X(ω)∈A}.\left\{\omega\in\Omega:X\left(\omega\right)\in A\right\}.

Let X:Ω→SX:\Omega\rightarrow S be a random variable on the probability space (Ω,F,P).\left(\Omega,\mathcal{F},\mathbb{P}\right). The pushforward probability measure of XX is the probability measure PX\mathbb{P}_X on (S,S)\left(S,\mathcal{S}\right) that assigns to each measurable subset of SS the probability of its inverse image.

PX(A):=P({ω∈Ω:X(ω)∈A})  ∀ A∈S\mathbb{P}_X\left(A\right) := \mathbb{P}\left(\left\{\omega\in\Omega:X\left(\omega\right)\in A\right\}\right)\,\,\forall\,A\in\mathcal{S}

The pushforward probability measure is also written PX(X∈A)\mathbb{P}_X\left(X\in A\right) or P(X∈A).\mathbb{P}\left(X\in A\right). The triple (S,S,PX)\left(S,\mathcal{S},\mathbb{P}_X\right) is a probability space, which allows the underlying probability space (Ω,F,P)\left(\Omega,\mathcal{F},\mathbb{P}\right) to be left implicit.

Let XX be a random variable with pushforward probability measure PX.\mathbb{P}_X. The probability distribution of XX is the probability measure PX\mathbb{P}_X on the state space.

Let XX be a real-valued random variable. The cumulative distribution function (cdf) of XX is the function FX:R→[0,1]F_X:\mathbb{R}\rightarrow\left[0,1\right] that gives the probability that XX is at most x.x.

FX(x):=PX(X≤x)  ∀ x∈RF_X\left(x\right):=\mathbb{P}_X\left(X\leq x\right)\,\,\forall\,x\in\mathbb{R}

The cdf has the following properties:

  • FXF_X is monotonically increasing;
  • FXF_X is right-continuous;
  • lim⁡x→−∞FX(x)=0\lim_{x\rightarrow-\infty}F_X\left(x\right)=0 and lim⁡x→∞FX(x)=1;\lim_{x\rightarrow\infty}F_X\left(x\right)=1;
  • the probability of a half-open interval is a difference of cdf values, PX(a1<X≤a2)=FX(a2)−FX(a1)\mathbb{P}_X\left(a_1<X\leq a_2\right)=F_X\left(a_2\right)-F_X\left(a_1\right) for a1<a2;a_1<a_2;
  • the cdf determines the probability distribution of X.X.

Let XX be a random variable with state space S.S. The random variable XX is a discrete random variable if and only if SS is countable, that is, S={x1,…,xn}S=\{x_1,\ldots,x_n\} is finite or S={x1,x2,…}S=\{x_1,x_2,\ldots\} is countably infinite. The state space of a discrete random variable is a discrete state space.

Let XX be a discrete random variable with state space S.S. The probability mass function (pmf) of XX is the function pX:S→[0,1]p_X:S\rightarrow\left[0,1\right] that gives the probability that XX takes each value.

pX(x):=PX(X=x)  ∀ x∈Sp_X\left(x\right):=\mathbb{P}_X\left(X=x\right)\,\,\forall\,x\in S

The pmf is extended to R\mathbb{R} by setting pX(x)=0p_X\left(x\right)=0 for x∈R∖S.x\in\mathbb{R}\setminus S. The pmf has the following properties:

pX(x)≥0  ∀ x∈Sp_X\left(x\right)\geq 0\,\,\forall\,x\in S ∑x∈SpX(x)=1\sum_{x\in S}p_X\left(x\right)=1 PX(X∈A)=∑x∈ApX(x)  ∀ A∈S\mathbb{P}_X\left(X\in A\right)=\sum_{x\in A}p_X\left(x\right)\,\,\forall\,A\in\mathcal{S}

The last property holds since the singleton sets {x},\{x\}, x∈S,x\in S, form a partition of SS into disjoint events.

Let XX be a discrete random variable with state space {x1,x2,…}\left\{x_1,x_2,\ldots\right\} where x1<x2<…,x_1<x_2<\ldots, pmf pXp_X and cdf FX.F_X. The cdf and pmf are related as follows: FX(x)=0F_X\left(x\right)=0 for x<x1,x<x_1, and

FX(x)=∑xi≤xpX(xi)  ∀ x≥x1,pX(xi)=FX(xi)−FX(xi−1)  ∀ i≥2.F_X\left(x\right)=\sum_{x_i\leq x}p_X\left(x_i\right)\,\,\forall\,x\geq x_1,\qquad p_X\left(x_i\right)=F_X\left(x_i\right)-F_X\left(x_{i-1}\right)\,\,\forall\,i\geq 2.

Let XX be a real-valued random variable. The random variable XX is a continuous random variable if and only if its cdf is a continuous function on R.\mathbb{R}.

Let XX be a real-valued random variable. The random variable XX is absolutely continuous if and only if there exists a function fX:R→[0,∞)f_X:\mathbb{R}\rightarrow\left[0,\infty\right) such that the cdf is the integral of fX.f_X.

FX(x)=∫−∞xfX(t) dt  ∀ x∈RF_X\left(x\right)=\int_{-\infty}^x f_X\left(t\right)\,dt\,\,\forall\,x\in\mathbb{R}

Let XX be an absolutely continuous random variable. The probability density function (pdf) of XX is the function fXf_X above. Below, a continuous random variable is taken to be absolutely continuous. The pdf has the following properties:

fX(x)≥0  ∀ x∈Rf_X\left(x\right)\geq 0\,\,\forall\,x\in\mathbb{R} ∫−∞∞fX(x) dx=1\int_{-\infty}^{\infty}f_X\left(x\right)\,dx=1 fX(x)=ddxFX(x)  wherever  FX  is  differentiablef_X\left(x\right)=\frac{d}{dx}F_X\left(x\right)\,\,\mathrm{wherever}\,\,F_X\,\,\mathrm{is}\,\,\mathrm{differentiable} PX(X∈A)=∫AfX(x) dx\mathbb{P}_X\left(X\in A\right)=\int_A f_X\left(x\right)\,dx

The cdf is differentiable wherever fXf_X is continuous. Continuous random variables have the following properties:

  • the probability that XX takes any single value is zero, PX(X=x)=0\mathbb{P}_X\left(X=x\right)=0 for all x∈R;x\in\mathbb{R};
  • the density fX(x)f_X\left(x\right) is not a probability and can exceed one;
  • including or excluding the endpoints of an interval does not change its probability, PX(x1≤X≤x2)=PX(x1<X<x2)=FX(x2)−FX(x1)\mathbb{P}_X\left(x_1\leq X\leq x_2\right)=\mathbb{P}_X\left(x_1<X<x_2\right)=F_X\left(x_2\right)-F_X\left(x_1\right) for x1<x2.x_1<x_2.

Let XX be a real-valued random variable with probability distribution PX.\mathbb{P}_X. The support of PX\mathbb{P}_X is the smallest closed set X⊆R\mathcal{X}\subseteq\mathbb{R} with PX(X∈X)=1.\mathbb{P}_X\left(X\in\mathcal{X}\right)=1. Equivalently, the support is the set of points whose every neighbourhood has positive probability.

X={x∈R:PX(x−ε<X<x+ε)>0  ∀ ε>0}\mathcal{X}=\left\{x\in\mathbb{R}:\mathbb{P}_X\left(x-\varepsilon<X<x+\varepsilon\right)>0\,\,\forall\,\varepsilon>0\right\}

Individual points of the support can have probability zero, and for a continuous random variable every point does. In practice the support of a discrete random variable is taken to be {x:pX(x)>0}\left\{x:p_X\left(x\right)>0\right\} and the support of a continuous random variable is taken to be {x:fX(x)>0},\left\{x:f_X\left(x\right)>0\right\}, which differ from the definition above by at most a set of probability zero.

Let X1X_1 and X2X_2 be random variables with a common state space. The random variables X1X_1 and X2X_2 are identically distributed if and only if they have the same probability distribution.

PX1(X1∈A)=PX2(X2∈A)  ∀ A\mathbb{P}_{X_1}\left(X_1\in A\right)=\mathbb{P}_{X_2}\left(X_2\in A\right)\,\,\forall\,A

Let X1,…,XnX_1,\ldots,X_n be real-valued random variables defined on a common probability space (Ω,F,P).\left(\Omega,\mathcal{F},\mathbb{P}\right). The random vector X‾=(X1,…,Xn)\underline{X}=\left(X_1,\ldots,X_n\right) is the function X‾:Ω→Rn\underline{X}:\Omega\rightarrow\mathbb{R}^n with components Xi.X_i.

Let X‾=(X1,…,Xn)\underline{X}=\left(X_1,\ldots,X_n\right) be a random vector. The joint distribution of X‾\underline{X} is its pushforward probability measure PX‾\mathbb{P}_{\underline{X}} on Rn.\mathbb{R}^n.

Let X‾=(X1,…,Xn)\underline{X}=\left(X_1,\ldots,X_n\right) be a random vector. The joint cumulative distribution function of X‾\underline{X} is the function that gives the probability that every component is at most the corresponding coordinate.

FX‾(x‾):=P(X1≤x1,…,Xn≤xn)  ∀ x‾∈RnF_{\underline{X}}\left(\underline{x}\right):=\mathbb{P}\left(X_1\leq x_1,\ldots,X_n\leq x_n\right)\,\,\forall\,\underline{x}\in\mathbb{R}^n

Let X‾=(X1,…,Xn)\underline{X}=\left(X_1,\ldots,X_n\right) be a random vector. The random vector X‾\underline{X} is a discrete random vector if and only if every component XiX_i is discrete.

Let X‾\underline{X} be a discrete random vector. The joint probability mass function of X‾\underline{X} is the function that gives the probability that the components take each tuple of values.

pX‾(x‾):=P(X1=x1,…,Xn=xn)p_{\underline{X}}\left(\underline{x}\right):=\mathbb{P}\left(X_1=x_1,\ldots,X_n=x_n\right)

The joint pmf sums to one over the state space.

Let X‾=(X1,…,Xn)\underline{X}=\left(X_1,\ldots,X_n\right) be a random vector. The random vector X‾\underline{X} is a continuous random vector if and only if there exists a function fX‾:Rn→[0,∞)f_{\underline{X}}:\mathbb{R}^n\rightarrow\left[0,\infty\right) such that for every measurable A⊆RnA\subseteq\mathbb{R}^n

P(X‾∈A)=∫AfX‾(x‾)dx‾.\mathbb{P}\left(\underline{X}\in A\right)=\int_A f_{\underline{X}}\left(\underline{x}\right)d\underline{x}.

Let X‾\underline{X} be a continuous random vector. The joint probability density function of X‾\underline{X} is the function fX‾f_{\underline{X}} above. The joint pdf integrates to one over Rn.\mathbb{R}^n.

Let X‾=(X1,…,Xn)\underline{X}=\left(X_1,\ldots,X_n\right) be a random vector. The marginal distribution of a component XiX_i is the probability distribution of XiX_i alone. The marginal distribution is recovered from the joint distribution by summing or integrating over the other components. For n=2n=2 the marginal pmf and marginal pdf of X1X_1 are

pX1(x1)=∑x2pX1,X2(x1,x2),fX1(x1)=∫−∞∞fX1,X2(x1,x2)dx2.p_{X_1}\left(x_1\right)=\sum_{x_2}p_{X_1,X_2}\left(x_1,x_2\right),\qquad f_{X_1}\left(x_1\right)=\int_{-\infty}^{\infty}f_{X_1,X_2}\left(x_1,x_2\right)dx_2.

Let X1X_1 and X2X_2 be random variables defined on a common probability space (Ω,F,P)\left(\Omega,\mathcal{F},\mathbb{P}\right) with state spaces (S1,S1)\left(S_1,\mathcal{S}_1\right) and (S2,S2),\left(S_2,\mathcal{S}_2\right), and A1∈S1A_1\in\mathcal{S}_1 and A2∈S2.A_2\in\mathcal{S}_2. The random variable events X1∈A1X_1\in A_1 and X2∈A2X_2\in A_2 are the events E1={ω:X1(ω)∈A1}E_1=\left\{\omega:X_1\left(\omega\right)\in A_1\right\} and E2={ω:X2(ω)∈A2}E_2=\left\{\omega:X_2\left(\omega\right)\in A_2\right\} in F,\mathcal{F}, so that PX1(X1∈A1)=P(E1)\mathbb{P}_{X_1}\left(X_1\in A_1\right)=\mathbb{P}\left(E_1\right) and PX2(X2∈A2)=P(E2).\mathbb{P}_{X_2}\left(X_2\in A_2\right)=\mathbb{P}\left(E_2\right). The joint random variable event (X1∈A1,X2∈A2)\left(X_1\in A_1,X_2\in A_2\right) is the intersection E1∩E2.E_1\cap E_2.

Let X1X_1 and X2X_2 be random variables defined on a common probability space. The bivariate probability distribution of the random vector (X1,X2)\left(X_1,X_2\right) is the joint pushforward probability measure.

PX1,X2(X1∈A1,X2∈A2):=P({ω∈Ω:X1(ω)∈A1,X2(ω)∈A2})\mathbb{P}_{X_1,X_2}\left(X_1\in A_1,X_2\in A_2\right):=\mathbb{P}\left(\left\{\omega\in\Omega:X_1\left(\omega\right)\in A_1,X_2\left(\omega\right)\in A_2\right\}\right)

Let X1X_1 and X2X_2 be random variables defined on a common probability space. The random variables X1X_1 and X2X_2 are independent if and only if every pair of random variable events is independent.

PX1,X2(X1∈A1,X2∈A2)=PX1(X1∈A1)PX2(X2∈A2)  ∀ A1∈S1,A2∈S2\mathbb{P}_{X_1,X_2}\left(X_1\in A_1,X_2\in A_2\right)=\mathbb{P}_{X_1}\left(X_1\in A_1\right)\mathbb{P}_{X_2}\left(X_2\in A_2\right)\,\,\forall\,A_1\in\mathcal{S}_1,A_2\in\mathcal{S}_2

Let X1,…,XnX_1,\ldots,X_n be random variables defined on a common probability space. The random variables are mutually independent if and only if the events X1∈A1,…,Xn∈AnX_1\in A_1,\ldots,X_n\in A_n are mutually independent for every choice of measurable sets A1,…,An.A_1,\ldots,A_n. The random variables X1,…,XnX_1,\ldots,X_n are mutually independent if and only if the joint cdf factorises as the product of the marginal cdfs for all x‾,\underline{x}, and equivalently if and only if the joint pmf or pdf factorises as the product of the marginal pmfs or pdfs.

FX‾(x‾)=∏i=1nFXi(xi)F_{\underline{X}}\left(\underline{x}\right)=\prod_{i=1}^n F_{X_i}\left(x_i\right) fX‾(x‾)=∏i=1nfXi(xi)f_{\underline{X}}\left(\underline{x}\right)=\prod_{i=1}^n f_{X_i}\left(x_i\right)

Let X1,…,XnX_1,\ldots,X_n be random variables defined on a common probability space. The random variables are independent and identically distributed (iid) if and only if they are mutually independent and identically distributed.

Let (X1,X2)\left(X_1,X_2\right) be a discrete random vector and x2x_2 be a value with pX2(x2)>0.p_{X_2}\left(x_2\right)>0. The conditional pmf of X1X_1 given X2=x2X_2=x_2 is the conditional probability of the event X1=x1X_1=x_1 given the event X2=x2.X_2=x_2.

pX1∣X2(x1∣x2):=pX1,X2(x1,x2)pX2(x2)p_{X_1\mid X_2}\left(x_1\mid x_2\right):=\frac{p_{X_1,X_2}\left(x_1,x_2\right)}{p_{X_2}\left(x_2\right)}

Let (X1,X2)\left(X_1,X_2\right) be a continuous random vector and x2x_2 be a value with fX2(x2)>0.f_{X_2}\left(x_2\right)>0. The conditional pdf of X1X_1 given X2=x2X_2=x_2 is the ratio of the joint pdf to the marginal pdf of X2,X_2, even though the event X2=x2X_2=x_2 has probability zero.

fX1∣X2(x1∣x2):=fX1,X2(x1,x2)fX2(x2)f_{X_1\mid X_2}\left(x_1\mid x_2\right):=\frac{f_{X_1,X_2}\left(x_1,x_2\right)}{f_{X_2}\left(x_2\right)}

Let (X1,X2)\left(X_1,X_2\right) be a random vector. The conditional distribution of X1X_1 given X2=x2X_2=x_2 is the probability distribution with the conditional pmf or pdf. Conditional distributions have the following properties:

  • the joint pdf factorises as fX1,X2=fX1∣X2fX2,f_{X_1,X_2}=f_{X_1\mid X_2}f_{X_2}, and likewise for pmfs;
  • X1X_1 and X2X_2 are independent if and only if the conditional distribution of X1X_1 given X2=x2X_2=x_2 equals the marginal distribution of X1X_1 for every x2;x_2;
  • Bayes’ theorem for densities follows in the same way as for events:
fX2∣X1(x2∣x1)=fX1∣X2(x1∣x2)fX2(x2)fX1(x1).f_{X_2\mid X_1}\left(x_2\mid x_1\right)=\frac{f_{X_1\mid X_2}\left(x_1\mid x_2\right)f_{X_2}\left(x_2\right)}{f_{X_1}\left(x_1\right)}.

Let XX be a real-valued random variable and g:R→Rg:\mathbb{R}\rightarrow\mathbb{R} be a function. The transformation of XX by gg is the random variable Y=g(X),Y=g(X), whose distribution is determined by that of XX through FY(y)=P(g(X)≤y).F_Y\left(y\right)=\mathbb{P}\left(g(X)\leq y\right). If XX is discrete then the pmf of YY sums the pmf of XX over the preimage of each value.

pY(y)=∑x: g(x)=ypX(x)p_Y\left(y\right)=\sum_{x:\,g(x)=y}p_X\left(x\right)

Let XX be a continuous random variable and g:R→Rg:\mathbb{R}\rightarrow\mathbb{R} be strictly monotonic and differentiable on the support of X.X. The change of variables formula states that the pdf of Y=g(X)Y=g(X) is the pdf of XX at the preimage scaled by the derivative of the inverse.

fY(y)=fX(g−1(y))∣ddyg−1(y)∣f_Y\left(y\right)=f_X\left(g^{-1}\left(y\right)\right)\left\lvert\frac{d}{dy}g^{-1}\left(y\right)\right\rvert

If gg is not monotonic the formula is applied separately on each interval where gg is monotonic and the contributions are summed. For example, Y=X2Y=X^2 has fY(y)=12y(fX(y)+fX(−y))f_Y\left(y\right)=\frac{1}{2\sqrt{y}}\left(f_X\left(\sqrt{y}\right)+f_X\left(-\sqrt{y}\right)\right) for y>0.y>0.

Let X‾\underline{X} be a continuous random vector and g‾:Rn→Rn\underline{g}:\mathbb{R}^n\rightarrow\mathbb{R}^n be a bijection that is continuously differentiable with continuously differentiable inverse. The multivariate change of variables formula states that the joint pdf of Y‾=g‾(X‾)\underline{Y}=\underline{g}\left(\underline{X}\right) is the joint pdf of X‾\underline{X} at the preimage scaled by the absolute value of the Jacobian determinant of the inverse, which follows from the change of variables theorem for multiple integrals.

fY‾(y‾)=fX‾(g‾−1(y‾))∣det⁡Jg‾−1(y‾)∣f_{\underline{Y}}\left(\underline{y}\right)=f_{\underline{X}}\left(\underline{g}^{-1}\left(\underline{y}\right)\right)\left\lvert\det J_{\underline{g}^{-1}}\left(\underline{y}\right)\right\rvert