Let (Ω,F,P) be a probability space and (S,S) be a measurable space. A random variable is a function X:Ω→S such that the inverse image of every measurable subset of S is an event.
{ω∈Ω:X(ω)∈A}∈F∀A∈S
Let X:Ω→S be a random variable. The set S is the state space of X and S is its σ-algebra. The random variable X is real-valued if and only if S⊆R. The notation X∈A denotes the event {ω∈Ω:X(ω)∈A}.
Let X:Ω→S be a random variable on the probability space (Ω,F,P). The pushforward probability measure of X is the probability measure PX on (S,S) that assigns to each measurable subset of S the probability of its inverse image.
PX(A):=P({ω∈Ω:X(ω)∈A})∀A∈S
The pushforward probability measure is also written PX(X∈A) or P(X∈A). The triple (S,S,PX) is a probability space, which allows the underlying probability space (Ω,F,P) to be left implicit.
Let X be a random variable with pushforward probability measure PX. The probability distribution of X is the probability measure PX on the state space.
Let X be a real-valued random variable. The cumulative distribution function (cdf) of X is the function FX:R→[0,1] that gives the probability that X is at most x.
FX(x):=PX(X≤x)∀x∈R
The cdf has the following properties:
FX is monotonically increasing;
FX is right-continuous;
limx→−∞FX(x)=0 and limx→∞FX(x)=1;
the probability of a half-open interval is a difference of cdf values, PX(a1<X≤a2)=FX(a2)−FX(a1) for a1<a2;
the cdf determines the probability distribution of X.
Let X be a random variable with state space S. The random variable X is a discrete random variable if and only if S is countable, that is, S={x1,…,xn} is finite or S={x1,x2,…} is countably infinite. The state space of a discrete random variable is a discrete state space.
Let X be a discrete random variable with state space S. The probability mass function (pmf) of X is the function pX:S→[0,1] that gives the probability that X takes each value.
pX(x):=PX(X=x)∀x∈S
The pmf is extended to R by setting pX(x)=0 for x∈R∖S. The pmf has the following properties:
pX(x)≥0∀x∈Sx∈S∑pX(x)=1PX(X∈A)=x∈A∑pX(x)∀A∈S
The last property holds since the singleton sets {x},x∈S, form a partition of S into disjoint events.
Let X be a discrete random variable with state space {x1,x2,…} where x1<x2<…, pmf pX and cdf FX. The cdf and pmf are related as follows: FX(x)=0 for x<x1, and
Let X be a real-valued random variable. The random variable X is a continuous random variable if and only if its cdf is a continuous function on R.
Let X be a real-valued random variable. The random variable X is absolutely continuous if and only if there exists a function fX:R→[0,∞) such that the cdf is the integral of fX.
FX(x)=∫−∞xfX(t)dt∀x∈R
Let X be an absolutely continuous random variable. The probability density function (pdf) of X is the function fX above. Below, a continuous random variable is taken to be absolutely continuous. The pdf has the following properties:
The cdf is differentiable wherever fX is continuous. Continuous random variables have the following properties:
the probability that X takes any single value is zero, PX(X=x)=0 for all x∈R;
the density fX(x) is not a probability and can exceed one;
including or excluding the endpoints of an interval does not change its probability, PX(x1≤X≤x2)=PX(x1<X<x2)=FX(x2)−FX(x1) for x1<x2.
Let X be a real-valued random variable with probability distribution PX. The support of PX is the smallest closed set X⊆R with PX(X∈X)=1. Equivalently, the support is the set of points whose every neighbourhood has positive probability.
X={x∈R:PX(x−ε<X<x+ε)>0∀ε>0}
Individual points of the support can have probability zero, and for a continuous random variable every point does. In practice the support of a discrete random variable is taken to be {x:pX(x)>0} and the support of a continuous random variable is taken to be {x:fX(x)>0}, which differ from the definition above by at most a set of probability zero.
Let X1 and X2 be random variables with a common state space. The random variables X1 and X2 are identically distributed if and only if they have the same probability distribution.
PX1(X1∈A)=PX2(X2∈A)∀A
Let X1,…,Xn be real-valued random variables defined on a common probability space (Ω,F,P). The random vectorX=(X1,…,Xn) is the function X:Ω→Rn with components Xi.
Let X=(X1,…,Xn) be a random vector. The joint distribution of X is its pushforward probability measure PX on Rn.
Let X=(X1,…,Xn) be a random vector. The joint cumulative distribution function of X is the function that gives the probability that every component is at most the corresponding coordinate.
FX(x):=P(X1≤x1,…,Xn≤xn)∀x∈Rn
Let X=(X1,…,Xn) be a random vector. The random vector X is a discrete random vector if and only if every component Xi is discrete.
Let X be a discrete random vector. The joint probability mass function of X is the function that gives the probability that the components take each tuple of values.
pX(x):=P(X1=x1,…,Xn=xn)
The joint pmf sums to one over the state space.
Let X=(X1,…,Xn) be a random vector. The random vector X is a continuous random vector if and only if there exists a function fX:Rn→[0,∞) such that for every measurable A⊆Rn
P(X∈A)=∫AfX(x)dx.
Let X be a continuous random vector. The joint probability density function of X is the function fX above. The joint pdf integrates to one over Rn.
Let X=(X1,…,Xn) be a random vector. The marginal distribution of a component Xi is the probability distribution of Xi alone. The marginal distribution is recovered from the joint distribution by summing or integrating over the other components. For n=2 the marginal pmf and marginal pdf of X1 are
Let X1 and X2 be random variables defined on a common probability space (Ω,F,P) with state spaces (S1,S1) and (S2,S2), and A1∈S1 and A2∈S2. The random variable eventsX1∈A1 and X2∈A2 are the events E1={ω:X1(ω)∈A1} and E2={ω:X2(ω)∈A2} in F, so that PX1(X1∈A1)=P(E1) and PX2(X2∈A2)=P(E2). The joint random variable event(X1∈A1,X2∈A2) is the intersection E1∩E2.
Let X1 and X2 be random variables defined on a common probability space. The bivariate probability distribution of the random vector (X1,X2) is the joint pushforward probability measure.
Let X1 and X2 be random variables defined on a common probability space. The random variables X1 and X2 are independent if and only if every pair of random variable events is independent.
Let X1,…,Xn be random variables defined on a common probability space. The random variables are mutually independent if and only if the events X1∈A1,…,Xn∈An are mutually independent for every choice of measurable sets A1,…,An. The random variables X1,…,Xn are mutually independent if and only if the joint cdf factorises as the product of the marginal cdfs for all x, and equivalently if and only if the joint pmf or pdf factorises as the product of the marginal pmfs or pdfs.
Let X1,…,Xn be random variables defined on a common probability space. The random variables are independent and identically distributed (iid) if and only if they are mutually independent and identically distributed.
Let (X1,X2) be a discrete random vector and x2 be a value with pX2(x2)>0. The conditional pmf of X1 given X2=x2 is the conditional probability of the event X1=x1 given the event X2=x2.
pX1∣X2(x1∣x2):=pX2(x2)pX1,X2(x1,x2)
Let (X1,X2) be a continuous random vector and x2 be a value with fX2(x2)>0. The conditional pdf of X1 given X2=x2 is the ratio of the joint pdf to the marginal pdf of X2, even though the event X2=x2 has probability zero.
fX1∣X2(x1∣x2):=fX2(x2)fX1,X2(x1,x2)
Let (X1,X2) be a random vector. The conditional distribution of X1 given X2=x2 is the probability distribution with the conditional pmf or pdf. Conditional distributions have the following properties:
the joint pdf factorises as fX1,X2=fX1∣X2fX2, and likewise for pmfs;
X1 and X2 are independent if and only if the conditional distribution of X1 given X2=x2 equals the marginal distribution of X1 for every x2;
Bayes’ theorem for densities follows in the same way as for events:
Let X be a real-valued random variable and g:R→R be a function. The transformation of X by g is the random variable Y=g(X), whose distribution is determined by that of X through FY(y)=P(g(X)≤y). If X is discrete then the pmf of Y sums the pmf of X over the preimage of each value.
pY(y)=x:g(x)=y∑pX(x)
Let X be a continuous random variable and g:R→R be strictly monotonic and differentiable on the support of X. The change of variables formula states that the pdf of Y=g(X) is the pdf of X at the preimage scaled by the derivative of the inverse.
fY(y)=fX(g−1(y))dydg−1(y)
If g is not monotonic the formula is applied separately on each interval where g is monotonic and the contributions are summed. For example, Y=X2 has fY(y)=2y1(fX(y)+fX(−y)) for y>0.
Let X be a continuous random vector and g:Rn→Rn be a bijection that is continuously differentiable with continuously differentiable inverse. The multivariate change of variables formula states that the joint pdf of Y=g(X) is the joint pdf of X at the preimage scaled by the absolute value of the Jacobian determinant of the inverse, which follows from the change of variables theorem for multiple integrals.