Let Θ be a set. A parametric family of distributions is a set {PX;θ:θ∈Θ} of probability distributions indexed by a scalar or vector θ. The index θ is the parameter of the family and the set Θ is its parameter space. The pmf or pdf of a member of the family is written pX(x;θ) or fX(x;θ). The notation X∼PX;θ states that X is distributed according to that member. The distributions below are the parametric families most used in probability models, each given with its support, pmf or pdf, mean, variance and, where it exists, mgf.
Let X be a real-valued random variable with cdf FX and u∈(0,1). The quantile function of X is the generalised inverse of its cdf.
FX−1(u):=inf{x∈R:FX(x)≥u}
The quantile function equals the ordinary inverse when FX is continuous and strictly increasing. The value FX−1(u) is the u-quantile of X. The median of X is the 21-quantile. The quartiles of X are the 41- and 43-quantiles.
Let n≥1. A random variable X has the discrete uniform distribution on {1,…,n}, written X∼U{1,…,n}, if and only if every value is equally likely.
pX(k)=n1∀k∈{1,…,n},E[X]=2n+1var[X]=12n2−1
Let p∈[0,1]. A random variable X has the Bernoulli distribution with parameter p, written X∼Bernoulli(p), if and only if it is the indicator of an event with probability p.
Let p∈[0,1]. A Bernoulli trial with success probability p is an experiment with two outcomes, success with probability p and failure with probability 1−p.
Let n∈N and p∈[0,1]. A random variable X has the binomial distribution with parameters n and p, written X∼Binomial(n,p), if and only if it is the number of successes in n independent Bernoulli trials with success probability p.
The binomial coefficient(kn)=k!(n−k)!n! counts the arrangements of k successes among n trials. The binomial distribution has the following properties:
a binomial random variable is a sum of n iid Bernoulli random variables;
the sum of independent Binomial(n1,p) and Binomial(n2,p) random variables is Binomial(n1+n2,p).
Let p∈(0,1]. A random variable X has the geometric distribution with parameter p, written X∼Geometric(p), if and only if it is the number of independent Bernoulli trials with success probability p up to and including the first success.
Let X be a random variable with state space {1,2,…}. The random variable X is memoryless if and only if P(X>m+k∣X>m)=P(X>k) for all m,k∈N. The geometric distribution is the only discrete distribution on {1,2,…} that is memoryless. An alternative convention counts the failures before the first success, which shifts the support to {0,1,…} and the mean to p1−p.
Let λ>0. A random variable X has the Poisson distribution with rate λ, written X∼Poisson(λ), if and only if
The sum of independent Poisson(λ1) and Poisson(λ2) random variables is Poisson(λ1+λ2).
Let Xn∼Binomial(n,pn) with npn→λ as n→∞. The Poisson limit theorem states that the binomial pmf converges to the Poisson pmf, pXn(k)→k!λke−λ for every k. The Poisson distribution therefore models counts of rare events among many opportunities.
Let a<b. A random variable X has the continuous uniform distribution on [a,b], written X∼U(a,b), if and only if its density is constant on [a,b].
Let X be a continuous random variable with cdf FX. The probability integral transform states that FX(X)∼U(0,1). Conversely, if U∼U(0,1) then FX−1(U) has cdf FX for any cdf FX.
Inverse transform sampling is the method that generates a random variable with cdf FX as FX−1(U) from a uniform random variable U∼U(0,1).
Let λ>0. A random variable X has the exponential distribution with rate λ, written X∼Exponential(λ), if and only if
Let X be a random variable with state space [0,∞). The random variable X is memoryless if and only if P(X>s+t∣X>s)=P(X>t) for all s,t≥0. The exponential distribution has the following properties:
it is the continuous analogue of the geometric distribution and the only continuous distribution on [0,∞) that is memoryless;
it models waiting times between events that occur at a constant rate;
the minimum of independent Exponential(λi) random variables is Exponential(∑iλi).
Let α>0. The gamma function at α is the improper integral
Γ(α):=∫0∞xα−1e−xdx.
The gamma function extends the factorial to real arguments and has the following properties:
Γ(α+1)=αΓ(α),Γ(n)=(n−1)!∀n∈N,Γ(21)=π
Let α>0 and β>0. A random variable X has the gamma distribution with shape α and rate β, written X∼Gamma(α,β), if and only if
The gamma distribution has the following properties:
the exponential distribution is Gamma(1,λ);
the sum of independent Gamma(αi,β) random variables with a common rate is Gamma(∑iαi,β), so the sum of n iid Exponential(λ) random variables is Gamma(n,λ);
the gamma distribution is also parametrised by its scaleθ=β1.
Let α>0 and β>0. The beta function at (α,β) is the integral
B(α,β):=∫01xα−1(1−x)β−1dx=Γ(α+β)Γ(α)Γ(β).
Let α>0 and β>0. A random variable X has the beta distribution with shape parameters α and β, written X∼Beta(α,β), if and only if
The chi-squared distribution has the following properties:
the sum of independent χk12 and χk22 random variables is χk1+k22;
the chi-squared distribution with k degrees of freedom is the distribution of the sum of squares of k iid standard normal random variables, defined below.
Let μ∈R and σ2>0. A random variable X has the normal distribution or Gaussian distribution with mean μ and variance σ2, written X∼N(μ,σ2), if and only if
The density integrates to one by the Gaussian integral, and the cdf has no closed form.
The standard normal distribution is N(0,1). Its pdf is written ϕ(z)=2π1e−2z2 and its cdf is written Φ(z)=∫−∞zϕ(t)dt. The normal distribution has the following properties:
it is symmetric about μ, so has skewness 0, and has excess kurtosis 0;
if X∼N(μ,σ2) then aX+b∼N(aμ+b,a2σ2), so its standardisation is σX−μ∼N(0,1) and FX(x)=Φ(σx−μ);
the sum of independent N(μ1,σ12) and N(μ2,σ22) random variables is N(μ1+μ2,σ12+σ22);
if Z∼N(0,1) then Z2∼χ12;
approximately 68%,95% and 99.7% of the probability lies within one, two and three standard deviations of the mean.
Let ν>0, and Z∼N(0,1) and V∼χν2 be independent. A random variable T has Student’s t-distribution with ν degrees of freedom, written T∼tν, if and only if it has the distribution of V/νZ.
it is symmetric about 0 with heavier tails than the standard normal distribution;
its k-th moment exists only for k<ν, and its mgf does not exist;
as ν→∞ the pdf of tν converges to ϕ;
it arises when a normal mean is standardised by an estimated rather than known standard deviation.
The Cauchy distribution is t1, with pdf π(1+t2)1. The Cauchy distribution has no mean.
Let d1,d2≥1, and U∼χd12 and V∼χd22 be independent. A random variable X has the F-distribution with d1 and d2 degrees of freedom, written X∼Fd1,d2, if and only if it has the distribution of V/d2U/d1.
it arises as the distribution of a ratio of independent variance estimates.
Let k≥2,n∈N and p=(p1,…,pk) with pi≥0 and ∑i=1kpi=1. A random vector X=(X1,…,Xk) has the multinomial distribution with parameters n and p, written X∼Multinomial(n,p), if and only if it counts how many of n independent trials, each with k possible outcomes with probabilities p, result in each outcome.
The multinomial distribution has the following properties:
each marginal Xi is Binomial(n,pi);
the binomial distribution is the case k=2;
the components are negatively correlated since they sum to n.
Let k≥1,μ∈Rk and Σ∈Rk×k be symmetric positive definite. A random vector X has the multivariate normal distribution with mean vector μ and covariance matrix Σ, written X∼Nk(μ,Σ), if and only if
Equivalently, X is multivariate normal if and only if every linear combination a⊺X has a univariate normal distribution. This equivalent definition also applies when Σ is only positive semi-definite and no density exists.
Let X∼Nk(μ,Σ). The Mahalanobis distance of X from μ is the square root of the quadratic form (X−μ)⊺Σ−1(X−μ). The multivariate normal distribution has the following properties:
AX+b∼Nm(Aμ+b,AΣA⊺) for A∈Rm×k and b∈Rm, so every marginal distribution of a subset of components is multivariate normal;
components are independent if and only if they are uncorrelated, so Σ diagonal implies independent components;
partitioning X=(X1,X2) with corresponding blocks of μ and Σ, the conditional distribution of X1 given X2=x2 is multivariate normal with mean μ1+Σ12Σ22−1(x2−μ2) and covariance matrix Σ11−Σ12Σ22−1Σ21;
if Z∼Nk(0,Ik) has iid standard normal components and Σ=LL⊺ is the Cholesky decomposition then μ+LZ∼Nk(μ,Σ), which is how multivariate normal random vectors are simulated;
the squared Mahalanobis distance has the χk2 distribution.
Let k≥2. The simplexΔk is the set of probability vectors with positive components.
Δk:={x∈Rk:xi>0,i=1∑kxi=1}
Let k≥2,α=(α1,…,αk) with αi>0 and α0=∑i=1kαi. A random vector X has the Dirichlet distribution with parameter α, written X∼Dirichlet(α), if and only if it takes values in the simplex with density
The Dirichlet distribution has the following properties:
it is the multivariate generalisation of the beta distribution, which is the case k=2, and models random probability vectors;
each marginal Xi is Beta(αi,α0−αi);
if Yi∼Gamma(αi,λ) are independent then ∑iYi1(Y1,…,Yk)∼Dirichlet(α).
Let {PX;θ:θ∈Θ} be a parametric family. The family is an exponential family if and only if its pmf or pdf can be written as
fX(x;θ)=h(x)exp(η(θ)⊺T(x)−A(θ))
for some functions h and T of x alone and η and A of θ alone. The vector η(θ) is the natural parameter, the vector T(x) is the vector of natural statistics and the function A(θ) is the log-partition function, which normalises the density. Exponential families have the following properties:
the Bernoulli distribution has η=loge1−pp,T(x)=x and A=loge(1+eη);
the normal distribution with known variance has η=σ2μ and T(x)=x;
the Bernoulli, binomial and multinomial (with n known), geometric, Poisson, exponential, gamma, beta, chi-squared, normal, multivariate normal and Dirichlet distributions are exponential families, whereas the uniform, t- and F-distributions are not;
the derivatives of the log-partition function with respect to the natural parameter give the moments of the natural statistics, E[T(X)]=∇A and cov[T(X)]=HA.