Let Θ\Theta be a set. A parametric family of distributions is a set {PX; θ:θ∈Θ}\left\{\mathbb{P}_{X;\,\theta}:\theta\in\Theta\right\} of probability distributions indexed by a scalar or vector θ.\theta. The index θ\theta is the parameter of the family and the set Θ\Theta is its parameter space. The pmf or pdf of a member of the family is written pX(x; θ)p_X\left(x;\,\theta\right) or fX(x; θ).f_X\left(x;\,\theta\right). The notation X∼PX; θX\sim\mathbb{P}_{X;\,\theta} states that XX is distributed according to that member. The distributions below are the parametric families most used in probability models, each given with its support, pmf or pdf, mean, variance and, where it exists, mgf.

Let XX be a real-valued random variable with cdf FXF_X and u∈(0,1).u\in\left(0,1\right). The quantile function of XX is the generalised inverse of its cdf.

FX−1(u):=inf⁡{x∈R:FX(x)≥u}F_X^{-1}\left(u\right):=\inf\left\{x\in\mathbb{R}:F_X\left(x\right)\geq u\right\}

The quantile function equals the ordinary inverse when FXF_X is continuous and strictly increasing. The value FX−1(u)F_X^{-1}\left(u\right) is the uu-quantile of X.X. The median of XX is the 12\frac{1}{2}-quantile. The quartiles of XX are the 14\frac{1}{4}- and 34\frac{3}{4}-quantiles.

Let n≥1.n\geq 1. A random variable XX has the discrete uniform distribution on {1,…,n},\{1,\ldots,n\}, written X∼U{1,…,n},X\sim\mathrm{U}\{1,\ldots,n\}, if and only if every value is equally likely.

pX(k)=1n  ∀ k∈{1,…,n},E[X]=n+12p_X\left(k\right)=\frac{1}{n}\,\,\forall\,k\in\{1,\ldots,n\},\qquad\mathbb{E}\left[X\right]=\frac{n+1}{2} var[X]=n2−112\mathrm{var}\left[X\right]=\frac{n^2-1}{12}

Let p∈[0,1].p\in\left[0,1\right]. A random variable XX has the Bernoulli distribution with parameter p,p, written X∼Bernoulli(p),X\sim\mathrm{Bernoulli}\left(p\right), if and only if it is the indicator of an event with probability p.p.

pX(k)=pk(1−p)1−k  ∀ k∈{0,1},E[X]=pp_X\left(k\right)=p^k\left(1-p\right)^{1-k}\,\,\forall\,k\in\{0,1\},\qquad\mathbb{E}\left[X\right]=p var[X]=p(1−p),MX(t)=1−p+pet\mathrm{var}\left[X\right]=p\left(1-p\right),\qquad M_X\left(t\right)=1-p+pe^t

Let p∈[0,1].p\in\left[0,1\right]. A Bernoulli trial with success probability pp is an experiment with two outcomes, success with probability pp and failure with probability 1−p.1-p.

Let n∈Nn\in\mathbb{N} and p∈[0,1].p\in\left[0,1\right]. A random variable XX has the binomial distribution with parameters nn and p,p, written X∼Binomial(n,p),X\sim\mathrm{Binomial}\left(n,p\right), if and only if it is the number of successes in nn independent Bernoulli trials with success probability p.p.

pX(k)=(nk)pk(1−p)n−k  ∀ k∈{0,…,n},E[X]=npp_X\left(k\right)=\binom{n}{k}p^k\left(1-p\right)^{n-k}\,\,\forall\,k\in\{0,\ldots,n\},\qquad\mathbb{E}\left[X\right]=np var[X]=np(1−p),MX(t)=(1−p+pet)n\mathrm{var}\left[X\right]=np\left(1-p\right),\qquad M_X\left(t\right)=\left(1-p+pe^t\right)^n

The binomial coefficient (nk)=n!k!(n−k)!\binom{n}{k}=\frac{n!}{k!\left(n-k\right)!} counts the arrangements of kk successes among nn trials. The binomial distribution has the following properties:

  • a binomial random variable is a sum of nn iid Bernoulli random variables;
  • the sum of independent Binomial(n1,p)\mathrm{Binomial}\left(n_1,p\right) and Binomial(n2,p)\mathrm{Binomial}\left(n_2,p\right) random variables is Binomial(n1+n2,p).\mathrm{Binomial}\left(n_1+n_2,p\right).

Let p∈(0,1].p\in\left(0,1\right]. A random variable XX has the geometric distribution with parameter p,p, written X∼Geometric(p),X\sim\mathrm{Geometric}\left(p\right), if and only if it is the number of independent Bernoulli trials with success probability pp up to and including the first success.

pX(k)=(1−p)k−1p  ∀ k∈{1,2,…},FX(k)=1−(1−p)kp_X\left(k\right)=\left(1-p\right)^{k-1}p\,\,\forall\,k\in\{1,2,\ldots\},\qquad F_X\left(k\right)=1-\left(1-p\right)^k E[X]=1p,var[X]=1−pp2\mathbb{E}\left[X\right]=\frac{1}{p},\qquad\mathrm{var}\left[X\right]=\frac{1-p}{p^2} MX(t)=pet1−(1−p)et  ∀ t<−log⁡e(1−p)M_X\left(t\right)=\frac{pe^t}{1-\left(1-p\right)e^t}\,\,\forall\,t<-\log_e\left(1-p\right)

Let XX be a random variable with state space {1,2,…}.\{1,2,\ldots\}. The random variable XX is memoryless if and only if P(X>m+k∣X>m)=P(X>k)\mathbb{P}\left(X>m+k\mid X>m\right)=\mathbb{P}\left(X>k\right) for all m,k∈N.m,k\in\mathbb{N}. The geometric distribution is the only discrete distribution on {1,2,…}\{1,2,\ldots\} that is memoryless. An alternative convention counts the failures before the first success, which shifts the support to {0,1,…}\{0,1,\ldots\} and the mean to 1−pp.\frac{1-p}{p}.

Let λ>0.\lambda>0. A random variable XX has the Poisson distribution with rate λ,\lambda, written X∼Poisson(λ),X\sim\mathrm{Poisson}\left(\lambda\right), if and only if

pX(k)=λke−λk!  ∀ k∈{0,1,2,…},E[X]=var[X]=λp_X\left(k\right)=\frac{\lambda^ke^{-\lambda}}{k!}\,\,\forall\,k\in\{0,1,2,\ldots\},\qquad\mathbb{E}\left[X\right]=\mathrm{var}\left[X\right]=\lambda MX(t)=exp⁡(λ(et−1)).M_X\left(t\right)=\exp\left(\lambda\left(e^t-1\right)\right).

The sum of independent Poisson(λ1)\mathrm{Poisson}\left(\lambda_1\right) and Poisson(λ2)\mathrm{Poisson}\left(\lambda_2\right) random variables is Poisson(λ1+λ2).\mathrm{Poisson}\left(\lambda_1+\lambda_2\right).

Let Xn∼Binomial(n,pn)X_n\sim\mathrm{Binomial}\left(n,p_n\right) with npn→λnp_n\rightarrow\lambda as n→∞.n\rightarrow\infty. The Poisson limit theorem states that the binomial pmf converges to the Poisson pmf, pXn(k)→λke−λk!p_{X_n}\left(k\right)\rightarrow\frac{\lambda^ke^{-\lambda}}{k!} for every k.k. The Poisson distribution therefore models counts of rare events among many opportunities.

Let a<b.a<b. A random variable XX has the continuous uniform distribution on [a,b],\left[a,b\right], written X∼U(a,b),X\sim\mathrm{U}\left(a,b\right), if and only if its density is constant on [a,b].\left[a,b\right].

fX(x)=1b−a  ∀ x∈[a,b],FX(x)=x−ab−a,E[X]=a+b2f_X\left(x\right)=\frac{1}{b-a}\,\,\forall\,x\in\left[a,b\right],\qquad F_X\left(x\right)=\frac{x-a}{b-a},\qquad\mathbb{E}\left[X\right]=\frac{a+b}{2} var[X]=(b−a)212,MX(t)=etb−etat(b−a)\mathrm{var}\left[X\right]=\frac{\left(b-a\right)^2}{12},\qquad M_X\left(t\right)=\frac{e^{tb}-e^{ta}}{t\left(b-a\right)}

Let XX be a continuous random variable with cdf FX.F_X. The probability integral transform states that FX(X)∼U(0,1).F_X\left(X\right)\sim\mathrm{U}\left(0,1\right). Conversely, if U∼U(0,1)U\sim\mathrm{U}\left(0,1\right) then FX−1(U)F_X^{-1}\left(U\right) has cdf FXF_X for any cdf FX.F_X.

Inverse transform sampling is the method that generates a random variable with cdf FXF_X as FX−1(U)F_X^{-1}\left(U\right) from a uniform random variable U∼U(0,1).U\sim\mathrm{U}\left(0,1\right).

Let λ>0.\lambda>0. A random variable XX has the exponential distribution with rate λ,\lambda, written X∼Exponential(λ),X\sim\mathrm{Exponential}\left(\lambda\right), if and only if

fX(x)=λe−λx  ∀ x≥0,FX(x)=1−e−λx,E[X]=1λf_X\left(x\right)=\lambda e^{-\lambda x}\,\,\forall\,x\geq 0,\qquad F_X\left(x\right)=1-e^{-\lambda x},\qquad\mathbb{E}\left[X\right]=\frac{1}{\lambda} var[X]=1λ2,MX(t)=λλ−t  ∀ t<λ.\mathrm{var}\left[X\right]=\frac{1}{\lambda^2},\qquad M_X\left(t\right)=\frac{\lambda}{\lambda-t}\,\,\forall\,t<\lambda.

Let XX be a random variable with state space [0,∞).\left[0,\infty\right). The random variable XX is memoryless if and only if P(X>s+t∣X>s)=P(X>t)\mathbb{P}\left(X>s+t\mid X>s\right)=\mathbb{P}\left(X>t\right) for all s,t≥0.s,t\geq 0. The exponential distribution has the following properties:

  • it is the continuous analogue of the geometric distribution and the only continuous distribution on [0,∞)\left[0,\infty\right) that is memoryless;
  • it models waiting times between events that occur at a constant rate;
  • the minimum of independent Exponential(λi)\mathrm{Exponential}\left(\lambda_i\right) random variables is Exponential(∑iλi).\mathrm{Exponential}\left(\sum_i\lambda_i\right).

Let α>0.\alpha>0. The gamma function at α\alpha is the improper integral

Γ(α):=∫0∞xα−1e−x dx.\Gamma\left(\alpha\right):=\int_0^{\infty}x^{\alpha-1}e^{-x}\,dx.

The gamma function extends the factorial to real arguments and has the following properties:

Γ(α+1)=α Γ(α),Γ(n)=(n−1)!  ∀ n∈N,Γ(12)=π\Gamma\left(\alpha+1\right)=\alpha\,\Gamma\left(\alpha\right),\qquad\Gamma\left(n\right)=\left(n-1\right)!\,\,\forall\,n\in\mathbb{N},\qquad\Gamma\left(\tfrac{1}{2}\right)=\sqrt{\pi}

Let α>0\alpha>0 and β>0.\beta>0. A random variable XX has the gamma distribution with shape α\alpha and rate β,\beta, written X∼Gamma(α,β),X\sim\mathrm{Gamma}\left(\alpha,\beta\right), if and only if

fX(x)=βαΓ(α)xα−1e−βx  ∀ x>0f_X\left(x\right)=\frac{\beta^{\alpha}}{\Gamma\left(\alpha\right)}x^{\alpha-1}e^{-\beta x}\,\,\forall\,x>0 E[X]=αβ,var[X]=αβ2\mathbb{E}\left[X\right]=\frac{\alpha}{\beta},\qquad\mathrm{var}\left[X\right]=\frac{\alpha}{\beta^2} MX(t)=(ββ−t)α  ∀ t<β.M_X\left(t\right)=\left(\frac{\beta}{\beta-t}\right)^{\alpha}\,\,\forall\,t<\beta.

The gamma distribution has the following properties:

  • the exponential distribution is Gamma(1,λ);\mathrm{Gamma}\left(1,\lambda\right);
  • the sum of independent Gamma(αi,β)\mathrm{Gamma}\left(\alpha_i,\beta\right) random variables with a common rate is Gamma(∑iαi,β),\mathrm{Gamma}\left(\sum_i\alpha_i,\beta\right), so the sum of nn iid Exponential(λ)\mathrm{Exponential}\left(\lambda\right) random variables is Gamma(n,λ);\mathrm{Gamma}\left(n,\lambda\right);
  • the gamma distribution is also parametrised by its scale θ=1β.\theta=\frac{1}{\beta}.

Let α>0\alpha>0 and β>0.\beta>0. The beta function at (α,β)\left(\alpha,\beta\right) is the integral

B(α,β):=∫01xα−1(1−x)β−1dx=Γ(α)Γ(β)Γ(α+β).B\left(\alpha,\beta\right):=\int_0^1 x^{\alpha-1}\left(1-x\right)^{\beta-1}dx=\frac{\Gamma\left(\alpha\right)\Gamma\left(\beta\right)}{\Gamma\left(\alpha+\beta\right)}.

Let α>0\alpha>0 and β>0.\beta>0. A random variable XX has the beta distribution with shape parameters α\alpha and β,\beta, written X∼Beta(α,β),X\sim\mathrm{Beta}\left(\alpha,\beta\right), if and only if

fX(x)=xα−1(1−x)β−1B(α,β)  ∀ x∈[0,1]f_X\left(x\right)=\frac{x^{\alpha-1}\left(1-x\right)^{\beta-1}}{B\left(\alpha,\beta\right)}\,\,\forall\,x\in\left[0,1\right] E[X]=αα+β\mathbb{E}\left[X\right]=\frac{\alpha}{\alpha+\beta} var[X]=αβ(α+β)2(α+β+1).\mathrm{var}\left[X\right]=\frac{\alpha\beta}{\left(\alpha+\beta\right)^2\left(\alpha+\beta+1\right)}.

The beta distribution has the following properties:

  • it models random proportions;
  • Beta(1,1)\mathrm{Beta}\left(1,1\right) is U(0,1);\mathrm{U}\left(0,1\right);
  • if Y1∼Gamma(α,λ)Y_1\sim\mathrm{Gamma}\left(\alpha,\lambda\right) and Y2∼Gamma(β,λ)Y_2\sim\mathrm{Gamma}\left(\beta,\lambda\right) are independent then Y1Y1+Y2∼Beta(α,β).\frac{Y_1}{Y_1+Y_2}\sim\mathrm{Beta}\left(\alpha,\beta\right).

Let k∈Nk\in\mathbb{N} with k≥1.k\geq 1. A random variable XX has the chi-squared distribution with kk degrees of freedom, written X∼χk2,X\sim\chi^2_k, if and only if X∼Gamma(k2,12).X\sim\mathrm{Gamma}\left(\frac{k}{2},\frac{1}{2}\right).

fX(x)=xk2−1e−x22k2Γ(k2)  ∀ x>0f_X\left(x\right)=\frac{x^{\frac{k}{2}-1}e^{-\frac{x}{2}}}{2^{\frac{k}{2}}\Gamma\left(\frac{k}{2}\right)}\,\,\forall\,x>0 E[X]=k,var[X]=2k,MX(t)=(1−2t)−k2  ∀ t<12\mathbb{E}\left[X\right]=k,\qquad\mathrm{var}\left[X\right]=2k,\qquad M_X\left(t\right)=\left(1-2t\right)^{-\frac{k}{2}}\,\,\forall\,t<\tfrac{1}{2}

The chi-squared distribution has the following properties:

  • the sum of independent χk12\chi^2_{k_1} and χk22\chi^2_{k_2} random variables is χk1+k22;\chi^2_{k_1+k_2};
  • the chi-squared distribution with kk degrees of freedom is the distribution of the sum of squares of kk iid standard normal random variables, defined below.

Let μ∈R\mu\in\mathbb{R} and σ2>0.\sigma^2>0. A random variable XX has the normal distribution or Gaussian distribution with mean μ\mu and variance σ2,\sigma^2, written X∼N(μ,σ2),X\sim\mathrm{N}\left(\mu,\sigma^2\right), if and only if

fX(x)=12πσ2exp⁡(−(x−μ)22σ2)  ∀ x∈Rf_X\left(x\right)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp\left(-\frac{\left(x-\mu\right)^2}{2\sigma^2}\right)\,\,\forall\,x\in\mathbb{R} E[X]=μ,var[X]=σ2,MX(t)=exp⁡(μt+12σ2t2).\mathbb{E}\left[X\right]=\mu,\qquad\mathrm{var}\left[X\right]=\sigma^2,\qquad M_X\left(t\right)=\exp\left(\mu t+\tfrac{1}{2}\sigma^2t^2\right).

The density integrates to one by the Gaussian integral, and the cdf has no closed form.

The standard normal distribution is N(0,1).\mathrm{N}\left(0,1\right). Its pdf is written ϕ(z)=12πe−z22\phi\left(z\right)=\frac{1}{\sqrt{2\pi}}e^{-\frac{z^2}{2}} and its cdf is written Φ(z)=∫−∞zϕ(t)dt.\Phi\left(z\right)=\int_{-\infty}^z\phi\left(t\right)dt. The normal distribution has the following properties:

  • it is symmetric about μ,\mu, so has skewness 0,0, and has excess kurtosis 0;0;
  • if X∼N(μ,σ2)X\sim\mathrm{N}\left(\mu,\sigma^2\right) then aX+b∼N(aμ+b,a2σ2),aX+b\sim\mathrm{N}\left(a\mu+b,a^2\sigma^2\right), so its standardisation is X−μσ∼N(0,1)\frac{X-\mu}{\sigma}\sim\mathrm{N}\left(0,1\right) and FX(x)=Φ(x−μσ);F_X\left(x\right)=\Phi\left(\frac{x-\mu}{\sigma}\right);
  • the sum of independent N(μ1,σ12)\mathrm{N}\left(\mu_1,\sigma_1^2\right) and N(μ2,σ22)\mathrm{N}\left(\mu_2,\sigma_2^2\right) random variables is N(μ1+μ2,σ12+σ22);\mathrm{N}\left(\mu_1+\mu_2,\sigma_1^2+\sigma_2^2\right);
  • if Z∼N(0,1)Z\sim\mathrm{N}\left(0,1\right) then Z2∼χ12;Z^2\sim\chi^2_1;
  • approximately 68%,68\%, 95%95\% and 99.7%99.7\% of the probability lies within one, two and three standard deviations of the mean.

Let ν>0,\nu>0, and Z∼N(0,1)Z\sim\mathrm{N}\left(0,1\right) and V∼χν2V\sim\chi^2_{\nu} be independent. A random variable TT has Student’s tt-distribution with ν\nu degrees of freedom, written T∼tν,T\sim t_{\nu}, if and only if it has the distribution of ZV/ν.\frac{Z}{\sqrt{V/\nu}}.

fT(t)=Γ(ν+12)νπ Γ(ν2)(1+t2ν)−ν+12  ∀ t∈Rf_T\left(t\right)=\frac{\Gamma\left(\frac{\nu+1}{2}\right)}{\sqrt{\nu\pi}\,\Gamma\left(\frac{\nu}{2}\right)}\left(1+\frac{t^2}{\nu}\right)^{-\frac{\nu+1}{2}}\,\,\forall\,t\in\mathbb{R} E[T]=0  (ν>1),var[T]=νν−2  (ν>2)\mathbb{E}\left[T\right]=0\,\,\left(\nu>1\right),\qquad\mathrm{var}\left[T\right]=\frac{\nu}{\nu-2}\,\,\left(\nu>2\right)

The tt-distribution has the following properties:

  • it is symmetric about 00 with heavier tails than the standard normal distribution;
  • its kk-th moment exists only for k<ν,k<\nu, and its mgf does not exist;
  • as ν→∞\nu\rightarrow\infty the pdf of tνt_{\nu} converges to ϕ;\phi;
  • it arises when a normal mean is standardised by an estimated rather than known standard deviation.

The Cauchy distribution is t1,t_1, with pdf 1π(1+t2).\frac{1}{\pi\left(1+t^2\right)}. The Cauchy distribution has no mean.

Let d1,d2≥1,d_1,d_2\geq 1, and U∼χd12U\sim\chi^2_{d_1} and V∼χd22V\sim\chi^2_{d_2} be independent. A random variable XX has the FF-distribution with d1d_1 and d2d_2 degrees of freedom, written X∼Fd1,d2,X\sim F_{d_1,d_2}, if and only if it has the distribution of U/d1V/d2.\frac{U/d_1}{V/d_2}.

fX(x)=1B(d12,d22)(d1d2)d12xd12−1(1+d1d2x)−d1+d22  ∀ x>0f_X\left(x\right)=\frac{1}{B\left(\frac{d_1}{2},\frac{d_2}{2}\right)}\left(\frac{d_1}{d_2}\right)^{\frac{d_1}{2}}x^{\frac{d_1}{2}-1}\left(1+\frac{d_1}{d_2}x\right)^{-\frac{d_1+d_2}{2}}\,\,\forall\,x>0 E[X]=d2d2−2  (d2>2)\mathbb{E}\left[X\right]=\frac{d_2}{d_2-2}\,\,\left(d_2>2\right)

The FF-distribution has the following properties:

  • if T∼tνT\sim t_{\nu} then T2∼F1,ν;T^2\sim F_{1,\nu};
  • if X∼Fd1,d2X\sim F_{d_1,d_2} then 1X∼Fd2,d1;\frac{1}{X}\sim F_{d_2,d_1};
  • it arises as the distribution of a ratio of independent variance estimates.

Let k≥2,k\geq 2, n∈Nn\in\mathbb{N} and p‾=(p1,…,pk)\underline{p}=\left(p_1,\ldots,p_k\right) with pi≥0p_i\geq 0 and ∑i=1kpi=1.\sum_{i=1}^k p_i=1. A random vector X‾=(X1,…,Xk)\underline{X}=\left(X_1,\ldots,X_k\right) has the multinomial distribution with parameters nn and p‾,\underline{p}, written X‾∼Multinomial(n,p‾),\underline{X}\sim\mathrm{Multinomial}\left(n,\underline{p}\right), if and only if it counts how many of nn independent trials, each with kk possible outcomes with probabilities p‾,\underline{p}, result in each outcome.

pX‾(x‾)=n!x1!⋯xk!∏i=1kpixi  ∀ x‾∈{0,…,n}k  with  ∑i=1kxi=np_{\underline{X}}\left(\underline{x}\right)=\frac{n!}{x_1!\cdots x_k!}\prod_{i=1}^k p_i^{x_i}\,\,\forall\,\underline{x}\in\{0,\ldots,n\}^k\,\,\mathrm{with}\,\,\sum_{i=1}^k x_i=n E[Xi]=npi,var[Xi]=npi(1−pi)\mathbb{E}\left[X_i\right]=np_i,\qquad\mathrm{var}\left[X_i\right]=np_i\left(1-p_i\right) cov[Xi,Xj]=−npipj  ∀ i≠j\mathrm{cov}\left[X_i,X_j\right]=-np_ip_j\,\,\forall\,i\neq j

The multinomial distribution has the following properties:

  • each marginal XiX_i is Binomial(n,pi);\mathrm{Binomial}\left(n,p_i\right);
  • the binomial distribution is the case k=2;k=2;
  • the components are negatively correlated since they sum to n.n.

Let k≥1,k\geq 1, μ‾∈Rk\underline{\mu}\in\mathbb{R}^k and Σ∈R k×k\Sigma\in\mathbb{R}^{\,k\times k} be symmetric positive definite. A random vector X‾\underline{X} has the multivariate normal distribution with mean vector μ‾\underline{\mu} and covariance matrix Σ,\Sigma, written X‾∼Nk(μ‾,Σ),\underline{X}\sim\mathrm{N}_k\left(\underline{\mu},\Sigma\right), if and only if

fX‾(x‾)=1(2π)k2∣Σ∣12exp⁡(−12(x‾−μ‾)⊺Σ−1(x‾−μ‾))  ∀ x‾∈Rkf_{\underline{X}}\left(\underline{x}\right)=\frac{1}{\left(2\pi\right)^{\frac{k}{2}}\lvert\Sigma\rvert^{\frac{1}{2}}}\exp\left(-\frac{1}{2}\left(\underline{x}-\underline{\mu}\right)^\intercal\Sigma^{-1}\left(\underline{x}-\underline{\mu}\right)\right)\,\,\forall\,\underline{x}\in\mathbb{R}^k MX‾(t‾)=E[et‾⊺X‾]=exp⁡(t‾⊺μ‾+12t‾⊺Σt‾).M_{\underline{X}}\left(\underline{t}\right)=\mathbb{E}\left[e^{\underline{t}^\intercal\underline{X}}\right]=\exp\left(\underline{t}^\intercal\underline{\mu}+\tfrac{1}{2}\underline{t}^\intercal\Sigma\underline{t}\right).

Equivalently, X‾\underline{X} is multivariate normal if and only if every linear combination a‾⊺X‾\underline{a}^\intercal\underline{X} has a univariate normal distribution. This equivalent definition also applies when Σ\Sigma is only positive semi-definite and no density exists.

Let X‾∼Nk(μ‾,Σ).\underline{X}\sim\mathrm{N}_k\left(\underline{\mu},\Sigma\right). The Mahalanobis distance of X‾\underline{X} from μ‾\underline{\mu} is the square root of the quadratic form (X‾−μ‾)⊺Σ−1(X‾−μ‾).\left(\underline{X}-\underline{\mu}\right)^\intercal\Sigma^{-1}\left(\underline{X}-\underline{\mu}\right). The multivariate normal distribution has the following properties:

  • AX‾+b‾∼Nm(Aμ‾+b‾,AΣA⊺)A\underline{X}+\underline{b}\sim\mathrm{N}_m\left(A\underline{\mu}+\underline{b},A\Sigma A^\intercal\right) for A∈R m×kA\in\mathbb{R}^{\,m\times k} and b‾∈Rm,\underline{b}\in\mathbb{R}^m, so every marginal distribution of a subset of components is multivariate normal;
  • components are independent if and only if they are uncorrelated, so Σ\Sigma diagonal implies independent components;
  • partitioning X‾=(X‾1,X‾2)\underline{X}=\left(\underline{X}_1,\underline{X}_2\right) with corresponding blocks of μ‾\underline{\mu} and Σ,\Sigma, the conditional distribution of X‾1\underline{X}_1 given X‾2=x‾2\underline{X}_2=\underline{x}_2 is multivariate normal with mean μ‾1+Σ12Σ22−1(x‾2−μ‾2)\underline{\mu}_1+\Sigma_{12}\Sigma_{22}^{-1}\left(\underline{x}_2-\underline{\mu}_2\right) and covariance matrix Σ11−Σ12Σ22−1Σ21;\Sigma_{11}-\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21};
  • if Z‾∼Nk(0‾,Ik)\underline{Z}\sim\mathrm{N}_k\left(\underline{0},I_k\right) has iid standard normal components and Σ=LL⊺\Sigma=LL^\intercal is the Cholesky decomposition then μ‾+LZ‾∼Nk(μ‾,Σ),\underline{\mu}+L\underline{Z}\sim\mathrm{N}_k\left(\underline{\mu},\Sigma\right), which is how multivariate normal random vectors are simulated;
  • the squared Mahalanobis distance has the χk2\chi^2_k distribution.

Let k≥2.k\geq 2. The simplex Δk\Delta_k is the set of probability vectors with positive components.

Δk:={x‾∈Rk:xi>0, ∑i=1kxi=1}\Delta_k:=\left\{\underline{x}\in\mathbb{R}^k:x_i>0,\,\sum_{i=1}^k x_i=1\right\}

Let k≥2,k\geq 2, α‾=(α1,…,αk)\underline{\alpha}=\left(\alpha_1,\ldots,\alpha_k\right) with αi>0\alpha_i>0 and α0=∑i=1kαi.\alpha_0=\sum_{i=1}^k\alpha_i. A random vector X‾\underline{X} has the Dirichlet distribution with parameter α‾,\underline{\alpha}, written X‾∼Dirichlet(α‾),\underline{X}\sim\mathrm{Dirichlet}\left(\underline{\alpha}\right), if and only if it takes values in the simplex with density

fX‾(x‾)=Γ(α0)∏i=1kΓ(αi)∏i=1kxiαi−1  ∀ x‾∈Δkf_{\underline{X}}\left(\underline{x}\right)=\frac{\Gamma\left(\alpha_0\right)}{\prod_{i=1}^k\Gamma\left(\alpha_i\right)}\prod_{i=1}^k x_i^{\alpha_i-1}\,\,\forall\,\underline{x}\in\Delta_k E[Xi]=αiα0\mathbb{E}\left[X_i\right]=\frac{\alpha_i}{\alpha_0} var[Xi]=αi(α0−αi)α02(α0+1)\mathrm{var}\left[X_i\right]=\frac{\alpha_i\left(\alpha_0-\alpha_i\right)}{\alpha_0^2\left(\alpha_0+1\right)} cov[Xi,Xj]=−αiαjα02(α0+1)  ∀ i≠j.\mathrm{cov}\left[X_i,X_j\right]=-\frac{\alpha_i\alpha_j}{\alpha_0^2\left(\alpha_0+1\right)}\,\,\forall\,i\neq j.

The Dirichlet distribution has the following properties:

  • it is the multivariate generalisation of the beta distribution, which is the case k=2,k=2, and models random probability vectors;
  • each marginal XiX_i is Beta(αi,α0−αi);\mathrm{Beta}\left(\alpha_i,\alpha_0-\alpha_i\right);
  • if Yi∼Gamma(αi,λ)Y_i\sim\mathrm{Gamma}\left(\alpha_i,\lambda\right) are independent then 1∑iYi(Y1,…,Yk)∼Dirichlet(α‾).\frac{1}{\sum_i Y_i}\left(Y_1,\ldots,Y_k\right)\sim\mathrm{Dirichlet}\left(\underline{\alpha}\right).

Let {PX; θ:θ∈Θ}\left\{\mathbb{P}_{X;\,\theta}:\theta\in\Theta\right\} be a parametric family. The family is an exponential family if and only if its pmf or pdf can be written as

fX(x; θ)=h(x)exp⁡(η‾(θ)⊺T‾(x)−A(θ))f_X\left(x;\,\theta\right)=h\left(x\right)\exp\left(\underline{\eta}\left(\theta\right)^\intercal\underline{T}\left(x\right)-A\left(\theta\right)\right)

for some functions hh and T‾\underline{T} of xx alone and η‾\underline{\eta} and AA of θ\theta alone. The vector η‾(θ)\underline{\eta}\left(\theta\right) is the natural parameter, the vector T‾(x)\underline{T}\left(x\right) is the vector of natural statistics and the function A(θ)A\left(\theta\right) is the log-partition function, which normalises the density. Exponential families have the following properties:

  • the Bernoulli distribution has η=log⁡ep1−p,\eta=\log_e\frac{p}{1-p}, T(x)=xT\left(x\right)=x and A=log⁡e(1+eη);A=\log_e\left(1+e^{\eta}\right);
  • the normal distribution with known variance has η=μσ2\eta=\frac{\mu}{\sigma^2} and T(x)=x;T\left(x\right)=x;
  • the Bernoulli, binomial and multinomial (with nn known), geometric, Poisson, exponential, gamma, beta, chi-squared, normal, multivariate normal and Dirichlet distributions are exponential families, whereas the uniform, tt- and FF-distributions are not;
  • the derivatives of the log-partition function with respect to the natural parameter give the moments of the natural statistics, E[T‾(X)]=∇A\mathbb{E}\left[\underline{T}\left(X\right)\right]=\nabla A and cov[T‾(X)]=HA.\mathrm{cov}\left[\underline{T}\left(X\right)\right]=H_A.