Let XX be a discrete real-valued random variable with pmf pXp_X and state space S.S. The expectation or mean of XX is the sum of its values weighted by their probabilities, provided the sum is absolutely convergent.

E[X]:=∑x∈Sx pX(x)\mathbb{E}\left[X\right]:=\sum_{x\in S}x\,p_X\left(x\right)

Let XX be a continuous random variable with pdf fX.f_X. The expectation or mean of XX is the integral of its values weighted by its density, provided the improper integral is absolutely convergent.

E[X]:=∫−∞∞x fX(x)dx\mathbb{E}\left[X\right]:=\int_{-\infty}^{\infty}x\,f_X\left(x\right)dx

If the sum or integral is not absolutely convergent then the expectation does not exist. The expectation is a property of the probability distribution of X,X, is often written μ,\mu, and need not be a value that XX can take.

Let XX be a real-valued random variable and g:R→Rg:\mathbb{R}\rightarrow\mathbb{R} be a function. The law of the unconscious statistician states that the expectation of the transformation g(X)g(X) can be computed from the distribution of XX without finding the distribution of g(X).g(X).

E[g(X)]=∑x∈Sg(x) pX(x),E[g(X)]=∫−∞∞g(x) fX(x)dx\mathbb{E}\left[g(X)\right]=\sum_{x\in S}g(x)\,p_X\left(x\right),\qquad\mathbb{E}\left[g(X)\right]=\int_{-\infty}^{\infty}g(x)\,f_X\left(x\right)dx

For a random vector X‾\underline{X} and g:Rn→Rg:\mathbb{R}^n\rightarrow\mathbb{R} the same holds with the joint pmf or pdf, E[g(X‾)]=∫Rng(x‾) fX‾(x‾)dx‾.\mathbb{E}\left[g(\underline{X})\right]=\int_{\mathbb{R}^n}g(\underline{x})\,f_{\underline{X}}\left(\underline{x}\right)d\underline{x}.

Let AA be an event. The indicator random variable of AA is the random variable 1A\mathbf{1}_A that is 11 if AA occurs and 00 otherwise.

Let X,YX,Y be random variables with finite expectations, a,b,c∈Ra,b,c\in\mathbb{R} and AA be an event. The expectation has the following properties:

E[c]=c,E[1A]=P(A)\mathbb{E}\left[c\right]=c,\qquad\mathbb{E}\left[\mathbf{1}_A\right]=\mathbb{P}\left(A\right) E[aX+bY+c]=a E[X]+b E[Y]+c\mathbb{E}\left[aX+bY+c\right]=a\,\mathbb{E}\left[X\right]+b\,\mathbb{E}\left[Y\right]+c if  X≤Y  then  E[X]≤E[Y]\mathrm{if}\,\,X\leq Y\,\,\mathrm{then}\,\,\mathbb{E}\left[X\right]\leq\mathbb{E}\left[Y\right] ∣E[X]∣≤E[∣X∣]\left\lvert\mathbb{E}\left[X\right]\right\rvert\leq\mathbb{E}\left[\lvert X\rvert\right] if  X  and  Y  are  independent  then  E[XY]=E[X]E[Y]\mathrm{if}\,\,X\,\,\mathrm{and}\,\,Y\,\,\mathrm{are}\,\,\mathrm{independent}\,\,\mathrm{then}\,\,\mathbb{E}\left[XY\right]=\mathbb{E}\left[X\right]\mathbb{E}\left[Y\right]

The second property is linearity of expectation, which holds whether or not XX and YY are independent.

A statement about random variables holds almost surely (a.s.) if and only if it holds with probability one.

Let XX be a random variable with finite mean μ.\mu. The variance of XX is the expected squared deviation from the mean.

var[X]:=E[(X−μ)2]=E[X2]−μ2\mathrm{var}\left[X\right]:=\mathbb{E}\left[\left(X-\mu\right)^2\right]=\mathbb{E}\left[X^2\right]-\mu^2

Let XX be a random variable with finite variance. The standard deviation of XX is the square root of its variance.

σ:=var[X]\sigma:=\sqrt{\mathrm{var}\left[X\right]}

The variance and standard deviation have the following properties for a,b∈R:a,b\in\mathbb{R}:

  • the variance is finite if and only if E[X2]\mathbb{E}\left[X^2\right] is finite;
  • var[X]≥0\mathrm{var}\left[X\right]\geq 0 with equality if and only if X=μX=\mu almost surely;
  • var[aX+b]=a2 var[X];\mathrm{var}\left[aX+b\right]=a^2\,\mathrm{var}\left[X\right];
  • the variance is measured in squared units of XX whereas the standard deviation is measured in the units of X.X.

Let XX be a random variable with mean μ\mu and standard deviation σ,\sigma, and k∈N.k\in\mathbb{N}. The kk-th moment of XX is E[Xk].\mathbb{E}\left[X^k\right]. The kk-th central moment of XX is E[(X−μ)k].\mathbb{E}\left[\left(X-\mu\right)^k\right]. The kk-th standardised moment of XX is E[(X−μσ)k].\mathbb{E}\left[\left(\frac{X-\mu}{\sigma}\right)^k\right]. Moments have the following properties:

  • the mean is the first moment and the variance is the second central moment;
  • if the kk-th moment is finite then every lower moment is finite.

Let XX be a random variable with finite third moment. The skewness of XX is its third standardised moment. The skewness measures the asymmetry of the distribution of X,X, and is zero for a distribution symmetric about its mean.

Let XX be a random variable with finite fourth moment. The kurtosis of XX is its fourth standardised moment. The excess kurtosis of XX is the kurtosis minus 3.3. The kurtosis measures the weight of the tails of the distribution relative to its centre.

Let XX be a random variable with mean μ\mu and standard deviation σ>0.\sigma>0. The standardisation of XX is the random variable Z=X−μσ,Z=\frac{X-\mu}{\sigma}, which has mean 00 and variance 1.1.

Let XX and YY be random variables with finite variances and means μX\mu_X and μY.\mu_Y. The covariance of XX and YY is the expected product of their deviations from their means.

cov[X,Y]:=E[(X−μX)(Y−μY)]=E[XY]−μXμY\mathrm{cov}\left[X,Y\right]:=\mathbb{E}\left[\left(X-\mu_X\right)\left(Y-\mu_Y\right)\right]=\mathbb{E}\left[XY\right]-\mu_X\mu_Y

The covariance has the following properties for a,b,c,d∈R:a,b,c,d\in\mathbb{R}:

cov[X,Y]=cov[Y,X],cov[X,X]=var[X]\mathrm{cov}\left[X,Y\right]=\mathrm{cov}\left[Y,X\right],\qquad\mathrm{cov}\left[X,X\right]=\mathrm{var}\left[X\right] cov[aX+b,cY+d]=ac cov[X,Y]\mathrm{cov}\left[aX+b,cY+d\right]=ac\,\mathrm{cov}\left[X,Y\right] var[X+Y]=var[X]+var[Y]+2 cov[X,Y]\mathrm{var}\left[X+Y\right]=\mathrm{var}\left[X\right]+\mathrm{var}\left[Y\right]+2\,\mathrm{cov}\left[X,Y\right]

Let XX and YY be random variables with finite variances. The random variables XX and YY are uncorrelated if and only if cov[X,Y]=0.\mathrm{cov}\left[X,Y\right]=0. Uncorrelated random variables have the following properties:

  • the variance of the sum of uncorrelated random variables is the sum of their variances;
  • independent random variables are uncorrelated;
  • uncorrelated random variables need not be independent: if XX is symmetric about 00 with finite third moment then XX and X2X^2 are uncorrelated but not independent.

Let XX and YY be random variables with finite positive variances. The correlation of XX and YY is their covariance divided by the product of their standard deviations.

ρX,Y:=cov[X,Y]σXσY=cov[X−μXσX,Y−μYσY]\rho_{X,Y}:=\frac{\mathrm{cov}\left[X,Y\right]}{\sigma_X\sigma_Y}=\mathrm{cov}\left[\frac{X-\mu_X}{\sigma_X},\frac{Y-\mu_Y}{\sigma_Y}\right]

The correlation has the following properties:

  • it is free of units;
  • −1≤ρX,Y≤1;-1\leq\rho_{X,Y}\leq 1;
  • ∣ρX,Y∣=1\lvert\rho_{X,Y}\rvert=1 if and only if Y=aX+bY=aX+b almost surely for some a≠0,a\neq 0, with the sign of aa equal to the sign of ρX,Y;\rho_{X,Y};
  • correlation measures linear dependence only.

Let X‾=(X1,…,Xn)\underline{X}=\left(X_1,\ldots,X_n\right) be a random vector whose components have finite variances. The mean vector of X‾\underline{X} is the vector of component means.

μ‾:=E[X‾],μi=E[Xi]\underline{\mu}:=\mathbb{E}\left[\underline{X}\right],\qquad\mu_i=\mathbb{E}\left[X_i\right]

Let X‾=(X1,…,Xn)\underline{X}=\left(X_1,\ldots,X_n\right) be a random vector whose components have finite variances and mean vector μ‾.\underline{\mu}. The covariance matrix of X‾\underline{X} is the matrix Σ=cov[X‾]∈R n×n\Sigma=\mathrm{cov}\left[\underline{X}\right]\in\mathbb{R}^{\,n\times n} of pairwise covariances of the components.

Σij:=cov[Xi,Xj],Σ=E[(X‾−μ‾)(X‾−μ‾)⊺]\Sigma_{ij}:=\mathrm{cov}\left[X_i,X_j\right],\qquad\Sigma=\mathbb{E}\left[\left(\underline{X}-\underline{\mu}\right)\left(\underline{X}-\underline{\mu}\right)^\intercal\right]

The covariance matrix has the following properties for A∈R m×nA\in\mathbb{R}^{\,m\times n} and b‾∈Rm:\underline{b}\in\mathbb{R}^m:

  • its diagonal holds the variances of the components;
  • it is symmetric and positive semi-definite, since a‾⊺Σa‾=var[a‾⊺X‾]≥0\underline{a}^\intercal\Sigma\underline{a}=\mathrm{var}\left[\underline{a}^\intercal\underline{X}\right]\geq 0 for every a‾∈Rn;\underline{a}\in\mathbb{R}^n;
  • it is positive definite unless some linear combination of the components is almost surely constant;
  • the affine transformation AX‾+b‾A\underline{X}+\underline{b} has mean vector Aμ‾+b‾A\underline{\mu}+\underline{b} and covariance matrix AΣA⊺.A\Sigma A^\intercal.

Let X‾=(X1,…,Xn)\underline{X}=\left(X_1,\ldots,X_n\right) be a random vector whose components have finite positive variances. The correlation matrix of X‾\underline{X} is the matrix with entries ρXi,Xj.\rho_{X_i,X_j}. The correlation matrix has ones on its diagonal and is the covariance matrix of the standardised components.

Let (X,Y)\left(X,Y\right) be a random vector and xx be a value with pX(x)>0p_X\left(x\right)>0 or fX(x)>0.f_X\left(x\right)>0. The conditional expectation of YY given X=xX=x is the expectation of the conditional distribution of YY given X=x.X=x.

E[Y∣X=x]:=∑yy pY∣X(y∣x),E[Y∣X=x]:=∫−∞∞y fY∣X(y∣x)dy\mathbb{E}\left[Y\mid X=x\right]:=\sum_y y\,p_{Y\mid X}\left(y\mid x\right),\qquad\mathbb{E}\left[Y\mid X=x\right]:=\int_{-\infty}^{\infty}y\,f_{Y\mid X}\left(y\mid x\right)dy

Let (X,Y)\left(X,Y\right) be a random vector and h(x)=E[Y∣X=x].h(x)=\mathbb{E}\left[Y\mid X=x\right]. The conditional expectation of YY given XX is the random variable obtained by evaluating hh at X.X.

E[Y∣X]:=h(X)\mathbb{E}\left[Y\mid X\right]:=h(X)

Let (X,Y)\left(X,Y\right) be a random vector with YY having finite expectation. The law of total expectation or tower property states that averaging the conditional expectation over XX recovers the expectation of Y.Y.

E[E[Y∣X]]=E[Y]\mathbb{E}\left[\mathbb{E}\left[Y\mid X\right]\right]=\mathbb{E}\left[Y\right]

Conditional expectation has the following properties:

  • it is linear in Y;Y;
  • E[g(X)Y∣X]=g(X) E[Y∣X]\mathbb{E}\left[g(X)Y\mid X\right]=g(X)\,\mathbb{E}\left[Y\mid X\right] for any function g;g;
  • if XX and YY are independent then E[Y∣X]=E[Y];\mathbb{E}\left[Y\mid X\right]=\mathbb{E}\left[Y\right];
  • among all functions of XX with finite variance, E[Y∣X]\mathbb{E}\left[Y\mid X\right] minimises the expected squared error E[(Y−g(X))2],\mathbb{E}\left[\left(Y-g(X)\right)^2\right], so it is the best predictor of YY from XX in mean square.

Let (X,Y)\left(X,Y\right) be a random vector with YY having finite variance. The conditional variance of YY given XX is the variance of the conditional distribution of YY given X.X.

var[Y∣X]:=E[(Y−E[Y∣X])2∣X]\mathrm{var}\left[Y\mid X\right]:=\mathbb{E}\left[\left(Y-\mathbb{E}\left[Y\mid X\right]\right)^2\mid X\right]

Let (X,Y)\left(X,Y\right) be a random vector with YY having finite variance. The law of total variance states that the variance of YY is the expected variance within values of XX plus the variance of the conditional means between values of X.X.

var[Y]=E[var[Y∣X]]+var[E[Y∣X]]\mathrm{var}\left[Y\right]=\mathbb{E}\left[\mathrm{var}\left[Y\mid X\right]\right]+\mathrm{var}\left[\mathbb{E}\left[Y\mid X\right]\right]

Let XX be a discrete random variable with state space {0,1,2,…}.\{0,1,2,\ldots\}. The probability generating function (pgf) of XX is the power series whose coefficients are the values of the pmf.

GX(s):=E[sX]=∑k=0∞pX(k)sk,∣s∣≤1G_X\left(s\right):=\mathbb{E}\left[s^X\right]=\sum_{k=0}^{\infty}p_X\left(k\right)s^k,\qquad\lvert s\rvert\leq 1

Let XX be a discrete random variable with state space {0,1,2,…}\{0,1,2,\ldots\} and k∈N.k\in\mathbb{N}. The kk-th factorial moment of XX is E[X(X−1)⋯(X−k+1)].\mathbb{E}\left[X\left(X-1\right)\cdots\left(X-k+1\right)\right]. The pgf has the following properties:

  • GX(1)=1;G_X\left(1\right)=1;
  • the pgf determines the distribution of X,X, since pX(k)=1k!GX(k)(0);p_X\left(k\right)=\frac{1}{k!}G_X^{(k)}\left(0\right);
  • the kk-th factorial moment is GX(k)(1),G_X^{(k)}\left(1\right), so E[X]=GX′(1)\mathbb{E}\left[X\right]=G_X'\left(1\right) and var[X]=GX′′(1)+GX′(1)−GX′(1)2;\mathrm{var}\left[X\right]=G_X''\left(1\right)+G_X'\left(1\right)-G_X'\left(1\right)^2;
  • if XX and YY are independent then GX+Y(s)=GX(s)GY(s).G_{X+Y}\left(s\right)=G_X\left(s\right)G_Y\left(s\right).

Let XX be a random variable. The moment generating function (mgf) of XX is the expectation of the exponential of tX,tX, provided this expectation is finite for all tt in some open interval containing 0.0.

MX(t):=E[etX]M_X\left(t\right):=\mathbb{E}\left[e^{tX}\right]

If no such interval exists then the mgf does not exist. The mgf has the following properties:

  • when it exists, every moment of XX is finite and is obtained by differentiating at 0,0, since the series expansion of the exponential gives MX(t)=∑k=0∞E[Xk]k!tkM_X\left(t\right)=\sum_{k=0}^{\infty}\frac{\mathbb{E}\left[X^k\right]}{k!}t^k and E[Xk]=MX(k)(0);\mathbb{E}\left[X^k\right]=M_X^{(k)}\left(0\right);
  • MX(0)=1M_X\left(0\right)=1 and MaX+b(t)=ebtMX(at);M_{aX+b}\left(t\right)=e^{bt}M_X\left(at\right);
  • if XX and YY are independent then MX+Y(t)=MX(t)MY(t).M_{X+Y}\left(t\right)=M_X\left(t\right)M_Y\left(t\right).

Let XX and YY be random variables whose mgfs exist. The uniqueness theorem states that if MXM_X and MYM_Y agree on an open interval containing 00 then XX and YY have the same distribution.

Let XX be a random variable whose mgf exists. The cumulant generating function of XX is the natural logarithm of the mgf.

KX(t):=log⁡eMX(t)K_X\left(t\right):=\log_e M_X\left(t\right)

Let XX be a random variable whose mgf exists and k≥1.k\geq 1. The kk-th cumulant of XX is the kk-th derivative of the cumulant generating function at 0.0.

κk:=KX(k)(0)\kappa_k:=K_X^{(k)}\left(0\right)

The cumulants have the following properties: κ1=μ,\kappa_1=\mu, κ2=σ2,\kappa_2=\sigma^2, κ3\kappa_3 is the third central moment and cumulants of independent random variables add.

Let XX be a random variable and ii be the imaginary unit. The characteristic function of XX is the expectation of the complex exponential of itX.itX.

φX(t):=E[eitX]=E[cos⁡(tX)]+i E[sin⁡(tX)]  ∀ t∈R\varphi_X\left(t\right):=\mathbb{E}\left[e^{itX}\right]=\mathbb{E}\left[\cos\left(tX\right)\right]+i\,\mathbb{E}\left[\sin\left(tX\right)\right]\,\,\forall\,t\in\mathbb{R}

The characteristic function has the following properties:

  • it exists for every random variable, since ∣eitX∣=1;\lvert e^{itX}\rvert=1;
  • φX(0)=1\varphi_X\left(0\right)=1 and ∣φX(t)∣≤1;\lvert\varphi_X\left(t\right)\rvert\leq 1;
  • φaX+b(t)=eitbφX(at);\varphi_{aX+b}\left(t\right)=e^{itb}\varphi_X\left(at\right);
  • if XX and YY are independent then φX+Y=φXφY;\varphi_{X+Y}=\varphi_X\varphi_Y;
  • if the kk-th moment of XX is finite then E[Xk]=i−kφX(k)(0);\mathbb{E}\left[X^k\right]=i^{-k}\varphi_X^{(k)}\left(0\right);
  • when the mgf exists, φX(t)=MX(it).\varphi_X\left(t\right)=M_X\left(it\right).

Let XX and YY be random variables. The inversion theorem states that the characteristic function determines the distribution uniquely, so XX and YY have the same distribution if and only if they have the same characteristic function. The characteristic function is the tool used to prove limit theorems about sums of random variables.