Let X be a discrete real-valued random variable with pmf pX and state space S. The expectation or mean of X is the sum of its values weighted by their probabilities, provided the sum is absolutely convergent.
E[X]:=x∈S∑xpX(x)
Let X be a continuous random variable with pdf fX. The expectation or mean of X is the integral of its values weighted by its density, provided the improper integral is absolutely convergent.
E[X]:=∫−∞∞xfX(x)dx
If the sum or integral is not absolutely convergent then the expectation does not exist. The expectation is a property of the probability distribution of X, is often written μ, and need not be a value that X can take.
Let X be a real-valued random variable and g:R→R be a function. The law of the unconscious statistician states that the expectation of the transformation g(X) can be computed from the distribution of X without finding the distribution of g(X).
E[g(X)]=x∈S∑g(x)pX(x),E[g(X)]=∫−∞∞g(x)fX(x)dx
For a random vector X and g:Rn→R the same holds with the joint pmf or pdf, E[g(X)]=∫Rng(x)fX(x)dx.
Let A be an event. The indicator random variable of A is the random variable 1A that is 1 if A occurs and 0 otherwise.
Let X,Y be random variables with finite expectations, a,b,c∈R and A be an event. The expectation has the following properties:
The second property is linearity of expectation, which holds whether or not X and Y are independent.
A statement about random variables holds almost surely (a.s.) if and only if it holds with probability one.
Let X be a random variable with finite mean μ. The variance of X is the expected squared deviation from the mean.
var[X]:=E[(X−μ)2]=E[X2]−μ2
Let X be a random variable with finite variance. The standard deviation of X is the square root of its variance.
σ:=var[X]
The variance and standard deviation have the following properties for a,b∈R:
the variance is finite if and only if E[X2] is finite;
var[X]≥0 with equality if and only if X=μ almost surely;
var[aX+b]=a2var[X];
the variance is measured in squared units of X whereas the standard deviation is measured in the units of X.
Let X be a random variable with mean μ and standard deviation σ, and k∈N. The k-th moment of X is E[Xk]. The k-th central moment of X is E[(X−μ)k]. The k-th standardised moment of X is E[(σX−μ)k]. Moments have the following properties:
the mean is the first moment and the variance is the second central moment;
if the k-th moment is finite then every lower moment is finite.
Let X be a random variable with finite third moment. The skewness of X is its third standardised moment. The skewness measures the asymmetry of the distribution of X, and is zero for a distribution symmetric about its mean.
Let X be a random variable with finite fourth moment. The kurtosis of X is its fourth standardised moment. The excess kurtosis of X is the kurtosis minus 3. The kurtosis measures the weight of the tails of the distribution relative to its centre.
Let X be a random variable with mean μ and standard deviation σ>0. The standardisation of X is the random variable Z=σX−μ, which has mean 0 and variance 1.
Let X and Y be random variables with finite variances and means μX and μY. The covariance of X and Y is the expected product of their deviations from their means.
cov[X,Y]:=E[(X−μX)(Y−μY)]=E[XY]−μXμY
The covariance has the following properties for a,b,c,d∈R:
Let X and Y be random variables with finite variances. The random variables X and Y are uncorrelated if and only if cov[X,Y]=0. Uncorrelated random variables have the following properties:
the variance of the sum of uncorrelated random variables is the sum of their variances;
independent random variables are uncorrelated;
uncorrelated random variables need not be independent: if X is symmetric about 0 with finite third moment then X and X2 are uncorrelated but not independent.
Let X and Y be random variables with finite positive variances. The correlation of X and Y is their covariance divided by the product of their standard deviations.
ρX,Y:=σXσYcov[X,Y]=cov[σXX−μX,σYY−μY]
The correlation has the following properties:
it is free of units;
−1≤ρX,Y≤1;
∣ρX,Y∣=1 if and only if Y=aX+b almost surely for some a=0, with the sign of a equal to the sign of ρX,Y;
correlation measures linear dependence only.
Let X=(X1,…,Xn) be a random vector whose components have finite variances. The mean vector of X is the vector of component means.
μ:=E[X],μi=E[Xi]
Let X=(X1,…,Xn) be a random vector whose components have finite variances and mean vector μ. The covariance matrix of X is the matrix Σ=cov[X]∈Rn×n of pairwise covariances of the components.
Σij:=cov[Xi,Xj],Σ=E[(X−μ)(X−μ)⊺]
The covariance matrix has the following properties for A∈Rm×n and b∈Rm:
its diagonal holds the variances of the components;
it is symmetric and positive semi-definite, since a⊺Σa=var[a⊺X]≥0 for every a∈Rn;
it is positive definite unless some linear combination of the components is almost surely constant;
the affine transformation AX+b has mean vector Aμ+b and covariance matrix AΣA⊺.
Let X=(X1,…,Xn) be a random vector whose components have finite positive variances. The correlation matrix of X is the matrix with entries ρXi,Xj. The correlation matrix has ones on its diagonal and is the covariance matrix of the standardised components.
Let (X,Y) be a random vector and x be a value with pX(x)>0 or fX(x)>0. The conditional expectation of Y given X=x is the expectation of the conditional distribution of Y given X=x.
Let (X,Y) be a random vector and h(x)=E[Y∣X=x]. The conditional expectation of Y given X is the random variable obtained by evaluating h at X.
E[Y∣X]:=h(X)
Let (X,Y) be a random vector with Y having finite expectation. The law of total expectation or tower property states that averaging the conditional expectation over X recovers the expectation of Y.
E[E[Y∣X]]=E[Y]
Conditional expectation has the following properties:
it is linear in Y;
E[g(X)Y∣X]=g(X)E[Y∣X] for any function g;
if X and Y are independent then E[Y∣X]=E[Y];
among all functions of X with finite variance, E[Y∣X] minimises the expected squared error E[(Y−g(X))2], so it is the best predictor of Y from X in mean square.
Let (X,Y) be a random vector with Y having finite variance. The conditional variance of Y given X is the variance of the conditional distribution of Y given X.
var[Y∣X]:=E[(Y−E[Y∣X])2∣X]
Let (X,Y) be a random vector with Y having finite variance. The law of total variance states that the variance of Y is the expected variance within values of X plus the variance of the conditional means between values of X.
var[Y]=E[var[Y∣X]]+var[E[Y∣X]]
Let X be a discrete random variable with state space {0,1,2,…}. The probability generating function (pgf) of X is the power series whose coefficients are the values of the pmf.
GX(s):=E[sX]=k=0∑∞pX(k)sk,∣s∣≤1
Let X be a discrete random variable with state space {0,1,2,…} and k∈N. The k-th factorial moment of X is E[X(X−1)⋯(X−k+1)]. The pgf has the following properties:
GX(1)=1;
the pgf determines the distribution of X, since pX(k)=k!1GX(k)(0);
the k-th factorial moment is GX(k)(1), so E[X]=GX′(1) and var[X]=GX′′(1)+GX′(1)−GX′(1)2;
if X and Y are independent then GX+Y(s)=GX(s)GY(s).
Let X be a random variable. The moment generating function (mgf) of X is the expectation of the exponential of tX, provided this expectation is finite for all t in some open interval containing 0.
MX(t):=E[etX]
If no such interval exists then the mgf does not exist. The mgf has the following properties:
when it exists, every moment of X is finite and is obtained by differentiating at 0, since the series expansion of the exponential gives MX(t)=∑k=0∞k!E[Xk]tk and E[Xk]=MX(k)(0);
MX(0)=1 and MaX+b(t)=ebtMX(at);
if X and Y are independent then MX+Y(t)=MX(t)MY(t).
Let X and Y be random variables whose mgfs exist. The uniqueness theorem states that if MX and MY agree on an open interval containing 0 then X and Y have the same distribution.
Let X be a random variable whose mgf exists. The cumulant generating function of X is the natural logarithm of the mgf.
KX(t):=logeMX(t)
Let X be a random variable whose mgf exists and k≥1. The k-th cumulant of X is the k-th derivative of the cumulant generating function at 0.
κk:=KX(k)(0)
The cumulants have the following properties: κ1=μ,κ2=σ2,κ3 is the third central moment and cumulants of independent random variables add.
Let X be a random variable and i be the imaginary unit. The characteristic function of X is the expectation of the complex exponential of itX.
φX(t):=E[eitX]=E[cos(tX)]+iE[sin(tX)]∀t∈R
The characteristic function has the following properties:
it exists for every random variable, since ∣eitX∣=1;
φX(0)=1 and ∣φX(t)∣≤1;
φaX+b(t)=eitbφX(at);
if X and Y are independent then φX+Y=φXφY;
if the k-th moment of X is finite then E[Xk]=i−kφX(k)(0);
when the mgf exists, φX(t)=MX(it).
Let X and Y be random variables. The inversion theorem states that the characteristic function determines the distribution uniquely, so X and Y have the same distribution if and only if they have the same characteristic function. The characteristic function is the tool used to prove limit theorems about sums of random variables.