Let X‾\underline{X} be a random sample distributed according to the population PX; θ.\mathbb{P}_{X;\,\theta}. An estimator for the parameter θ\theta is a statistic whose value θ^\hat{\theta} is intended to approximate θ:\theta:

θ^=t(X‾).\hat{\theta}=t\left(\underline{X}\right).

A realised value of an estimator is called an estimate and is evaluated for an observed random sample x‾:\underline{x}:

θ^=t(x‾).\hat{\theta}=t\left(\underline{x}\right).

The bias b(θ^)b\left(\hat{\theta}\right) of an estimator θ^\hat{\theta} is the difference between the expected value of its sampling distribution and the true value of the parameter θ:\theta:

b(θ^)=E[θ^]−θ.b\left(\hat{\theta}\right)=\mathbb{E}\left[\hat{\theta}\right]-\theta.

An estimator is unbiased if the expected value of its sampling distribution is equal to the parameter being estimated:

E[θ^]=θ⇔b(θ^)=0.\mathbb{E}\left[\hat{\theta}\right]=\theta\Leftrightarrow b\left(\hat{\theta}\right)=0.

The standard error se(θ^)se\left(\hat{\theta}\right) of an estimator θ^\hat{\theta} is the standard deviation of its sampling distribution:

se(θ^)=var[θ^].se\left(\hat{\theta}\right)=\sqrt{\mathrm{var}\left[\hat{\theta}\right]}.

The mean squared error MSE(θ^)MSE\left(\hat{\theta}\right) of an estimator θ^\hat{\theta} is defined as a function of the parameter θ:\theta:

MSE(θ^)=E[(θ^−θ)2]=b2(θ^)+var[θ^]\begin{align*} MSE\left(\hat{\theta}\right)&=\mathbb{E}\left[\left(\hat{\theta}-\theta\right)^2\right]\\ &=b^2\left(\hat{\theta}\right)+\mathrm{var}\left[\hat{\theta}\right] \end{align*}

and captures the trade-off between the bias and variance of an estimator.

The bias, standard error and mean squared error are all defined in terms of the moments of the estimator’s sampling distribution. However, since the random samples are drawn from the population PX; θ\mathbb{P}_{X;\,\theta} then these quantities can also be calculated as functions of the moments of PX; θ.\mathbb{P}_{X;\,\theta}.

Let θ^n=tn(X1,…,Xn)\hat{\theta}_n=t_n\left(X_1,\ldots,X_n\right) be an estimator of θ\theta computed from a random sample of size n,n, for each n∈N.n\in\mathbb{N}. The sequence of estimators is consistent for θ\theta if and only if it converges in probability to the true parameter value, whatever that value is.

θ^n→pθ  ∀ θ∈Θ\hat{\theta}_n\xrightarrow{p}\theta\,\,\forall\,\theta\in\Theta

It is strongly consistent if the convergence is almost sure. Consistency states that with enough data the estimator is close to the truth with high probability, and is the minimal requirement of a sensible estimator. By Markov’s inequality applied to (θ^n−θ)2,\left(\hat{\theta}_n-\theta\right)^2, an estimator whose mean squared error tends to zero is consistent.

P(∣θ^n−θ∣>ε)≤MSE(θ^n)ε2=b2(θ^n)+var[θ^n]ε2→0\mathbb{P}\left(\left\lvert\hat{\theta}_n-\theta\right\rvert>\varepsilon\right)\leq\frac{MSE\left(\hat{\theta}_n\right)}{\varepsilon^2}=\frac{b^2\left(\hat{\theta}_n\right)+\mathrm{var}\left[\hat{\theta}_n\right]}{\varepsilon^2}\rightarrow 0

An estimator is therefore consistent if its bias and variance both tend to zero, in which case it is asymptotically unbiased. The sample mean is consistent for the population mean by the weak law of large numbers, and strongly consistent by the strong law, and the sample variance is consistent for the population variance. Consistency and unbiasedness are distinct: X1X_1 is an unbiased but inconsistent estimator of the population mean, whereas 1n∑i=1n(Xi−X‾n)2\frac{1}{n}\sum_{i=1}^n\left(X_i-\overline{X}_n\right)^2 is a biased but consistent estimator of the population variance. By the continuous mapping theorem, if θ^n\hat{\theta}_n is consistent for θ\theta and gg is continuous then g(θ^n)g\left(\hat{\theta}_n\right) is consistent for g(θ).g\left(\theta\right).

A sequence of estimators is asymptotically normal if and only if its standardised error converges in distribution to a normal distribution,

n(θ^n−θ)→dN(0,v(θ)),\sqrt{n}\left(\hat{\theta}_n-\theta\right)\xrightarrow{d}\mathrm{N}\left(0,v\left(\theta\right)\right),

where v(θ)v\left(\theta\right) is the asymptotic variance. An asymptotically normal estimator is consistent, and its sampling distribution is approximately N(θ,v(θ)n)\mathrm{N}\left(\theta,\frac{v\left(\theta\right)}{n}\right) for large n,n, which is what the interval estimates and tests of the following notes are built on. The sample mean is asymptotically normal with v=σ2v=\sigma^2 by the central limit theorem, and the delta method gives the asymptotic normality of smooth functions of asymptotically normal estimators.

Let the population PX; θ\mathbb{P}_{X;\,\theta} have pmf or pdf fX; θ(x; θ)f_{X;\,\theta}\left(x;\,\theta\right) with scalar parameter θ\theta in an open parameter space, and suppose the regularity conditions hold: the support of fX; θf_{X;\,\theta} does not depend on θ,\theta, fX; θf_{X;\,\theta} is twice differentiable in θ\theta and differentiation with respect to θ\theta and integration or summation over xx may be interchanged. The score is the derivative of the log pmf or pdf with respect to the parameter,

s(θ; x):=∂∂θlog⁡efX; θ(x; θ),s\left(\theta;\,x\right):=\frac{\partial}{\partial\theta}\log_e f_{X;\,\theta}\left(x;\,\theta\right),

which measures how sensitive the log probability of the observation xx is to the parameter. The score has expectation zero under the population, E[s(θ; X)]=0,\mathbb{E}\left[s\left(\theta;\,X\right)\right]=0, since ∫∂∂θfX; θ(x; θ)dx=∂∂θ∫fX; θ(x; θ)dx=0.\int\frac{\partial}{\partial\theta}f_{X;\,\theta}\left(x;\,\theta\right)dx=\frac{\partial}{\partial\theta}\int f_{X;\,\theta}\left(x;\,\theta\right)dx=0.

The Fisher information I(θ)I\left(\theta\right) of a single observation is the variance of the score, which under the regularity conditions is also the negative expected second derivative of the log pmf or pdf.

I(θ):=var[s(θ; X)]=E[s(θ; X)2]=−E[∂2∂θ2log⁡efX; θ(X; θ)]I\left(\theta\right):=\mathrm{var}\left[s\left(\theta;\,X\right)\right]=\mathbb{E}\left[s\left(\theta;\,X\right)^2\right]=-\mathbb{E}\left[\frac{\partial^2}{\partial\theta^2}\log_e f_{X;\,\theta}\left(X;\,\theta\right)\right]

The Fisher information measures how much an observation reveals about θ:\theta: a large value means that the distribution changes rapidly with θ,\theta, so nearby parameter values are easy to distinguish. Since the score of a random sample is the sum of the scores of its iid observations, the Fisher information of a random sample of size nn is n I(θ).n\,I\left(\theta\right). For a vector parameter θ‾∈Rk\underline{\theta}\in\mathbb{R}^k the score is the gradient ∇θ‾log⁡efX; θ‾\nabla_{\underline{\theta}}\log_e f_{X;\,\underline{\theta}} and the Fisher information matrix I(θ‾)∈R k×kI\left(\underline{\theta}\right)\in\mathbb{R}^{\,k\times k} is its covariance matrix, which equals the negative expected Hessian matrix of the log pmf or pdf.

The Cramér–Rao lower bound states that under the regularity conditions the variance of any unbiased estimator θ^\hat{\theta} of θ\theta based on a random sample of size nn is at least the reciprocal of the Fisher information of the sample.

var[θ^]≥1n I(θ)\mathrm{var}\left[\hat{\theta}\right]\geq\frac{1}{n\,I\left(\theta\right)}

More generally, an estimator θ^\hat{\theta} with bias b(θ)b\left(\theta\right) satisfies var[θ^]≥(1+b′(θ))2n I(θ),\mathrm{var}\left[\hat{\theta}\right]\geq\frac{\left(1+b'\left(\theta\right)\right)^2}{n\,I\left(\theta\right)}, an unbiased estimator of g(θ)g\left(\theta\right) satisfies var[g^]≥g′(θ)2n I(θ)\mathrm{var}\left[\hat{g}\right]\geq\frac{g'\left(\theta\right)^2}{n\,I\left(\theta\right)} and, for a vector parameter, the covariance matrix of an unbiased estimator satisfies cov[θ‾^]−1nI(θ‾)−1\mathrm{cov}\left[\hat{\underline{\theta}}\right]-\frac{1}{n}I\left(\underline{\theta}\right)^{-1} is positive semi-definite. The bound follows from the Cauchy–Schwarz inequality applied to θ^−θ\hat{\theta}-\theta and the score of the sample. A biased estimator can have smaller variance, and even smaller mean squared error, than the bound allows for unbiased estimators.

An unbiased estimator whose variance attains the Cramér–Rao lower bound for every θ∈Θ\theta\in\Theta is efficient. The efficiency of an unbiased estimator is the ratio of the bound to its variance,

e(θ^):=1n I(θ) var[θ^]∈(0,1],e\left(\hat{\theta}\right):=\frac{1}{n\,I\left(\theta\right)\,\mathrm{var}\left[\hat{\theta}\right]}\in\left(0,1\right],

so an efficient estimator has efficiency one. The relative efficiency of two unbiased estimators θ^1\hat{\theta}_1 and θ^2\hat{\theta}_2 is the ratio of their variances var[θ^2]var[θ^1],\frac{\mathrm{var}\left[\hat{\theta}_2\right]}{\mathrm{var}\left[\hat{\theta}_1\right]}, which is the factor by which the sample size using θ^2\hat{\theta}_2 must be increased to match the precision of θ^1.\hat{\theta}_1. For example, for a normal population the sample mean is efficient for μ\mu whereas the sample median has asymptotic variance πσ22,\frac{\pi\sigma^2}{2}, so its relative efficiency with respect to the sample mean is 2π≈0.64.\frac{2}{\pi}\approx 0.64. An efficient estimator need not exist, and when it does it is unique. An asymptotically normal estimator is asymptotically efficient if its asymptotic variance equals 1I(θ),\frac{1}{I\left(\theta\right)}, the smallest asymptotic variance that a consistent and asymptotically normal estimator can have at all but a negligible set of parameter values.