Let X‾\underline{X} be a random sample from the population PX; θ.\mathbb{P}_{X;\,\theta}. An estimator of θ\theta is a statistic θ^=t(X‾)\hat{\theta}=t\left(\underline{X}\right) whose value is intended to approximate θ.\theta.

Let θ^=t(X‾)\hat{\theta}=t\left(\underline{X}\right) be an estimator of θ\theta and x‾\underline{x} be an observed random sample. An estimate of θ\theta is the realised value θ^=t(x‾)\hat{\theta}=t\left(\underline{x}\right) of the estimator.

Let θ^\hat{\theta} be an estimator of θ\theta with finite expectation. The bias of θ^\hat{\theta} is the difference between the expectation of its sampling distribution and the true value of the parameter.

b(θ^):=E[θ^]−θb\left(\hat{\theta}\right):=\mathbb{E}\left[\hat{\theta}\right]-\theta

Let θ^\hat{\theta} be an estimator of θ.\theta. The estimator θ^\hat{\theta} is unbiased if and only if the expectation of its sampling distribution equals the parameter.

E[θ^]=θ⇔b(θ^)=0\mathbb{E}\left[\hat{\theta}\right]=\theta\Leftrightarrow b\left(\hat{\theta}\right)=0

Let θ^\hat{\theta} be an estimator of θ\theta with finite variance. The standard error of θ^\hat{\theta} is the standard deviation of its sampling distribution.

se(θ^):=var[θ^]se\left(\hat{\theta}\right):=\sqrt{\mathrm{var}\left[\hat{\theta}\right]}

Let θ^\hat{\theta} be an estimator of θ\theta with finite variance. The mean squared error of θ^\hat{\theta} is the expected squared difference between the estimator and the parameter, which is the sum of the squared bias and the variance.

MSE(θ^):=E[(θ^−θ)2]=b2(θ^)+var[θ^]\begin{align*} MSE\left(\hat{\theta}\right)&:=\mathbb{E}\left[\left(\hat{\theta}-\theta\right)^2\right]\\ &=b^2\left(\hat{\theta}\right)+\mathrm{var}\left[\hat{\theta}\right] \end{align*}

The mean squared error captures the trade-off between the bias and the variance of an estimator. The bias, standard error and mean squared error are defined in terms of the moments of the sampling distribution of the estimator, and since the random sample is drawn from the population PX; θ\mathbb{P}_{X;\,\theta} they can also be calculated as functions of the moments of PX; θ.\mathbb{P}_{X;\,\theta}.

Let θ^n=tn(X1,…,Xn)\hat{\theta}_n=t_n\left(X_1,\ldots,X_n\right) be an estimator of θ\theta computed from a random sample of size n,n, for each n∈N.n\in\mathbb{N}. The sequence of estimators is consistent for θ\theta if and only if it converges in probability to the true parameter value, whatever that value is.

θ^n→pθ  ∀ θ∈Θ\hat{\theta}_n\xrightarrow{p}\theta\,\,\forall\,\theta\in\Theta

Let θ^n\hat{\theta}_n be a sequence of estimators of θ.\theta. The sequence is strongly consistent for θ\theta if and only if it converges almost surely to the true parameter value.

Let θ^n\hat{\theta}_n be a sequence of estimators of θ.\theta. The sequence is asymptotically unbiased if and only if its bias tends to zero as n→∞.n\rightarrow\infty. Consistency has the following properties:

  • consistency states that with enough data the estimator is close to the truth with high probability, and is the minimal requirement of a sensible estimator;
  • by Markov’s inequality applied to (θ^n−θ)2,\left(\hat{\theta}_n-\theta\right)^2, an estimator whose mean squared error tends to zero is consistent, since P(∣θ^n−θ∣>ε)≤MSE(θ^n)ε2=b2(θ^n)+var[θ^n]ε2;\mathbb{P}\left(\left\lvert\hat{\theta}_n-\theta\right\rvert>\varepsilon\right)\leq\frac{MSE\left(\hat{\theta}_n\right)}{\varepsilon^2}=\frac{b^2\left(\hat{\theta}_n\right)+\mathrm{var}\left[\hat{\theta}_n\right]}{\varepsilon^2};
  • an estimator is therefore consistent if it is asymptotically unbiased and its variance tends to zero;
  • the sample mean is consistent for the population mean by the weak law of large numbers, and strongly consistent by the strong law, and the sample variance is consistent for the population variance;
  • consistency and unbiasedness are distinct: X1X_1 is an unbiased but inconsistent estimator of the population mean, whereas 1n∑i=1n(Xi−X‾n)2\frac{1}{n}\sum_{i=1}^n\left(X_i-\overline{X}_n\right)^2 is a biased but consistent estimator of the population variance;
  • by the continuous mapping theorem, if θ^n\hat{\theta}_n is consistent for θ\theta and gg is continuous then g(θ^n)g\left(\hat{\theta}_n\right) is consistent for g(θ).g\left(\theta\right).

Let θ^n\hat{\theta}_n be a sequence of estimators of θ.\theta. The sequence is asymptotically normal if and only if its standardised error converges in distribution to a normal distribution.

n(θ^n−θ)→dN(0,v(θ))\sqrt{n}\left(\hat{\theta}_n-\theta\right)\xrightarrow{d}\mathrm{N}\left(0,v\left(\theta\right)\right)

Let θ^n\hat{\theta}_n be an asymptotically normal sequence of estimators of θ\theta with limiting variance v(θ).v\left(\theta\right). The asymptotic variance of θ^n\hat{\theta}_n is v(θ).v\left(\theta\right). Asymptotic normality has the following properties:

  • an asymptotically normal estimator is consistent;
  • the sampling distribution of an asymptotically normal estimator is approximately N(θ,v(θ)n)\mathrm{N}\left(\theta,\frac{v\left(\theta\right)}{n}\right) for large n,n, which is what the interval estimates and tests of the following notes are built on;
  • the sample mean is asymptotically normal with v=σ2v=\sigma^2 by the central limit theorem;
  • the delta method gives the asymptotic normality of smooth functions of asymptotically normal estimators.

Let PX; θ\mathbb{P}_{X;\,\theta} be a population with pmf or pdf fX; θ(x; θ)f_{X;\,\theta}\left(x;\,\theta\right) and scalar parameter θ\theta in an open parameter space. The regularity conditions are the following conditions on the population:

  • the support of fX; θf_{X;\,\theta} does not depend on θ;\theta;
  • fX; θf_{X;\,\theta} is twice differentiable in θ;\theta;
  • differentiation with respect to θ\theta and integration or summation over xx may be interchanged.

Let PX; θ\mathbb{P}_{X;\,\theta} be a population with pmf or pdf fX; θ(x; θ)f_{X;\,\theta}\left(x;\,\theta\right) satisfying the regularity conditions. The score of an observation xx is the derivative of the log pmf or pdf with respect to the parameter.

s(θ; x):=∂∂θlog⁡efX; θ(x; θ)s\left(\theta;\,x\right):=\frac{\partial}{\partial\theta}\log_e f_{X;\,\theta}\left(x;\,\theta\right)

The score measures how sensitive the log probability of the observation xx is to the parameter. The score has expectation zero under the population, E[s(θ; X)]=0,\mathbb{E}\left[s\left(\theta;\,X\right)\right]=0, since ∫∂∂θfX; θ(x; θ)dx=∂∂θ∫fX; θ(x; θ)dx=0.\int\frac{\partial}{\partial\theta}f_{X;\,\theta}\left(x;\,\theta\right)dx=\frac{\partial}{\partial\theta}\int f_{X;\,\theta}\left(x;\,\theta\right)dx=0.

Let PX; θ\mathbb{P}_{X;\,\theta} be a population satisfying the regularity conditions. The Fisher information of a single observation is the variance of the score.

I(θ):=var[s(θ; X)]=E[s(θ; X)2]=−E[∂2∂θ2log⁡efX; θ(X; θ)]I\left(\theta\right):=\mathrm{var}\left[s\left(\theta;\,X\right)\right]=\mathbb{E}\left[s\left(\theta;\,X\right)^2\right]=-\mathbb{E}\left[\frac{\partial^2}{\partial\theta^2}\log_e f_{X;\,\theta}\left(X;\,\theta\right)\right]

The last equality holds under the regularity conditions. The Fisher information has the following properties:

  • it measures how much an observation reveals about θ:\theta: a large value means that the distribution changes rapidly with θ,\theta, so nearby parameter values are easy to distinguish;
  • since the score of a random sample is the sum of the scores of its iid observations, the Fisher information of a random sample of size nn is n I(θ).n\,I\left(\theta\right).

Let PX; θ‾\mathbb{P}_{X;\,\underline{\theta}} be a population with vector parameter θ‾∈Rk\underline{\theta}\in\mathbb{R}^k satisfying the regularity conditions. The score is the gradient ∇θ‾log⁡efX; θ‾.\nabla_{\underline{\theta}}\log_e f_{X;\,\underline{\theta}}. The Fisher information matrix I(θ‾)∈R k×kI\left(\underline{\theta}\right)\in\mathbb{R}^{\,k\times k} is the covariance matrix of the score, which equals the negative expected Hessian matrix of the log pmf or pdf.

Let PX; θ\mathbb{P}_{X;\,\theta} be a population satisfying the regularity conditions and θ^\hat{\theta} be an unbiased estimator of θ\theta based on a random sample of size n.n. The Cramér–Rao lower bound states that the variance of θ^\hat{\theta} is at least the reciprocal of the Fisher information of the sample.

var[θ^]≥1n I(θ)\mathrm{var}\left[\hat{\theta}\right]\geq\frac{1}{n\,I\left(\theta\right)}

The Cramér–Rao lower bound has the following properties:

  • it follows from the Cauchy–Schwarz inequality applied to θ^−θ\hat{\theta}-\theta and the score of the sample;
  • an estimator θ^\hat{\theta} with bias b(θ)b\left(\theta\right) satisfies var[θ^]≥(1+b′(θ))2n I(θ);\mathrm{var}\left[\hat{\theta}\right]\geq\frac{\left(1+b'\left(\theta\right)\right)^2}{n\,I\left(\theta\right)};
  • an unbiased estimator g^\hat{g} of g(θ)g\left(\theta\right) satisfies var[g^]≥g′(θ)2n I(θ);\mathrm{var}\left[\hat{g}\right]\geq\frac{g'\left(\theta\right)^2}{n\,I\left(\theta\right)};
  • for a vector parameter, the covariance matrix of an unbiased estimator satisfies that cov[θ‾^]−1nI(θ‾)−1\mathrm{cov}\left[\hat{\underline{\theta}}\right]-\frac{1}{n}I\left(\underline{\theta}\right)^{-1} is positive semi-definite;
  • a biased estimator can have smaller variance, and even smaller mean squared error, than the bound allows for unbiased estimators.

Let θ^\hat{\theta} be an unbiased estimator of θ.\theta. The estimator θ^\hat{\theta} is efficient if and only if its variance attains the Cramér–Rao lower bound for every θ∈Θ.\theta\in\Theta. An efficient estimator need not exist, and when it does it is unique.

Let θ^\hat{\theta} be an unbiased estimator of θ.\theta. The efficiency of θ^\hat{\theta} is the ratio of the Cramér–Rao lower bound to its variance.

e(θ^):=1n I(θ) var[θ^]∈(0,1]e\left(\hat{\theta}\right):=\frac{1}{n\,I\left(\theta\right)\,\mathrm{var}\left[\hat{\theta}\right]}\in\left(0,1\right]

An efficient estimator has efficiency one.

Let θ^1\hat{\theta}_1 and θ^2\hat{\theta}_2 be unbiased estimators of θ.\theta. The relative efficiency of θ^1\hat{\theta}_1 with respect to θ^2\hat{\theta}_2 is the ratio of their variances.

var[θ^2]var[θ^1]\frac{\mathrm{var}\left[\hat{\theta}_2\right]}{\mathrm{var}\left[\hat{\theta}_1\right]}

The relative efficiency is the factor by which the sample size using θ^2\hat{\theta}_2 must be increased to match the precision of θ^1.\hat{\theta}_1. For example, for a normal population the sample mean is efficient for μ\mu whereas the sample median has asymptotic variance πσ22,\frac{\pi\sigma^2}{2}, so the relative efficiency of the sample median with respect to the sample mean is 2π≈0.64.\frac{2}{\pi}\approx 0.64.

Let θ^n\hat{\theta}_n be an asymptotically normal sequence of estimators of θ.\theta. The sequence is asymptotically efficient if and only if its asymptotic variance equals 1I(θ).\frac{1}{I\left(\theta\right)}. This is the smallest asymptotic variance that a consistent and asymptotically normal estimator can have at all but a negligible set of parameter values.