Estimators
Let be a random sample from the population An estimator of is a statistic whose value is intended to approximate
Let be an estimator of and be an observed random sample. An estimate of is the realised value of the estimator.
Let be an estimator of with finite expectation. The bias of is the difference between the expectation of its sampling distribution and the true value of the parameter.
Let be an estimator of The estimator is unbiased if and only if the expectation of its sampling distribution equals the parameter.
Let be an estimator of with finite variance. The standard error of is the standard deviation of its sampling distribution.
Let be an estimator of with finite variance. The mean squared error of is the expected squared difference between the estimator and the parameter, which is the sum of the squared bias and the variance.
The mean squared error captures the trade-off between the bias and the variance of an estimator. The bias, standard error and mean squared error are defined in terms of the moments of the sampling distribution of the estimator, and since the random sample is drawn from the population they can also be calculated as functions of the moments of
Let be an estimator of computed from a random sample of size for each The sequence of estimators is consistent for if and only if it converges in probability to the true parameter value, whatever that value is.
Let be a sequence of estimators of The sequence is strongly consistent for if and only if it converges almost surely to the true parameter value.
Let be a sequence of estimators of The sequence is asymptotically unbiased if and only if its bias tends to zero as Consistency has the following properties:
- consistency states that with enough data the estimator is close to the truth with high probability, and is the minimal requirement of a sensible estimator;
- by Markov’s inequality applied to an estimator whose mean squared error tends to zero is consistent, since
- an estimator is therefore consistent if it is asymptotically unbiased and its variance tends to zero;
- the sample mean is consistent for the population mean by the weak law of large numbers, and strongly consistent by the strong law, and the sample variance is consistent for the population variance;
- consistency and unbiasedness are distinct: is an unbiased but inconsistent estimator of the population mean, whereas is a biased but consistent estimator of the population variance;
- by the continuous mapping theorem, if is consistent for and is continuous then is consistent for
Let be a sequence of estimators of The sequence is asymptotically normal if and only if its standardised error converges in distribution to a normal distribution.
Let be an asymptotically normal sequence of estimators of with limiting variance The asymptotic variance of is Asymptotic normality has the following properties:
- an asymptotically normal estimator is consistent;
- the sampling distribution of an asymptotically normal estimator is approximately for large which is what the interval estimates and tests of the following notes are built on;
- the sample mean is asymptotically normal with by the central limit theorem;
- the delta method gives the asymptotic normality of smooth functions of asymptotically normal estimators.
Let be a population with pmf or pdf and scalar parameter in an open parameter space. The regularity conditions are the following conditions on the population:
- the support of does not depend on
- is twice differentiable in
- differentiation with respect to and integration or summation over may be interchanged.
Let be a population with pmf or pdf satisfying the regularity conditions. The score of an observation is the derivative of the log pmf or pdf with respect to the parameter.
The score measures how sensitive the log probability of the observation is to the parameter. The score has expectation zero under the population, since
Let be a population satisfying the regularity conditions. The Fisher information of a single observation is the variance of the score.
The last equality holds under the regularity conditions. The Fisher information has the following properties:
- it measures how much an observation reveals about a large value means that the distribution changes rapidly with so nearby parameter values are easy to distinguish;
- since the score of a random sample is the sum of the scores of its iid observations, the Fisher information of a random sample of size is
Let be a population with vector parameter satisfying the regularity conditions. The score is the gradient The Fisher information matrix is the covariance matrix of the score, which equals the negative expected Hessian matrix of the log pmf or pdf.
Let be a population satisfying the regularity conditions and be an unbiased estimator of based on a random sample of size The Cramér–Rao lower bound states that the variance of is at least the reciprocal of the Fisher information of the sample.
The Cramér–Rao lower bound has the following properties:
- it follows from the Cauchy–Schwarz inequality applied to and the score of the sample;
- an estimator with bias satisfies
- an unbiased estimator of satisfies
- for a vector parameter, the covariance matrix of an unbiased estimator satisfies that is positive semi-definite;
- a biased estimator can have smaller variance, and even smaller mean squared error, than the bound allows for unbiased estimators.
Let be an unbiased estimator of The estimator is efficient if and only if its variance attains the Cramér–Rao lower bound for every An efficient estimator need not exist, and when it does it is unique.
Let be an unbiased estimator of The efficiency of is the ratio of the Cramér–Rao lower bound to its variance.
An efficient estimator has efficiency one.
Let and be unbiased estimators of The relative efficiency of with respect to is the ratio of their variances.
The relative efficiency is the factor by which the sample size using must be increased to match the precision of For example, for a normal population the sample mean is efficient for whereas the sample median has asymptotic variance so the relative efficiency of the sample median with respect to the sample mean is
Let be an asymptotically normal sequence of estimators of The sequence is asymptotically efficient if and only if its asymptotic variance equals This is the smallest asymptotic variance that a consistent and asymptotically normal estimator can have at all but a negligible set of parameter values.