Point Estimation
Let be a random sample of size from the population and The -th sample moment of is the statistic given by the average of the -th powers of the observations.
The -th sample moment is an unbiased estimator of the -th population moment
Let be a random sample from the population where contains parameters that can be expressed in terms of the first moments of the population. The method of moments is the estimation method that equates the first sample moments with the corresponding population moments and solves the resulting system of equations for The method of moments estimator of is the solution of this system.
Let be an observed random sample of size from the population with joint pmf or pdf The likelihood function of given is the joint pmf or pdf of the observed random sample viewed as a function of
Let be an observed random sample of iid observations from the population with pmf or pdf The likelihood function is the product of the pmfs or pdfs of the observations.
If the observations are independent but not identically distributed, with the -th observation having pmf or pdf then the likelihood function is
Let be an observed random sample with likelihood function The log-likelihood function of given is the natural logarithm of the likelihood function.
The likelihood function often contains exponential terms and products, which the log-likelihood function turns into sums.
Let be an observed random sample with likelihood function The maximum likelihood estimate (MLE) of is the value of that maximises the likelihood function for the given
Because the logarithm is strictly increasing and the likelihood is positive, the maximum likelihood estimate also maximises the log-likelihood function, which is often more practical to maximise.
Let be a random sample from the population The maximum likelihood estimator (also MLE) of is the estimator whose value at each observed random sample is the maximum likelihood estimate.
Let be the maximum likelihood estimator of under the regularity conditions and some further smoothness conditions. The maximum likelihood estimator has the following asymptotic properties, which make it the default estimator in parametric models:
- it is consistent,
- it is asymptotically normal with asymptotic variance equal to the reciprocal of the Fisher information, so it is asymptotically efficient;
- it is invariant: if is the maximum likelihood estimate of then is the maximum likelihood estimate of for any function
The maximum likelihood estimator is in general biased for finite For example, the maximum likelihood estimate of a normal variance is rather than the unbiased sample variance.
Let be an observed random sample with log-likelihood function and maximum likelihood estimate The observed Fisher information is the negative second derivative of the log-likelihood function at the maximum likelihood estimate.
The observed Fisher information is a consistent estimate of and gives the approximate standard error