If a parameter θ∈Θ\theta\in\Theta can be expressed in terms of the moments of the population PX; θ\mathbb{P}_{X;\,\theta} then θ\theta can be estimated by equating the population moments with their corresponding sample moments using the method of moments:

  1. Given a random sample X‾\underline{X} distributed according to the population PX; θ\mathbb{P}_{X;\,\theta} and assuming θ\theta contains kk parameters to be estimated, the first kk sample moments are calculated.
  2. The sample moments are equated with their corresponding population moments.
  3. The system of equations is solved to find the unknown θ.\theta.

Given an observed random sample x‾\underline{x} of size nn from a population PX; θ\mathbb{P}_{X;\,\theta} with pdf fX; θ(x; θ),f_{X;\,\theta}\left(x;\,\theta\right), the likelihood function L(θ∣x‾)\mathcal{L}\left(\theta\mid\underline{x}\right) is the joint pmf or pdf of x‾\underline{x} expressed in terms of θ:\theta:

L(θ∣x‾)=fX‾; θ(x‾; θ)\mathcal{L}\left(\theta\mid\underline{x}\right)=f_{\underline{X};\,\theta}\left(\underline{x};\,\theta\right)

Given that X‾\underline{X} contains iid random variables then the likelihood can be expressed as:

L(θ∣x‾)=∏i=1nfX; θ(xi; θ)\mathcal{L}\left(\theta\mid\underline{x}\right)=\prod_{i=1}^n f_{X;\,\theta}\left(x_i;\,\theta\right)

In the case where X‾\underline{X} contains independent but not identically distributed random variables, the likelihood is:

L(θ∣x‾)=∏i=1nfX; θ,i(xi; θ)\mathcal{L}\left(\theta\mid\underline{x}\right)=\prod_{i=1}^n f_{X;\,\theta,i}\left(x_i;\,\theta\right)

The maximum likelihood estimate (MLE) θ^(x‾)\hat{\theta}\left(\underline{x}\right) is the value of θ\theta that maximises the likelihood for a given x‾:\underline{x}:

θ^(x‾)=arg max⁡θ∈Θ L(θ∣x‾)\hat{\theta}\left(\underline{x}\right)=\operatorname*{arg\,max}_{\theta\in\Theta}\,\mathcal{L}\left(\theta\mid\underline{x}\right)

The likelihood function often contains exponential terms. The log-likelihood function is defined as:

ℓ(θ∣x‾)=log⁡eL(θ∣x‾)\ell\left(\theta\mid\underline{x}\right)=\log_e\mathcal{L}\left(\theta\mid\underline{x}\right)

Because the logarithm is a strictly increasing function and the likelihood is positive then θ^(x‾)\hat{\theta}\left(\underline{x}\right) also maximises the log-likelihood ℓ(θ∣x‾)\ell\left(\theta\mid\underline{x}\right) and is often more practical to maximise:

θ^(x‾)=arg max⁡θ∈Θ ℓ(θ∣x‾)\hat{\theta}\left(\underline{x}\right)=\operatorname*{arg\,max}_{\theta\in\Theta}\,\ell\left(\theta\mid\underline{x}\right)

The maximum likelihood estimator (also MLE) is the estimator θ^(X‾).\hat{\theta}\left(\underline{X}\right).

Under the regularity conditions and some further smoothness, the maximum likelihood estimator has the following asymptotic properties, which make it the default estimator in parametric models:

  • it is consistent, θ^(X‾)→pθ;\hat{\theta}\left(\underline{X}\right)\xrightarrow{p}\theta;
  • it is asymptotically normal with asymptotic variance equal to the reciprocal of the Fisher information, n(θ^(X‾)−θ)→dN(0,1I(θ)),\sqrt{n}\left(\hat{\theta}\left(\underline{X}\right)-\theta\right)\xrightarrow{d}\mathrm{N}\left(0,\frac{1}{I\left(\theta\right)}\right), so it is asymptotically efficient;
  • it is invariant: if θ^\hat{\theta} is the maximum likelihood estimate of θ\theta then g(θ^)g\left(\hat{\theta}\right) is the maximum likelihood estimate of g(θ)g\left(\theta\right) for any function g.g.

The maximum likelihood estimator is in general biased for finite n;n; for example, the maximum likelihood estimate of a normal variance is 1n∑i=1n(xi−x‾)2\frac{1}{n}\sum_{i=1}^n\left(x_i-\overline{x}\right)^2 rather than the unbiased sample variance. The observed Fisher information −ℓ′′(θ^∣x‾)-\ell''\left(\hat{\theta}\mid\underline{x}\right) is a consistent estimate of n I(θ)n\,I\left(\theta\right) and gives the approximate standard error se(θ^)≈(−ℓ′′(θ^∣x‾))−12.se\left(\hat{\theta}\right)\approx\left(-\ell''\left(\hat{\theta}\mid\underline{x}\right)\right)^{-\frac{1}{2}}.