Let X‾\underline{X} be a random sample of size nn from the population PX; θ\mathbb{P}_{X;\,\theta} and k≥1.k\geq 1. The kk-th sample moment of X‾\underline{X} is the statistic given by the average of the kk-th powers of the observations.

mk:=1n∑i=1nXikm_k:=\frac{1}{n}\sum_{i=1}^n X_i^k

The kk-th sample moment is an unbiased estimator of the kk-th population moment E[Xk].\mathbb{E}\left[X^k\right].

Let X‾\underline{X} be a random sample from the population PX; θ,\mathbb{P}_{X;\,\theta}, where θ\theta contains kk parameters that can be expressed in terms of the first kk moments of the population. The method of moments is the estimation method that equates the first kk sample moments with the corresponding population moments and solves the resulting system of equations for θ.\theta. The method of moments estimator of θ\theta is the solution of this system.

Let x‾\underline{x} be an observed random sample of size nn from the population PX; θ\mathbb{P}_{X;\,\theta} with joint pmf or pdf fX‾; θ(x‾; θ).f_{\underline{X};\,\theta}\left(\underline{x};\,\theta\right). The likelihood function of θ\theta given x‾\underline{x} is the joint pmf or pdf of the observed random sample viewed as a function of θ.\theta.

L(θ∣x‾):=fX‾; θ(x‾; θ)\mathcal{L}\left(\theta\mid\underline{x}\right):=f_{\underline{X};\,\theta}\left(\underline{x};\,\theta\right)

Let x‾\underline{x} be an observed random sample of iid observations from the population with pmf or pdf fX; θ(x; θ).f_{X;\,\theta}\left(x;\,\theta\right). The likelihood function is the product of the pmfs or pdfs of the observations.

L(θ∣x‾)=∏i=1nfX; θ(xi; θ)\mathcal{L}\left(\theta\mid\underline{x}\right)=\prod_{i=1}^n f_{X;\,\theta}\left(x_i;\,\theta\right)

If the observations are independent but not identically distributed, with the ii-th observation having pmf or pdf fX; θ,i,f_{X;\,\theta,i}, then the likelihood function is L(θ∣x‾)=∏i=1nfX; θ,i(xi; θ).\mathcal{L}\left(\theta\mid\underline{x}\right)=\prod_{i=1}^n f_{X;\,\theta,i}\left(x_i;\,\theta\right).

Let x‾\underline{x} be an observed random sample with likelihood function L(θ∣x‾).\mathcal{L}\left(\theta\mid\underline{x}\right). The log-likelihood function of θ\theta given x‾\underline{x} is the natural logarithm of the likelihood function.

ℓ(θ∣x‾):=log⁡eL(θ∣x‾)\ell\left(\theta\mid\underline{x}\right):=\log_e\mathcal{L}\left(\theta\mid\underline{x}\right)

The likelihood function often contains exponential terms and products, which the log-likelihood function turns into sums.

Let x‾\underline{x} be an observed random sample with likelihood function L(θ∣x‾).\mathcal{L}\left(\theta\mid\underline{x}\right). The maximum likelihood estimate (MLE) of θ\theta is the value of θ\theta that maximises the likelihood function for the given x‾.\underline{x}.

θ^(x‾):=arg max⁡θ∈Θ L(θ∣x‾)\hat{\theta}\left(\underline{x}\right):=\operatorname*{arg\,max}_{\theta\in\Theta}\,\mathcal{L}\left(\theta\mid\underline{x}\right)

Because the logarithm is strictly increasing and the likelihood is positive, the maximum likelihood estimate also maximises the log-likelihood function, which is often more practical to maximise.

θ^(x‾)=arg max⁡θ∈Θ ℓ(θ∣x‾)\hat{\theta}\left(\underline{x}\right)=\operatorname*{arg\,max}_{\theta\in\Theta}\,\ell\left(\theta\mid\underline{x}\right)

Let X‾\underline{X} be a random sample from the population PX; θ.\mathbb{P}_{X;\,\theta}. The maximum likelihood estimator (also MLE) of θ\theta is the estimator θ^(X‾)\hat{\theta}\left(\underline{X}\right) whose value at each observed random sample is the maximum likelihood estimate.

Let θ^(X‾)\hat{\theta}\left(\underline{X}\right) be the maximum likelihood estimator of θ\theta under the regularity conditions and some further smoothness conditions. The maximum likelihood estimator has the following asymptotic properties, which make it the default estimator in parametric models:

  • it is consistent, θ^(X‾)→pθ;\hat{\theta}\left(\underline{X}\right)\xrightarrow{p}\theta;
  • it is asymptotically normal with asymptotic variance equal to the reciprocal of the Fisher information, n(θ^(X‾)−θ)→dN(0,1I(θ)),\sqrt{n}\left(\hat{\theta}\left(\underline{X}\right)-\theta\right)\xrightarrow{d}\mathrm{N}\left(0,\frac{1}{I\left(\theta\right)}\right), so it is asymptotically efficient;
  • it is invariant: if θ^\hat{\theta} is the maximum likelihood estimate of θ\theta then g(θ^)g\left(\hat{\theta}\right) is the maximum likelihood estimate of g(θ)g\left(\theta\right) for any function g.g.

The maximum likelihood estimator is in general biased for finite n.n. For example, the maximum likelihood estimate of a normal variance is 1n∑i=1n(xi−x‾)2\frac{1}{n}\sum_{i=1}^n\left(x_i-\overline{x}\right)^2 rather than the unbiased sample variance.

Let x‾\underline{x} be an observed random sample with log-likelihood function ℓ\ell and maximum likelihood estimate θ^.\hat{\theta}. The observed Fisher information is the negative second derivative of the log-likelihood function at the maximum likelihood estimate.

−ℓ′′(θ^∣x‾)-\ell''\left(\hat{\theta}\mid\underline{x}\right)

The observed Fisher information is a consistent estimate of n I(θ),n\,I\left(\theta\right), and gives the approximate standard error se(θ^)≈(−ℓ′′(θ^∣x‾))−12.se\left(\hat{\theta}\right)\approx\left(-\ell''\left(\hat{\theta}\mid\underline{x}\right)\right)^{-\frac{1}{2}}.