The posterior distribution is the complete answer to a Bayesian inference problem, but it is usually summarised by a point estimate, an interval or a decision. These summaries are derived from decision theory, in which the consequences of each choice are assessed under the posterior distribution.

Let Θ\Theta be a parameter space. A set of actions is a set A\mathcal{A} of choices available to the decision maker.

Let Θ\Theta be a parameter space and A\mathcal{A} be a set of actions. A loss function is a function L:Θ×A→[0,∞)L:\Theta\times\mathcal{A}\rightarrow\left[0,\infty\right) where L(θ,a)L\left(\theta,a\right) is the loss incurred by taking action aa when the true parameter value is θ.\theta. The utility of an action is the negative of its loss, so minimising loss and maximising utility are equivalent.

Let LL be a loss function, a∈Aa\in\mathcal{A} be an action and π(θ∣x‾)\pi\left(\theta\mid\underline{x}\right) be a posterior distribution. The posterior expected loss of aa is the expected loss of aa under the posterior distribution.

ρ(a∣x‾):=E[L(θ,a)∣x‾]=∫ΘL(θ,a)π(θ∣x‾)dθ\rho\left(a\mid\underline{x}\right):=\mathbb{E}\left[L\left(\theta,a\right)\mid\underline{x}\right]=\int_{\Theta}L\left(\theta,a\right)\pi\left(\theta\mid\underline{x}\right)d\theta

Let LL be a loss function and π(θ∣x‾)\pi\left(\theta\mid\underline{x}\right) be a posterior distribution. A Bayes action is an action that minimises the posterior expected loss.

a∗(x‾):=arg min⁡a∈Aρ(a∣x‾)a^*\left(\underline{x}\right):=\operatorname*{arg\,min}_{a\in\mathcal{A}}\rho\left(a\mid\underline{x}\right)

Let A\mathcal{A} be a set of actions. A decision rule is a function δ\delta that maps each possible data set x‾\underline{x} to an action δ(x‾)∈A.\delta\left(\underline{x}\right)\in\mathcal{A}.

Let LL be a loss function and δ\delta be a decision rule. The frequentist risk of δ\delta at θ\theta is its expected loss over repeated sampling with θ\theta fixed.

R(θ,δ):=E[L(θ,δ(X‾))∣θ]R\left(\theta,\delta\right):=\mathbb{E}\left[L\left(\theta,\delta\left(\underline{X}\right)\right)\mid\theta\right]

Let LL be a loss function, δ\delta be a decision rule and π(θ)\pi\left(\theta\right) be a prior distribution. The Bayes risk of δ\delta is its frequentist risk averaged over the prior.

r(π,δ):=∫ΘR(θ,δ)π(θ)dθr\left(\pi,\delta\right):=\int_{\Theta}R\left(\theta,\delta\right)\pi\left(\theta\right)d\theta

Let LL be a loss function and π(θ)\pi\left(\theta\right) be a prior distribution. A Bayes rule is a decision rule that minimises the Bayes risk. The decision rule that takes the Bayes action for every data set is a Bayes rule, so acting optimally conditional on the observed data is also optimal on average over data sets.

Let LL be a loss function on Θ×Θ.\Theta\times\Theta. A Bayes estimator of θ\theta is a Bayes action for the decision problem with A=Θ.\mathcal{A}=\Theta. The Bayes estimator depends on the loss function.

The squared error loss is the loss function L(θ,a)=(θ−a)2.L\left(\theta,a\right)=\left(\theta-a\right)^2. The Bayes estimator under squared error loss is the posterior mean.

θ^=E[θ∣x‾]=∫Θθ π(θ∣x‾)dθ\hat{\theta}=\mathbb{E}\left[\theta\mid\underline{x}\right]=\int_{\Theta}\theta\,\pi\left(\theta\mid\underline{x}\right)d\theta

Let π(θ∣x‾)\pi\left(\theta\mid\underline{x}\right) be a posterior distribution. The posterior variance of θ\theta is the variance of the posterior distribution, var[θ∣x‾].\mathrm{var}\left[\theta\mid\underline{x}\right]. The posterior variance is the posterior expected loss of the posterior mean under squared error loss. The posterior standard deviation of θ\theta is var[θ∣x‾],\sqrt{\mathrm{var}\left[\theta\mid\underline{x}\right]}, which is the Bayesian analogue of the standard error.

The absolute error loss is the loss function L(θ,a)=∣θ−a∣.L\left(\theta,a\right)=\lvert\theta-a\rvert. The posterior median of θ\theta is the median of the posterior distribution. The Bayes estimator under absolute error loss is the posterior median.

Let ε>0.\varepsilon>0. The zero-one loss is the loss function L(θ,a)=1{∣θ−a∣>ε}.L\left(\theta,a\right)=\mathbf{1}_{\{\lvert\theta-a\rvert>\varepsilon\}}.

Let π(θ∣x‾)\pi\left(\theta\mid\underline{x}\right) be a posterior distribution with pmf or pdf. The posterior mode or maximum a posteriori (MAP) estimate of θ\theta is the value that maximises the posterior pmf or pdf.

θ^MAP:=arg max⁡θ∈Θπ(θ∣x‾)=arg max⁡θ∈Θ(ℓ(θ∣x‾)+log⁡eπ(θ))\hat{\theta}_{\mathrm{MAP}}:=\operatorname*{arg\,max}_{\theta\in\Theta}\pi\left(\theta\mid\underline{x}\right)=\operatorname*{arg\,max}_{\theta\in\Theta}\left(\ell\left(\theta\mid\underline{x}\right)+\log_e\pi\left(\theta\right)\right)

The Bayes estimator under zero-one loss is the posterior mode in the limit ε→0.\varepsilon\rightarrow 0.

Let ℓ\ell be a log-likelihood function and p(θ)p\left(\theta\right) be a penalty function. A penalised maximum likelihood estimate is a value of θ\theta that maximises ℓ(θ∣x‾)−p(θ).\ell\left(\theta\mid\underline{x}\right)-p\left(\theta\right). The MAP estimate has the following properties:

  • it is the maximum likelihood estimate under a flat prior;
  • it is a penalised maximum likelihood estimate with penalty −log⁡eπ(θ):-\log_e\pi\left(\theta\right): a normal prior centred at zero penalises θ2\theta^2 and a Laplace prior penalises ∣θ∣,\lvert\theta\rvert, which are the penalties of ridge and lasso regularisation.

Point estimates have the following invariance properties under reparametrisation:

  • the posterior median is invariant under monotonic reparametrisation;
  • the posterior mean is invariant only under affine reparametrisation;
  • the posterior mode is invariant under neither, since the change of variables factor moves the maximum of the density.

Let π(θ∣x‾)\pi\left(\theta\mid\underline{x}\right) be a posterior distribution and α∈(0,1).\alpha\in\left(0,1\right). A credible set of credible level 1−α1-\alpha is a set C⊆ΘC\subseteq\Theta that has posterior probability 1−α.1-\alpha.

P(θ∈C∣x‾)=∫Cπ(θ∣x‾)dθ=1−α\mathbb{P}\left(\theta\in C\mid\underline{x}\right)=\int_C\pi\left(\theta\mid\underline{x}\right)d\theta=1-\alpha

A credible interval is a credible set that is an interval. Credible sets have the following properties:

  • a credible interval contains θ\theta with posterior probability 1−α1-\alpha given the observed data, which is the statement that a confidence interval is often mistakenly taken to make;
  • a confidence interval, in contrast, has probability 1−α1-\alpha of containing θ\theta over repeated samples before the data are observed;
  • credible sets of a given level are not unique.

Let π(θ∣x‾)\pi\left(\theta\mid\underline{x}\right) be a posterior distribution with quantile function Fθ∣x‾−1F^{-1}_{\theta\mid\underline{x}} and α∈(0,1).\alpha\in\left(0,1\right). The equal-tailed credible interval of level 1−α1-\alpha is the credible interval that places probability α2\frac{\alpha}{2} in each tail of the posterior distribution.

[Fθ∣x‾−1(α2),Fθ∣x‾−1(1−α2)]\left[F^{-1}_{\theta\mid\underline{x}}\left(\tfrac{\alpha}{2}\right),F^{-1}_{\theta\mid\underline{x}}\left(1-\tfrac{\alpha}{2}\right)\right]

The equal-tailed credible interval transforms correctly under monotonic reparametrisation.

Let π(θ∣x‾)\pi\left(\theta\mid\underline{x}\right) be a posterior distribution with pdf and α∈(0,1).\alpha\in\left(0,1\right). The highest posterior density (HPD) region of level 1−α1-\alpha is the set of parameter values with posterior density above a threshold kk chosen to give the required level.

CHPD:={θ∈Θ:π(θ∣x‾)≥k},P(θ∈CHPD∣x‾)=1−αC_{\mathrm{HPD}}:=\left\{\theta\in\Theta:\pi\left(\theta\mid\underline{x}\right)\geq k\right\},\qquad\mathbb{P}\left(\theta\in C_{\mathrm{HPD}}\mid\underline{x}\right)=1-\alpha

The HPD region has the following properties:

  • it is the smallest set of its level;
  • it is an interval when the posterior is unimodal and may be a union of intervals when the posterior is multimodal;
  • it is not invariant under reparametrisation;
  • it coincides with the equal-tailed credible interval for a symmetric unimodal posterior.

Let z1−α2z_{1-\frac{\alpha}{2}} be the (1−α2)\left(1-\frac{\alpha}{2}\right)-quantile of the standard normal distribution. By the Bernstein–von Mises theorem both the equal-tailed credible interval and the HPD region are approximately θ^±z1−α2(n I(θ^))−12\hat{\theta}\pm z_{1-\frac{\alpha}{2}}\left(n\,I\left(\hat{\theta}\right)\right)^{-\frac{1}{2}} for large n.n.

Let H0:θ∈Θ0H_0:\theta\in\Theta_0 and H1:θ∈Θ1H_1:\theta\in\Theta_1 be hypotheses with Θ0∪Θ1=Θ\Theta_0\cup\Theta_1=\Theta and π(θ∣x‾)\pi\left(\theta\mid\underline{x}\right) be a posterior distribution. The posterior probability of a hypothesis is the posterior probability of its parameter set.

P(H0∣x‾)=∫Θ0π(θ∣x‾)dθ,P(H1∣x‾)=1−P(H0∣x‾)\mathbb{P}\left(H_0\mid\underline{x}\right)=\int_{\Theta_0}\pi\left(\theta\mid\underline{x}\right)d\theta,\qquad\mathbb{P}\left(H_1\mid\underline{x}\right)=1-\mathbb{P}\left(H_0\mid\underline{x}\right)

Let L0L_0 be the loss for wrongly rejecting H0H_0 and L1L_1 be the loss for wrongly accepting H0.H_0. The Bayes action is to reject H0H_0 if and only if the posterior odds exceed the ratio of the losses.

P(H1∣x‾)P(H0∣x‾)>L0L1\frac{\mathbb{P}\left(H_1\mid\underline{x}\right)}{\mathbb{P}\left(H_0\mid\underline{x}\right)}>\frac{L_0}{L_1}

Bayesian hypothesis testing has the following properties:

  • the hypotheses are treated symmetrically and H0H_0 can be accepted, in contrast to a frequentist test;
  • the posterior probability of a composite hypothesis is a direct measure of its plausibility rather than a pp-value;
  • a simple hypothesis H0:θ=θ0H_0:\theta=\theta_0 has posterior probability zero under a continuous prior, so testing it requires a prior that places positive probability on θ0,\theta_0, which is treated as a comparison of two models.

Let X~\tilde{X} be a future observation with posterior predictive distribution f(x~∣x‾).f\left(\tilde{x}\mid\underline{x}\right). The posterior predictive mean of X~\tilde{X} is the mean of the posterior predictive distribution.

E[X~∣x‾]=E[E[X~∣θ]∣x‾]\mathbb{E}\left[\tilde{X}\mid\underline{x}\right]=\mathbb{E}\left[\mathbb{E}\left[\tilde{X}\mid\theta\right]\mid\underline{x}\right]

The posterior predictive mean is the Bayes prediction of X~\tilde{X} under squared error loss.

Let X~\tilde{X} be a future observation with posterior predictive distribution and α∈(0,1).\alpha\in\left(0,1\right). A predictive interval of level 1−α1-\alpha is an interval with posterior predictive probability 1−α.1-\alpha. By the law of total variance

var[X~∣x‾]=E[var[X~∣θ]∣x‾]+var[E[X~∣θ]∣x‾],\mathrm{var}\left[\tilde{X}\mid\underline{x}\right]=\mathbb{E}\left[\mathrm{var}\left[\tilde{X}\mid\theta\right]\mid\underline{x}\right]+\mathrm{var}\left[\mathbb{E}\left[\tilde{X}\mid\theta\right]\mid\underline{x}\right],

which adds the variability of the observation to the uncertainty about the parameter, so a predictive interval is wider than a credible interval for E[X~∣θ].\mathbb{E}\left[\tilde{X}\mid\theta\right].