Model Choice

Let $M_1,\ldots,M_K$ be candidate models for the data $\underline{x},$ where model $M_k$ has parameter $\theta_k\in\Theta_k,$ likelihood $f\left(\underline{x}\mid\theta_k,M_k\right)$ and prior $\pi\left(\theta_k\mid M_k\right).$ Model uncertainty is uncertainty about which model generated the data, and Bayesian inference treats it like any other uncertainty by placing a prior distribution $\mathbb{P}\left(M_k\right)$ over the models, with $\sum_{k=1}^K\mathbb{P}\left(M_k\right)=1.$ The marginal likelihood of a model is the probability it assigns to the observed data after integrating out its parameters,

$$f\left(\underline{x}\mid M_k\right)=\int_{\Theta_k}f\left(\underline{x}\mid\theta_k,M_k\right)\pi\left(\theta_k\mid M_k\right)d\theta_k,$$

and the posterior model probability follows from Bayes’ theorem.

$$\mathbb{P}\left(M_k\mid\underline{x}\right)=\frac{f\left(\underline{x}\mid M_k\right)\mathbb{P}\left(M_k\right)}{\sum_{j=1}^K f\left(\underline{x}\mid M_j\right)\mathbb{P}\left(M_j\right)}$$

The Bayes factor in favour of model $M_1$ over model $M_2$ is the ratio of their marginal likelihoods,

$$B_{12}:=\frac{f\left(\underline{x}\mid M_1\right)}{f\left(\underline{x}\mid M_2\right)}=\frac{\mathbb{P}\left(M_1\mid\underline{x}\right)/\mathbb{P}\left(M_2\mid\underline{x}\right)}{\mathbb{P}\left(M_1\right)/\mathbb{P}\left(M_2\right)},$$

which is the factor by which the data update the prior odds of the two models into the posterior odds and so measures the evidence the data provide for one model over the other independently of the prior model probabilities. A Bayes factor of $1$ is no evidence, and by convention values of $B_{12}$ above about $3,$ $10$ and $100$ are read as positive, strong and decisive evidence for $M_1.$ For two simple hypotheses the Bayes factor is the likelihood ratio, and a simple null hypothesis $H_0:\theta=\theta_0$ is tested against $H_1:\theta\neq\theta_0$ by comparing the model with $\theta$ fixed at $\theta_0$ to the model with a prior on $\theta.$

The marginal likelihood is the prior predictive density of the observed data, so it rewards a model for predicting the data well before seeing them. A model with more parameters or a more diffuse prior spreads its prior predictive distribution over more possible data sets and so assigns lower probability to any particular one unless the extra flexibility is needed, so the Bayes factor automatically penalises complexity, a property known as Occam’s razor. This penalty makes the Bayes factor sensitive to the prior on the parameters: as the prior on $\theta$ under $H_1$ becomes more diffuse the marginal likelihood of $H_1$ tends to zero and the Bayes factor favours $H_0$ whatever the data, even when a frequentist test rejects $H_0$ decisively, which is the Jeffreys–Lindley paradox. Improper priors cannot be used for the parameters that differ between models, since the marginal likelihood is then defined only up to an arbitrary constant. Marginal likelihoods are rarely available in closed form and are estimated by importance sampling or bridge sampling, or approximated using the Laplace approximation, which as $n\rightarrow\infty$ with $d_k$ parameters gives

$$\log_e f\left(\underline{x}\mid M_k\right)\approx\ell\left(\hat{\theta}_k\mid\underline{x}\right)-\frac{d_k}{2}\log_e n.$$

The Bayesian information criterion (BIC) is minus twice this approximation,

$$\mathrm{BIC}_k:=-2\,\ell\left(\hat{\theta}_k\mid\underline{x}\right)+d_k\log_e n,$$

so that $\frac{1}{2}\left(\mathrm{BIC}_2-\mathrm{BIC}_1\right)\approx\log_e B_{12}$ and the model with the smallest BIC has approximately the largest marginal likelihood. The BIC depends on the prior only through terms that do not grow with $n$ and is consistent in the sense of selecting the true model with probability tending to one when it is among the candidates.

Bayesian model averaging (BMA) makes inferences and predictions by averaging over the models with their posterior probabilities as weights rather than conditioning on a single chosen model. The posterior predictive distribution of a future observation and the posterior distribution of a quantity $\Delta$ that has the same meaning in every model, such as a prediction or an effect size, are

$$f\left(\tilde{x}\mid\underline{x}\right)=\sum_{k=1}^K f\left(\tilde{x}\mid\underline{x},M_k\right)\mathbb{P}\left(M_k\mid\underline{x}\right)$$ $$\pi\left(\Delta\mid\underline{x}\right)=\sum_{k=1}^K\pi\left(\Delta\mid\underline{x},M_k\right)\mathbb{P}\left(M_k\mid\underline{x}\right).$$

Model averaging propagates model uncertainty into the final inference, so its intervals are wider and better calibrated than those from the single best model, and its predictions have lower expected loss under the logarithmic score than those of any single model when the true model is among the candidates.

Model selection chooses a single model, which is the Bayes action under the zero-one loss that charges one unit for choosing any model other than the true one: the model with the largest posterior probability, or equivalently the largest Bayes factor against each alternative when the prior model probabilities are equal. When no candidate model is believed to be true, models are instead compared by how well they predict new data. The log predictive density of a new observation under a model is $\log_e f\left(\tilde{x}\mid\underline{x},M_k\right),$ and the expected log predictive density is its expectation over the true distribution of $\tilde{X},$ which is estimated without new data by leave-one-out cross-validation,

$$\mathrm{elpd}_{\mathrm{loo}}:=\sum_{i=1}^n\log_e f\left(x_i\mid\underline{x}_{-i},M_k\right),$$

where $\underline{x}_{-i}$ is the data with the $i$-th observation removed, or approximated by an information criterion. The deviance of a model at parameter value $\theta$ is $D\left(\theta\right):=-2\log_e f\left(\underline{x}\mid\theta\right),$ and the deviance information criterion (DIC) is the posterior mean deviance plus a penalty equal to the effective number of parameters $p_D,$ the difference between the posterior mean deviance and the deviance at the posterior mean,

$$\mathrm{DIC}:=\overline{D}+p_D,\qquad\overline{D}:=\mathbb{E}\left[D\left(\theta\right)\mid\underline{x}\right]$$ $$p_D:=\overline{D}-D\left(\mathbb{E}\left[\theta\mid\underline{x}\right]\right)$$

which is computed directly from posterior draws. The widely applicable information criterion (WAIC) is a more stable alternative that uses the pointwise posterior predictive densities,

$$\mathrm{WAIC}:=-2\left(\sum_{i=1}^n\log_e\mathbb{E}\left[f\left(x_i\mid\theta\right)\mid\underline{x}\right]-\sum_{i=1}^n\mathrm{var}\left[\log_e f\left(x_i\mid\theta\right)\mid\underline{x}\right]\right),$$

in which the first sum is the log pointwise predictive density and the second is the effective number of parameters, and it asymptotically equals $-2\,\mathrm{elpd}_{\mathrm{loo}}.$ Lower values of DIC and WAIC indicate better expected predictive performance. Unlike the Bayes factor these criteria are insensitive to diffuse priors and do not require the true model to be among the candidates, but they do not deliver posterior probabilities of the models.

Model comparison only ranks the candidates, so a model should also be checked against the data on its own. Let $\underline{x}^{\mathrm{rep}}$ be a replicated data set, a hypothetical new data set of the same size and structure as $\underline{x}$ drawn from the posterior predictive distribution,

$$f\left(\underline{x}^{\mathrm{rep}}\mid\underline{x}\right)=\int_{\Theta}f\left(\underline{x}^{\mathrm{rep}}\mid\theta\right)\pi\left(\theta\mid\underline{x}\right)d\theta,$$

which is simulated by drawing $\theta^{(s)}$ from the posterior and then $\underline{x}^{\mathrm{rep}(s)}$ from the population distribution at $\theta^{(s)}.$ Posterior predictive checking compares the observed data with the replicated data sets: if the model is adequate the observed data should look like a typical draw from the posterior predictive distribution. A test quantity $T\left(\underline{x},\theta\right)$ is a function of the data, and possibly the parameter, chosen to capture a feature of the data that the model should reproduce, such as the proportion of zeros, the maximum, the autocorrelation or a goodness-of-fit statistic. The posterior predictive $p$-value is the posterior probability that the test quantity of replicated data is at least as extreme as that of the observed data,

$$p_B:=\mathbb{P}\left(T\left(\underline{x}^{\mathrm{rep}},\theta\right)\geq T\left(\underline{x},\theta\right)\mid\underline{x}\right)\approx\frac{1}{S}\sum_{s=1}^S\mathbf{1}_{\left\{T\left(\underline{x}^{\mathrm{rep}(s)},\theta^{(s)}\right)\geq T\left(\underline{x},\theta^{(s)}\right)\right\}},$$

and values near $0$ or $1$ indicate that the model does not reproduce that feature of the data. The posterior predictive $p$-value is not a frequentist $p$-value and is not uniformly distributed under the model, since the data are used both to fit the model and to assess it, which makes it conservative; it is a measure of discrepancy rather than a formal test, and graphical comparisons of the observed data with replicated data sets are often more informative than a single number. A prior predictive check draws replicated data from the prior predictive distribution instead and shows whether the prior and likelihood together produce plausible data before any are observed, which is how prior choices are calibrated. A sensitivity analysis repeats the analysis under alternative priors and model specifications to assess how much the conclusions depend on choices that the data cannot settle.