We empirically show that Bayesian inference can be inconsistent under misspecification in simple linear regression problems, both in a model averaging/selection and in a Bayesian ridge regression setting. We use the standard linear model, which assumes homoskedasticity, whereas the data are heteroskedastic, and observe that the posterior puts its mass on ever more high-dimensional models as the sample size increases. To remedy the problem, we equip the likelihood in Bayes' theorem with an exponent called the learning rate, and we propose the {\em Safe Bayesian\/} method to learn the learning rate from the data. SafeBayes tends to select small learning rates as soon the standard posterior is not `cumulatively concentrated', and its results on our data are quite encouraging.

Cornell University Library
arXiv.org e-Print archive
Safe Statistics
Algorithms and Complexity

Grünwald, P.D, & van Ommen, M. (2014). Inconsistency of Bayesian inference for misspecified linear models, and a proposal for repairing it. arXiv.org e-Print archive. Cornell University Library.