Bayesian Methods for Neural Networks: Theory and Applications

David J C MacKay

Neural networks are parameterized non-linear models used for empirical regression and classification modelling. Their flexibility makes them able to discover more general relationships in data than traditional statistical models.

Bayesian probability theory provides a unifying framework for data modeling which offers several benefits. First, the {\bf overfitting problem} can be solved by using Bayesian methods to control model complexity. Bayesian model comparison can be used for example to optimize weight decay rates, and to infer automatically which are the relevant input variables for a problem. % Second, probabilistic modelling handles uncertainty in a natural manner. There is a unique prescription, {\bf marginalization}, for incorporating uncertainty about parameters into predictions; this procedure yields better predictions. Third, we can define more sophisticated probabilistic models which are able to extract more information from data.

I will discuss the practical application of these methods to a problem involving the prediction of a series of building energy loads from a series of environmental variables.

I will also discuss the application of Bayesian methods to neural network classifiers.

Finally I will describe the implementation of a probabilistic regression model in BUGS. BUGS is a program that carries out Bayesian inference on statistical problems using a simulation technique known as Gibbs sampling.


How to cite this document.
It's not been published in a journal. You may find that this published article is sufficiently similar that you can cite it:
 KEY            ="MacKay",
 AUTHOR         ="D. J. C.  MacKay",
 TITLE          ="Probable Networks and Plausible Predictions --- A
		  Review of Practical {B}ayesian Methods for Supervised
		  Neural Networks",
 journal ="Network: Computation in Neural Systems",
 volume = 6,
 YEAR           =1995,
 pages = "469-505",
 ANNOTE ="Date submitted: 1994; Date accepted: 1994; Collaborating
		  institutes: none"}

If not, cite the URL of this abstract page.

David MacKay's: home page, publications. bibtex file.