Ensemble Learning and Evidence Maximization
David J C MacKay
Ensemble learning by variational free energy minimization is a
tool introduced to neural networks by Hinton and van Camp in
which learning is described in terms of the optimization of an
ensemble of parameter vectors. The optimized ensemble is an
approximation to the posterior probability distribution of the
parameters. This tool has now been applied to a variety of
statistical inference problems.
In this paper I study a linear regression model with both
parameters and hyperparameters.
I demonstrate that the evidence approximation for the
optimization of regularization constants can be derived in detail
from a free energy minimization viewpoint.