Compactness of variational approximations

David J C MacKay, Richard Turner, and Maneesh Sahani

Let's approximate a complicated distribution P(x) by a simpler distribution Q(x), possibly a separable distribution. It's often the case that variational free energy minimization (also known as mean field) leads to an approximating distribution Q that is `more compact' than the true distribution.
Is there in fact a theorem that we could prove along the lines of `optimized $Q$ is always more compact'?
We show, with a counterexample, that the folk theorem about variational approximations being `more compact' is not always true.

postscript (Cambridge UK).

postscript (Canada mirror).

pdf (Cambridge UK).

pdf (Canada mirror).

All postscript files are compressed with gzip - see this page for advice about gzip, if needed.


related publications.
David MacKay's: home page, publications. bibtex file.
Canadian mirrors: home page, publications. bibtex file.