Statistical Theory for Expectation Propagation and Variational Message Passing on Factor Graphs

Lead CI: Matt Wand

Expectation propagation (EP) and variational message passing (VMP) are two protocols for performing approximate inference in Bayesian hierarchical models. They have the advantage of scaling up well to large models and sample sizes (a.k.a. `big models', `big data'), as well as high velocity data (a.k.a. `fast data', `streaming data'). These two so-called inference engines are the basis on which the software framework Infer.NET (http://research.microsoft.com/infernet).

The factor graph representation of EP and VMP is not well known, especially in Statistics, and may well be restricted to some mid-2000s in-house technical reports by Infer.NET head developers T. Minka and J.M. Winn. However, it provides an attractive means by which a large array of models, particularly those involving exponential family distributions, can be accommodated.

The statistical accuracy of VMP and, especially, EP has received very little attention. The folklore is that EP, whilst harder to implement, achieves a `higher order' level of accuracy compared with VMP. Proposer Wand had a brief conversation with M.I. Jordan (University of California at Berkeley, USA, and ACEMS board member) to this effect in October 2011. D.M. Titteringon (University of Glasgow) has written a small amount on this topic, focussing on a very simple finite mixture model. An in-progress paper - hopefully to be submitted later in 2014 - by P.G. Hall, A. Huang, J.T. Ormerod and M.P. Wand (Note: Hall and Wand are ACEMS chief investigators and Ormerod is an ACEMS associate investigator) derived path-breaking statistical theory for EP in a specific case.

In early 2014, proposer Wand and PhD student, S.I. Kim, are preparing a paper that provides details on the factor graph approach version of EP and VMP for a specific simple example, where the data are a univariate random sample and the parameters are scalar. Subsequent work will investigate vector parameter extensions such as regression.

Chief investigators Hall and Wand have a 2010s track record of research approximately of this type with two 2011 papers, one in `Statistica Sinica' and one in `The Annals of Statistics' on the statistical properties of Gaussian variational approximation - whichis loosely related to VMP.