[Lisa_seminaires] Nov 11th, 10:30 - UdeM-McGill-MITACS seminar by Tijmen Tieleman

Pascal Lamblin lamblinp at iro.umontreal.ca
Ven 7 Nov 17:22:12 EST 2008


Hello everyone, and sorry for the late notice,

Next MITACS seminar of will be on Tuesday, November 11th, 10:30 at UdeM
(room AA-3195). WARNING: Notice the unusual time and day!

Title:    Using fast weights to improve Persistent Contrastive Divergence
Speaker:  Tijmen Tieleman 
Date:     Tuesday, November 11th, 10:30 (a.m.)
Place:    Room 3195 (pav. André-Aisenstadt, UdeM)

Abstract:
The most commonly used learning algorithm for restricted Boltzmann
machines is contrastive divergence which starts a Markov chain at a data
point and runs the chain for only a few iterations to get a cheap, low
variance estimate of the sufficient statistics under the model. Tieleman
(2008) showed that better learning can be achieved by estimating the
model's statistics using a small set of persistent "fantasy particles"
that are not reinitialized to data points after each weight update. With
sufficiently small weight updates, the fantasy particles represent the
equilibrium distribution accurately but to explain why the method works
with much larger weight updates it is necessary to consider the
interaction between the weight updates and the Markov chain. We show
that the weight updates force the Markov chain to mix fast, and using
this insight we develop an even faster mixing chain that uses an
auxiliary set of "fast weights" to implement a temporary overlay on the
energy landscape. The fast weights learn rapidly but also decay rapidly
and do not contribute to the normal energy landscape that defines the
model.

This is joint work with Geoffrey Hinton.


I Hope to see you there,
-- 
Pascal


Plus d'informations sur la liste de diffusion Lisa_seminaires