[Lisa_teatalk] [Lisa_labo] ICML-practice talks: Thursday, June 13th, AA3195, @ 2:00PM

Yoshua Bengio bengioy at iro.umontreal.ca
Wed Jun 12 14:27:50 EDT 2013


Actually for Razvan it is a full ICML oral.

-- Yoshua

On 2013-06-12, at 13:57, Guillaume Desjardins wrote:

> Please join us tomorrow afternoon (2:00-3:30PM) for a special round of
> ICML practice talks. Speakers will be Razvan Pascanu (remotely via
> Google Hangouts) and Yoshua Bengio (for both a workshop talk, and a 5
> min ICML spotlight). Titles and abstract below.
> 
> The talk will be held in AA3195 as usual.
> 
> 
> Speaker: Razvan Pascanu
> Title: On the difficulty of training Recurrent Neural Networks
> 
> There are two widely known issues with properly training Recurrent
> Neural Networks, the vanishing and the exploding gradient problems
> detailed in Bengio et al. (1994). In this paper we attempt to improve
> the understanding of the underlying issues by exploring these problems
> from an analytical, a geometric and a dynamical systems perspective.
> Our analysis is used to justify a simple yet effective solution. We
> propose a gradient norm clipping strategy to deal with exploding
> gradients and a soft constraint for the vanishing gradients problem.
> We validate empirically our hypothesis and proposed solutions in the
> experimental section.
> 
> 
> Speaker: Yoshua Bengio (workshop talk)
> Title: From Latent Anonymous Variables to Deep Stochastic Networks
> 
> Graphical models with anonymous latent variables (such as Markov
> Random Fields and variants of Boltzmann machines) are very powerful
> models which have had a great impact in machine learning. We propose
> here to consider a type of models over the joint distribution of
> observed variables that shares many of the properties of these
> anonymous latent variable models, but without the need for potentially
> hurtful approximate inference or approximation of the partition
> function during training (or both, like in Deep Boltzmann Machines).
> The only approximation is function approximation.  The proposed deep
> stochastic networks can provably estimate the underlying joint
> distribution of the data (or a conditional, if discriminant learning
> is preferred), in the sense that they are consistent estimators of
> this joint or conditional distribution.  However, unlike the usual
> probabilistic models, they only represent the learned distribution
> implicitly, through the convergent distribution of a Markov chain.
> Like them, though, they can handle missing values and structured
> output. Unlike them, they can be trained by straightforward gradient
> descent in a deep (possibly recurrent) network in which noise is
> injected. Most of the impressive experimental progress in deep
> learning in recent years (in particular for speech and object
> recognition) has been with with deep supervised networks, sometimes
> with noise injected (dropout). Interestingly, the same kinds of
> architectures can be used to train the proposed unsupervised (or
> structured output) models.
> 
> 
> Speaker: Yoshua Bengio (ICML spotlight).
> Better Mixing via Deep Representations
> 
> It has been hypothesized, and supported with experimental evidence,
> that deeper representations, when well trained, tend to do a better
> job at disentangling the underlying factors of variation. We study the
> following related conjecture: better representations, in the sense of
> better disentangling, can be exploited to produce Markov chains that
> mix faster between modes. Consequently, mixing between modes would be
> more efficient at higher levels of representation. To better
> understand this, we propose a secondary conjecture: the higher-level
> samples fill more uniformly the space they occupy and the high-density
> manifolds tend to unfold when represented at higher levels. The paper
> discusses these hypotheses and tests them experimentally through
> visualization and measurements of mixing between modes and
> interpolating between samples.
> _______________________________________________
> Lisa_labo mailing list
> Lisa_labo at iro.umontreal.ca
> https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo



More information about the Lisa_teatalk mailing list